I've re-compiled with --enable-dlmalloc and it didn't make a difference. I'm still leaking memory. The cachemgr lists 427621 KB in use, but only 260926 KB accounted for. Here's the cachemgr output:
Squid Object Cache: Version 2.5.STABLE4-20030929 Start Time: Wed, 01 Oct 2003 03:58:41 GMT Current Time: Thu, 02 Oct 2003 15:07:46 GMT Connection information for squid: Number of clients accessing cache: 0 Number of HTTP requests received: 3500368 Number of ICP messages received: 378876 Number of ICP messages sent: 379091 Number of queued ICP replies: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 1659.7 Average ICP messages per minute since start: 359.4 Select loop called: 17631204 times, 7.177 ms avg Cache information for squid: Request Hit Ratios: 5min: 68.9%, 60min: 67.2% Byte Hit Ratios: 5min: 9.5%, 60min: 19.1% Request Memory Hit Ratios: 5min: 6.0%, 60min: 7.5% Request Disk Hit Ratios: 5min: 12.9%, 60min: 13.8% Storage Swap size: 31896476 KB Storage Mem size: 98292 KB Mean Object Size: 14.27 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.01847 0.02592 Cache Misses: 0.17711 0.17711 Cache Hits: 0.01309 0.01648 Near Hits: 0.10281 0.10857 Not-Modified Replies: 0.01387 0.01648 DNS Lookups: 0.00464 0.00669 ICP Queries: 0.00541 0.00575 Resource usage for squid: UP Time: 126544.845 seconds CPU Time: 24888.760 seconds CPU Usage: 19.67% CPU Usage, 5 minute avg: 64.31% CPU Usage, 60 minute avg: 54.81% Process Data Segment Size via sbrk(): 427621 KB Maximum Resident Size: 0 KB Page faults with physical i/o: 837 Memory usage for squid via mallinfo(): Total space in arena: 427621 KB Ordinary blocks: 405719 KB 102690 blks Small blocks: 0 KB 0 blks Holding blocks: 17336 KB 9 blks Free Small blocks: 0 KB Free Ordinary blocks: 21901 KB Total in use: 423055 KB 95% Total free: 21901 KB 5% Total size: 444957 KB Memory accounted for: Total accounted: 260926 KB memPoolAlloc calls: 427293143 memPoolFree calls: 422443907 File descriptor usage for squid: Maximum number of file descriptors: 4096 Largest file desc currently in use: 997 Number of file desc currently in use: 877 Files queued for open: 0 Available number of file descriptors: 3219 Reserved number of file descriptors: 100 Store Disk files open: 3 Internal Data Structures: 2241473 StoreEntries 18942 StoreEntries with MemObjects 18926 Hot Object Cache Items 2234980 on-disk objects -----Original Message----- From: Mike Mitchell Sent: Monday, September 29, 2003 6:23 PM To: [EMAIL PROTECTED] Subject: Squid memory leak? I've been chasing what looks like a memory leak for quite some time. I've seen it all through the Squid 2.X releases, and it still is present in 2.5STABLE4. I've seen it on machines running HP-UX 10.20 and on Red Hat Linux. Currently I'm running on three identical Dell 2650 servers, each with dual 1.8 GHz processors, 2 GB of RAM, and two 20 GB cache partitions. They are all running Red Hat ES 2.1, kernel 2.4.9-e.25smp. Squid is built with these configuration options: Squid Cache: Version 2.5.STABLE4 configure options: --prefix=/local/proxy/squid --enable-cache-digests --enable-underscores --with-pthreads --enable-storeio=aufs,ufs --enable-removal-policies=heap --enable-gnuregex With a full cache, Squid memory utilization starts at about 250 MB. It then grows at about 300 MB per week. The graph at ftp://ftp.sas.com/pub/nam/squid-mem.png shows how the memory grows. We see about 80 requests per second during business hours. The graph ftp://ftp.sas.com/pub/nam/squid-reqs.png shows our requests per second. We use DNS round-robin to load balance requests between the three systems. Each squid process forwards most requests on to a Trend Interscan VirusWall process listening on port 8080 on the same server. It also sends the URLs through the "adzap" redirector. ACLs are used to decide which URLs are sent through the redirector and also which URLs are sent to the virus scanner. Here's some squid information output from the cachemgr.cgi: =========================================================== Squid Object Cache: Version 2.5.STABLE4 Start Time: Sat, 20 Sep 2003 19:29:16 GMT Current Time: Mon, 29 Sep 2003 21:14:57 GMT Connection information for squid: Number of clients accessing cache: 0 Number of HTTP requests received: 17264989 Number of ICP messages received: 1961488 Number of ICP messages sent: 1961585 Number of queued ICP replies: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 1321.4 Average ICP messages per minute since start: 300.3 Select loop called: 90171815 times, 8.694 ms avg Cache information for squid: Request Hit Ratios: 5min: 62.3%, 60min: 65.9% Byte Hit Ratios: 5min: 15.8%, 60min: 21.1% Request Memory Hit Ratios: 5min: 4.5%, 60min: 4.2% Request Disk Hit Ratios: 5min: 18.4%, 60min: 12.5% Storage Swap size: 31876696 KB Storage Mem size: 98348 KB Mean Object Size: 14.58 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.02317 0.02317 Cache Misses: 0.16775 0.16775 Cache Hits: 0.01309 0.01469 Near Hits: 0.09736 0.10857 Not-Modified Replies: 0.01387 0.01469 DNS Lookups: 0.00464 0.00464 ICP Queries: 0.00316 0.00541 Resource usage for squid: UP Time: 783941.680 seconds CPU Time: 118154.720 seconds CPU Usage: 15.07% CPU Usage, 5 minute avg: 65.24% CPU Usage, 60 minute avg: 60.68% Process Data Segment Size via sbrk(): 556455 KB Maximum Resident Size: 0 KB Page faults with physical i/o: 33606 Memory usage for squid via mallinfo(): Total space in arena: 556455 KB Ordinary blocks: 491460 KB 532257 blks Small blocks: 0 KB 0 blks Holding blocks: 19384 KB 12 blks Free Small blocks: 0 KB Free Ordinary blocks: 64995 KB Total in use: 510844 KB 89% Total free: 64995 KB 11% Total size: 575839 KB Memory accounted for: Total accounted: 263830 KB memPoolAlloc calls: 2122989169 memPoolFree calls: 2118393830 File descriptor usage for squid: Maximum number of file descriptors: 4096 Largest file desc currently in use: 877 Number of file desc currently in use: 465 Files queued for open: 0 Available number of file descriptors: 3631 Reserved number of file descriptors: 100 Store Disk files open: 4 Internal Data Structures: 2199786 StoreEntries 13566 StoreEntries with MemObjects 13530 Hot Object Cache Items 2186456 on-disk objects =================================================== What is most interesting from this is that mallinfo says 556455 KB is in the arena, but squid can only account for 263830 KB of it. Here's the interesting bits of our configuration: =================================================== http_port 80 8000 cache_peer 127.0.0.1 parent 8080 7 no-query no-digest default cache_peer inetgw02.unx.sas.com sibling 80 3130 proxy-only cache_peer inetgw03.unx.sas.com sibling 80 3130 proxy-only cache_peer inetgw02.unx.sas.com parent 8080 7 no-query no-digest round-robin cache_peer inetgw03.unx.sas.com parent 8080 7 no-query no-digest round-robin cache_mem 96 MB cache_swap_low 95 cache_swap_high 97 maximum_object_size 700 MB maximum_object_size_in_memory 32 KB cache_replacement_policy heap LFUDA memory_replacement_policy heap GDSF cache_dir aufs /cache/cache1/squid 16384 53 254 cache_dir aufs /cache/cache2/squid 16384 53 254 cache_store_log none emulate_httpd_log on acl all src 0.0.0.0/0.0.0.0 acl noAds myport 80 acl Reads method GET HEAD acl CIDR_A src 10.0.0.0/255.0.0.0 acl virus_proto proto HTTP acl novirus-url urlpath_regex -i \.gif(\?.*)?$ \.jpg(\?.*)?$ \.png(\?.*)?$ ident_lookup_access allow CIDR_A ident_lookup_access deny all memory_pools on log_icp_queries off client_db off always_direct allow novirus-url always_direct allow !virus_proto never_direct allow all strip_query_terms off redirect_program /opt/adzap/wrapzap redirect_children 10 redirector_access deny !Reads redirector_access allow noAds redirector_bypass on =================================================== Here's some messages that are showing up in the cache.log file. =================================================== 2003/09/29 16:38:26| sslReadServer: FD 578: read failure: (104) Connection reset by peer 2003/09/29 16:39:00| sslReadServer: FD 574: read failure: (104) Connection reset by peer 2003/09/29 16:41:44| sslReadServer: FD 560: read failure: (104) Connection reset by peer 2003/09/29 16:42:14| sslReadServer: FD 485: read failure: (104) Connection reset by peer 2003/09/29 16:42:54| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:42:55| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:42:59| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:03| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:07| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:08| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:17| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:21| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:31| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:32| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:33| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:34| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:46| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:43:47| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:44:00| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' 2003/09/29 16:44:00| urlParse: Illegal character in hostname 'inetgw01.unx.sas.com:80inetgw01.unx.sas.com' =================================================== Does anyone have an idea of why I'm consuming so much memory? Am I just running into memory fragmentation issues? Should I be linking with a different version of malloc? Would the configuration option '--enable-dlmalloc' help? I've worked around this problem by having a cron job check how large the squid process is. If it grows too large I restart it. Currently it gets restarted twice a month. ------------------------------------------------------------------------------ Mike Mitchell [EMAIL PROTECTED] (919) 677-8000 X16793