Re: [squid-users] Squid uses way too much RAM and starts swapping ...
Hi, Any news on this topic? Unfortunately, RAM is running full within days and at the moment, our workaround is to do a reboot ... We would appreciate any other solution! thanks, peter On Wed, May 11, 2011 at 3:47 PM, guest01 gues...@gmail.com wrote: On Wed, May 11, 2011 at 10:47 AM, Amos Jeffries squ...@treenet.co.nz wrote: On 11/05/11 19:19, guest01 wrote: Hi, I am currently using squid 3.1.12 as forward-proxy without harddisk-caching (only RAM is used for caching). Each server is running on RHEL5.5 and is pretty strong (16 CPUs, 28GB RAM), but each servers starts swapping a few days after start. The workaround at the moment is to reboot the server once a week, which I don't really like. But swapping leads to serious side effects, e.g. performance troubles, ... way too much swapping: http://imageshack.us/m/52/6149/memoryday.png I already read a lot of posts and mails for similar problems, but unfortunately, I was not able to solve this problem. I added following infos to my squid.conf-file: # cache specific settings cache_replacement_policy heap LFUDA cache_mem 1600 MB memory_replacement_policy heap LFUDA maximum_object_size_in_memory 2048 KB memory_pools off cache_swap_low 85 cache_swap_high 90 (There are four squid instances per server, which means that 1600*4 = 6400MB RAM used for caching, which is not even 1/4 of the total available amount of RAM. Plenty enough, don't you think?) Not that is for HTTP object caching, emphasis on *caching* and HTTP object. In-transit objects and non-HTTP caches (Ip cache, domain name cache, persistent connections cache, client database, via/fwd database, network performance cache, auth caches, external ACL caches) and the indexes for all those caches use other memory. Then again they should all be using no more than a few GB combined. So you may have hit a new leak (all the known ones are resolved before 3.1.12). Ok, very strange. But at least it is reproducible, it takes about a week until squid is starting to swap ... http://img191.imageshack.us/img191/9615/memorymonth.png Very strange are the negative values (Memory usage for squid via mallinfo():) from the output below. Maybe that is a reason for running out of RAM? mallinfo() sucks badly when going above 2GB of RAM. It can be ignored. The section underneath it Memory accounted for: is Squids own accounting and more of a worry. It should not have had negatives since before 3.1.10. HTTP/1.0 200 OK Server: squid/3.1.12 Mime-Version: 1.0 Date: Wed, 11 May 2011 07:06:10 GMT Content-Type: text/plain Expires: Wed, 11 May 2011 07:06:10 GMT Last-Modified: Wed, 11 May 2011 07:06:10 GMT X-Cache: MISS from xlsqip03_1 Via: 1.0 xlsqip03_1 (squid/3.1.12) Connection: close Squid Object Cache: Version 3.1.12 Start Time: Wed, 27 Apr 2011 11:01:13 GMT Current Time: Wed, 11 May 2011 07:06:10 GMT Connection information for squid: Number of clients accessing cache: 1671 Number of HTTP requests received: 16144359 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 810.3 Average ICP messages per minute since start: 0.0 Select loop called: 656944758 times, 1.820 ms avg Cache information for squid: Hits as % of all requests: 5min: 17.4%, 60min: 18.2% Hits as % of bytes sent: 5min: 45.6%, 60min: 39.9% Memory hits as % of hit requests: 5min: 86.1%, 60min: 88.9% Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Storage Swap size: 0 KB Storage Swap capacity: 0.0% used, 0.0% free Storage Mem size: 1622584 KB Storage Mem capacity: 100.0% used, 0.0% free Okay 1.6 GB of RAM used for caching HTTP objects. Fully used. Mean Object Size: 0.00 KB Problem #1. It *may* be Squid not accounting for the memory objects in the mean. Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.01648 0.01235 Cache Misses: 0.05046 0.04277 Cache Hits: 0.00091 0.00091 Near Hits: 0.01469 0.01745 Not-Modified Replies: 0.0 0.00091 DNS Lookups: 0.00190 0.00190 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time: 1195497.286 seconds CPU Time: 22472.507 seconds CPU Usage: 1.88% CPU Usage, 5 minute avg: 5.38% CPU Usage, 60 minute avg: 5.44% Process Data Segment Size via sbrk(): 3145032 KB Maximum Resident Size: 0 KB Page faults with physical
Re: [squid-users] Squid uses way too much RAM and starts swapping ...
On 30/05/11 20:07, guest01 wrote: Hi, Any news on this topic? Unfortunately, RAM is running full within days and at the moment, our workaround is to do a reboot ... We would appreciate any other solution! thanks, peter :) I was just looking at this bug again today. :( still no seriously good ideas. Is there any way at all you can run one of these Squid under valgrind and get a report of what its memory is doing? Even if valgrind is just built in to squid. The info report gains a lot of extra memory stats and leak report without having to stop squid running. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.7 and 3.1.12.1
Re: [squid-users] Squid uses way too much RAM and starts swapping ...
ok, I can at least try to start it under valgrind, which I have never heard before. Do I just start squid under valgrind and send you the logfile? Do you need any special options? On Mon, May 30, 2011 at 2:15 PM, Amos Jeffries squ...@treenet.co.nz wrote: On 30/05/11 20:07, guest01 wrote: Hi, Any news on this topic? Unfortunately, RAM is running full within days and at the moment, our workaround is to do a reboot ... We would appreciate any other solution! thanks, peter :) I was just looking at this bug again today. :( still no seriously good ideas. Is there any way at all you can run one of these Squid under valgrind and get a report of what its memory is doing? Even if valgrind is just built in to squid. The info report gains a lot of extra memory stats and leak report without having to stop squid running. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.7 and 3.1.12.1
Re: [squid-users] Squid uses way too much RAM and starts swapping ...
On 31/05/11 03:20, guest01 wrote: ok, I can at least try to start it under valgrind, which I have never heard before. Do I just start squid under valgrind and send you the logfile? Do you need any special options? Squid need valgrind support built in when this is done. Apart from that yes it is just run Squid using valgrind like you would a debugger. Once it gets to some unusual amount of RAM usage shutdown Squid and send in the resulting valgrind log. We shall see if anything interesting pops up there. Thank you Amos On Mon, May 30, 2011 at 2:15 PM, Amos Jeffriessqu...@treenet.co.nz wrote: On 30/05/11 20:07, guest01 wrote: Hi, Any news on this topic? Unfortunately, RAM is running full within days and at the moment, our workaround is to do a reboot ... We would appreciate any other solution! thanks, peter :) I was just looking at this bug again today. :( still no seriously good ideas. Is there any way at all you can run one of these Squid under valgrind and get a report of what its memory is doing? Even if valgrind is just built in to squid. The info report gains a lot of extra memory stats and leak report without having to stop squid running. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.7 and 3.1.12.1 -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.12 Beta testers wanted for 3.2.0.7 and 3.1.12.1
Re: [squid-users] Squid uses way too much RAM and starts swapping ...
On 11/05/11 19:19, guest01 wrote: Hi, I am currently using squid 3.1.12 as forward-proxy without harddisk-caching (only RAM is used for caching). Each server is running on RHEL5.5 and is pretty strong (16 CPUs, 28GB RAM), but each servers starts swapping a few days after start. The workaround at the moment is to reboot the server once a week, which I don't really like. But swapping leads to serious side effects, e.g. performance troubles, ... way too much swapping: http://imageshack.us/m/52/6149/memoryday.png I already read a lot of posts and mails for similar problems, but unfortunately, I was not able to solve this problem. I added following infos to my squid.conf-file: # cache specific settings cache_replacement_policy heap LFUDA cache_mem 1600 MB memory_replacement_policy heap LFUDA maximum_object_size_in_memory 2048 KB memory_pools off cache_swap_low 85 cache_swap_high 90 (There are four squid instances per server, which means that 1600*4 = 6400MB RAM used for caching, which is not even 1/4 of the total available amount of RAM. Plenty enough, don't you think?) Not that is for HTTP object caching, emphasis on *caching* and HTTP object. In-transit objects and non-HTTP caches (Ip cache, domain name cache, persistent connections cache, client database, via/fwd database, network performance cache, auth caches, external ACL caches) and the indexes for all those caches use other memory. Then again they should all be using no more than a few GB combined. So you may have hit a new leak (all the known ones are resolved before 3.1.12). Very strange are the negative values (Memory usage for squid via mallinfo():) from the output below. Maybe that is a reason for running out of RAM? mallinfo() sucks badly when going above 2GB of RAM. It can be ignored. The section underneath it Memory accounted for: is Squids own accounting and more of a worry. It should not have had negatives since before 3.1.10. HTTP/1.0 200 OK Server: squid/3.1.12 Mime-Version: 1.0 Date: Wed, 11 May 2011 07:06:10 GMT Content-Type: text/plain Expires: Wed, 11 May 2011 07:06:10 GMT Last-Modified: Wed, 11 May 2011 07:06:10 GMT X-Cache: MISS from xlsqip03_1 Via: 1.0 xlsqip03_1 (squid/3.1.12) Connection: close Squid Object Cache: Version 3.1.12 Start Time: Wed, 27 Apr 2011 11:01:13 GMT Current Time: Wed, 11 May 2011 07:06:10 GMT Connection information for squid: Number of clients accessing cache: 1671 Number of HTTP requests received: 16144359 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 810.3 Average ICP messages per minute since start:0.0 Select loop called: 656944758 times, 1.820 ms avg Cache information for squid: Hits as % of all requests: 5min: 17.4%, 60min: 18.2% Hits as % of bytes sent:5min: 45.6%, 60min: 39.9% Memory hits as % of hit requests: 5min: 86.1%, 60min: 88.9% Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0% Storage Swap size: 0 KB Storage Swap capacity: 0.0% used, 0.0% free Storage Mem size: 1622584 KB Storage Mem capacity: 100.0% used, 0.0% free Okay 1.6 GB of RAM used for caching HTTP objects. Fully used. Mean Object Size: 0.00 KB Problem #1. It *may* be Squid not accounting for the memory objects in the mean. Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.01648 0.01235 Cache Misses: 0.05046 0.04277 Cache Hits:0.00091 0.00091 Near Hits: 0.01469 0.01745 Not-Modified Replies: 0.0 0.00091 DNS Lookups: 0.00190 0.00190 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:1195497.286 seconds CPU Time: 22472.507 seconds CPU Usage: 1.88% CPU Usage, 5 minute avg:5.38% CPU Usage, 60 minute avg: 5.44% Process Data Segment Size via sbrk(): 3145032 KB Maximum Resident Size: 0 KB Page faults with physical i/o: 8634 Memory usage for squid via mallinfo(): Total space in arena: -1049140 KB Ordinary blocks: -1277813 KB 87831 blks Small blocks: 0 KB 0 blks Holding blocks: 2240 KB 5 blks Free Small blocks: 0 KB Free Ordinary blocks: 228673 KB Total in use: -1275574 KB 122% Total free:228674 KB -22% Total size:-1046900 KB Memory accounted for: Total accounted:
