Re: [squid-users] What is the max number of Squirm redirect_children?
On Mon, Nov 21, 2011 at 10:35 PM, Amos Jeffries wrote: Also in the squid.conf: acl toSquirm url_regex ^http://www\.google\..*/(search|images)\? url_rewrite_access allow toSquirm url_rewrite_access deny all ... will make Squid not even bother to send URLs to Squirm if they wont be changed. Meaning your total traffic can be higher and not bottleneck at the URL-rewrite step. Hmmm... now I am wondering whether I could achieve the same effect through a Perl script to call via redirect_program... You could. Squirm is faster than the perl alternatives IIRC. Clever. Thank you. I think I'll then stick with Squirm with the Squid optimization you suggested. Regards, L.
Re: [squid-users] What is the max number of Squirm redirect_children?
Thanks Amos. As for the title question: You are the only one who knows that. It depends entirely on how much RAM your system has and how much is being used (by everything running). The number which can run on your system alongside Squid and the OS and everything else without causing the system to swap. The server on which Squid runs has about 3 Gb of RAM and sports a 3 Ghz processor. I'm testing it right now with no network connection (I can't do live tests for the moment). Spawning 80 instances of Squirm makes the machine crawl for a few minutes, but eventually everything becomes reusable and there are no page-ins/page-outs, according to vmstat. Squid Cache (Version 3.1.7): Terminated abnormally. Please try a more recent 3.1 release. We have done a lot towards small efficiencies this year. Unfortunately I can't upgrade right now, but I hope I'm able to do it soon. I'd also look at what Squirm is doing and try to reduce a few things ... * the number of helper lookups. With url_rewrite_access directive ACLs * the work Squid does handling responses. By sending empty response back for no-change, and using 3xx redirect responses instead of re-write responses. You may also be able to remove some uses of Squirm entirely by using deny_info redirection. I use Squirm uniquely to force SafeSearch on Google via these regex patterns: regexi ^(http://www\.google\..*/search\?.*) \1safe=active regexi ^(http://www\.google\..*/images\?.*) \1safe=active Hmmm... now I am wondering whether I could achieve the same effect through a Perl script to call via redirect_program... Thanks for your time, and best regards, L.
Re: [squid-users] What is the max number of Squirm redirect_children?
On Mon, 21 Nov 2011 16:44:04 +0100, Leonardo wrote: snip I'd also look at what Squirm is doing and try to reduce a few things ... * the number of helper lookups. With url_rewrite_access directive ACLs * the work Squid does handling responses. By sending empty response back for no-change, and using 3xx redirect responses instead of re-write responses. You may also be able to remove some uses of Squirm entirely by using deny_info redirection. I use Squirm uniquely to force SafeSearch on Google via these regex patterns: regexi ^(http://www\.google\..*/search\?.*) \1safe=active regexi ^(http://www\.google\..*/images\?.*) \1safe=active Also in the squid.conf: acl toSquirm url_regex ^http://www\.google\..*/(search|images)\? url_rewrite_access allow toSquirm url_rewrite_access deny all ... will make Squid not even bother to send URLs to Squirm if they wont be changed. Meaning your total traffic can be higher and not bottleneck at the URL-rewrite step. Hmmm... now I am wondering whether I could achieve the same effect through a Perl script to call via redirect_program... You could. Squirm is faster than the perl alternatives IIRC. Amos
[squid-users] What is the max number of Squirm redirect_children?
Dear all, I have a Squid transparent proxy which uses 40 Squirm URL rewriter processes. Everything worked fine until today, when Squid crashed after an error 179 pending requests queued: 2011/11/15 13:55:41| WARNING: All redirector processes are busy. 2011/11/15 13:55:41| WARNING: 40 pending requests queued 2011/11/15 13:55:41| Consider increasing the number of redirector processes in your config file. 2011/11/15 14:05:41| WARNING: All redirector processes are busy. 2011/11/15 14:05:41| WARNING: 179 pending requests queued 2011/11/15 14:05:41| storeDirWriteCleanLogs: Starting... 2011/11/15 14:05:41| WARNING: Closing open FD 190 2011/11/15 14:05:41| 65536 entries written so far. 2011/11/15 14:05:41|131072 entries written so far. 2011/11/15 14:05:41|196608 entries written so far. 2011/11/15 14:05:41|262144 entries written so far. 2011/11/15 14:05:41|327680 entries written so far. 2011/11/15 14:05:41|393216 entries written so far. 2011/11/15 14:05:41|458752 entries written so far. 2011/11/15 14:05:41|524288 entries written so far. 2011/11/15 14:05:41|589824 entries written so far. 2011/11/15 14:05:42| Finished. Wrote 633017 entries. 2011/11/15 14:05:42| Took 1.49 seconds (426234.18 entries/sec). FATAL: Too many queued redirector requests Squid Cache (Version 3.1.7): Terminated abnormally. CPU Usage: 91414.241 seconds = 55144.038 user + 36270.203 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 8 Memory usage for squid via mallinfo(): total space in arena: 1007116 KB Ordinary blocks: 1003347 KB 100368 blks Small blocks: 0 KB 1 blks Holding blocks: 27164 KB 8 blks Free Small blocks: 0 KB Free Ordinary blocks:3768 KB Total in use: 1030511 KB 102% Total free: 3768 KB 0% Restarting gave this error: 2011/11/15 14:14:02| WARNING: Disk space over limit: 11037556 KB 1024 KB 2011/11/15 14:14:02| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 10: (2) No such file or directory 2011/11/15 14:14:02| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 12: (2) No such file or directory 2011/11/15 14:14:02| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 14: (2) No such file or directory 2011/11/15 14:14:02| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 16: (2) No such file or directory 2011/11/15 14:14:02| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 18: (2) No such file or directory (... snipped 60 entries similar as the one above ... ) 2011/11/15 14:14:05| WARNING: All redirector processes are busy. 2011/11/15 14:14:05| WARNING: 40 pending requests queued 2011/11/15 14:14:05| Consider increasing the number of redirector processes in your config file. I tried increasing redirect_children to 300 in squid.conf, but since each children spawns as a process, this brought the system to its knees. Even with no active HTTP clients. What is the max/reasonable number of Squirm redirect_children? These two lines also puzzle me. I could rebuild the cache to get rid of it, but I'd like to know why this occurred: 2011/11/15 14:14:02| WARNING: Disk space over limit: 11037556 KB 1024 KB Total in use: 1030511 KB 102% Thanks a lot for your thoughts, Leonardo
Re: [squid-users] What is the max number of Squirm redirect_children?
On Tue, 15 Nov 2011 16:59:46 +0100, Leonardo wrote: Dear all, As for the title question: You are the only one who knows that. It depends entirely on how much RAM your system has and how much is being used (by everything running). The number which can run on your system alongside Squid and the OS and everything else without causing the system to swap. I have a Squid transparent proxy which uses 40 Squirm URL rewriter processes. Everything worked fine until today, when Squid crashed after an error 179 pending requests queued: 2011/11/15 13:55:41| WARNING: All redirector processes are busy. 2011/11/15 13:55:41| WARNING: 40 pending requests queued 2011/11/15 13:55:41| Consider increasing the number of redirector processes in your config file. 2011/11/15 14:05:41| WARNING: All redirector processes are busy. 2011/11/15 14:05:41| WARNING: 179 pending requests queued snip FATAL: Too many queued redirector requests Squid Cache (Version 3.1.7): Terminated abnormally. Please try a more recent 3.1 release. We have done a lot towards small efficiencies this year. CPU Usage: 91414.241 seconds = 55144.038 user + 36270.203 sys Maximum Resident Size: 0 KB Page faults with physical i/o: 8 Memory usage for squid via mallinfo(): total space in arena: 1007116 KB Ordinary blocks: 1003347 KB 100368 blks Small blocks: 0 KB 1 blks Holding blocks: 27164 KB 8 blks Free Small blocks: 0 KB Free Ordinary blocks:3768 KB Total in use: 1030511 KB 102% Total free: 3768 KB 0% Restarting gave this error: 2011/11/15 14:14:02| WARNING: Disk space over limit: 11037556 KB 1024 KB 2011/11/15 14:14:02| IpIntercept.cc(137) NetfilterInterception: NF getsockopt(SO_ORIGINAL_DST) failed on FD 10: (2) No such file or directory Err. Connections apparently being dropped out of the NAT system records. If these continue that will need to be investigated. Since there seem to be only 60 it could just be a hiccup in the network traffic. (... snipped 60 entries similar as the one above ... ) 2011/11/15 14:14:05| WARNING: All redirector processes are busy. 2011/11/15 14:14:05| WARNING: 40 pending requests queued 2011/11/15 14:14:05| Consider increasing the number of redirector processes in your config file. I tried increasing redirect_children to 300 in squid.conf, but since each children spawns as a process, this brought the system to its knees. Even with no active HTTP clients. What is the max/reasonable number of Squirm redirect_children? Check your cache_mem and normal RAM usage for Squid. The helper fork will consume that much virtual memory on each fork. A big jump like 40-300 is probably not a good idea, but smaller jump 40-100 maybe okay. I'd also look at what Squirm is doing and try to reduce a few things ... * the number of helper lookups. With url_rewrite_access directive ACLs * the work Squid does handling responses. By sending empty response back for no-change, and using 3xx redirect responses instead of re-write responses. You may also be able to remove some uses of Squirm entirely by using deny_info redirection. These two lines also puzzle me. I could rebuild the cache to get rid of it, but I'd like to know why this occurred: 2011/11/15 14:14:02| WARNING: Disk space over limit: 11037556 KB 1024 KB Total in use: 1030511 KB 102% I suspect it has something to do with 2GB files in the cache. The older Squid had some cache size accounting issues with individual files 2GB or total size 2TB doing 32-bit wrap and using way too much disk space. Amos
