[squid-users] squid performance tunning
I have some Dell 1950 servers dedicated to squid in my production environment. Each with 16GB RAM and 300G disk As the website traffic grows, the load of squid becomes high at high traffic time. Average load is higher than 10. Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.01 0.06 0.13 1.23 1.4528.87 0.004.13 2.19 0.04 sda1 0.00 0.01 0.06 0.11 1.23 1.4531.59 0.004.52 2.40 0.04 sdb 0.07 0.07 0.01 0.01 0.33 0.3259.88 0.00 19.75 15.74 0.03 sdc 0.00 2.08 9.13 104.4481.30 1066.74 20.22 0.50 11.95 1.73 19.63 avg-cpu: %user %nice %system %iowait %steal %idle 3.500.003.75 24.340.00 68.41 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 70.00 70.00 3.50 sda1 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 70.00 70.00 3.50 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdc 0.00 0.00 21.50 186.00 204.00 3106.25 31.9117.76 100.55 2.63 54.65 Here is the squidclient mgr:info output Squid Object Cache: Version 3.1.12 Start Time: Sun, 14 Aug 2011 19:39:15 GMT Current Time: Thu, 18 Aug 2011 04:41:20 GMT Connection information for squid: Number of clients accessing cache: 77651 Number of HTTP requests received: 40449309 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 8319.3 Average ICP messages per minute since start:0.0 Select loop called: 476454933 times, 0.612 ms avg Cache information for squid: Hits as % of all requests: 5min: 30.7%, 60min: 32.1% Hits as % of bytes sent:5min: 40.5%, 60min: 43.2% Memory hits as % of hit requests: 5min: 88.3%, 60min: 88.8% Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% Storage Swap size: 120792244 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 5191632 KB Storage Mem capacity: 100.0% used, 0.0% free Mean Object Size: 20.61 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.00865 0.00865 Cache Misses: 0.01035 0.01035 Cache Hits:0.0 0.0 Near Hits: 0.00091 0.00091 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.0 0.0 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:291725.519 seconds CPU Time: 37204.391 seconds CPU Usage: 12.75% CPU Usage, 5 minute avg:19.42% CPU Usage, 60 minute avg: 18.20% Process Data Segment Size via sbrk(): 1012440 KB Maximum Resident Size: 28552368 KB Page faults with physical i/o: 2957 Memory usage for squid via mallinfo(): Total space in arena: -1265560 KB Ordinary blocks: -1308538 KB 264611 blks Small blocks: 0 KB 0 blks Holding blocks: 20708 KB 9 blks Free Small blocks: 0 KB Free Ordinary blocks: 42978 KB Total in use: -1287830 KB 103% Total free: 42978 KB -3% Total size:-1244852 KB Memory accounted for: Total accounted: -1781767 KB 143% memPool accounted: 6606841 KB -531% memPool unaccounted: -7851693 KB 0% memPoolAlloc calls: 10008474163 memPoolFree calls: 10065124847 File descriptor usage for squid: Maximum number of file descriptors: 20480 Largest file desc currently in use: 4828 Number of file desc currently in use: 4703 Files queued for open: 178 Available number of file descriptors: 15599 Reserved number of file descriptors: 100 Store Disk files open: 22 Internal Data Structures: 5860834 StoreEntries 256880 StoreEntries with MemObjects 256646 Hot Object Cache Items 5860661 on-disk objects related parameters cache_mem 5120 MB maximum_object_size 51200 KB maximum_object_size_in_memory 1024 KB log_icp_queries off cache_swap_low 90 cache_swap_high 95 hosts_file /etc/squid/hosts cache_dir aufs /export/squid/cache 131072 32 256 Is there any idea I can
Re: [squid-users] squid performance tunning
W dniu 2011-08-18 08:19, Chen Bangzhong pisze: I have some Dell 1950 servers dedicated to squid in my production environment. Each with 16GB RAM and 300G disk As the website traffic grows, the load of squid becomes high at high traffic time. Average load is higher than 10. Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.01 0.06 0.13 1.23 1.4528.87 0.004.13 2.19 0.04 sda1 0.00 0.01 0.06 0.11 1.23 1.4531.59 0.004.52 2.40 0.04 sdb 0.07 0.07 0.01 0.01 0.33 0.3259.88 0.00 19.75 15.74 0.03 sdc 0.00 2.08 9.13 104.4481.30 1066.74 20.22 0.50 11.95 1.73 19.63 avg-cpu: %user %nice %system %iowait %steal %idle 3.500.003.75 24.340.00 68.41 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 70.00 70.00 3.50 sda1 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 70.00 70.00 3.50 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdc 0.00 0.00 21.50 186.00 204.00 3106.25 31.9117.76 100.55 2.63 54.65 Here is the squidclient mgr:info output Squid Object Cache: Version 3.1.12 Start Time: Sun, 14 Aug 2011 19:39:15 GMT Current Time: Thu, 18 Aug 2011 04:41:20 GMT Connection information for squid: Number of clients accessing cache: 77651 Number of HTTP requests received: 40449309 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 8319.3 Average ICP messages per minute since start:0.0 Select loop called: 476454933 times, 0.612 ms avg Cache information for squid: Hits as % of all requests: 5min: 30.7%, 60min: 32.1% Hits as % of bytes sent:5min: 40.5%, 60min: 43.2% Memory hits as % of hit requests: 5min: 88.3%, 60min: 88.8% Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% Storage Swap size: 120792244 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 5191632 KB Storage Mem capacity: 100.0% used, 0.0% free Mean Object Size: 20.61 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.00865 0.00865 Cache Misses: 0.01035 0.01035 Cache Hits:0.0 0.0 Near Hits: 0.00091 0.00091 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.0 0.0 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:291725.519 seconds CPU Time: 37204.391 seconds CPU Usage: 12.75% CPU Usage, 5 minute avg:19.42% CPU Usage, 60 minute avg: 18.20% Process Data Segment Size via sbrk(): 1012440 KB Maximum Resident Size: 28552368 KB Page faults with physical i/o: 2957 Memory usage for squid via mallinfo(): Total space in arena: -1265560 KB Ordinary blocks: -1308538 KB 264611 blks Small blocks: 0 KB 0 blks Holding blocks: 20708 KB 9 blks Free Small blocks: 0 KB Free Ordinary blocks: 42978 KB Total in use: -1287830 KB 103% Total free: 42978 KB -3% Total size:-1244852 KB Memory accounted for: Total accounted: -1781767 KB 143% memPool accounted: 6606841 KB -531% memPool unaccounted: -7851693 KB 0% memPoolAlloc calls: 10008474163 memPoolFree calls: 10065124847 File descriptor usage for squid: Maximum number of file descriptors: 20480 Largest file desc currently in use: 4828 Number of file desc currently in use: 4703 Files queued for open: 178 Available number of file descriptors: 15599 Reserved number of file descriptors: 100 Store Disk files open: 22 Internal Data Structures: 5860834 StoreEntries 256880 StoreEntries with MemObjects 256646 Hot Object Cache Items 5860661 on-disk objects related parameters cache_mem 5120 MB maximum_object_size 51200 KB maximum_object_size_in_memory 1024 KB log_icp_queries off cache_swap_low 90
Re: [squid-users] squid performance tunning
Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.00865 0.00865 Cache Misses: 0.01035 0.01035 Cache Hits: 0.0 0.0 Near Hits: 0.00091 0.00091 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.0 0.0 ICP Queries: 0.0 0.0 Response time is reasonable at this time, while, peak time capture is good for performance tunning. Try atop 1 at peak time, this magic tool can clear about bottleneck. Try multi-instance, which can improve throughput dramaticlly. Docs's here: http://wiki.squid-cache.org/MultipleInstances CARP is another choice for extreme perf demand. http://wiki.squid-cache.org/ConfigExamples/ExtremeCarpFrontend
Re: [squid-users] squid performance tunning
My cached objects will expire after 10 minutes. Cache-Control:max-age=600 I don't know why there are so many disk writes and there are so many objects on disk. In addtion, Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% is very low. Can I increase the cache_mem？ or not use disk cache at all? 2011/8/18 Łukasz Makowski lukasz.makow...@itsoft.pl: W dniu 2011-08-18 08:19, Chen Bangzhong pisze: I have some Dell 1950 servers dedicated to squid in my production environment. Each with 16GB RAM and 300G disk As the website traffic grows, the load of squid becomes high at high traffic time. Average load is higher than 10. Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.01 0.06 0.13 1.23 1.45 28.87 0.00 4.13 2.19 0.04 sda1 0.00 0.01 0.06 0.11 1.23 1.45 31.59 0.00 4.52 2.40 0.04 sdb 0.07 0.07 0.01 0.01 0.33 0.32 59.88 0.00 19.75 15.74 0.03 sdc 0.00 2.08 9.13 104.44 81.30 1066.74 20.22 0.50 11.95 1.73 19.63 avg-cpu: %user %nice %system %iowait %steal %idle 3.50 0.00 3.75 24.34 0.00 68.41 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 70.00 70.00 3.50 sda1 0.00 0.00 0.50 0.00 2.00 0.00 8.00 0.04 70.00 70.00 3.50 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdc 0.00 0.00 21.50 186.00 204.00 3106.25 31.91 17.76 100.55 2.63 54.65 Here is the squidclient mgr:info output Squid Object Cache: Version 3.1.12 Start Time: Sun, 14 Aug 2011 19:39:15 GMT Current Time: Thu, 18 Aug 2011 04:41:20 GMT Connection information for squid: Number of clients accessing cache: 77651 Number of HTTP requests received: 40449309 Number of ICP messages received: 0 Number of ICP messages sent: 0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 8319.3 Average ICP messages per minute since start: 0.0 Select loop called: 476454933 times, 0.612 ms avg Cache information for squid: Hits as % of all requests: 5min: 30.7%, 60min: 32.1% Hits as % of bytes sent: 5min: 40.5%, 60min: 43.2% Memory hits as % of hit requests: 5min: 88.3%, 60min: 88.8% Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% Storage Swap size: 120792244 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 5191632 KB Storage Mem capacity: 100.0% used, 0.0% free Mean Object Size: 20.61 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min 60 min: HTTP Requests (All): 0.00865 0.00865 Cache Misses: 0.01035 0.01035 Cache Hits: 0.0 0.0 Near Hits: 0.00091 0.00091 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.0 0.0 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time: 291725.519 seconds CPU Time: 37204.391 seconds CPU Usage: 12.75% CPU Usage, 5 minute avg: 19.42% CPU Usage, 60 minute avg: 18.20% Process Data Segment Size via sbrk(): 1012440 KB Maximum Resident Size: 28552368 KB Page faults with physical i/o: 2957 Memory usage for squid via mallinfo(): Total space in arena: -1265560 KB Ordinary blocks: -1308538 KB 264611 blks Small blocks: 0 KB 0 blks Holding blocks: 20708 KB 9 blks Free Small blocks: 0 KB Free Ordinary blocks: 42978 KB Total in use: -1287830 KB 103% Total free: 42978 KB -3% Total size: -1244852 KB Memory accounted for: Total accounted: -1781767 KB 143% memPool accounted: 6606841 KB -531% memPool unaccounted: -7851693 KB 0% memPoolAlloc calls: 10008474163 memPoolFree calls: 10065124847 File descriptor usage for squid: Maximum number of file descriptors: 20480 Largest file desc currently in use: 4828 Number of file desc currently in use: 4703 Files queued for open: 178 Available number of file descriptors: 15599 Reserved number of file
Re: [squid-users] squid performance tunning
2011/8/18 Chen Bangzhong bangzh...@gmail.com: My cached objects will expire after 10 minutes. Cache-Control:max-age=600 Static content like pictures should cache longer, like 1 day, 86400. I don't know why there are so many disk writes and there are so many objects on disk. In addtion, Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% is very low. Maybe cause by disk read timeout. You used too much disk space, you can shrink it a little by a little, until disk busy percentage reduced to 80% or lower. Can I increase the cache_mem？ or not use disk cache at all? I used all memory I can use :-)
Re: [squid-users] squid performance tunning
On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: My cached objects will expire after 10 minutes. Cache-Control:max-age=600 Static content like pictures should cache longer, like 1 day, 86400. Could also be a whole year. If you control the origin website, set caching times as large as reasonably possible for each object. With revalidate settings relevant to its likely replacement needs. And always send a correct ETag. With those details Squid and other caches will take care of reducing caching times to suit the network and disk needs and updates/revalidation to suit your needs. So please set it large. I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Likely your 10-minute age is affecting this in a big way. The cache will have a lot of storable object which are stale. Next request they will be fetched into memory, then replaced by a revalidation REFRESH (near-HIT) response, which writes new data back to disk later. In addtion, Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% is very low. Maybe cause by disk read timeout. You used too much disk space, you can shrink it a little by a little, until disk busy percentage reduced to 80% or lower. Your Squid version is one which will promote HIT objects from disk and service repeat HITs from memory. Which reducing that disk-hit % a lot more than earlier squid versions would show it as. Can I increase the cache_mem？ or not use disk cache at all? I used all memory I can use :-) Indeed, the more the merrier. Unless it is swapping under high load. If that happens Squid speed goes terrible almost immediately. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: My cached objects will expire after 10 minutes. Cache-Control:max-age=600 Static content like pictures should cache longer, like 1 day, 86400. Could also be a whole year. If you control the origin website, set caching times as large as reasonably possible for each object. With revalidate settings relevant to its likely replacement needs. And always send a correct ETag. With those details Squid and other caches will take care of reducing caching times to suit the network and disk needs and updates/revalidation to suit your needs. So please set it large. I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Likely your 10-minute age is affecting this in a big way. The cache will have a lot of storable object which are stale. Next request they will be fetched into memory, then replaced by a revalidation REFRESH (near-HIT) response, which writes new data back to disk later. In addtion, Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% is very low. Maybe cause by disk read timeout. You used too much disk space, you can shrink it a little by a little, until disk busy percentage reduced to 80% or lower. Your Squid version is one which will promote HIT objects from disk and service repeat HITs from memory. Which reducing that disk-hit % a lot more than earlier squid versions would show it as. Can I increase the cache_mem？ or not use disk cache at all? I used all memory I can use :-) Indeed, the more the merrier. Unless it is swapping under high load. If that happens Squid speed goes terrible almost immediately. Actually I disabled swap at all, and use a script to start squid process immediately when killed by OS. OS will kill squid when OOM(Out of memory).
Re: [squid-users] squid performance tunning
thanks you Amos and Drunkard. My website hosts novels, That's, user can read novel there. The pages are not truely static contents, so I can only cache them for 10 minutes. My squids serve both non-cachable requests (works like nginx) and cachable-requests (10 min cache). So 60% cache miss is reasonable. It is not a good design, but we can't do more now. Another point is, only hot novels are read by users. Crawlers/robots will push many objects to cache. These objects are rarely read by user and will expire after 10 minutes. If the http response header indicates it is not cachable(eg: max-age=0), will squid save the response in RAM or disk? My guess is squid will discard the response. If the http response header indicates it is cachable(eg: max-age=600), squid will save it in the cache_mem. If the object is larger than maximum_object_size_in_memory, it will be written to disk. Can you tell me when will squid save the object to disk? When will squid delete the staled objects? 2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: My cached objects will expire after 10 minutes. Cache-Control:max-age=600 Static content like pictures should cache longer, like 1 day, 86400. Could also be a whole year. If you control the origin website, set caching times as largeas reasonably possible for each object. With revalidate settings relevant to its likely replacement needs. And always send a correct ETag. With those details Squid and other caches will take care of reducing caching times to suit the network and disk needs and updates/revalidation to suit your needs. So please set it large. I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Likely your 10-minute age is affecting this in a big way. The cache will have a lot of storable object which are stale. Next request they will be fetched into memory, then replaced by a revalidation REFRESH (near-HIT) response, which writes new data back to disk later. In addtion, Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% is very low. Maybe cause by disk read timeout. You used too much disk space, you can shrink it a little by a little, until disk busy percentage reduced to 80% or lower. Your Squid version is one which will promote HIT objects from disk and service repeat HITs from memory. Which reducing that disk-hit % a lot more than earlier squid versions would show it as. Can I increase the cache_mem？ or not use disk cache at all? I used all memory I can use :-) Indeed, the more the merrier. Unless it is swapping under high load. If that happens Squid speed goes terrible almost immediately. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: My cached objects will expire after 10 minutes. Cache-Control:max-age=600 Static content like pictures should cache longer, like 1 day, 86400. Could also be a whole year. If you control the origin website, set caching times as large as reasonably possible for each object. With revalidate settings relevant to its likely replacement needs. And always send a correct ETag. With those details Squid and other caches will take care of reducing caching times to suit the network and disk needs and updates/revalidation to suit your needs. So please set it large. I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Will a cache deny matching those non-storable objects suppress storing them to disk? And HTTP header 'Cache-Control: no-store' ? Likely your 10-minute age is affecting this in a big way. The cache will have a lot of storable object which are stale. Next request they will be fetched into memory, then replaced by a revalidation REFRESH (near-HIT) response, which writes new data back to disk later. In addtion, Disk hits as % of hit requests: 5min: 1.6%, 60min: 1.9% is very low. Maybe cause by disk read timeout. You used too much disk space, you can shrink it a little by a little, until disk busy percentage reduced to 80% or lower. Your Squid version is one which will promote HIT objects from disk and service repeat HITs from memory. Which reducing that disk-hit % a lot more than earlier squid versions would show it as. Can I increase the cache_mem？ or not use disk cache at all? I used all memory I can use :-) Indeed, the more the merrier. Unless it is swapping under high load. If that happens Squid speed goes terrible almost immediately. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10 Thanks, Kaiwang
Re: [squid-users] squid performance tunning
Mean Object Size: 20.61 K maximum_object_size_in_memory 1024 KB So most objects will be save in RAM first, still can't explain why there are so many disk writes. avg-cpu: %user %nice %system %iowait %steal %idle 1.520.001.636.950.00 89.91 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.01 0.06 0.13 1.24 1.4528.96 0.004.16 2.20 0.04 sda1 0.00 0.01 0.06 0.11 1.24 1.4531.69 0.004.55 2.41 0.04 sdb 0.07 0.07 0.01 0.01 0.33 0.3159.88 0.00 19.77 15.75 0.03 sdc 0.00 2.08 9.16 104.9681.61 1071.39 20.21 0.575.02 1.73 19.75 avg-cpu: %user %nice %system %iowait %steal %idle 2.380.003.38 10.380.00 83.88 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdc 0.00 4.50 11.00 293.00 104.00 3768.50 25.48 7.26 23.88 1.92 58.30 avg-cpu: %user %nice %system %iowait %steal %idle 3.250.002.633.880.00 90.24 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdc 0.00 0.50 15.50 94.50 150.00 644.2514.44 0.423.79 1.95 21.50 avg-cpu: %user %nice %system %iowait %steal %idle 3.000.002.883.380.00 90.75 Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.000.00 0.00 0.00 sdc 0.00 4.00 13.50 241.50 134.00 1609.75 13.68 0.893.37 0.76 19.50 在 2011年8月18日 下午6:50，Chen Bangzhong bangzh...@gmail.com 写道： thanks you Amos and Drunkard. My website hosts novels, That's, user can read novel there. The pages are not truely static contents, so I can only cache them for 10 minutes. My squids serve both non-cachable requests (works like nginx) and cachable-requests (10 min cache). So 60% cache miss is reasonable. It is not a good design, but we can't do more now. Another point is, only hot novels are read by users. Crawlers/robots will push many objects to cache. These objects are rarely read by user and will expire after 10 minutes. If the http response header indicates it is not cachable(eg: max-age=0), will squid save the response in RAM or disk? My guess is squid will discard the response. If the http response header indicates it is cachable(eg: max-age=600), squid will save it in the cache_mem. If the object is larger than maximum_object_size_in_memory, it will be written to disk. Can you tell me when will squid save the object to disk? When will squid delete the staled objects? 2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: My cached objects will expire after 10 minutes. Cache-Control:max-age=600 Static content like pictures should cache longer, like 1 day, 86400. Could also be a whole year. If you control the origin website, set caching times as largeas reasonably possible for each object. With revalidate settings relevant to its likely replacement needs. And always send a correct ETag. With those details Squid and other caches will take care of reducing caching times to suit the network and disk needs and updates/revalidation to suit your needs. So please set it large. I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Likely your 10-minute age is affecting this in a big way. The cache will have a lot of storable object which
Re: [squid-users] squid performance tunning
On 18/08/11 22:50, Chen Bangzhong wrote: thanks you Amos and Drunkard. My website hosts novels, That's, user can read novel there. The pages are not truely static contents, so I can only cache them for 10 minutes. My squids serve both non-cachable requests (works like nginx) and cachable-requests (10 min cache). So 60% cache miss is reasonable. It is not a good design, but we can't do more now. Oh well. Good luck wishes on that side of the problem. Another point is, only hot novels are read by users. Crawlers/robots will push many objects to cache. These objects are rarely read by user and will expire after 10 minutes. If the http response header indicates it is not cachable(eg: max-age=0), will squid save the response in RAM or disk? My guess is squid will discard the response. Correct. It will discard the response AND anything it has already cached for that URL. For non-hot objects this will not be a major problem. But may raise disk I/O a bit as the existing old stored content gets kicked out. Which might actually be a good thing, emptying space in the cache early. Or wasted I/O. It's not clear exactly which. If the http response header indicates it is cachable(eg: max-age=600), squid will save it in the cache_mem. If the object is larger than maximum_object_size_in_memory, it will be written to disk. Yes. Can you tell me when will squid save the object to disk? When will squid delete the staled objects? Stale objects are deleted at the point they are detected as stale and no longer usable (ie a request has been made for it and updated replacement has arrived from the web server). Or if they are the oldest object stored and more cache space is needed for newer objects. Other than tuning your existing setup there are two things I think you may be interested in. The first is a Measurement Factory project which involves altering Squid to completely bypass the cache storage when an object can't be cached or re-used by other clients. Makes them faster to process, and avoids dropping cached objects to make room. Combining this with a cache deny rule identifying those annoying robots as non-cacheable would allow you to store only the real users traffic needs. This is a slightly longer-term project, AFAIK it is not ready for production use (might be wrong). At minimum TMF are possibly needing sponsorship assistance to progress it faster. Contact Alex Rousskov about possibilities there, http://www.measurement-factory.com/contact.html The second thing is an alternative squid configuration which would emulate that behaviour immediately using two Squid instances. Basically; configure a new second instance as a non-caching gateway which all requests go to first. That could pass the robots and other easily detected non-cacheable requests straight to the web servers for service. While passing the other potentially cacheable requests to your current Squid instance, where storage and cache fetches happen more often without the robots. The gateway squid would have a much smaller footprint since it needs no memory for caching or indexing, and no disk usage at all. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
On 18/08/11 22:56, Chen Bangzhong wrote: Mean Object Size: 20.61 K maximum_object_size_in_memory 1024 KB So most objects will be save in RAM first, still can't explain why there are so many disk writes. Well, I would check the HTTP response headers there. Make sure they are containing Content-Length: header. If that is missing Squid is forced to assume it will have infinite length and require disk backing for the object until it is finished arriving. The Mean Object Size: metric is measured on completely received and stored objects. So does not really account for unknown length objects or non-cacheable previous objects. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
On 18/08/11 22:53, Kaiwang Chen wrote: 2011/8/18 Amos Jeffriessqu...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: snip I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Will a cache deny matching those non-storable objects suppress storing them to disk? And HTTP header 'Cache-Control: no-store' ? no-store header and cache deny directive have the same effect on your Squid. Both erase existing stored objects and erase the newely received one _after_ it is finished transfer. The difference is that the header applies everywhere receiving the object. The cache access control is limited to that one Squid instance testing it. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
thanks. Before I try the gateway squid solution, I want to change one of my squid to use memory cache only. I have 16GB RAM. now cache_mem is set to 5GB. I will try to increase it to 12GB and set cache_dir to null schma. I do this because I am sure that my hot objects can be saved in RAM, non-hot objects created by robots will stale and the memory will be reused. Is that all I need to set squid to be a memory cache? 2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 18/08/11 22:50, Chen Bangzhong wrote: thanks you Amos and Drunkard. My website hosts novels, That's, user can read novel there. The pages are not truely static contents, so I can only cache them for 10 minutes. My squids serve both non-cachable requests (works like nginx) and cachable-requests (10 min cache). So 60% cache miss is reasonable. It is not a good design, but we can't do more now. Oh well. Good luck wishes on that side of the problem. Another point is, only hot novels are read by users. Crawlers/robots will push many objects to cache. These objects are rarely read by user and will expire after 10 minutes. If the http response header indicates it is not cachable(eg: max-age=0), will squid save the response in RAM or disk? My guess is squid will discard the response. Correct. It will discard the response AND anything it has already cached for that URL. For non-hot objects this will not be a major problem. But may raise disk I/O a bit as the existing old stored content gets kicked out. Which might actually be a good thing, emptying space in the cache early. Or wasted I/O. It's not clear exactly which. If the http response header indicates it is cachable(eg: max-age=600), squid will save it in the cache_mem. If the object is larger than maximum_object_size_in_memory, it will be written to disk. Yes. Can you tell me when will squid save the object to disk? When will squid delete the staled objects? Stale objects are deleted at the point they are detected as stale and no longer usable (ie a request has been made for it and updated replacement has arrived from the web server). Or if they are the oldest object stored and more cache space is needed for newer objects. Other than tuning your existing setup there are two things I think you may be interested in. The first is a Measurement Factory project which involves altering Squid to completely bypass the cache storage when an object can't be cached or re-used by other clients. Makes them faster to process, and avoids dropping cached objects to make room. Combining this with a cache deny rule identifying those annoying robots as non-cacheable would allow you to store only the real users traffic needs. This is a slightly longer-term project, AFAIK it is not ready for production use (might be wrong). At minimum TMF are possibly needing sponsorship assistance to progress it faster. Contact Alex Rousskov about possibilities there, http://www.measurement-factory.com/contact.html The second thing is an alternative squid configuration which would emulate that behaviour immediately using two Squid instances. Basically; configure a new second instance as a non-caching gateway which all requests go to first. That could pass the robots and other easily detected non-cacheable requests straight to the web servers for service. While passing the other potentially cacheable requests to your current Squid instance, where storage and cache fetches happen more often without the robots. The gateway squid would have a much smaller footprint since it needs no memory for caching or indexing, and no disk usage at all. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 18/08/11 22:53, Kaiwang Chen wrote: 2011/8/18 Amos Jeffriessqu...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: snip I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Will a cache deny matching those non-storable objects suppress storing them to disk? And HTTP header 'Cache-Control: no-store' ? no-store header and cache deny directive have the same effect on your Squid. Both erase existing stored objects and erase the newely received one _after_ it is finished transfer. The difference is that the header applies everywhere receiving the object. The cache access control is limited to that one Squid instance testing it. Great. What about Cache-Control: max-age=0 and Cache-Control: no-cache responses? Does squid store them, hoping it is cheaper to make a validatation than to fetch a whole fresh object? Which souce code files describe the logic to deal with such cases? Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10 Thanks, Kaiwang
Re: [squid-users] squid performance tunning
在 2011年8月18日 下午9:07，Amos Jeffries squ...@treenet.co.nz 写道： On 18/08/11 22:56, Chen Bangzhong wrote: Mean Object Size: 20.61 K maximum_object_size_in_memory 1024 KB So most objects will be save in RAM first, still can't explain why there are so many disk writes. Well, I would check the HTTP response headers there. Make sure they are containing Content-Length: header. If that is missing Squid is forced to assume it will have infinite length and require disk backing for the object until it is finished arriving. Will squid require disk backing despite of the object size, even it is smaller than the receive buffer? Not sure what is the default size of receive buffer, is it one of these? read_ahead_gap 16 KB tcp_recv_bufsize 0 bytes The Mean Object Size: metric is measured on completely received and stored objects. So does not really account for unknown length objects or non-cacheable previous objects. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10 Thanks, Kaiwang
Re: [squid-users] squid performance tunning
On 19/08/11 02:40, Kaiwang Chen wrote: 2011/8/18 Amos Jeffriessqu...@treenet.co.nz: On 18/08/11 22:53, Kaiwang Chen wrote: 2011/8/18 Amos Jeffriessqu...@treenet.co.nz: On 18/08/11 19:40, Drunkard Zhang wrote: 2011/8/18 Chen Bangzhong: snip I don't know why there are so many disk writes and there are so many objects on disk. All traffic goes through either RAM cache or if its bigger than maximum_object_size_in_memory will go through disks. From that info report ~60% of your traffic bytes are MISS responses. A large portion of that MISS traffic is likely not storable, so will be written to cache then discarded immediately. Squid is overall mostly-write with its disk behaviour. Will a cache deny matching those non-storable objects suppress storing them to disk? And HTTP header 'Cache-Control: no-store' ? no-store header and cache deny directive have the same effect on your Squid. Both erase existing stored objects and erase the newely received one _after_ it is finished transfer. The difference is that the header applies everywhere receiving the object. The cache access control is limited to that one Squid instance testing it. Great. What about Cache-Control: max-age=0 and Cache-Control: no-cache responses? Does squid store them, max-age=0, that means discard immediately. Same as no-store to Squid. no-cache on responses is borderline. I can't seem to find anything relevant to no-cache kicking off a refresh. The HTTP/1.1 support results show it acting like no-store when last tested. So probably not usable yet. Luckily there is an overlap with the must-revalidate response directive. You can send that on the reply instead. hoping it is cheaper to make a validatation than to fetch a whole fresh object? Which souce code files describe the logic to deal with such cases? If the object has not actually changed, the server sends 304 instead of a new object, and there is an ETag to identify that object both machines are talking about is identical. Then yes, revalidation is much smaller. Squid does not (yet) send If-None-Match on revalidations (accepts and relay it but does not create it), so there are a number of possible cases where revalidation fails to be smaller. src/client_side_reply.cc cacheHit() handles the reply when an object is found in storage (to determin if its usable, obsolete, or simply old). That makes use of various other process*() code and src/refresh.cc does the revalidation calculations. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
On 19/08/11 02:10, Chen Bangzhong wrote: thanks. Before I try the gateway squid solution, I want to change one of my squid to use memory cache only. I have 16GB RAM. now cache_mem is set to 5GB. I will try to increase it to 12GB and set cache_dir to null schma. I do this because I am sure that my hot objects can be saved in RAM, non-hot objects created by robots will stale and the memory will be reused. Is that all I need to set squid to be a memory cache? You have squid-3.1, so only comment out the cache_dir lines and set cache_mem to something large. null dir schema no longer exists. Remember that cache_mem still has an index to account for and the usual active traffic buffering stays present. Also that reconfigure will wipe the RAM cache to empty. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
Amos, I want to find out what is filling my disk at 2-3MB/s. If there is no cache related information in the response header, will squid write the response to the disk? In squid wiki, I found the following sentences: Responses with Cache-Control: Private are NOT cachable. Responses with Cache-Control: No-Cache are NOT cachable. Responses with Cache-Control: No-Store are NOT cachable. Responses for requests with an Authorization header are cachable ONLY if the reponse includes Cache-Control: Public. The following HTTP status codes are cachable: 200 OK 203 Non-Authoritative Information 300 Multiple Choices 301 Moved Permanently 410 Gone My question is: If there is no Cache-control related information, such as the following header Server nginx/0.8.54 DateThu, 18 Aug 2011 15:56:29 GMT Content-Typeapplication/json; charset=UTF-8 Content-Length 1218 X-Cache MISS from zw12squid.my.com X-Cache-Lookup MISS from zw12squid.my.com:80 Via 1.0 zw12squid.my.com (squid/3.1.12) Connection keep-alive will squid save it to disk? Can you give me a detailed description about when will squid save the object to disk? thanks a lot for your kind help. 2011/8/18 Amos Jeffries squ...@treenet.co.nz: On 19/08/11 02:10, Chen Bangzhong wrote: thanks. Before I try the gateway squid solution, I want to change one of my squid to use memory cache only. I have 16GB RAM. now cache_mem is set to 5GB. I will try to increase it to 12GB and set cache_dir to null schma. I do this because I am sure that my hot objects can be saved in RAM, non-hot objects created by robots will stale and the memory will be reused. Is that all I need to set squid to be a memory cache? You have squid-3.1, so only comment out the cache_dir lines and set cache_mem to something large. null dir schema no longer exists. Remember that cache_mem still has an index to account for and the usual active traffic buffering stays present. Also that reconfigure will wipe the RAM cache to empty. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
On 19/08/11 02:59, Kaiwang Chen wrote: 在 2011年8月18日 下午9:07，Amos Jeffriessqu...@treenet.co.nz 写道： On 18/08/11 22:56, Chen Bangzhong wrote: Mean Object Size: 20.61 K maximum_object_size_in_memory 1024 KB So most objects will be save in RAM first, still can't explain why there are so many disk writes. Well, I would check the HTTP response headers there. Make sure they are containing Content-Length: header. If that is missing Squid is forced to assume it will have infinite length and require disk backing for the object until it is finished arriving. Will squid require disk backing despite of the object size, even it is smaller than the receive buffer? _require_ it. No. Do it that way due to old code, yes maybe. The amount of data waiting to be processed does not matter much. Could be zero bytes chunked encoded and a set of followup pipelined response headers. Until it is processed and stored somewhere Squid can't tell if its some bytes that happened to appear early, or the whole thing. The packet size, read_ahead_gap, and the receive buffer size (dynamic! 1-64KB), and cache_dir min/max values all have an effect in that area. I believe it picks a cache area before continuing to read more bytes (but not completely certain). If the cache_dir all have small maximum size limits and RAM looks bigger it will go there. In fact cache_dir usage for backing being practically welded in 3.1 series with large cache_mem have been showing signs of memory-backing instead on occasion. The other dev have projects underway to eliminate all that confusion in 3.2 anyways. Not sure what is the default size of receive buffer, is it one of these? read_ahead_gap 16 KB sliding window of bytes to buffer unsent to the client. Mostly unrelated to the receive buffer. When in effect its the minimum buffer size. tcp_recv_bufsize 0 bytes The tcp_recv_bufsize is the maximum amount per read cycle (0 being use the OS sysctl details, which is usually 4KB). Default buffer is hard-coded as 1KB for most of 3.1 series. 4KB for older and newer releases (slow-start algorithm from 1KB turned out to be bad for speed on MB sized objects and no benefit for small ones). Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
Re: [squid-users] squid performance tunning
On 19/08/11 03:58, Chen Bangzhong wrote: Amos, I want to find out what is filling my disk at 2-3MB/s. If there is no cache related information in the response header, will squid write the response to the disk? In squid wiki, I found the following sentences: Responses with Cache-Control: Private are NOT cachable. Responses with Cache-Control: No-Cache are NOT cachable. Responses with Cache-Control: No-Store are NOT cachable. Responses for requests with an Authorization header are cachable ONLY if the reponse includes Cache-Control: Public. The following HTTP status codes are cachable: 200 OK 203 Non-Authoritative Information 300 Multiple Choices 301 Moved Permanently 410 Gone My question is: If there is no Cache-control related information, such as the following header Server nginx/0.8.54 DateThu, 18 Aug 2011 15:56:29 GMT Content-Typeapplication/json; charset=UTF-8 Content-Length 1218 X-Cache MISS from zw12squid.my.com X-Cache-Lookup MISS from zw12squid.my.com:80 Via 1.0 zw12squid.my.com (squid/3.1.12) Connection keep-alive will squid save it to disk? No. It has a small Content-Length. Will store to RAM. But your RAM cache is running at 100% full, so something old will be pushed out to disk and this fills the empty gap. Lack of Cache-Control and Expires: headers means on the nest request for its URL your refresh_pattern rules will be tested against the URL and whichever one matches will be used to determine whether its served or revalidated. The only thing that could feed that algorithm is Date: when produced and current time, so Squid is unlikely to get it right of the two are very similar or very different. Probably leading to a revalidation or new request anyway. Can you give me a detailed description about when will squid save the object to disk? When it can't be saved to RAM cache_mem area. * cache_mem is full = least-popular object goes to disk. * object bigger than maximum_object_size_in_memory = goes to disk * object smaller than minimum_object_size_in_memory AND a cache_dir can accept it = goes to disk * object unknown length = goes to disk. Maybe RAM as well. Those are the cases I know about. There may be others. We know disk I/O happens far more often than it reasonably should in Squid. The newer releases since 2.6 and 3.0 are being improved to avoid it and increase traffic speeds, but progress is slow and irregular. You were going to try the memory-only caching. I think that was a good idea for your 88% RAM-hit vs 1% disk-hit ratios. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.14 Beta testers wanted for 3.2.0.10
