RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
Disabled ipv6 from the OS and also from Squid. Extended RAM to 300MB (Double than before). And browsing speed/response is much faster than before. But www.music.com.bd and www.djmaza.com is still like before. What should I do? Regards, Saiful From: saifu...@hotmail.com To: squid-users@squid-cache.org Date: Tue, 25 Jan 2011 17:17:30 +1100 Subject: RE: [squid-users] Some pages loading very slow in 3.1.10 Stable As advised earlier, I have disabled ipv6 support while recompiling squid, but the problem still exist. Do you want me to disable ipv6 in the OS and try again? Regards, Saiful Date: Mon, 24 Jan 2011 18:39:42 -0200 From: marcus.k...@urlfilterdb.com To: squ...@treenet.co.nz CC: squid-users@squid-cache.org Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable Amos Jeffries wrote: On 24/01/11 23:09, Michael Hendrie wrote: On 24/01/2011, at 8:17 PM, Saiful Alam wrote: OK I have kept your suggestion in my mind, but right now I'm not in a position to buy two HDD's. May be I can afford to buy 15 days later. For the time being, my prime problem is the loading of two major sites from where my users download mp3. Those are www.music.com.bd and www.djmaza.com Seems to load fine for me but that doesn't mean your slow = my fine. I had issues with some random sites being slow with 3.1.10 and tracked it down to squid trying to get records for the problem sites (or objects pulled from other sites). Not sure why this was occurring as IPv6 is not enabled on the OS. I didn't investigate too much and just recompiled with --disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue. Seems like you actually had IPv6 partially enabled in the OS, and maybe a break in DNS or MTU. When Squid 3.1.10 starts up it probes the OS network capabilities to see if IPv6 connections can be made. When they are possible it enables things like to use those connections. --disable-ipv6 merely sets the result of that test to always be false. With a reasonably fast DNS response time (under a half second) lookups will not be noticeable. I am one of those who live in Brazil and most lookups are slow since the international lines from Brazil to the USA are not properly sized. The log of the DNS server shows that the lookup for the A record is always preceded by the lookup of the record. What is the point of doing record lookups and trying to use the result and then reverting to A record lookups and succeeding with the connect() ? It is a lot of overhead for systems connected to slowish WANs and systems with *many* connections. The latter category might get a performance increase if Squid automagically detects that there is no IPv6 router or uses a new configuration option 'network_has_ipv6_router'. I did not find options to configure bind/named to ignore lookups either so I would love to see Squid have the new option. With working MTU there will be almost zero lag from opening and attempting IPv6 connections on an IPv4-only network. Don't know the reason, but music.com.bd loads very slow. And in firebug i see that the problem persists while loading 3 ads from ads.clicksor.com and some facebook widgets. Can you please check There you have the problem by the looks of it. Ad servers are very bad for being slow. They usually do a lot of processing or slow operations in the background before replying. Due to their tracking desires they do not permit proxies to cache and speed up their results. Some are more noticeable than others. Facebook is designed in a similar way which also suffers from these heavy processing delay problems on the APIs. But they do seem to be emitting useful cache controls on the static bits to avoid that. You have a choice: put up with it or block those URL from being fetched. and try to load these two domains if you're running a Squid 3.1.X version and see if everything is alright from your end. Amos
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
On Jan 24, 2011, at 3:39 PM, Marcus Kool wrote: I did not find options to configure bind/named to ignore lookups either so I would love to see Squid have the new option. man named if your running bind 9 you can force it to operate in v4 only. named -4 OPTIONS -4 Use IPv4 only even if the host machine is capable of IPv6. -4 and -6 are mutually exclusive. -6 Use IPv6 only even if the host machine is capable of IPv4. -4 and -6 are mutually exclusive.
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
it's not a dns problem... you can make sure and test the round trip and check the problem.. if you will insert these servers into your squid host to the hosts file you are done with DNS just make sure it's updated in every day with a nice script or manualy. this is one side of the problem we can then take it out of the picture... after you will make sure it's not it.. (and im sure it's not a DNS issue 99. percents) then raise your ram for like at least the bandwidth usage of your interfaces.. let say it's a 1GB in and 1GB out... give squid it... you must understand that if you have 8GB of ram for this machine and the purpose of the machine is to serve fast.. make a use of every piece of fast hardware you have to make it happened!!! if let say there are 3 different files that takes 50 MB and 10 people are downloading them you'd better serve them from your ram and not from you HD. and in any case your squid will serve only very static content so. make a choice man: a. want to serve and fast. b. just want them to get it faster then they will be mad to shout at you most of the time. HD costs like 20-30 dollars? a cheap and old sata will cost 15 dollars? buy 3-4 of the most cheapest HD you can and try to put all of them in raid 0 and you will see much more performance in a way you couldn't imagine(with adding some more ram available for squid) by the way. is there a nice and good software that can make web caching for 1 hits for 3MB file ? i mean like something that someone used? i can use nc to hit the server... On 25/01/2011 15:41, donovan jeffrey j wrote: On Jan 24, 2011, at 3:39 PM, Marcus Kool wrote: I did not find options to configure bind/named to ignore lookups either so I would love to see Squid have the new option. man named if your running bind 9 you can force it to operate in v4 only. named -4 OPTIONS -4 Use IPv4 only even if the host machine is capable of IPv6. -4 and -6 are mutually exclusive. -6 Use IPv6 only even if the host machine is capable of IPv4. -4 and -6 are mutually exclusive.
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
On Tue, 25 Jan 2011 23:42:02 +0200, Eliezer wrote: it's not a dns problem... you can make sure and test the round trip and check the problem.. if you will insert these servers into your squid host to the hosts file you are done with DNS just make sure it's updated in every day with a nice script or manualy. this is one side of the problem we can then take it out of the picture... Once up on a time back in teh original request this was a DNS and disk and origin server combo problem. The DNS part is now resolved AFAICT. Apparently the disk bit too. Now it is just down to the origin server being weird and the thread going way off track. after you will make sure it's not it.. (and im sure it's not a DNS issue 99. percents) then raise your ram for like at least the bandwidth usage of your interfaces.. let say it's a 1GB in and 1GB out... give squid it... you must understand that if you have 8GB of ram for this machine and the purpose of the machine is to serve fast.. make a use of every piece of fast hardware you have to make it happened!!! if let say there are 3 different files that takes 50 MB and 10 people are downloading them you'd better serve them from your ram and not from you HD. and in any case your squid will serve only very static content so. make a choice man: a. want to serve and fast. b. just want them to get it faster then they will be mad to shout at you most of the time. HD costs like 20-30 dollars? a cheap and old sata will cost 15 dollars? buy 3-4 of the most cheapest HD you can and try to put all of them in raid 0 and you will see much more performance in a way you couldn't imagine(with adding some more ram available for squid) Not RAID-0. Worst possible config. Cheap drives are best with a JBOD configuration, one cache_dir each. by the way. is there a nice and good software that can make web caching for 1 hits for 3MB file ? i mean like something that someone used? i can use nc to hit the server... 10K HITS ? over how long a period? Squid can suck in that many within a minute or so on modern server hardware. Then overload a 100Mbps NIC sending the file data from a cache_mem. Have not seen it replicated on Gbit-E cards yet. Amos
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
.pal.seabone.net.www 172.16.10.9.1408: Flags [.], cksum 0xdc7e (correct), seq 1, ack 956, win 7648, length 0 13:12:05.955120 IP (tos 0x0, ttl 127, id 2284, offset 0, flags [DF], proto TCP (6), length 40) 192.168.96.165.57362 195.122.152.64.www: Flags [.], cksum 0xda34 (correct), seq 0, ack 8761, win 64240, length 0 13:12:05.955148 IP (tos 0x0, ttl 64, id 18790, offset 0, flags [DF], proto TCP (6), length 972) 195.122.152.64.www 192.168.96.165.57362: Flags [P.], cksum 0x02b4 (correct), seq 8761:9693, ack 0, win 6432, length 932 13:12:05.955605 IP (tos 0x0, ttl 126, id 20667, offset 0, flags [DF], proto TCP (6), length 40) ^C172.16.106.3.49956 213.254.248.146.www: Flags [.], cksum 0x696e (correct), seq 348423304, ack 3320774375, win 16425, length 0 226 packets captured 107740 packets received by filter 107433 packets dropped by kernel Regards, Saiful From: saifu...@hotmail.com To: rm...@free.fr CC: squid-users@squid-cache.org Date: Mon, 24 Jan 2011 18:06:02 +1100 Subject: RE: [squid-users] Some pages loading very slow in 3.1.10 Stable Thanks for your prompt reply. Well I am not much experienced admin in linux or in terms of squid, and therefore I havent installed wireshark/tshark/tcpdump in squid yet, but I will install it now to go in deep. My previous version of squid was 2.7 Stable downloaded from aptitude which was installed in a different machine, and not in this new machine. My current configuration in the current PC (Core i7) is a squid 3.1.10 custom compiled by me with only the ZPH and the Linux Net Filter enabled. I haven't used any other parameters while configuring it. Default Cache_Dir is splitted into three directories: UFS /media/cache/storage501 51400 16 256 UFS /media/cache/storage502 51400 16 256 UFS /media/cache/storage503 51400 16 256 and cache_mem is 150MB Sometimes during peak hours while the network load is high, we do experience very slow browsing (in all websites) which is usually resolved after a quick restart of the squid process/service. Thanks Best Regards, Shishir Date: Mon, 24 Jan 2011 07:48:12 +0100 From: rm...@free.fr To: saifu...@hotmail.com CC: rm...@free.fr Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable Hi Saiful, On same plateform, do you have a old squid version working better? what is your previous squid version please? can you start network capture like wireshark/tshark/tcpdump on your squid please? and run firebug for example and compare last and your previous squid version please? can you compare squid cpu usage last and previous version please? Regards Rmkml On Mon, 24 Jan 2011, Saiful Alam wrote: Hello Everyone, My Squid Configuration is pretty much default, except the fact, that I have added some refresh_patterns myself and collected from internet in order to get more hits. The server is a Squid3.1.10 (in 3128 intercept transparent) running on Ubuntu 10.10 and comprises of Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz. I would say that everything is fine, except some pages are loading very very slow. For example, www.music.com.bd / www.djmaza.com ... I will try to give more examples as I find more, but for the time being these two sites are very important for my clients. Don't understand actually whats wrong. If this has anything to do with the squid.conf file, I will attach it with the next reply. For the time being here's my refresh_patterns. refresh_pattern ^http://.*\.facebook.com.*\.(jpg|gif|png|swf|wav|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.fbcdn.net.*\.(jpg|gif|png|swf|wav|mp3|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.yimg.com.*\.(jpg|gif|png|swf|wav|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.ytimg.com.*\.(jpg|gif|png|swf|wav|mp3|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.gstatic.com.*\.(jpg|gif|png) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.google.com.*\.(jpg|gif|png) 10080 300% 40320 reload-into-ims refresh_pattern dsebd\.org.*\.(jpg|gif|png) 600 50% 3000 refresh_pattern windowsupdate.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims refresh_pattern download.microsoft.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims refresh_pattern ^http://.*\.cnn\.com 1000 50% 4320 override-lastmod refresh_pattern ^http://news\.bbc\.co\.uk 1000 50% 4320 override-lastmod refresh_pattern microsoft 1080 150% 10080 override-lastmod refresh_pattern msn\.com 4320 150% 10080 override-lastmod refresh_pattern ^http://.*FIDO 360 1000% 480 refresh_pattern \.r[0-9][0-0]$ 10080 150% 40320 refresh_pattern ^http://.*\.asis$ 1440 50% 20160 refresh_pattern -i \.art$ 10080 150% 43200 refresh_pattern -i \.qtm$ 10080 150% 40320 reload-into-ims refresh_pattern -i \.viv$ 10080 150% 40320 reload-into-ims refresh_pattern -i
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
TRIED AUFS, but didn't get better performance, while
researching in the web, I read everywhere that AUFS is better than UFS
in terms of performance,but I don't know why I get bad performance with
this.
Processor is Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz
Motherboard is Intel® Desktop Board DH55TC
RAM is 2x4GB=8GB DDR3
DISK 1 = 250GB Hitachi SATA HDP72502 {[USED ONLY FOR UBUNTU SYSTEM BOOT]}
DISK 2 = 500GB Hitachi SATA HDS72105 {[USED FOR CACHE DIRECTORIES ONLY, MOUNTED
ON /MEDIA/CACHE FILESYSTEM EXT4]}
Normally my Disk I/O never goes more than 15% and I would say the average is
about 3-4%.
For bandwidth monitoring I usually see iptraf which is also good, but surely
I'll try ifstat next time.
At peak hours (which is between 10pm - 2am GMT +0600), we have around 500
clients connected (approx)
I have tried apt-get install squid3 (which is the default 3.1.6 in apt
repository) and found the performance of 3.1.10 (my custom configuration) is
better than the 3.1.6.
Regards,
Saiful
Date: Mon, 24 Jan 2011 10:36:14 +0200
From: elie...@ec.hadorhabaac.com
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
It's a small peace and most of the answers are not really suppose to there.
the first thing is that your cache is not just cache it's a store house..
it;'s not bad but you can try to change the ufs to aufs..
can get better performance.
what are the specification of the machine?
core i7 ? ram? what disk? array of disks?
did you tried to ping from the machine or WGET?
if it's debian you can install ifstat that can give you real-rime
bandwidth usage and it might be cause of something else that is not
related to squid..
if you have even like 25 clients downloading obsessively mp3 files for
like 10 or more minutes in this time your I\O of your hard drive will rise..
also..
you can monitor the access to the squid folders onfly while you have the
problem and to understand what is causing it..
if it's CPU load or DISK I\O load .. or other stuff.
by the way.. can you try the ubuntu squid3 stable?
im using squid3 stable on ubuntu 10.04 on an Intel Atom D450 machine
with cache of 40GB and it's taking the load very nicely.
On 24/01/2011 09:12, Saiful Alam wrote:
Some results of TCPDUMP in -vv mode.
13:12:04.191180 IP (tos 0x0, ttl 127, id 2750, offset 0, flags [DF], proto
TCP (6), length 40)
172.16.80.2.1155 77.67.29.42.www: Flags [.], cksum 0x6de4 (correct), seq
1127903567, ack 4192021369, win 64700, length 0
13:12:04.192822 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto
TCP (6), length 823)
www-12-02-snc5.facebook.com.www 10.16.63.123.3714: Flags [P.], cksum
0xc789 (correct), seq 4235393899:4235394682, ack 3661185770, win 14600,
length 783
13:12:04.831628 IP (tos 0x0, ttl 127, id 29487, offset 0, flags [DF], proto
TCP (6), length 40)
172.16.80.6.rmiregistry cds174.lax.llnw.net.www: Flags [.], cksum 0xfa45
(correct), seq 0, ack 460801, win 65535, length 0
13:12:05.355300 IP (tos 0x0, ttl 127, id 2265, offset 0, flags [DF], proto
TCP (6), length 40)
192.168.96.165.57362 195.122.152.64.www: Flags [.], cksum 0xfc6c
(correct), seq 1875916651, ack 4162436021, win 64240, length 0
13:12:05.355322 IP (tos 0x0, ttl 64, id 18785, offset 0, flags [DF], proto
TCP (6), length 1500)
195.122.152.64.www 192.168.96.165.57362: Flags [.], cksum 0x0f8f
(correct), seq 4381:5841, ack 0, win 6432, length 1460
13:12:05.355328 IP (tos 0x0, ttl 127, id 8325, offset 0, flags [DF], proto
TCP (6), length 40)
172.16.112.124.cvspserver 8.27.4.126.www: Flags [.], cksum 0xfe6f
(correct), seq 0, ack 662461, win 65535, length 0
13:12:05.655764 IP (tos 0x0, ttl 127, id 5393, offset 0, flags [DF], proto
TCP (6), length 40)
172.16.64.10.2308 co63.reliablehosting.com.www: Flags [.], cksum 0x1b43
(correct), seq 63164814, ack 3610294926, win 65535, length 0
13:12:05.655789 IP (tos 0x0, ttl 64, id 65532, offset 0, flags [DF], proto
TCP (6), length 1500)
co63.reliablehosting.com.www 172.16.64.10.2308: Flags [P.], cksum 0x4e5f
(correct), seq 1461:2921, ack 0, win 17184, length 1460
13:12:05.655798 IP (tos 0x0, ttl 64, id 65533, offset 0, flags [DF], proto
TCP (6), length 1215)
co63.reliablehosting.com.www 172.16.64.10.2308: Flags [P.], cksum 0xa23e
(correct), seq 2921:4096, ack 0, win 17184, length 1175
13:12:05.656066 IP (tos 0x0, ttl 126, id 42247, offset 0, flags [DF], proto
TCP (6), length 73)
172.16.30.6.1575 79-110-88-28.gfsrv.net.15103: Flags [P.], cksum 0x207b
(correct), seq 22088:22121, ack 55870, win 64475, length 33
13:12:05.656081 IP (tos 0x0, ttl 126, id 42248, offset 0, flags [DF], proto
TCP (6), length 51)
172.16.30.6.1575 79-110-88-28.gfsrv.net.15103: Flags [P.], cksum 0xd9a7
(correct), seq 22121
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
are you kdding me?
500 clients?
if most of the clients are just doing almost nothing just downloading
one page of 2 MB..
how much is your HD I\O ?? in this case?(in speed MBps) ?
so first.. change the UFS to AUFS you dont need to do anything to the
cache it self cause it the same system just with Async options on.
and just buy two more of these 500GB drives and put all three of them in
raid 0 or 5.
or first try 0 on two and then add another one to then if it goes smoothly.
this will give you a lot more speed.
i dont now the cause but it looks like or connectivity/dns or I\O problem.
i'v tried to use iptraf but ifstat just gives you the numbers of every
Interface you have in aginst out traffic meter in a simple way.
On 24/01/2011 11:02, Saiful Alam wrote:
TRIED AUFS, but didn't get better performance, while
researching in the web, I read everywhere that AUFS is better than UFS
in terms of performance,but I don't know why I get bad performance with
this.
Processor is Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz
Motherboard is Intel® Desktop Board DH55TC
RAM is 2x4GB=8GB DDR3
DISK 1 = 250GB Hitachi SATA HDP72502 {[USED ONLY FOR UBUNTU SYSTEM BOOT]}
DISK 2 = 500GB Hitachi SATA HDS72105 {[USED FOR CACHE DIRECTORIES ONLY, MOUNTED
ON /MEDIA/CACHE FILESYSTEM EXT4]}
Normally my Disk I/O never goes more than 15% and I would say the average is
about 3-4%.
For bandwidth monitoring I usually see iptraf which is also good, but surely
I'll try ifstat next time.
At peak hours (which is between 10pm - 2am GMT +0600), we have around 500
clients connected (approx)
I have tried apt-get install squid3 (which is the default 3.1.6 in apt
repository) and found the performance of 3.1.10 (my custom configuration) is
better than the 3.1.6.
Regards,
Saiful
Date: Mon, 24 Jan 2011 10:36:14 +0200
From: elie...@ec.hadorhabaac.com
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
It's a small peace and most of the answers are not really suppose to there.
the first thing is that your cache is not just cache it's a store house..
it;'s not bad but you can try to change the ufs to aufs..
can get better performance.
what are the specification of the machine?
core i7 ? ram? what disk? array of disks?
did you tried to ping from the machine or WGET?
if it's debian you can install ifstat that can give you real-rime
bandwidth usage and it might be cause of something else that is not
related to squid..
if you have even like 25 clients downloading obsessively mp3 files for
like 10 or more minutes in this time your I\O of your hard drive will rise..
also..
you can monitor the access to the squid folders onfly while you have the
problem and to understand what is causing it..
if it's CPU load or DISK I\O load .. or other stuff.
by the way.. can you try the ubuntu squid3 stable?
im using squid3 stable on ubuntu 10.04 on an Intel Atom D450 machine
with cache of 40GB and it's taking the load very nicely.
On 24/01/2011 09:12, Saiful Alam wrote:
Some results of TCPDUMP in -vv mode.
13:12:04.191180 IP (tos 0x0, ttl 127, id 2750, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.80.2.1155 77.67.29.42.www: Flags [.], cksum 0x6de4 (correct), seq
1127903567, ack 4192021369, win 64700, length 0
13:12:04.192822 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto TCP
(6), length 823)
www-12-02-snc5.facebook.com.www 10.16.63.123.3714: Flags [P.], cksum 0xc789
(correct), seq 4235393899:4235394682, ack 3661185770, win 14600, length 783
13:12:04.831628 IP (tos 0x0, ttl 127, id 29487, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.80.6.rmiregistry cds174.lax.llnw.net.www: Flags [.], cksum 0xfa45
(correct), seq 0, ack 460801, win 65535, length 0
13:12:05.355300 IP (tos 0x0, ttl 127, id 2265, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.96.165.57362 195.122.152.64.www: Flags [.], cksum 0xfc6c (correct),
seq 1875916651, ack 4162436021, win 64240, length 0
13:12:05.355322 IP (tos 0x0, ttl 64, id 18785, offset 0, flags [DF], proto TCP
(6), length 1500)
195.122.152.64.www 192.168.96.165.57362: Flags [.], cksum 0x0f8f (correct),
seq 4381:5841, ack 0, win 6432, length 1460
13:12:05.355328 IP (tos 0x0, ttl 127, id 8325, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.112.124.cvspserver 8.27.4.126.www: Flags [.], cksum 0xfe6f (correct),
seq 0, ack 662461, win 65535, length 0
13:12:05.655764 IP (tos 0x0, ttl 127, id 5393, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.64.10.2308 co63.reliablehosting.com.www: Flags [.], cksum 0x1b43
(correct), seq 63164814, ack 3610294926, win 65535, length 0
13:12:05.655789 IP (tos 0x0, ttl 64, id 65532, offset 0, flags [DF], proto TCP
(6), length 1500)
co63.reliablehosting.com.www 172.16.64.10.2308: Flags [P.], cksum 0x4e5f
(correct), seq 1461:2921, ack 0, win 17184, length
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
OK I have kept your suggestion in my mind, but right now I'm not in a position
to buy two HDD's. May be I can afford to buy 15 days later. For the time being,
my prime problem is the loading of two major sites from where my users download
mp3. Those are
www.music.com.bd and www.djmaza.com
Don't know the reason, but music.com.bd loads very slow. And in firebug i see
that the problem persists while loading 3 ads from ads.clicksor.com and some
facebook widgets. Can you please check and try to load these two domains if
you're running a Squid 3.1.X version and see if everything is alright from your
end.
Regards,
Saiful
Date: Mon, 24 Jan 2011 11:41:04 +0200
From: elie...@ec.hadorhabaac.com
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
are you kdding me?
500 clients?
if most of the clients are just doing almost nothing just downloading
one page of 2 MB..
how much is your HD I\O ?? in this case?(in speed MBps) ?
so first.. change the UFS to AUFS you dont need to do anything to the
cache it self cause it the same system just with Async options on.
and just buy two more of these 500GB drives and put all three of them in
raid 0 or 5.
or first try 0 on two and then add another one to then if it goes smoothly.
this will give you a lot more speed.
i dont now the cause but it looks like or connectivity/dns or I\O problem.
i'v tried to use iptraf but ifstat just gives you the numbers of every
Interface you have in aginst out traffic meter in a simple way.
On 24/01/2011 11:02, Saiful Alam wrote:
TRIED AUFS, but didn't get better performance, while
researching in the web, I read everywhere that AUFS is better than UFS
in terms of performance,but I don't know why I get bad performance with
this.
Processor is Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz
Motherboard is Intel® Desktop Board DH55TC
RAM is 2x4GB=8GB DDR3
DISK 1 = 250GB Hitachi SATA HDP72502 {[USED ONLY FOR UBUNTU SYSTEM BOOT]}
DISK 2 = 500GB Hitachi SATA HDS72105 {[USED FOR CACHE DIRECTORIES ONLY,
MOUNTED ON /MEDIA/CACHE FILESYSTEM EXT4]}
Normally my Disk I/O never goes more than 15% and I would say the average
is about 3-4%.
For bandwidth monitoring I usually see iptraf which is also good, but
surely I'll try ifstat next time.
At peak hours (which is between 10pm - 2am GMT +0600), we have around 500
clients connected (approx)
I have tried apt-get install squid3 (which is the default 3.1.6 in apt
repository) and found the performance of 3.1.10 (my custom configuration)
is better than the 3.1.6.
Regards,
Saiful
Date: Mon, 24 Jan 2011 10:36:14 +0200
From: elie...@ec.hadorhabaac.com
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
It's a small peace and most of the answers are not really suppose to there.
the first thing is that your cache is not just cache it's a store house..
it;'s not bad but you can try to change the ufs to aufs..
can get better performance.
what are the specification of the machine?
core i7 ? ram? what disk? array of disks?
did you tried to ping from the machine or WGET?
if it's debian you can install ifstat that can give you real-rime
bandwidth usage and it might be cause of something else that is not
related to squid..
if you have even like 25 clients downloading obsessively mp3 files for
like 10 or more minutes in this time your I\O of your hard drive will
rise..
also..
you can monitor the access to the squid folders onfly while you have the
problem and to understand what is causing it..
if it's CPU load or DISK I\O load .. or other stuff.
by the way.. can you try the ubuntu squid3 stable?
im using squid3 stable on ubuntu 10.04 on an Intel Atom D450 machine
with cache of 40GB and it's taking the load very nicely.
On 24/01/2011 09:12, Saiful Alam wrote:
Some results of TCPDUMP in -vv mode.
13:12:04.191180 IP (tos 0x0, ttl 127, id 2750, offset 0, flags [DF],
proto TCP (6), length 40)
172.16.80.2.1155 77.67.29.42.www: Flags [.], cksum 0x6de4 (correct), seq
1127903567, ack 4192021369, win 64700, length 0
13:12:04.192822 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto
TCP (6), length 823)
www-12-02-snc5.facebook.com.www 10.16.63.123.3714: Flags [P.], cksum
0xc789 (correct), seq 4235393899:4235394682, ack 3661185770, win 14600,
length 783
13:12:04.831628 IP (tos 0x0, ttl 127, id 29487, offset 0, flags [DF],
proto TCP (6), length 40)
172.16.80.6.rmiregistry cds174.lax.llnw.net.www: Flags [.], cksum 0xfa45
(correct), seq 0, ack 460801, win 65535, length 0
13:12:05.355300 IP (tos 0x0, ttl 127, id 2265, offset 0, flags [DF],
proto TCP (6), length 40)
192.168.96.165.57362 195.122.152.64.www: Flags
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
On 24/01/2011, at 8:17 PM, Saiful Alam wrote:
OK I have kept your suggestion in my mind, but right now I'm not in a
position to buy two HDD's. May be I can afford to buy 15 days later. For the
time being, my prime problem is the loading of two major sites from where my
users download mp3. Those are
www.music.com.bd and www.djmaza.com
Seems to load fine for me but that doesn't mean your slow = my fine.
I had issues with some random sites being slow with 3.1.10 and tracked it
down to squid trying to get records for the problem sites (or objects
pulled from other sites). Not sure why this was occurring as IPv6 is not
enabled on the OS. I didn't investigate too much and just recompiled with
--disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue.
Don't know the reason, but music.com.bd loads very slow. And in firebug i see
that the problem persists while loading 3 ads from ads.clicksor.com and some
facebook widgets. Can you please check and try to load these two domains if
you're running a Squid 3.1.X version and see if everything is alright from
your end.
Regards,
Saiful
Date: Mon, 24 Jan 2011 11:41:04 +0200
From: elie...@ec.hadorhabaac.com
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
are you kdding me?
500 clients?
if most of the clients are just doing almost nothing just downloading
one page of 2 MB..
how much is your HD I\O ?? in this case?(in speed MBps) ?
so first.. change the UFS to AUFS you dont need to do anything to the
cache it self cause it the same system just with Async options on.
and just buy two more of these 500GB drives and put all three of them in
raid 0 or 5.
or first try 0 on two and then add another one to then if it goes smoothly.
this will give you a lot more speed.
i dont now the cause but it looks like or connectivity/dns or I\O problem.
i'v tried to use iptraf but ifstat just gives you the numbers of every
Interface you have in aginst out traffic meter in a simple way.
On 24/01/2011 11:02, Saiful Alam wrote:
TRIED AUFS, but didn't get better performance, while
researching in the web, I read everywhere that AUFS is better than UFS
in terms of performance,but I don't know why I get bad performance with
this.
Processor is Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz
Motherboard is Intel® Desktop Board DH55TC
RAM is 2x4GB=8GB DDR3
DISK 1 = 250GB Hitachi SATA HDP72502 {[USED ONLY FOR UBUNTU SYSTEM BOOT]}
DISK 2 = 500GB Hitachi SATA HDS72105 {[USED FOR CACHE DIRECTORIES ONLY,
MOUNTED ON /MEDIA/CACHE FILESYSTEM EXT4]}
Normally my Disk I/O never goes more than 15% and I would say the average
is about 3-4%.
For bandwidth monitoring I usually see iptraf which is also good, but
surely I'll try ifstat next time.
At peak hours (which is between 10pm - 2am GMT +0600), we have around 500
clients connected (approx)
I have tried apt-get install squid3 (which is the default 3.1.6 in apt
repository) and found the performance of 3.1.10 (my custom configuration)
is better than the 3.1.6.
Regards,
Saiful
Date: Mon, 24 Jan 2011 10:36:14 +0200
From: elie...@ec.hadorhabaac.com
To: squid-users@squid-cache.org
Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
It's a small peace and most of the answers are not really suppose to there.
the first thing is that your cache is not just cache it's a store house..
it;'s not bad but you can try to change the ufs to aufs..
can get better performance.
what are the specification of the machine?
core i7 ? ram? what disk? array of disks?
did you tried to ping from the machine or WGET?
if it's debian you can install ifstat that can give you real-rime
bandwidth usage and it might be cause of something else that is not
related to squid..
if you have even like 25 clients downloading obsessively mp3 files for
like 10 or more minutes in this time your I\O of your hard drive will
rise..
also..
you can monitor the access to the squid folders onfly while you have the
problem and to understand what is causing it..
if it's CPU load or DISK I\O load .. or other stuff.
by the way.. can you try the ubuntu squid3 stable?
im using squid3 stable on ubuntu 10.04 on an Intel Atom D450 machine
with cache of 40GB and it's taking the load very nicely.
On 24/01/2011 09:12, Saiful Alam wrote:
Some results of TCPDUMP in -vv mode.
13:12:04.191180 IP (tos 0x0, ttl 127, id 2750, offset 0, flags [DF],
proto TCP (6), length 40)
172.16.80.2.1155 77.67.29.42.www: Flags [.], cksum 0x6de4 (correct), seq
1127903567, ack 4192021369, win 64700, length 0
13:12:04.192822 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto
TCP (6), length 823)
www-12-02-snc5.facebook.com.www 10.16.63.123.3714: Flags [P
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
On 24/01/11 22:02, Saiful Alam wrote:
TRIED AUFS, but didn't get better performance, while
researching in the web, I read everywhere that AUFS is better than
UFS in terms of performance,but I don't know why I get bad
performance with this. Processor is Intel(R) Core(TM) i7 CPU
870 @ 2.93GHz Motherboard is Intel® Desktop Board DH55TC
UFS performs one file operation per cache_dir over many small steps in a
single series.
AUFS operates by using multiple threads per cache_dir to execute the
disk I/O. Taking better advantage of OS disk scheduling mechanisms etc.
RAM is 2x4GB=8GB DDR3 DISK 1 = 250GB Hitachi SATA HDP72502
{[USED ONLY FOR UBUNTU SYSTEM BOOT]} DISK 2 = 500GB Hitachi SATA
HDS72105 {[USED FOR CACHE DIRECTORIES ONLY, MOUNTED ON /MEDIA/CACHE
FILESYSTEM EXT4]}
Having only one disk with multiple cache_dir on it is likely the source
of any slow disk problems you see. Squid uses its cache_dir in a
random-access mostly-write fashion with all cache_dir accessed in parallel.
When more than one UFS-based (ie UFS, AUFS or DiskD) dirs exists on the
same drive their I/O will collide with each other causing I/O signals
and other processor 'page faults' to slow the machine disk IO down.
Sometimes a lot.
Normally my Disk I/O never goes more than 15% and I would say the
average is about 3-4%. For bandwidth monitoring I usually see iptraf
which is also good, but surely I'll try ifstat next time. At peak
hours (which is between 10pm - 2am GMT +0600), we have around 500
clients connected (approx)
I have tried apt-get install squid3 (which is the default 3.1.6 in
apt repository) and found the performance of 3.1.10 (my custom
configuration) is better than the 3.1.6.
Regards, Saiful
Using webkit developer tools or firebug plugin to firefox you can get a
trace of all the objects on a particular page along with their fetch
times and total page rendering time.
In my experience most slow pages are composed of vast amounts of code
which needs to be run. The longer this takes to render the worse the
page appears to be in the eyes of clients.
The fetch time is the only part relevant to Squid. This should be also
shown by the transfer time column in your access.log. Once you have
identified the particular delayed requests you can look at why they are
delayed.
Amos
--
Please be using
Current Stable Squid 2.7.STABLE9 or 3.1.10
Beta testers wanted for 3.2.0.4
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
On 24/01/11 23:09, Michael Hendrie wrote: On 24/01/2011, at 8:17 PM, Saiful Alam wrote: OK I have kept your suggestion in my mind, but right now I'm not in a position to buy two HDD's. May be I can afford to buy 15 days later. For the time being, my prime problem is the loading of two major sites from where my users download mp3. Those are www.music.com.bd and www.djmaza.com Seems to load fine for me but that doesn't mean your slow = my fine. I had issues with some random sites being slow with 3.1.10 and tracked it down to squid trying to get records for the problem sites (or objects pulled from other sites). Not sure why this was occurring as IPv6 is not enabled on the OS. I didn't investigate too much and just recompiled with --disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue. Seems like you actually had IPv6 partially enabled in the OS, and maybe a break in DNS or MTU. When Squid 3.1.10 starts up it probes the OS network capabilities to see if IPv6 connections can be made. When they are possible it enables things like to use those connections. --disable-ipv6 merely sets the result of that test to always be false. With a reasonably fast DNS response time (under a half second) lookups will not be noticeable. With working MTU there will be almost zero lag from opening and attempting IPv6 connections on an IPv4-only network. Don't know the reason, but music.com.bd loads very slow. And in firebug i see that the problem persists while loading 3 ads from ads.clicksor.com and some facebook widgets. Can you please check There you have the problem by the looks of it. Ad servers are very bad for being slow. They usually do a lot of processing or slow operations in the background before replying. Due to their tracking desires they do not permit proxies to cache and speed up their results. Some are more noticeable than others. Facebook is designed in a similar way which also suffers from these heavy processing delay problems on the APIs. But they do seem to be emitting useful cache controls on the static bits to avoid that. You have a choice: put up with it or block those URL from being fetched. and try to load these two domains if you're running a Squid 3.1.X version and see if everything is alright from your end. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.10 Beta testers wanted for 3.2.0.4
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
Hi Amor, I have recompiled Squid with the following parameters. ./configure --prefix=/usr/local/squid --enable-async-io=8 --enable-storeio=ufs,aufs,diskd --enable-removal-policies=lru,heap --enable-delay-pools --enable-cache-digests --enable-underscores --enable-icap-client --enable-follow-x-forwarded-for --enable-arp-acl --enable-esi --enable-zph-qos --disable-translation --with-large-files --with-filedescriptors=65536 --disable-ipv6 --enable-linux-netfilter DISK I/O is just like before, on average 5% I would say. I am using the default removal policy, not heap. Well after recompiling it, I can see the Median DNS Lookup time reduced by 10 times are the browsing experience is improved than before. Thank you for that. But those two domains i.e www.music.com.bd and www.djmaza.com problem is not solved. I cannot determine exactly what is stopping it to load the site fast, indeed it loads may be 10 minutes later. To my (amateur) understanding, the problem is with the ads.clicksor.com and the facebook widgets it tries to load, but I am not 100% sure. Can anyone try to load the site again and see if the problem persists, or how should I exempt those URLs from being caching. I think the URLS are :- 1) http://ads.clicksor.com/showAd.php?pid=102683adtype=1sid=152482zone= 2) http://ads.clicksor.com/showAd.php?pid=102683adtype=9sid=152482zone= 3) http://www.facebook.com/ajax/connect/activity_widget.php?__a=1site=www.music.com.bdwidth=350height=225header=falsecolorscheme=lightborder_color=%23006a4epost_form_id=7c21c99b9bd83930bb871649bf465fcfuser=50560nb_activities=5newest=0 URL 3 waiting time sometimes goes more than 800ms. Please advise. Regards, Shishir Date: Tue, 25 Jan 2011 01:33:10 +1300 From: squ...@treenet.co.nz To: squid-users@squid-cache.org Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable On 24/01/11 23:09, Michael Hendrie wrote: On 24/01/2011, at 8:17 PM, Saiful Alam wrote: OK I have kept your suggestion in my mind, but right now I'm not in a position to buy two HDD's. May be I can afford to buy 15 days later. For the time being, my prime problem is the loading of two major sites from where my users download mp3. Those are www.music.com.bd and www.djmaza.com Seems to load fine for me but that doesn't mean your slow = my fine. I had issues with some random sites being slow with 3.1.10 and tracked it down to squid trying to get records for the problem sites (or objects pulled from other sites). Not sure why this was occurring as IPv6 is not enabled on the OS. I didn't investigate too much and just recompiled with --disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue. Seems like you actually had IPv6 partially enabled in the OS, and maybe a break in DNS or MTU. When Squid 3.1.10 starts up it probes the OS network capabilities to see if IPv6 connections can be made. When they are possible it enables things like to use those connections. --disable-ipv6 merely sets the result of that test to always be false. With a reasonably fast DNS response time (under a half second) lookups will not be noticeable. With working MTU there will be almost zero lag from opening and attempting IPv6 connections on an IPv4-only network. Don't know the reason, but music.com.bd loads very slow. And in firebug i see that the problem persists while loading 3 ads from ads.clicksor.com and some facebook widgets. Can you please check There you have the problem by the looks of it. Ad servers are very bad for being slow. They usually do a lot of processing or slow operations in the background before replying. Due to their tracking desires they do not permit proxies to cache and speed up their results. Some are more noticeable than others. Facebook is designed in a similar way which also suffers from these heavy processing delay problems on the APIs. But they do seem to be emitting useful cache controls on the static bits to avoid that. You have a choice: put up with it or block those URL from being fetched. and try to load these two domains if you're running a Squid 3.1.X version and see if everything is alright from your end. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.10 Beta testers wanted for 3.2.0.4
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
Dear Everyone, It is now 12AM here (GMT +0600) and very peak time for our ISP. Most of our users are now online. I took a snapshot from my cachemgr to show you guys. Squid Object Cache: Version 3.1.10 Start Time:Mon, 24 Jan 2011 13:58:44 GMT Current Time:Mon, 24 Jan 2011 18:04:49 GMT Connection information for squid: Number of clients accessing cache: 570 Number of HTTP requests received: 1468986 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 5969.4 Average ICP messages per minute since start:0.0 Select loop called: 36711620 times, 0.402 ms avg Cache information for squid: Hits as % of all requests: 5min: 35.1%, 60min: 35.6% Hits as % of bytes sent:5min: 25.6%, 60min: 31.4% Memory hits as % of hit requests: 5min: 4.5%, 60min: 3.9% Disk hits as % of hit requests: 5min: 63.3%, 60min: 65.4% Storage Swap size: 142110940 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 152228 KB Storage Mem capacity: 100.1% used, 0.0% free Mean Object Size: 18.00 KB Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.44492 0.44492 Cache Misses: 0.76407 0.80651 Cache Hits:0.01309 0.01745 Near Hits: 0.35832 0.37825 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.06963 0.09971 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:14765.225 seconds CPU Time: 2012.280 seconds CPU Usage: 13.63% CPU Usage, 5 minute avg:14.37% CPU Usage, 60 minute avg: 15.29% Process Data Segment Size via sbrk(): 1778280 KB Maximum Resident Size: 7213216 KB Page faults with physical i/o: 3 Memory usage for squid via mallinfo(): Total space in arena: 1778412 KB Ordinary blocks: 1750711 KB 20784 blks Small blocks: 0 KB 0 blks Holding blocks: 44844 KB 13 blks Free Small blocks: 0 KB Free Ordinary blocks: 27700 KB Total in use: 179 KB 98% Total free: 27700 KB 2% Total size:1823256 KB Memory accounted for: Total accounted: 1325755 KB 73% memPool accounted: 1325755 KB 73% memPool unaccounted: 497500 KB 27% memPoolAlloc calls: 373493434 memPoolFree calls: 375838488 File descriptor usage for squid: Maximum number of file descriptors: 65535 Largest file desc currently in use: 4467 Number of file desc currently in use: 3790 Files queued for open: 0 Available number of file descriptors: 61745 Reserved number of file descriptors: 100 Store Disk files open: 60 Internal Data Structures: 7896282 StoreEntries 16070 StoreEntries with MemObjects 15496 Hot Object Cache Items 7895225 on-disk objects Please see and check if there is anything which is not NORMAL, or might be reason for problems discussed below. Regards Saiful From: saifu...@hotmail.com To: squid-users@squid-cache.org Subject: RE: [squid-users] Some pages loading very slow in 3.1.10 Stable Date: Tue, 25 Jan 2011 05:00:30 +1100 Hi Amor, I have recompiled Squid with the following parameters. ./configure --prefix=/usr/local/squid --enable-async-io=8 --enable-storeio=ufs,aufs,diskd --enable-removal-policies=lru,heap --enable-delay-pools --enable-cache-digests --enable-underscores --enable-icap-client --enable-follow-x-forwarded-for --enable-arp-acl --enable-esi --enable-zph-qos --disable-translation --with-large-files --with-filedescriptors=65536 --disable-ipv6 --enable-linux-netfilter DISK I/O is just like before, on average 5% I would say. I am using the default removal policy, not heap. Well after recompiling it, I can see the Median DNS Lookup time reduced by 10 times are the browsing experience is improved than before. Thank you for that. But those two domains i.e www.music.com.bd and www.djmaza.com problem is not solved. I cannot determine exactly what is stopping it to load the site fast, indeed it loads may be 10 minutes later. To my (amateur) understanding, the problem is with the ads.clicksor.com and the facebook widgets it tries to load, but I am not 100% sure. Can anyone try to load the site again and see if the problem persists, or how
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
Amos Jeffries wrote: On 24/01/11 23:09, Michael Hendrie wrote: On 24/01/2011, at 8:17 PM, Saiful Alam wrote: OK I have kept your suggestion in my mind, but right now I'm not in a position to buy two HDD's. May be I can afford to buy 15 days later. For the time being, my prime problem is the loading of two major sites from where my users download mp3. Those are www.music.com.bd and www.djmaza.com Seems to load fine for me but that doesn't mean your slow = my fine. I had issues with some random sites being slow with 3.1.10 and tracked it down to squid trying to get records for the problem sites (or objects pulled from other sites). Not sure why this was occurring as IPv6 is not enabled on the OS. I didn't investigate too much and just recompiled with --disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue. Seems like you actually had IPv6 partially enabled in the OS, and maybe a break in DNS or MTU. When Squid 3.1.10 starts up it probes the OS network capabilities to see if IPv6 connections can be made. When they are possible it enables things like to use those connections. --disable-ipv6 merely sets the result of that test to always be false. With a reasonably fast DNS response time (under a half second) lookups will not be noticeable. I am one of those who live in Brazil and most lookups are slow since the international lines from Brazil to the USA are not properly sized. The log of the DNS server shows that the lookup for the A record is always preceded by the lookup of the record. What is the point of doing record lookups and trying to use the result and then reverting to A record lookups and succeeding with the connect() ? It is a lot of overhead for systems connected to slowish WANs and systems with *many* connections. The latter category might get a performance increase if Squid automagically detects that there is no IPv6 router or uses a new configuration option 'network_has_ipv6_router'. I did not find options to configure bind/named to ignore lookups either so I would love to see Squid have the new option. With working MTU there will be almost zero lag from opening and attempting IPv6 connections on an IPv4-only network. Don't know the reason, but music.com.bd loads very slow. And in firebug i see that the problem persists while loading 3 ads from ads.clicksor.com and some facebook widgets. Can you please check There you have the problem by the looks of it. Ad servers are very bad for being slow. They usually do a lot of processing or slow operations in the background before replying. Due to their tracking desires they do not permit proxies to cache and speed up their results. Some are more noticeable than others. Facebook is designed in a similar way which also suffers from these heavy processing delay problems on the APIs. But they do seem to be emitting useful cache controls on the static bits to avoid that. You have a choice: put up with it or block those URL from being fetched. and try to load these two domains if you're running a Squid 3.1.X version and see if everything is alright from your end. Amos
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
mån 2011-01-24 klockan 18:39 -0200 skrev Marcus Kool: I did not find options to configure bind/named to ignore lookups either so I would love to see Squid have the new option. It does. a) If Squid is built without IPv6 support b) If the host where Squid runs do not have IPv6 support at all. Regards Henrik
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
On Tue, 25 Jan 2011 05:08:00 +1100, Saiful Alam saifu...@hotmail.com wrote: Dear Everyone, It is now 12AM here (GMT +0600) and very peak time for our ISP. Most of our users are now online. I took a snapshot from my cachemgr to show you guys. Squid Object Cache: Version 3.1.10 Start Time:Mon, 24 Jan 2011 13:58:44 GMT Current Time:Mon, 24 Jan 2011 18:04:49 GMT Connection information for squid: Number of clients accessing cache: 570 Number of HTTP requests received: 1468986 Number of ICP messages received:0 Number of ICP messages sent:0 Number of queued ICP replies: 0 Number of HTCP messages received: 0 Number of HTCP messages sent: 0 Request failure ratio: 0.00 Average HTTP requests per minute since start: 5969.4 Average ICP messages per minute since start:0.0 Select loop called: 36711620 times, 0.402 ms avg Cache information for squid: Hits as % of all requests: 5min: 35.1%, 60min: 35.6% Hits as % of bytes sent:5min: 25.6%, 60min: 31.4% Memory hits as % of hit requests: 5min: 4.5%, 60min: 3.9% Disk hits as % of hit requests: 5min: 63.3%, 60min: 65.4% Storage Swap size: 142110940 KB Storage Swap capacity: 90.0% used, 10.0% free Storage Mem size: 152228 KB Storage Mem capacity: 100.1% used, 0.0% free Mean Object Size: 18.00 KB 152 MB of RAM shared between 570 users doe not look so good. If you can increase that things should improve a bit. Requests given to unlinkd: 0 Median Service Times (seconds) 5 min60 min: HTTP Requests (All): 0.44492 0.44492 Cache Misses: 0.76407 0.80651 Cache Hits:0.01309 0.01745 Near Hits: 0.35832 0.37825 Not-Modified Replies: 0.0 0.0 DNS Lookups: 0.06963 0.09971 ICP Queries: 0.0 0.0 Resource usage for squid: UP Time:14765.225 seconds CPU Time: 2012.280 seconds CPU Usage: 13.63% CPU Usage, 5 minute avg:14.37% CPU Usage, 60 minute avg: 15.29% Process Data Segment Size via sbrk(): 1778280 KB Maximum Resident Size: 7213216 KB Page faults with physical i/o: 3 Memory usage for squid via mallinfo(): Total space in arena: 1778412 KB Ordinary blocks: 1750711 KB 20784 blks Small blocks: 0 KB 0 blks Holding blocks: 44844 KB 13 blks Free Small blocks: 0 KB Free Ordinary blocks: 27700 KB Total in use: 179 KB 98% Total free: 27700 KB 2% Total size:1823256 KB Memory accounted for: Total accounted: 1325755 KB 73% memPool accounted: 1325755 KB 73% memPool unaccounted: 497500 KB 27% memPoolAlloc calls: 373493434 memPoolFree calls: 375838488 File descriptor usage for squid: Maximum number of file descriptors: 65535 Largest file desc currently in use: 4467 Number of file desc currently in use: 3790 Files queued for open: 0 Available number of file descriptors: 61745 Reserved number of file descriptors: 100 Store Disk files open: 60 Internal Data Structures: 7896282 StoreEntries 16070 StoreEntries with MemObjects 15496 Hot Object Cache Items 7895225 on-disk objects Please see and check if there is anything which is not NORMAL, or might be reason for problems discussed below. Regards Saiful From: saifu...@hotmail.com To: squid-users@squid-cache.org Subject: RE: [squid-users] Some pages loading very slow in 3.1.10 Stable Date: Tue, 25 Jan 2011 05:00:30 +1100 Hi Amor, I have recompiled Squid with the following parameters. ./configure --prefix=/usr/local/squid --enable-async-io=8 --enable-storeio=ufs,aufs,diskd --enable-removal-policies=lru,heap --enable-delay-pools --enable-cache-digests --enable-underscores --enable-icap-client --enable-follow-x-forwarded-for --enable-arp-acl --enable-esi --enable-zph-qos --disable-translation --with-large-files --with-filedescriptors=65536 --disable-ipv6 --enable-linux-netfilter DISK I/O is just like before, on average 5% I would say. I am using the default removal policy, not heap. Well after recompiling it, I can see the Median DNS Lookup time reduced by 10 times are the browsing experience is improved than before. Thank you for that. But those two domains i.e www.music.com.bd and www.djmaza.com problem is not solved. I cannot determine exactly what is stopping it to load the site fast, indeed it loads may be 10 minutes later. To my (amateur) understanding, the problem is with the ads.clicksor.com
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
As advised earlier, I have disabled ipv6 support while recompiling squid, but the problem still exist. Do you want me to disable ipv6 in the OS and try again? Regards, Saiful Date: Mon, 24 Jan 2011 18:39:42 -0200 From: marcus.k...@urlfilterdb.com To: squ...@treenet.co.nz CC: squid-users@squid-cache.org Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable Amos Jeffries wrote: On 24/01/11 23:09, Michael Hendrie wrote: On 24/01/2011, at 8:17 PM, Saiful Alam wrote: OK I have kept your suggestion in my mind, but right now I'm not in a position to buy two HDD's. May be I can afford to buy 15 days later. For the time being, my prime problem is the loading of two major sites from where my users download mp3. Those are www.music.com.bd and www.djmaza.com Seems to load fine for me but that doesn't mean your slow = my fine. I had issues with some random sites being slow with 3.1.10 and tracked it down to squid trying to get records for the problem sites (or objects pulled from other sites). Not sure why this was occurring as IPv6 is not enabled on the OS. I didn't investigate too much and just recompiled with --disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue. Seems like you actually had IPv6 partially enabled in the OS, and maybe a break in DNS or MTU. When Squid 3.1.10 starts up it probes the OS network capabilities to see if IPv6 connections can be made. When they are possible it enables things like to use those connections. --disable-ipv6 merely sets the result of that test to always be false. With a reasonably fast DNS response time (under a half second) lookups will not be noticeable. I am one of those who live in Brazil and most lookups are slow since the international lines from Brazil to the USA are not properly sized. The log of the DNS server shows that the lookup for the A record is always preceded by the lookup of the record. What is the point of doing record lookups and trying to use the result and then reverting to A record lookups and succeeding with the connect() ? It is a lot of overhead for systems connected to slowish WANs and systems with *many* connections. The latter category might get a performance increase if Squid automagically detects that there is no IPv6 router or uses a new configuration option 'network_has_ipv6_router'. I did not find options to configure bind/named to ignore lookups either so I would love to see Squid have the new option. With working MTU there will be almost zero lag from opening and attempting IPv6 connections on an IPv4-only network. Don't know the reason, but music.com.bd loads very slow. And in firebug i see that the problem persists while loading 3 ads from ads.clicksor.com and some facebook widgets. Can you please check There you have the problem by the looks of it. Ad servers are very bad for being slow. They usually do a lot of processing or slow operations in the background before replying. Due to their tracking desires they do not permit proxies to cache and speed up their results. Some are more noticeable than others. Facebook is designed in a similar way which also suffers from these heavy processing delay problems on the APIs. But they do seem to be emitting useful cache controls on the static bits to avoid that. You have a choice: put up with it or block those URL from being fetched. and try to load these two domains if you're running a Squid 3.1.X version and see if everything is alright from your end. Amos
Re: [squid-users] Some pages loading very slow in 3.1.10 Stable
On 24/01/2011, at 11:03 PM, Amos Jeffries wrote: On 24/01/11 23:09, Michael Hendrie wrote: On 24/01/2011, at 8:17 PM, Saiful Alam wrote: snip I had issues with some random sites being slow with 3.1.10 and tracked it down to squid trying to get records for the problem sites (or objects pulled from other sites). Not sure why this was occurring as IPv6 is not enabled on the OS. I didn't investigate too much and just recompiled with --disable-ipv6 as it wasn't needed. Doing so resolved my slow sites issue. Seems like you actually had IPv6 partially enabled in the OS, and maybe a break in DNS or MTU. When Squid 3.1.10 starts up it probes the OS network capabilities to see if IPv6 connections can be made. When they are possible it enables things like to use those connections. --disable-ipv6 merely sets the result of that test to always be false. With a reasonably fast DNS response time (under a half second) lookups will not be noticeable. With working MTU there will be almost zero lag from opening and attempting IPv6 connections on an IPv4-only network. Sorry to to hijack thread but thought I'd post my findings on this as it may be useful to other users. Thanks to Amos for comments, investigation shows that simply telling RHEL/CENTOS (5.5) not to enable IPv6 with NETWORKING_IPV6=no in /etc/sysconfig/network is not enough to disable IPv6. Article at http://www.cyberciti.biz/faq/redhat-centos-disable-ipv6-networking/ covers what is required. After doing this and recompiling squid this time without the --disable-ipv6 option, squid no longer issues lookup requests. snip Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.10 Beta testers wanted for 3.2.0.4
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
Thanks for your prompt reply. Well I am not much experienced admin in linux or in terms of squid, and therefore I havent installed wireshark/tshark/tcpdump in squid yet, but I will install it now to go in deep. My previous version of squid was 2.7 Stable downloaded from aptitude which was installed in a different machine, and not in this new machine. My current configuration in the current PC (Core i7) is a squid 3.1.10 custom compiled by me with only the ZPH and the Linux Net Filter enabled. I haven't used any other parameters while configuring it. Default Cache_Dir is splitted into three directories: UFS /media/cache/storage501 51400 16 256 UFS /media/cache/storage502 51400 16 256 UFS /media/cache/storage503 51400 16 256 and cache_mem is 150MB Sometimes during peak hours while the network load is high, we do experience very slow browsing (in all websites) which is usually resolved after a quick restart of the squid process/service. Thanks Best Regards, Shishir Date: Mon, 24 Jan 2011 07:48:12 +0100 From: rm...@free.fr To: saifu...@hotmail.com CC: rm...@free.fr Subject: Re: [squid-users] Some pages loading very slow in 3.1.10 Stable Hi Saiful, On same plateform, do you have a old squid version working better? what is your previous squid version please? can you start network capture like wireshark/tshark/tcpdump on your squid please? and run firebug for example and compare last and your previous squid version please? can you compare squid cpu usage last and previous version please? Regards Rmkml On Mon, 24 Jan 2011, Saiful Alam wrote: Hello Everyone, My Squid Configuration is pretty much default, except the fact, that I have added some refresh_patterns myself and collected from internet in order to get more hits. The server is a Squid3.1.10 (in 3128 intercept transparent) running on Ubuntu 10.10 and comprises of Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz. I would say that everything is fine, except some pages are loading very very slow. For example, www.music.com.bd / www.djmaza.com ... I will try to give more examples as I find more, but for the time being these two sites are very important for my clients. Don't understand actually whats wrong. If this has anything to do with the squid.conf file, I will attach it with the next reply. For the time being here's my refresh_patterns. refresh_pattern ^http://.*\.facebook.com.*\.(jpg|gif|png|swf|wav|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.fbcdn.net.*\.(jpg|gif|png|swf|wav|mp3|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.yimg.com.*\.(jpg|gif|png|swf|wav|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.ytimg.com.*\.(jpg|gif|png|swf|wav|mp3|3gp|flv|swf|wmv|zip|rar) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.gstatic.com.*\.(jpg|gif|png) 10080 300% 40320 reload-into-ims refresh_pattern ^http://.*\.google.com.*\.(jpg|gif|png) 10080 300% 40320 reload-into-ims refresh_pattern dsebd\.org.*\.(jpg|gif|png) 600 50% 3000 refresh_pattern windowsupdate.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims refresh_pattern download.microsoft.com/.*\.(cab|exe) 4320 100% 43200 reload-into-ims refresh_pattern ^http://.*\.cnn\.com 1000 50% 4320 override-lastmod refresh_pattern ^http://news\.bbc\.co\.uk 1000 50% 4320 override-lastmod refresh_pattern microsoft 1080 150% 10080 override-lastmod refresh_pattern msn\.com 4320 150% 10080 override-lastmod refresh_pattern ^http://.*FIDO 360 1000% 480 refresh_pattern \.r[0-9][0-0]$ 10080 150% 40320 refresh_pattern ^http://.*\.asis$ 1440 50% 20160 refresh_pattern -i \.art$ 10080 150% 43200 refresh_pattern -i \.qtm$ 10080 150% 40320 reload-into-ims refresh_pattern -i \.viv$ 10080 150% 40320 reload-into-ims refresh_pattern -i \.pdf$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.jpg$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.png$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.gif$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.flv$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.ico$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.jpeg$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.x-flv$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.swf$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.wav$ 10080 300% 40320 reload-into-ims refresh_pattern -i \.mp3$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.mp4$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.zip$ 10080 300% 40320 override-expire override-lastmod reload-into-ims ignore-reload refresh_pattern -i \.rar$ 10080 300% 40320 override-expire override-lastmod reload
RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
Some results of TCPDUMP in -vv mode.
13:12:04.191180 IP (tos 0x0, ttl 127, id 2750, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.80.2.1155 77.67.29.42.www: Flags [.], cksum 0x6de4 (correct), seq
1127903567, ack 4192021369, win 64700, length 0
13:12:04.192822 IP (tos 0x0, ttl 64, id 4692, offset 0, flags [DF], proto TCP
(6), length 823)
www-12-02-snc5.facebook.com.www 10.16.63.123.3714: Flags [P.], cksum
0xc789 (correct), seq 4235393899:4235394682, ack 3661185770, win 14600, length
783
13:12:04.831628 IP (tos 0x0, ttl 127, id 29487, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.80.6.rmiregistry cds174.lax.llnw.net.www: Flags [.], cksum 0xfa45
(correct), seq 0, ack 460801, win 65535, length 0
13:12:05.355300 IP (tos 0x0, ttl 127, id 2265, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.96.165.57362 195.122.152.64.www: Flags [.], cksum 0xfc6c
(correct), seq 1875916651, ack 4162436021, win 64240, length 0
13:12:05.355322 IP (tos 0x0, ttl 64, id 18785, offset 0, flags [DF], proto TCP
(6), length 1500)
195.122.152.64.www 192.168.96.165.57362: Flags [.], cksum 0x0f8f
(correct), seq 4381:5841, ack 0, win 6432, length 1460
13:12:05.355328 IP (tos 0x0, ttl 127, id 8325, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.112.124.cvspserver 8.27.4.126.www: Flags [.], cksum 0xfe6f
(correct), seq 0, ack 662461, win 65535, length 0
13:12:05.655764 IP (tos 0x0, ttl 127, id 5393, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.64.10.2308 co63.reliablehosting.com.www: Flags [.], cksum 0x1b43
(correct), seq 63164814, ack 3610294926, win 65535, length 0
13:12:05.655789 IP (tos 0x0, ttl 64, id 65532, offset 0, flags [DF], proto TCP
(6), length 1500)
co63.reliablehosting.com.www 172.16.64.10.2308: Flags [P.], cksum 0x4e5f
(correct), seq 1461:2921, ack 0, win 17184, length 1460
13:12:05.655798 IP (tos 0x0, ttl 64, id 65533, offset 0, flags [DF], proto TCP
(6), length 1215)
co63.reliablehosting.com.www 172.16.64.10.2308: Flags [P.], cksum 0xa23e
(correct), seq 2921:4096, ack 0, win 17184, length 1175
13:12:05.656066 IP (tos 0x0, ttl 126, id 42247, offset 0, flags [DF], proto TCP
(6), length 73)
172.16.30.6.1575 79-110-88-28.gfsrv.net.15103: Flags [P.], cksum 0x207b
(correct), seq 22088:22121, ack 55870, win 64475, length 33
13:12:05.656081 IP (tos 0x0, ttl 126, id 42248, offset 0, flags [DF], proto TCP
(6), length 51)
172.16.30.6.1575 79-110-88-28.gfsrv.net.15103: Flags [P.], cksum 0xd9a7
(correct), seq 22121:22132, ack 55870, win 64475, length 11
13:12:05.949960 IP (tos 0x0, ttl 126, id 5909, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.10.9.1408 GGCpmo1.pal.seabone.net.www: Flags [.], cksum 0xfe1a
(correct), seq 3933402590, ack 133923280, win 65535, length 0
13:12:05.949983 IP (tos 0x0, ttl 126, id 31989, offset 0, flags [DF], proto TCP
(6), length 52)
172.16.52.6.62360 c-n180-u0282-208.webazilla.com.www: Flags [.], cksum
0x0bbb (correct), seq 0, ack 1119485, win 64240, options [nop,nop,sack 1
{1120945:1129705}], length 0
13:12:05.949997 IP (tos 0x0, ttl 127, id 1202, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.200.42.1454 146.82.204.207.www: Flags [.], cksum 0x5a1a (correct),
seq 0, ack 1965427, win 65375, length 0
13:12:05.950985 IP (tos 0x0, ttl 126, id 36156, offset 0, flags [DF], proto TCP
(6), length 40)
172.16.30.7.1795 cds17.sin.llnw.net.www: Flags [.], cksum 0x779d
(correct), seq 4079975438, ack 3825116412, win 65535, length 0
13:12:05.951132 IP (tos 0x0, ttl 126, id 5910, offset 0, flags [DF], proto TCP
(6), length 996)
172.16.10.9.1408 GGCpmo1.pal.seabone.net.www: Flags [P.], cksum 0xf7ba
(correct), seq 0:956, ack 1, win 65535, length 956
13:12:05.951152 IP (tos 0x0, ttl 64, id 20092, offset 0, flags [DF], proto TCP
(6), length 40)
GGCpmo1.pal.seabone.net.www 172.16.10.9.1408: Flags [.], cksum 0xdc7e
(correct), seq 1, ack 956, win 7648, length 0
13:12:05.955120 IP (tos 0x0, ttl 127, id 2284, offset 0, flags [DF], proto TCP
(6), length 40)
192.168.96.165.57362 195.122.152.64.www: Flags [.], cksum 0xda34
(correct), seq 0, ack 8761, win 64240, length 0
13:12:05.955148 IP (tos 0x0, ttl 64, id 18790, offset 0, flags [DF], proto TCP
(6), length 972)
195.122.152.64.www 192.168.96.165.57362: Flags [P.], cksum 0x02b4
(correct), seq 8761:9693, ack 0, win 6432, length 932
13:12:05.955605 IP (tos 0x0, ttl 126, id 20667, offset 0, flags [DF], proto TCP
(6), length 40)
^C 172.16.106.3.49956 213.254.248.146.www: Flags [.], cksum 0x696e
(correct), seq 348423304, ack 3320774375, win 16425, length 0
226 packets captured
107740 packets received by filter
107433 packets dropped by kernel
Regards,
Saiful
From: saifu...@hotmail.com
To: rm...@free.fr
CC: squid-users@squid-cache.org
Date: Mon, 24 Jan 2011 18:06:02 +1100
Subject: RE: [squid-users] Some pages loading very slow in 3.1.10 Stable
