Re: [squid-users] Amount of Bandwidth squid can handle
Thanks for your reply I posted to mailing list so that other admins share their experiences about maximum bandwidth they could handle with squid, and finally gather the configs and spec of hardware for good reference :) Thanks for more guidance or statistic ;) Best Regards Nima Chavoshi On Thu, Jan 7, 2010 at 2:18 PM, Kinkie wrote: > On Thu, Jan 7, 2010 at 11:22 AM, nima chavooshi wrote: >> Thanks for your attention >> I want more statistic about bandwidth and hardware you can handle with squid. > > We (the developers) would like that too :) > We're trying to collect such statistics in > http://wiki.squid-cache.org/KnowledgeBase/Benchmarks . Any > contribution is welcome. > > -- > /kinkie > -- N.Chavoshi
Re: [squid-users] Amount of Bandwidth squid can handle
On Thu, Jan 7, 2010 at 12:35 PM, Travel Factory S.r.l. wrote: > >> On Thu, Jan 7, 2010 at 11:22 AM, nima chavooshi >> wrote: >> > Thanks for your attention >> > I want more statistic about bandwidth and hardware you can >> handle with squid. >> >> We (the developers) would like that too :) > > Everytime I try to think about how to benchmark squid I can't find a way that > can mimick real working conditions... > Infact I'd like to test different configurations (ram/hd) but it is almost > impossible to replicate the exact conditions (disk usage, buffers, internet > speed) > > One of the ideas is to take the access_log of a day (after completely zapping > the cache) and then have two components: a client that asks for the pages at > the time present in the log, and a server that intercepts all the squid > request that replies with the data (same size) and correct answers 302, 304 > also from the log We're not really interested in benchmarks in fact, but real-world useage data. Those are much more valuable to system administrators in defining what they need to address a real-world situation. There are projects underway to help with this efforts. See for instance http://wiki.squid-cache.org/Features/BenchmarkCacheMgrPage and http://wiki.squid-cache.org/PerformanceMeasure . Any contribution in terms of ideas, requirements, wishes, and of course code, is welcome. -- /kinkie
Re: [squid-users] Amount of Bandwidth squid can handle
On Thu, Jan 7, 2010 at 11:22 AM, nima chavooshi wrote: > Thanks for your attention > I want more statistic about bandwidth and hardware you can handle with squid. We (the developers) would like that too :) We're trying to collect such statistics in http://wiki.squid-cache.org/KnowledgeBase/Benchmarks . Any contribution is welcome. -- /kinkie
Re: [squid-users] Amount of Bandwidth squid can handle
> Apparently Wikimedia is doing 100-250Mbit/s per Squid server, according > to this presentation: > http://www.nedworks.org/~mark/presentations/san/Wikimedia%20architecture.pdf > > 55 Squid servers currently, plus 20 waiting for setup > • ~ 1 000 HTTP requests/s per server, up to 2 500 > under stress > • ~ 100 - 250 Mbit/s per server > • ~ 14 000 - 32 000 open connections per server That's a reverse proxy scenario though. A forward proxy's numbers are expected to be different (and lower). -- /kinkie
Re: [squid-users] Amount of Bandwidth squid can handle
On 6-1-2010 20:28, nima chavooshi wrote: > Hi > First of all thanks for sharing your experience on this mailing list. > I intend to install squid as forward cache in few companies with high > HTTP traffic almost 60 or 80 or 100Mb. > Can squid handle this amount of traffic??of course I do not have any > idea about selecting hardware yet. > May you tell me maximum of bandwidth you could handle with squid?it's > so good if you give me spec of your hardware that run squid on high > traffic. > > Thanks in advance > > -- > N.Chavoshi Apparently Wikimedia is doing 100-250Mbit/s per Squid server, according to this presentation: http://www.nedworks.org/~mark/presentations/san/Wikimedia%20architecture.pdf 55 Squid servers currently, plus 20 waiting for setup • ~ 1 000 HTTP requests/s per server, up to 2 500 under stress • ~ 100 - 250 Mbit/s per server • ~ 14 000 - 32 000 open connections per server -- With kind regards, Angelo Höngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist -- NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 a.hong...@netmatch.nl www.netmatch.nl --
Re: [squid-users] Amount of Bandwidth squid can handle
To build on Shawn's comments - I've handled peak loads in forward cacheing in the several hundred requests per second per Squid server, with 3.0-STABLE13 through 17 and some older 2.6 servers, as part of a smartphone company web interface. Servers were 4 GB dual Xeon quad core, running FreeBSD something for the 2.6 servers and CentOS 5.2 for the 3.0 servers we were moving towards. There were four disks in use - OS, Logs, Cache 1, and Cache 2, with no redundancy. We operated in larger cache groups initially but pared back to pairs and triplets due to operational management concerns, over time. Total cache hit rate was slightly over 50%. Peak benchmarking performance was over 600 hits/sec/server with a production log sample workload, we saw about a third to half of that as actual operational peaks (and were trying to keep margins of 2.0 from benchmarked perf to max production load). We did 100k and 1m request benchmark runs with medium sized IP pools making the queries for testing, so it was pretty good load testing, though the test harness was not optimal. -george william herbert george.herb...@gmail.com On Wed, Jan 6, 2010 at 8:14 PM, Shawn Wright wrote: > > We've been running Squid 2.6 for 5+ years with a 10Mb full duplex connection > serving ~650 active users. It has handled peak loads of 60-90 req/sec without > issue, which represents a fully utilized 10Mb link (managed with delay > pools). Last month we upgraded to a full 1Gb (yes 100x speed increase!) on a > trial basis. During a one week trial, we saw about 2-3x bandwidth use (or > 20-30Mbps sustained average) with little affect on the proxy server load. > During tests we were able to manage speedtest results of 250-300Mbps from a > single Gb connected host to Speakeasy's Seattle test node, and saw no > difference between going direct or via squid. We were also able to achieve a > full 100Mbps speed result on each of 4 simultaneous hosts tested via squid > (each was using 100Mb NIC). So far, the only issue we have seen is a problem > our log files exceeding 2Gb in less than 24 hours, which required a > re-compile to add the '--with-large-files' option. > Still far short of the 60-100Mb rates you mention (are these peak or > sustained?), but our server appears to have plenty of breathing room left, > and is modest by today's standards: > > Dell PE2850 with Dual Quad Xeons > Ubuntu 6.06 32bit, 4Gb RAM > 6x 15K 72Gb SCSI drives, 4 for cache, 1 for logs, one for system, running XFS > Squid 2.6stable20 > Single Gb NIC in use. > Lots of ACLs (300,000 lines), delay pools, all clients authenticated via AD > > I expect we will need to do more tuning since opening up the bandwidth, but > so far, things are going fine. Prior to this week's re-compile, the system > was running 24x7 since April 08. :-) > > Hope this helps. > > -- > > Shawn Wright > I.T. Manager, Shawnigan Lake School > http://www.shawnigan.ca > > > - Original Message - > From: "nima chavooshi" > To: squid-users@squid-cache.org > Sent: Wednesday, January 6, 2010 11:28:23 AM GMT -08:00 US/Canada Pacific > Subject: [squid-users] Amount of Bandwidth squid can handle > > Hi > First of all thanks for sharing your experience on this mailing list. > I intend to install squid as forward cache in few companies with high > HTTP traffic almost 60 or 80 or 100Mb. > Can squid handle this amount of traffic??of course I do not have any > idea about selecting hardware yet. > May you tell me maximum of bandwidth you could handle with squid?it's > so good if you give me spec of your hardware that run squid on high > traffic. > > Thanks in advance > > -- > N.Chavoshi > -- -george william herbert george.herb...@gmail.com
Re: [squid-users] Amount of Bandwidth squid can handle
We've been running Squid 2.6 for 5+ years with a 10Mb full duplex connection serving ~650 active users. It has handled peak loads of 60-90 req/sec without issue, which represents a fully utilized 10Mb link (managed with delay pools). Last month we upgraded to a full 1Gb (yes 100x speed increase!) on a trial basis. During a one week trial, we saw about 2-3x bandwidth use (or 20-30Mbps sustained average) with little affect on the proxy server load. During tests we were able to manage speedtest results of 250-300Mbps from a single Gb connected host to Speakeasy's Seattle test node, and saw no difference between going direct or via squid. We were also able to achieve a full 100Mbps speed result on each of 4 simultaneous hosts tested via squid (each was using 100Mb NIC). So far, the only issue we have seen is a problem our log files exceeding 2Gb in less than 24 hours, which required a re-compile to add the '--with-large-files' option. Still far short of the 60-100Mb rates you mention (are these peak or sustained?), but our server appears to have plenty of breathing room left, and is modest by today's standards: Dell PE2850 with Dual Quad Xeons Ubuntu 6.06 32bit, 4Gb RAM 6x 15K 72Gb SCSI drives, 4 for cache, 1 for logs, one for system, running XFS Squid 2.6stable20 Single Gb NIC in use. Lots of ACLs (300,000 lines), delay pools, all clients authenticated via AD I expect we will need to do more tuning since opening up the bandwidth, but so far, things are going fine. Prior to this week's re-compile, the system was running 24x7 since April 08. :-) Hope this helps. -- Shawn Wright I.T. Manager, Shawnigan Lake School http://www.shawnigan.ca - Original Message - From: "nima chavooshi" To: squid-users@squid-cache.org Sent: Wednesday, January 6, 2010 11:28:23 AM GMT -08:00 US/Canada Pacific Subject: [squid-users] Amount of Bandwidth squid can handle Hi First of all thanks for sharing your experience on this mailing list. I intend to install squid as forward cache in few companies with high HTTP traffic almost 60 or 80 or 100Mb. Can squid handle this amount of traffic??of course I do not have any idea about selecting hardware yet. May you tell me maximum of bandwidth you could handle with squid?it's so good if you give me spec of your hardware that run squid on high traffic. Thanks in advance -- N.Chavoshi
