It's trivial to run a wget or curl on the same server that the squid proxy is
on and access pages through it, directing the output to /dev/null, in order to
prime the cache. But there's no explicit way to tell squid to "please pull this
URL into your cache" without an actual HTTP request for tha
Another good use for cache clustering is when you need squid to scale beyond a
single core* on your proxy server; you can run multiple squid instances on a
single box and cluster them to avoid duplicate storage.
-C
*Yes, I know I/O is threaded, but the main event loop is not...
On May 1, 2010,
To split the hairs a bit further, squid's core is mostly single threaded, but
does have disk i/o processes that are spawned out. So going from single to dual
core will give you gains for that reason in addition to the benefit of other
system processes no longer competing with squid for timeslice
All,
Forgive me if I'm misremembering, but I have a vague recollection of some
discussion on this list of some individuals or organization using squid's
offline mode as a way to bring a (admittedly limited) subset of internet
content to a remote internet-inaccessible areas - a bookmobile for th
One thing we've noticed is that until the COSS store is rebuilt, no objects get
saved to it (each attempt shows up as SO_FAIL in cache.log). Complicating this
further is the fact that -F doesn't seem to work for COSS rebuilds - I know I
opened a bug on this but I can't seem to find it at the mom
It can. Best bet is to install via the MacPorts packaging system, but normal
configure/make/make install works for me too.
-C
On Dec 16, 2009, at 8:45 PM, Jeff Pang wrote:
> Can squid be installed and run on a mac os box? Thanks.
>
> --
> Jeff Pang
> http://home.arcor.de/pangj/
Barring the development of an in-code fix, I think your best bet is to
take the config generator route you describe, but then run the
generator from your squid initscript triggered by "start" or "reload".
Beyond pushing out an updated text file to your boxes and building the
cache_peer line
Hi,
Had a quick question re: the output of the mgr:carp squidclient query.
Below is a sample of what we're seeing on our child squid:
Hostname Hash Multiplier Factor Actual
http-cache-1a ba95c958 1.00 0.17 0.169094
http-cache-1b d6cf90fc 1.00 0.17
If you need multiple SSL certs, you need a different IP/tcp port combo
for each certificate.
If all your backend servers are within a single domain, a wildcard
cert may do the trick.
-C
On Jun 30, 2009, at 3:07 PM, Mario Remy Almeida wrote:
Hi All,
Would like to know if its possible to
Are the requests coming in at *exactly* the same time? If not,
collapsed_fowarding should help if you're running 2.x...
http://www.squid-cache.org/Versions/v2/2.7/cfgman/collapsed_forwarding.html
-C
On Jun 29, 2009, at 8:33 PM, Judson wrote:
I'm working with an application that generates a
It's really a squid issue, not an Adobe issue, assuming that you're
viewing the .pdf in-browser via the Reader plugin (as opposed to
downloading, then opening)...
http://www.squid-cache.org/bugs/show_bug.cgi?id=2639
The issue is that the Acrobat plugin requests multiple byte ranges of
the
Take a careful look at the stale-if-error Cache-control header, as
described below:
http://tools.ietf.org/html/draft-nottingham-http-stale-if-error-01
In a nutshell, this allows you to force squid to serve up objects if
the origin is down, even if those objects are stale, for a
configurabl
Since we're on the subject, a couple points to share...
1. If you're delivering unique objects based on the User-Agent, make
sure that you either mark the content no-cache, or make sure that each
different object has a unique ETag. IIRC, Adding Vary: User-Agent is
mandatory as well if you'r
There is no code in squid to "transform" content inside the cache
beyond headers. The development path for content transformation (of
which gzip compression is one of many potential examples of) is via
ICAP services (3.0 and above) and ECAP plugins (3.1).
That said, squid is 100% open sourc
Squid isn't going to do that natively; this is the domain of ICAP or
ECAP (supported in 3.0 and 3.1beta, respectively).
http://wiki.squid-cache.org/Features/ICAP
-C
On Jun 5, 2009, at 8:08 AM, Luis Daniel Lucio Quiroz wrote:
hi Squids,
I wonder to know if there is a way to reduce size of o
As explained to me, the issue is that retrieving successive 4K blocks
of an object in memory takes non-linear amounts of CPU time - it takes
X cycles to retrieve the first 4K, but 2*X to retrieve the second 4K,
3*X to retrieve the third 4K, etc. etc.
-C
On May 24, 2009, at 12:19 PM, Dror G
I can attest to stable performance under squid 2.7STABLE. The only
outstanding bugs I've observed are:
1. COSS is always dirty when squid is shut down, so it must be rebuilt
every startup
2. -F flag does not wait for COSS rebuild before binding ports
Item 2 has a patch pending, coming to a
A couple lessons learned from my end, both in my own experience and
picked up from various squid-users threads...
I've said this before, but never underestimate the value of kernel
page cache. If you need to scale the box, put in as much RAM as you
can afford.
Also, as has been said befor
One common mistake (I made it myself with my first rewriter, in fact)
is to buffer the helper's STDOUT. Make sure you either disable
buffering entirely (in perl, you can do this with a "$| = 1;"
statement) or make sure you flush() STDOUT after printing OK/ERR.
-C
On May 12, 2009, at 9:23 A
Squid indexes content by its URL, so in all honesty, the best way to
get an object into squid's storage is to just request it through the
proxy. This is easily scriptable via the curl and wget command-line
tools, or frameworks like perl's LWP.
-C
On May 9, 2009, at 10:04 PM, Laurent Luce w
On May 6, 2009, at 8:14 PM, Amos Jeffries wrote:
Hi,
I've noticed a behavior in CARP failover (on 2.7) that I was
wondering
if someone could explain.
In my test environment, I have a non-caching squid configured with
multiple CARP parent caches - two servers, three per box (listening
on
Hi,
I've noticed a behavior in CARP failover (on 2.7) that I was wondering
if someone could explain.
In my test environment, I have a non-caching squid configured with
multiple CARP parent caches - two servers, three per box (listening on
ports 1080/1081/1082, respectively, for a total of
I'm not putting the include file path in double quotes under
2.7STABLE6 and things are working fine here (although I can see a file
path with, say, spaces not working). Is 3.x any different in this
regard?
You do need an absolute path, however; using a relative path will
result in the fil
ntry in place for this.
-C
On Apr 22, 2009, at 10:00 AM, Chris Woodfield wrote:
...and sure enough, it's right there in -h output...
cache$ /usr/local/squid/sbin/squid -h
...
-FDon't serve any requests until store is rebuilt.
...
/me goes to write "I
le requests until the
store is rebuilt - its the "-F" flag.
I'm fixing the store rebuild times in Lusca-HEAD at the moment and
this includes writing some new COSS rebuild-from-index, rebuild-from-
log
and rebuild-from-rawdevice tools.
Adrian
On Wed, Apr 22, 2009, Chris Woodfie
On Apr 22, 2009, at 4:56 AM, Amos Jeffries wrote:
Chris Woodfield wrote:
So I'm running with COSS under 2.7STABLE6, we've noticed (as I can
see others have, teh Googles tell me so) that the COSS rebuild a.
happens every time squid is restarted, and b. takes quite a while
i
So I'm running with COSS under 2.7STABLE6, we've noticed (as I can see
others have, teh Googles tell me so) that the COSS rebuild a. happens
every time squid is restarted, and b. takes quite a while if the COSS
stripes are large. However, I've noticed that while the stripes are
being rebuil
On Apr 17, 2009, at 7:45 PM, Amos Jeffries wrote:
Chris Woodfield wrote:
Hi,
We've noticed that when a request is sent that has multiple byte
ranges in the Range: header, the behavior is not what one would
expect.
If one requests multiple byte ranges that are sequential and d
Hi,
We've noticed that when a request is sent that has multiple byte
ranges in the Range: header, the behavior is not what one would expect.
If one requests multiple byte ranges that are sequential and do not
overlap (i.e. Range: bytes=1-20,30-50), the response is the expected
206 Partial
So now that this behavior has a name, I looked and noticed that per
the 2.7 docs, collapsed_forwarding defaults to off, and isn't enabled
in our config either. Does running squid in reverse proxy mode
implicitly turn this on?
-C
On Apr 10, 2009, at 12:26 AM, Amos Jeffries wrote:
Hi,
I've noticed that either by design or as a side-effect of squid's
caching that if I request the same object from multiple clients at the
same time, squid will effectively "multiplex" the transfer - that is,
use a single transfer from origin to feed the object to each client as
it rece
Easy answer - run sarg as a crontab.
Harder answer - compile in SNMP support, and run RRDTool/Cacti/etc to
generate charts based on squid's SNMP counters for the relevant metrics.
-C
On Apr 2, 2009, at 7:01 AM, sameer shinde wrote:
On Wed, Apr 1, 2009 at 10:32 PM, Eric Valencia > wrote:
I'
One performance-specific 2.7 question - I recall hearing mention of an
issue with 2.6 where larger objects held in mem_cache required
exponentially more CPU cycles to serve up (i.e. n cycles for a 4KB
object, n*2 for an 8KB object, n*4 for a 12KB object, etc). Does
anyone know if this issue
Looking at the response headers, it's possible that the URL isn't
being cached at all by squid because there's no Last-Modified or Cache-
Control: max-age header. IIRC (Amos, correct me if I'm wrong here),
Squid will assume that content without a Last-Modified header is
dynamic content and n
Think of read_ahead_gap as a maximum buffer size for incoming data
from origin sites to squid.
The main benefit of read_ahead_gap is that if your origin is burst or
prone to stalling, squid will buffer more of the incoming data stream
in order to deliver content more smoothly to clients. If
This is from a 64-bit 2.7STABLE6 instance, as part of the squidclient
mgr:info output. We're seeing values similar to these on every squid
we operate.
Request Hit Ratios: 5min: 97.7%, 60min: 97.3%
Byte Hit Ratios:5min: 93.2%, 60min: 95.8%
Request Memory Hit
squid doesn't support http 1.1 from cache to client. squid 2.7
supports 1.1 from cache to origin servers, but cannot pass through
chunked transfer-encodings. (It's the lack of support for this that
prevents it from advertising 1.0 to clients). However, just about
every other 1.1 function is
Hi,
Had a question about squid's CARP implementation.
Let's say I have a farm of squids sitting behind an SLB, and behind
those I have a set of parent caches. If I were to enable CARP on the
front-end caches, is the hash algorithm deterministic enough to result
in a URL request seen by mor
On Mar 16, 2009, at 8:13 PM, Amos Jeffries wrote:
To elaborate, squid should restart new url_rewrite_program instances
when the number of live children falls to <= 50% of the configured
number. So once 8 processes out of 15 die, squid should launch a
whole
new set of 15. You'll then have 23
On Mar 16, 2009, at 9:07 PM, Amos Jeffries wrote:
Hi,
I'm looking into setting up cache peering - I currently have small
sets of reverse-proxy squids sitting behind a load balancer, with no
URI hashing or other content-based switching in play (thanks to a
nice
bug/feature in Foundry's IOS t
You can use them together, but you can't bind squid to the standard
SNMP UDP/161 port if snmpd is also bound to that port.
In my setup, I have snmpd configured to proxy requests for squid's MIB
to squid, which is listening on localhost:1610:
squid.conf:
acl snmpcommunity snmp_community foob
Hi,
I'm looking into setting up cache peering - I currently have small
sets of reverse-proxy squids sitting behind a load balancer, with no
URI hashing or other content-based switching in play (thanks to a nice
bug/feature in Foundry's IOS that prevents "graceful" rehashing when
new serve
To elaborate, squid should restart new url_rewrite_program instances
when the number of live children falls to <= 50% of the configured
number. So once 8 processes out of 15 die, squid should launch a whole
new set of 15. You'll then have 23 url_rewriter processes, but squid
will launch 15
2.7 supports the "stale-if-error" cache-control directive, which will
accomplish this goal. The only caveat (AFAIK) is that it will only
continue to serve objects if origin returns a 500 server error or if
the origin is unreachable; if the origin returns a 404 it will flush
the object and p
Hi,
We've noticed that when downloading files through squid 2.7 (as a
reverse proxy) via MSIE 7 or 8 beta, MSIE will not resume partially-
completed downloads, which it does normally when downloading from
other web servers. This appears to be MSIE-specific behavior - the
second request has
Hi, this is a quick one -
Does squid properly grok the date format of an Expires: header where
the time zone is specified numerically (i.e. "-0400") instead of by
letter code (i.e. "EDT")?
-C
Hi,
We've been noticing lately that the logrotation process is taking
longer and longer as our caches fill up - currently, with ~18 million
on-disk objects, we've seen it take as long as 12 seconds, during
which time squid is not answering queries.
Searching on this issue found the follow
Hi,
Can someone confirm whether Expires: or Cache-control: max-age
parameters take precedence when both are present in an object's
headers? My assumption would be Cache-control: max-age would be
preferred, but we're seeing some behavior that suggests otherwise.
Specifically, we're seeing
Personally, I think 8 cores is most likely overkill unless you've got
some seriously cpu-intensive helper apps - squid on its own tends to
block on I/O far earlier than it blocks on CPU. You're more likely to
get more bang for the buck by going with 4 cores and extra RAM which
the kernel wi
Squid does not do transfer encoding of objects on its own; however, it
will cache and serve multiple encodings of the same object *as
delivered to the cache by the origin* in order to send the requested
encoding to clients. This is dependent on the origin server sending
the proper Vary: Acc
squidpurge works, but it's hardly ideal, especially on squids with big
disks...in my testing on a box with 3x1TB cache_dirs, it took 15
minutes to run and thrashed the disks pretty hard while it was
running, affecting response time for production traffic.
The reason for this is that squid s
Hi,
We're testing a migration from 2.6 to 2.7, primarily to get the HTTP/
1.1 header in squid-to-origin requests in a reverse proxy environment.
Per our test plan, we migrated one of two servers sitting behind a
load balancer to 2.7STABLE3 with the patch for bug 2406 applied; that
bugfix a
, 2008, at 6:38 PM, Henrik Nordstrom wrote:
On ons, 2008-07-16 at 09:34 +0200, Kinkie wrote:
On Tue, Jul 15, 2008 at 11:03 PM, Mark Nottingham <[EMAIL PROTECTED]
inc.com> wrote:
+1 - both cases are useful.
On 10/07/2008, at 1:15 AM, Chris Woodfield wrote:
Bug filed, #2406.
As I annota
It's pretty easy to grab the URL of the logo and configure jesred to
replace it with a URL pointing to your own logo. However, that's no
guarantee that Google won't change their logo URL at any time - at
which point your rewrite will break until you fix your rule to match.
-C
On Jul 17, 20
While more/faster is always better, that's not what I'm arguing - I'll
take the fastest CPU budget allows and as many cores as they'll let me
buy :) In particular, more CPU *does* help when you have dozens of
helper apps such as authenticators or url rewriters.
I think the point of the argu
Hi,
One thing to keep in mind is that in my experience, it makes sense to
not only get fast disks, but put as much RAM in the box you can
afford. Now *don't* give this all the squid via the mem_cache config;
let the OS use the spare memory for caching disk reads. This will spee
Additional
I seem to remember having a similar problem when calling a URL
rewriter with command-line arguments; I solved it by having squid call
a shell script instead that had the actual rewriter + arguments on an
exec line. I later rewrote the helper app to read a config file in
lieu of command-line
o pre-
rewrite if desired. If no rewriter is configured, this element would
print null.
-C
On Jul 8, 2008, at 6:36 PM, Henrik Nordstrom wrote:
On tis, 2008-07-08 at 16:47 -0400, Chris Woodfield wrote:
I've noticed that squid 2.7STABLE3 logs incoming URLs differently
than
2.6 did wh
Hi,
I've noticed that squid 2.7STABLE3 logs incoming URLs differently than
2.6 did when using a url_rewrite_program. It appears that under 2.6,
the URL logged was pre-rewrite, under 2.7 it's the URL returned by the
rewriter. This presents problems as I have the potential for a large
numbe
So we're looking to upgrade from 2.6 to 2.7, primarily to get the HTTP/
1.1 header support. I realize that the full 1.1 spec is not completely
implemented, but are there any real "Danger, Will Robinson!"
implications?
Specifically, is there any functionality or access to content that
would
Replying a bit late to this...
One interesting thing that the Flash player will do is that if it
finds itself unable to connect directly on 1935, 80, or 443 (for a
site, that, let's say, is blocking 1935 and proxying 80/443 with
squid), it will attempt to tunnel RTMP data inside HTTP GETs a
Hi,
I've had a few folks ask me what exactly is missing from squid
(2.6STABLE18 in our case) that prevents it from advertising itself as
an HTTP/1.1 client. We've run across the occasional origin server
that, for whatever reason, is throwing away *all* 1.1-specific headers
- including Hos
ks for the help.
-C
On Apr 11, 2008, at 6:22 PM, Henrik Nordstrom wrote:
fre 2008-04-11 klockan 12:43 -0400 skrev Chris Woodfield:
Further poking suggests that this is due to how the object is
delivered when it is first loaded into the cache. For example, when I
purge the object from the ser
More to the point, would a vendor only be obligated to provide the
source code directly (as opposed to providing a link to, say, squid's
main source repository) if the vendor made modifications to said
source code? Or is the vendor obligated to provide the source code
directly regardless, e
Hi,
We have a farm of squid servers (2.6STABLE18) sitting behind a load
balancer, all with identical configurations but obviously their own
cache disks. We're noticing that some servers are honoring "Accept-
Encoding: gzip" on particular objects, but not others. The effect is,
some servers
For our purposes (reverse proxy usage) we don't see any missing
features from squid 3 that we would need - however, we'd like to see
the code base mature some more before we trust it in production. Same
reason that smart folks don't deploy new Cisco IOS trains until it
hits the 3rd or 4th r
A near hit is a validated cache miss - the object was stale, but squid
did a Get with If-Modified-Since to the origin and received a 304 Not
Modified, which resets the refresh timer on the object. You'll see
these as TCP_REFRESH_HIT in the access log.
-C
On Mar 10, 2008, at 12:18 AM, J. P
I'll readily admit that I Am Not A Developer, but I'm wondering if
this could be something that could be worked incrementally - finding
easy-to-cleave-off subsystems that can be moved to separate threads
similarly to how asyncio was. The most obvious one I can think of is
the front-end clie
at are added or removed
from cache storage?
-C
On Mar 4, 2008, at 10:52 PM, Adrian Chadd wrote:
Check to see if the object is actually in cache. I bet that the
RELEASE line you're seeing is the temporary store entry that was
created purely to return the 304 message.
Adrian
On Tue,
I haven't tried this myself, but can't you just have two cache-peer
lines with the same host but different port numbers?
-C
On Mar 4, 2008, at 5:11 PM, Nick Duda wrote:
I seem to be stumped. I need to reverse proxy for one internal server
that listens on both 80 and 443. How can I configure
Hi,
We recently added the "reload-into-ims" directive to our squid config
after noticing that a large number of queries were coming in with No-
Cache set, killing our cache efficiency. We have a relatively short
max-age set, working on the assumption that the If-Modified-Since will
keep th
We decided that we wanted to cache files up to 5GB (DVD ISOs), and in
the process noticed this compile-time flag:
--enable-large-cache-files Enable support for large cache files (>2GB).
WARNING: on-disk cache format is changed by
this option
We're running a 64-bit
First off, when you run squid -k rotate, squid sends EOF to all
rewriter children and starts new ones. This is coded behavior,
although it's a simple patch to disable if you want - just remove the
calls to redirectShutdown() and redirectInit() from the mainRotate()
function.
As far as why
Before you go and say "wow, I want more MEM_HIT, increase mem_cache!",
remember that if you have a decent amount of spare memory, the kernel
will use that as page cache, so it's a good change that frequenty-
requested items will, in fact, be served from RAM (via the page cache)
even if it's
Your best bet, in my experience is simply to run tcpdump on the box
and examine the results with Wireshark...
-C
On Feb 7, 2008, at 3:14 PM, mista_eng wrote:
How can I see the actual headers being sent/received during the
dying file
request? I have already performed a "less /var/log/squid/a
On Feb 6, 2008, at 3:44 AM, Adrian Chadd wrote:
I see Apache can also do reverse proxy, which was surprising to me,
or
is it not quite the same thing?
Sort of. :)
Apache's ProxyPass module performs similar proxying functionality, but
without squid's caching abilities, so you won't ge
8, at 6:57 AM, Amos Jeffries wrote:
Chris Woodfield wrote:
Hi all,
I'm facing an issue where we'd like to implement cache peering on
our squid farms, primarily to leverage the combined disk capacity
across all our boxes into a larger cache. I would presume that this
requires the u
Hi all,
I'm facing an issue where we'd like to implement cache peering on our
squid farms, primarily to leverage the combined disk capacity across
all our boxes into a larger cache. I would presume that this requires
the use of the proxy-only directive to avoid content duplication.
Howeve
I just put a squid system with url_rewriter children into production.
Alongside this we have a script that regularly runs squid -k rotate,
then FTPs the log.1 files to a remote site for backup/processing.
The issue I've noticed is that every time squid -k rotate is run,
squid also stops and
Interesting. What sort of size threshold do you see where performance
begins to drop off? Is it just a matter of larger objects reducing
hitrate (due to few objects being cacheable in memory) or a bottleneck
in squid itself that causes issues?
-C
On Jan 28, 2008, at 8:00 PM, Adrian Chadd w
Hi,
Reading the squid FAQ, it's obvious to me that putting cache_dirs on a
RAID (particularly RAID5) has serious performance penalties and is
highly discouraged. However, what's not as clear is how squid deals
with single-disk failures and whether or not it handles failures
gracefully eno
This does bring an interesting question - is it possible to give squid
*too much* memory?
My theoretical setup would be an uber-box (32GB RAM, multi-TB of disk)
running 64-bit squid and with mem_cache set to something in the
25-30GB range (as high as we can without swap risk), with a
maxi
I did, setting the handler to "exit 0", the rewriter now catches
EOF as an error in ReadWrite and dies properly as it should.
-C
On Jan 18, 2008, at 7:46 PM, Chris Woodfield wrote:
Hi all,
I'm working on a custom url_rewriter helper app, written in perl,
that translates incoming UR
Hi all,
I'm working on a custom url_rewriter helper app, written in perl, that
translates incoming URL requests to custom URLs using a config file
that is loaded at startup and is polled for changes at regular
intervals.
In order to handle the update polling, the script uses POE for event
84 matches
Mail list logo
