Re: GRSEC and Varnish
On Tue, Feb 02, 2010 at 04:44:48PM +0100, Bernardf FRIT wrote: Hi, I'am running : - varnishd (varnish-2.0.4) Why not 2.0.6? and it appears that the grsec Kernel repeatedly and unexpectedly sends signal 11 to the varnishd child. grsec seems to just report that a segfault occurred. SIGSEG would be a strange signal to send in any event. You want to fetch yourself a core-dump of this. However, before we get into that, I'd like to know what parameters you start Varnish with, and the general setup. VCL too, if possible. -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgpFwMaQMo7wh.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Cache utilization?
On Wed, Nov 18, 2009 at 01:02:32PM +, Poul-Henning Kamp wrote: In message f066603d-901b-4e4d-ac37-e5890c963...@gyldendal.dk, =?iso-8859-1?Q? Lars_J=F8rgensen?= writes: Hi, Obvious question but I can't find the answer anywhere: How do I know how much of my cache is utilized, i.e. is my cache file big enough? You can see if it is too small, because you will get LRU kill activity in varnishstat. There is not really a good way to see if your cache is to big, and apart from diskspace, there is no disadvantage to that. Well, I usually look at resident memory usage. That, and/or the allocated/free memory in varnishstat: If it never gets closed to filling, reducing the size could make sense. Or to turn it around: Buying more memory isn't likely to help you. (Though LRU would've told you that too.) -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgp4CPdtL0Olm.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnishlog not behaving as expected (by me)
On Wed, Jan 13, 2010 at 11:24:07AM +, Alex Hooper wrote: I'm having a bit of trouble using varnishlog. I am using: /usr/local/bin/varnishlog -o RxURL ^wp-admin$ All your urls start with /, so that will never match. Try ^/wp-admin and you should have better luck. (...) '/usr/local/bin/varnishlog' -i RxURL shows me the requests coming in. '/usr/local/bin/varnishlog -i RxURL|grep wp-admin' shows that there are generally no requests for wp-admin. Use the above approach instead. And how much traffic do you have? When you pipe it (for instance to grep), varnishlog will be buffered. -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgp4XQWKf4WCu.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: vim syntax
On Tue, Nov 24, 2009 at 07:57:54PM +0200, Elan Ruusamäe wrote: @phk mentioned in the irc, that somebody has written varnish syntax for vim. i tried to search the archives but found no info about it. anyone knows? would be pity to start writing one from scratch if somebody already spend some time of yours on it. I've heard rumours too, but never seen it. Since it's close to C, I usually get by with C-syntax. There are some differences (ie: set foo = bar, not foo = bar, and ~...) but I suppose using C as a base will save you a lot of time. If you do write one, drop it here and I wouldn't mind testing it, I do work with VCL from time to time ;) -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgpRDNFNWUitK.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange different behavior
On Wed, Jan 13, 2010 at 06:45:26PM +0100, Bernardf FRIT wrote: Hi, I'm facing a pretty strange issue. URLs not supposed to be cached are in cache when requested by firefox but not by lwp. (...) Vary: Accept-Encoding,User-Agent This is why. Vary: means this page will look different depending on the following client headers. Your backend is telling Varnish to expect different variants of the page depending on what Accept-Encoding AND User-Agent header the client provides. Accept-Encoding is normal and sane (ie: if one client supports gzip and an other does not, one client will get a compressed variant of the page and the other client the uncompressed one). Vary on User-Agent is generally bad, and you should Just Fix That [tm]. Due to Vary: Accept-Encoding, lwp-request and Firefox will still get different variants, but you can supply the Accept-Encoding header to lwp-request, but if you want the content, you'll need to pipe it through gunzip (headers are still readable). -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgpvY82yV0SD3.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strange 302 redirect to google.com
On Mon, Jan 11, 2010 at 01:57:27PM +0200, Maurice wrote: i have a problem with a strange 302 redirect to google.(com|ro) just by visiting http://dir.acasa.ro on google chrome and ie, on firefox everything works fine. the strange part is that on certain ips (not listed in the vcl so nothing fancy) the page works on all browsers. is it possible that varnish caches the 302 redirects and on a get index request and spits out 302? but why redirect to google? I just had a look at this issue and can't reproduce it, nor do I see any good reason for why it would happen based on what you provide. Varnish can/does cache 302s, but that wouldn't explain why they are there in the first place. The varnishlog you provide does have any Location: header that links to Google, and after running through the list, none of those sites refer to Google either. And I can promise you that Varnish does not have any ties to Google out of the box. How consistently does this happen? What other systems are involved (loadbalancers,etc)? What sort of pattern do you see when it does happen? If you can reproduce it, seeing output from GET/HEAD and/or firebug might help reveal the underlying issue too. -- Kristian Lyngstøl Redpill Linpro AS Mob: +47 99014497 pgpThttlgNTbp.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Saint mode
Hi Anders,
On Wed, Nov 18, 2009 at 02:52:39PM +0100, Anders Nordby wrote:
We spoke briefly about in the Varnish meeting. Unfortunately I can not
remember the conclusion(s), so I ask again:
1) How to check in vcl_recv if a request was restarted saintmode? You
mentioned something about checking TTL? Does it matter if that request
is handled with pass or lookup? I want to avoid mixing up restarts due
to saint mode and restarts due to what should have been normal passes in
vcl_fetch (IMO) but which are restarts to handle it in vcl_recv to avoid
hit for pass problems.
In vcl_recv we don't have a ttl, so you can't specifically check. You could
do something simple like:
sub vcl_fetch {
if (beresp.status != 200) {
set req.http.X-Saint-restart = 1;
set beresp.saintmode = 1m;
restart;
}
...
}
That way you can easily check in recv. It's not perfect, but unless we have
the TTL, that's the only way.
In vcl_hit you can check if obj.ttl is negative, if it is, then you are
using a graced object.
2) How do use saint mode when backend fails to respond, covering
everything from no response/connection to backend just resetting the
connection? As far as my syslogging shows, connection failures are adressed
in vcl_error and easy/possible to catch in vcl_fetch?
That's an excellent question. Implementation-wise saint-mode can handle
this, but we loose a bit too much information before we reach vcl_error, it
seems.
Saint-mode uses both the backend and the objecthead(reference) to work, and
we detach it a bit too soon.
PHK: Do you have any idea how to solve this? We've talked about making more
information available in vcl_error before, or just using vcl_fetch...
Cheers from Gyoda, Saitama, Japan.
Cheers :) Just finished 2.5 weeks of various forms of training and
traveling myself, trying to catch up on mail and whatnot. Apologies for the
somewhat delayed response.
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
pgpQegDLZ7t5K.pgp
Description: PGP signature
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Benchmarking and stress testing Varnish
As some of you might already know, I regular stress tests of Varnish, most of the time it's a matter of testing the same thing over and over to ensure that there aren't any huge surprises during the development of Varnish (we run nightly builds and stress tests of trunk - they need to be as predictable as possible to have any value). However, I also do occasional tests which involve trying to find the various breaking points of Varnish. Last time I did this, I used roughly 42 cpu cores spread over about 30 computers to generate traffic against a single Varnish server on our quad xenon core machine. The result thus far was that all of the clients ran far too close to 100% cpu usage, and Varnish was serving between 140 and 150 thousand requests per second. The reason I'm telling you this is because I'm looking for input on aspects that should be tested next time I do this, which will most likely be during Christmas (far easier to borrow machine power). So far on my list, I've got: 1. Test performance over some time period when pages are evicted more frequently. (ie: X thousand pages requested repeatedly, but random expiry time). 2. Test with fewer requests per session (this is somewhat hard because the clients tend to turn into the bottleneck). 3. Test raw hot-hit cache rate (more of what I did before - get a high number). 4. Test raw performance with a huge data set that's bound to be swapped out. 5. Various tests of -sfile versus -smalloc and large datasets, combined with critbit/classic tests. 6. Find some reasonably optimal settings, then fidget around trying to find a worst-case scenario. 7. Test the effect of session lingering with really slow clients. One thing that should be noted is that the test server is limited to 1gbit, which means that for raw req/s, we're basically forced to use tiny pages, or we just end up starving the network. The goal is to test theories regarding performance, stability and predictability. Basically find the breaking points, what's good and what's not, and what we have to care about and what we can silently ignore. As you can see, the list is getting long and this is off the top of my head, but any suggestions are welcome. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpLMdbIyku9d.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish stuck on stresstest/approved by real traffic
(Excessive trimming ahead. Whoohoo) On Tue, Nov 03, 2009 at 11:51:22AM +0100, Václav Bílek wrote: When testing varnish throughput and scalability I have found strange varnish behavior. What's the cpu load at that point? Also: use sess_linger. No session_linger == kaboom when things get too loaded. It's 50ms default in 2.0.5/trunk, but set to 0ms in 2.0.4 and previous. The behaviour in trunk is slightly different/better, but it's still worth using it in 2.0.4. - Kristian pgpAz5VGK4rBe.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: rewrite redirects
On Fri, Aug 21, 2009 at 03:52:12PM +0200, Guenter Wildmann wrote: Hello! Can varnish rewrite redirects from the backend (equivalent to ProxyPassReverse in apache)? If so, how is it done? Assuming the redirects are 302/301s; yes. Check for them in vcl_fetch, then do the magic there, but you may have to elaborate as to what precisely you want. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpIBnCc36GXW.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
On Mon, Sep 21, 2009 at 02:55:07PM +0300, Roi Avinoam wrote: At Metacafe we're testing the integration with Varnish, and I was tasked with benchmarking our Varnish setup. I intentionally over-flooded the server with requests, in an attempt to see how the system will behave under extensive traffic. Surprisingly, the server ran out of swap and crashed. That seems mighty strange. What sort of tests did you do? In out configuration, -s file,/var/lib/varnish/varnish_storage.bin,1G. Does it mean Varnish shouldn't use more than 1GB of the virtual memory? Is there any other way to limit the memory/storage usage? If you are using -s file and you have 4GB of memory, you are telling Varnish to create a _file_ of 1GB, and it's up to the kernel what it keeps in memory or not. If you actually run out of memory with this setup, you've either hit a bug (need more details first), or you're doing something strange like having the mmaped file (/var/lib/varnish/) in tmpfs with a sizelimit less than 1GB or something along those lines. But I need more details to say anything for certain. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgphXG7YeoQJX.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish virtual memory usage
On Wed, Oct 21, 2009 at 03:07:00PM +0200, Roi Avinoam wrote: Thanks for your reply. 1. What I did was create 100 simultaneous processes, and each process requested the same page (with 'curl'): a. Once with the exact same URL - which resulted in a 99.9% hit-ratio and VERY high performance on varnish. b. And once with a random key that changes the URL (something like 'index.php?rand=193837364'), thus forcing Varnish to hit the backend, and store the multiple objects in memory. c. A combination of the two - in an attempt to maintain a 60-70% hit-ratio. What we saw is that the kernel simply filled all of the RAM *and* the swap until it crashed. When did it fill up and how fast? 2. I'm sorry, but I'm still confused about the mmaped file. If it's limited to 1G, Varnish shouldn't use more than 1G of virtual memory, correct? Or in our setup - 1G of RAM? The -s parameter only applies to storage, which is to say that it's an approximiation and there will be some additional overhead which is not stored there, ie datastructures for threads, sessions, backends etc. So with -s ..., 1G, you should expect a little bit over 1GB, but not nearly as much as you describe. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpBkeZKEmnEc.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Restructuring the Wiki
As we talked about during the VUG meeting, the Wiki could use some love. Unless anyone has any strong objections, I'll restructure it shortly. I'll probably rewrite, rename and add a considerable amount of information, so I apologize in advance for the inconvenience. I hope to avoid removing data unless it's incorrect, obsolete or otherwise unwanted. I did a quick tour of the Wiki, and figured I'd divide the information into categories similar to the following (these are my raw notes, so fill in your own blanks): Wiki - Overview - News - Varnish Major.minor.patch - ? - Features - Architect Notes - Version N - Current - Frequently declined - Shopping List - Planned - FAQ - Source - Community - IRC / ML - Documentation - Introduction - Reference documentation - Options - CLI - VCL - Request Flowcharts - Guides - Getting started - Preparing for production - OS tweaks - Tips and tricks for hit-rate - Examples - Reference examples (VCL examples demonstrating specific features) - General VCL example-snippets - Complete examples - Reviewed - Raw - Tuning examples (Think various parameter-tweaks) - Develop - Debugging Varnish - (most of what is under DeveloperResources) - Resources - Varnish-cache.com - Commercial support - Accelerating wordpress with varnish, etc - trouble log - void? Or fix? - who uses... ? This is likely to change during the actual work, but it should give you an idea at what I'm aiming at. My goal is to make it easy to find what you are looking for and make the front page useful both for first time visitors and for those of us who use the Wiki on a daily basis. Knowing myself and how I've done wiki-restructuring in the past, I expect the majority of the work will happen in one or two evenings, so thou has been warned. So again: comments, objections etc. are welcome, as you're the people who will use this wiki and hopefully help improve it. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpVZaH830w4q.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish User Group Meeting 2009-09
On Fri, Aug 07, 2009 at 12:08:38PM +0200, Tollef Fog Heen wrote: On September 21st and 22nd, the first Varnish User Group meeting will be held, in Canonical Ltd's offices in Millbank Tower, London, UK. Please see http://varnish.projects.linpro.no/wiki/200909UserGroupMeeting A little update, since we seem to have forgotten to mention it: We will begin at 09:00 London-time and keep going through the day. Canonical have been kind enough to lend us the meeting room we'll be using. See you there :) -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpgI9QiwPkTW.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: travel: eurobsdcon2009 and vug1
On Thu, Sep 17, 2009 at 09:10:21AM +0200, Eivind Bengtsson wrote: What is the exact address (Millbank tower - yes but which floor/company :-)) Which hotel are you guys staying at ? (can you recommend) I can't answer for everyone, but I'm staying at Best Western Corona Hotel. I believe Tollef is too (?). As for location: It's Canonical's offices, if memory serves me right, that's the 28th floor. But I might be mistaken on the floor number. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp6KTMB8uQAZ.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: trunk and obj.ttl obj.grace obj.http.set-cookie
On Thu, Sep 17, 2009 at 05:02:37PM +0200, Václav Bílek wrote: I have tried trunk releas and hit problem vith VLC which worked in 2.0.4... Variable 'obj.grace' not accessible in method 'vcl_fetch' Variable 'obj.http.set-cookie' not accessible in method 'vcl_fetch' Variable 'obj.ttl' not accessible in method 'vcl_fetch' did the syntax changed? Yup, in fetch, most of what used to be 'obj' is now available as 'beresp'. So: beresp.grace, beresp.http.*, beresp.ttl in your case. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpgjbM5NAXUi.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: #551: Varnish Crash: Missing errorhandling code in HSH_Prepare(), cache_hash.c line 188
On Wed, Sep 16, 2009 at 09:54:25AM -0700, Ken Brownfield wrote: Ah, I stand corrected. But I was definitely having random crashes when I enabled the vcl_fetch() section below: (...) I'm a bit loathe to reenable this to get a full stacktrace and gdb output, but if there's really nothing wrong with this I might consider it. Nothing wrong with it, but my first guess would be obj_workspace being overloaded, which would look similar to #551 unless you know exactly what to look for. Also, using trunk (a couple weeks ago) I can't reference obj in vcl_fetch() at all, which I assumed was an intentional side-step of the #310 bug. Nope, it's just renamed beresp. For now, it's still the same thing for all intents and purposes, but the idea is to only fetch headers so as to be able to go from fetch to pipe in some unknown future. - Kristian pgpZgO6nxYTwZ.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Content-Range header problem
Hi, On Wed, Sep 16, 2009 at 01:16:20PM -0400, Nathan Kinkade wrote: For some reason Varnish (2.0.4) doesn't seem to be respecting the Content-Range header, so our users are unable to resume large downloads that somehow got interrupted. Is this expected and normal behavior, or should this work? Do we need to add a rule in vcl_recv that passes any incoming requests containing a Content-Range header directly to the backend? We currently do not have support for range-headers, so you'll have to pipe these requests. Note that pass wont do the trick, since it'll still be cached/buffered in varnish (but immediately discarded). -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpjLt5mA5sEr.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Content-Range header problem
On Wed, Sep 16, 2009 at 01:36:00PM -0400, Nathan Kinkade wrote: Thanks. The resolution is simple, but I just wanted to be sure. Always nice to verify :) By the way, I just read the labs.cc.o post [1] from when you started using Varnish and I'm curious as to how you're doing now with regards to these issues? [1] http://labs.creativecommons.org/2008/04/03/varnish-cache-at-cc/ -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpvljKm80niY.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.ttl derived?
On Mon, Sep 14, 2009 at 12:43:20PM -0700, David Birdsong wrote:
On Mon, Sep 14, 2009 at 12:38 PM, David Birdsong
david.birds...@gmail.com wrote:
awesome, thanks. this explains poor cache-hit ratio.
On Mon, Sep 14, 2009 at 12:22 PM, Kristian Lyngstol
krist...@redpill-linpro.com wrote:
On Mon, Sep 14, 2009 at 11:57:25AM -0700, David Birdsong wrote:
How is obj.ttl derived when both Expires and max-age is set by the
backends?
We had a case where the backend was setting Expires to 60seconds after
the request and max-age was 5184000. Additionally in vcl_fetch:
sub vcl_fetch {
if (obj.ttl 9000s) {
set obj.ttl = 9000s;
}
}
What would obj.ttl be set to given the Expires and max-age contradiction?
If s-maxage is set, use that as default, otherwise use max-age as default.
If Expires is sooner than the default, use that.
So in your example, the ttl should be 60s.
oh, to clarify...
would the stanza in vcl_fetch override the obj.ttl which was set by
the Expire header?
sub vcl_fetch {
if (obj.ttl 9000s) {
set obj.ttl = 9000s;
}
}
or do I have the order wrong?
Ah, yes, it would, I misread the as , sorry about that.
But are you sure you're getting hits at all?
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
pgpEcbDgVNBo8.pgp
Description: PGP signature
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: obj.ttl derived?
On Mon, Sep 14, 2009 at 12:52:15PM -0700, David Birdsong wrote: (snip snip) But are you sure you're getting hits at all? varnistat shows hitrate around .54 for 10 and 1000 second averages. N expired objects is anywhere between 10-40 second though and varnishd has only been running for about 12 hours. i expect the hit rate to drop sharply as peak traffic continues to diminish for the day. Depending on the content, that might be a very low hit-rate. i think a 9000s obj.ttl is a complete mistake on our part. so i've followed some of the steps found here http://varnish.projects.linpro.no/wiki/VCLExampleLongerCaching on 1 of our servers. i'm going to compare it in a day or so. why do you ask though? It's fairly common to have pages that aren't cached. If I'm going to give you a single tip right now, it's to look at the output of: varnishtop -i TxURL That's a sorted list of what urls Varnish requests from your backend (in other words: cache misses). Most items should have a 1 or less next to them. The number represents a decaying average, and you should know why if you have items there that have double-digit representation. It should also give you an indication of whether this is a problem with your TTL or a few pages that aren't cached at all, all though the pass/hitpass counter in varnishstat will tell you that too. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpAbFlAYDjWN.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: returning expired data
On Thu, Sep 10, 2009 at 09:04:07AM +0200, Václav Bílek wrote:
Do not know if it is related but this sometime apears in the log:
Not likely to be related, no.
varnishd[12160]: Child (12161) died signal=6
varnishd[12160]: Child (12161) Panic message: Assert error in Tcheck(),
cache.h line 648:#012 Condition((t.e) != 0) not true. thread =
(cache-worker)sp = 0x7fd7a1952008 {#012 fd = 1213, id = 1213, xid =
905469220,#012 client = 213.220.224.168:1871,#012 step = STP_PIPE,#012
handling = pipe,#012 err_code = 400, err_reason = (null),#012 ws =
0x7fd7a1952078 { #012id = sess,#012{s,f,r,e} =
{0x7fd7a1952808,,+601,(nil),+16384},#012 },#012worker =
0x7fd83b365be0 {#012},#012vcl = {#012 srcname = {#012
input,#012Default,#012 },#012},#012},#012
varnishd[12160]: Child cleanup complete
varnishd[12160]: child (3092) Started
varnishd[12160]: Child (3092) said Closed fds: 4 5 8 9 11 12
varnishd[12160]: Child (3092) said Child starts
varnishd[12160]: Child (3092) said managed to mmap 30010953728 bytes of
30010953728
varnishd[12160]: Child (3092) said Ready
This is some sort of bug or mishap; Varnish is throwing an assert error and
essentially emptying your cache before the child is restarted. In your
case, this happens in STP_PIPE, so when you're piping some data.
Václav Bílek napsal(a):
I have a problem that varnish sometimes returns expired data.
The ttl of objects is from 1 to 10 seconds but varnish retuned objects
older than tens of minutes.
Grace is set to 60s.
default ttl to 60s.
Can you attach all the header-data you have regarding this issue?
Age header of such old object had negative value...
Age: -6643 or
Age: -4803
Looks fairly strange
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
pgpbak3hXKVcW.pgp
Description: PGP signature
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Child Died
On Wed, Sep 02, 2009 at 10:12:00AM -0400, maillis...@gmail.com wrote: I just started my first instance of varnish in production. Within 12 hours, there were alerts from our monitoring system that Varnish was taking 90% of the cpu. Right after that, I find these messages in /var/log/messages, several times over a 2 minute period: Did you check syslog for assert errors too? varnishd[12461]: Child (20086) not responding to ping, killing it. The child restarted, and the stats and cache all disappeared. This is a machine with 8 gigs of ram and a pair of slightly older quad core xeons. The storage method is file with a 50 gig limit. At its peak, the machine is serving around 40 requests a second, about 5000k a second. The configs are the defaults. What should my first steps be to troubleshoot this? Is there a likely culprit? The first I'd do is check syslog for assert errors. If it's being killed in the same place, something must be wrong (... ). Secondly, I'd check the value of cli_timeout. This default has changed over time, but a very busy varnish can be slow to reply to pings from the management thread, and thus get killed needlessly. You can check it with the telnet interface or «varnishadm -T localhost:yourmangementport param.show cli_timeout». The new default is 10s, which should be enough, though it still might be too low for extremely busy threads. You may also want to supply a varnishstat -1 (after varnish has had a chance to warm up) and any custom VCL to the list. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpzGhT4qnAZT.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: cant get restart; to fetch and deliver from other backend on HTTP error
On Thu, Sep 03, 2009 at 03:25:32PM +0200, Gerald Leier wrote: On Thu, 2009-09-03 at 11:00 +0200, Kristian Lyngstol wrote: On Thu, Aug 27, 2009 at 04:29:13PM +0200, Gerald Leier wrote: After setting up 2 servers(one returning the requested page the other returning 500 errors) i tested a bit but i have some bug in there i cant get a grip on. (snip snip) All I can say is that your VCL looks sane. I'm not ruling out that Varnish might be a factor here, but I'd have to play with the setup a bit to see what happens. Perhaps a packet capture between the varnish server and test backends could reveal what's going on? -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp6KzykrqfCD.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: cant get restart; to fetch and deliver from other backend on HTTP error
On Thu, Aug 27, 2009 at 04:29:13PM +0200, Gerald Leier wrote: After setting up 2 servers(one returning the requested page the other returning 500 errors) i tested a bit but i have some bug in there i cant get a grip on. after the first node returns a http 500 error varnish continues with the second nodehere is the part where it stops doing what i want: (...) 11 TxHeader b X-Forwarded-For: 10.21.1.40 11 BackendClose b test2 10 VCL_call c error (...) If the connection failed, like it seems to have done here, you do not end up in VCL fetch. Note how varnish doesn't receive any headers from the web server (no RxHeader for the second backend). Varnish is unable to connect properly to your second server and that's what's causing problems. Varnishstat will probably reveal backend failures. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp9IS7cm68en.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish memory consumption issues?
On Sat, Aug 29, 2009 at 12:09:15AM -0400, pub crawler wrote: Hello, new to Varnish. We have been running Varnish for about 5 days now. So far, excellent product. We have a potential issue and I haven't seen anything like this before. We just restarted Varnish - we have a 1GB cache file on disk. When I run top, I see Varnish is using : 10m RES 153g VIRT I'll leave a more detail explanation to PHK or someone more experienced with the deep magics of a VM, but the 10m is the real memory consumption which I'm sure top/free can confirm. Though 153G seem very high for VIRT. What I usually see is VIRT = 2x cache size, or thereabouts, but that might not be conclusive. Do you care to paste your startup arguments and parameters? -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp3jBn43z5mb.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish munin plugin trouble
On Fri, Jul 17, 2009 at 11:04:20AM +0200, Norman Leutner wrote: Thanks for the hint, it's been the PATH... [varnish_*] env.varnishstat /usr/local/bin/varnishstat works fine now. Glad to hear it :) I found a small mistake in your plugin, when using multiple varnish installations on a server: 645 $arg .= -l $varnishname; should be 645 $arg .= -n $varnishname; -l # Lists the available fields to use Fixed, nice catch. Have you ever thought of writing a plugin which catches the most frequent requests passed to the apache server? You can achieve this with varnishtop: varnishtop -i TxURL It'll list all backend requests and sort by which is most frequently requested. Fairly useful. This is realtime, though, so it'll have to keep running. If we want a script to gather this data over time, it'll essentially be the same as varnishtop but a daemon of sorts. So far, I've settled with varnishtop. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpEbdTU3FXm8.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
New munin plugin (Was: Varnish munin plugin trouble)
On Fri, Jul 17, 2009 at 01:03:07PM +0200, Dag-Erling Smørgrav wrote: Norman Leutner n.leut...@all2e.com writes: [to Kristian Lyngstol] Your're the one who wrote that plugin... I was about to say no, that was me - then I found out that my code had been replaced. Kristian, what was wrong with the existing plugin? More to the point, what was so wrong about it that you didn't even bother to discuss the matter with me before you blew it away? The problem with the old plugin was that it didn't let the user see the relationship between relevant statistics. With the growing number of statistics available, it simply generated too many graphs which made it difficult to see the bigger picture. The current implementation combines related graphs and still includes all information if the user wants it (ie: 9 or so aspects aren't linked by default). The previous plugin wasn't horribly big and it would've been more work to re-factor it the way we wanted it. It simply didn't strike me that it would be an issue replacing it. Or that I had to discuss it with you. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgptiiJzq0VEk.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish munin plugin trouble
On Thu, Jul 16, 2009 at 03:31:01PM -0400, Norman Leutner wrote: Trying manually using munin-run I'm getting the expected results. # munin-run varnish_hit_rate client_req.value 30714 (...) Using the munin telnet interface I get no data... # telnet localhost 4949 Trying 127.0.0.1... Connected to localhost.localdomain. Escape character is '^]'. # munin node at .com fetch varnish_hit_rate . Any hints ? This is most commonly caused by differences in run-time parameters, typically PATH or similar. Where is the varnishstat binary located on your system (this can be defined by setting the 'varnishstat' variable) ? -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpqdUi9kONKz.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Fwd: varnish 2.0.4 backend errors
On Wed, Jul 15, 2009 at 10:01:08AM +0200, Lazy wrote: (...) setting 500 as min threads didn't make any diference, funny thing is that failed requests are in logged in apache as succesful Sorry if I didn't make myself clear: The output from varnishstat didn't indicate that threads was the issue causing 503s. It was more of a tuning-comment than a shot at solving the real problem. Sorry about the confusion. 09:23:50.563934 IP x.x.x.x.51235 x.x.x.x.88: S 1066103134:1066103134(0) win 32792 mss 16396,sackOK,timestamp 552835403 0,nop,wscale 7 () Is it possible for you to mail me the raw cap-file? (Feel free to obfuscate the IPs, but I'm really more of an ethereal-man than tcpdump-output...). Feel free to drop it directly to me, if that makes it easier. successful POST in apache logs, in varnishlog 14 Backend c 15 default default (...) The request must've started earlier, as this is just the backend request. Do you have the entire transaction available? (...) restart shouldn't make another backend connection if current one got closed ? It should, but it should be noted that: 1. We don't know _why_ it fails. 2. Restart in vcl_error is brand new, so there might still be some issues. You could try to turn on vcl-trace. (-p vcl_trace=on). Also, could you post your VCL in it's entirety? (If you already did, I must've missed it.) -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp6OC1UYDneP.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: How many simultanious users
On Wed, Jul 15, 2009 at 11:38:27AM +0200, Lazy wrote: I'm trying to figure out how many simultaneous users a single 8 core machine with local apache running as a backend can handle assumming that all the requests are cached. This is actually very difficult to test, as you often end up with client-side issues. Or issues that are irrelevant to a production environment. I've done extensive stress testing, and getting the load up on the Varnish-server is not trivial. Our stress-testing rig is using a single dual-core opteron _clocked down_ to 1ghz as Varnish-server, and roughly 8-12 cpu cores spread over 3-5 servers as 'clients'. And it's able to handle 18-19k req/s consistently (these are 1-3byte pages of cache hits). That should give you an idea of the synthetic performance Varnish can offer. testing with ab on a slow 100Mbps link shows 2500 hit/s, locally i got 12 000 hit/s with over 200Mbps traffic How big are the pages requested? Are they all hits? What's the load on the client and server? assuming that each user loads 40 files in 1 minute we get 12000*60/40=18 000 users per minute Is it possible to get half of that 18k users/per minute in real word ignoring the amounts of traffic it will generate ? I'd say so, but it depends on how big the data set is. If you can store it in memory, varnish is ridiculously fast. I also wouldn't recommend relying on a single Varnish for more than a few thousand requests per second. If something goes wrong (suddenly getting lots of misses for instance), it will quickly spread. For comparison, I'm looking at a box with roughly 0.4 load serving 2000req/s as we speak, and that's on 2xDual Core Opteron 2212. Going by those numbers, it should theoretically be able to handle almost ten times as many requests if Varnish scaled as a straight line. That'd give you roughly 18000 req/s at peak (give or take a little...) Now you're talking about 8 cores, that should be 36k req/s. That's _not_ unrealistic, from what we've seen in synthetic tests. If each client requires 40 items, that means roughly 900 clients _per second_. Or 54k in a minute. This math is all rough estimates, but the foundation is production sites and real traffic patterns. The problem is that getting your Varnish to deal with 36k req/s is rather difficult, and you quickly run into network issues and similar. And at 36k req/s you can hardly take any amount of backend traffic or delays before it all falls over. For now it's only theoretical question, but we would like to estimate how many machines will it take to handle this kind of load. Another question how to scale varnish, I'm thinking about setting a 2 loadbalancers whitch will take care of sessions getting to the same server, and 3x8 core machines for www + varnish or maybe 2x4 core loadbalancers with varnish and 3x8 core machines for www. I would be possible to use varnish as a loadbalancer with some http cookie trickery. I wouldn't recommend using Varnish to implement sticky-sessions, even though it might be possible. What I've seen people do, though, is put apache with mod_proxy _behind_ varnish, and let that deal with sticky-sessions, then varnish only have to know what to cache and what not to cache. (And for varnish, there'll be only one backend). -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpkttXdoKe8V.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Fwd: varnish 2.0.4 backend errors
On Wed, Jul 15, 2009 at 12:14:21PM +0200, Lazy wrote:
2009/7/15 Kristian Lyngstol krist...@redpill-linpro.com:
You could try to turn on vcl-trace. (-p vcl_trace=on).
right now i have something 100-200 req/s won't it kill the server and
I can't restart varnish now so I will have try to enable it at runtime
Ok, I'm not entirely sure if that'll work, but it's worth a try (and it
should cause any damage anyway).
Also, could you post your VCL in it's entirety? (If you already did, I
must've missed it.)
sub vcl_recv {
(...)
if (req.http.Cache-Control ~ no-cache) {
pass;
You probably do not want this; It will mean that every time a _client_
sends no-cache, it's a pass. That'll be quite often (on refresh, for
instance), and the refreshed item wont get cached, so it wont refresh the
cache.
(I'll have a look at the cap when I find some time. Could take a few
hours).
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
pgpH7KLi3pVfs.pgp
Description: PGP signature
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Question about Cache Purge
On Thu, Jul 09, 2009 at 02:17:35PM -0500, Brian Pan wrote: Hi all, I have a newbie question regarding 'purging the cache.' When I run the following command to purge the root domain of my website, varnishadm -T localhost:80 purge.url ^/$ Are you sure the admin interface is at port 80? That sounds very unlikely. How do you start varnishd? Varnish can listen to _two_ ports, one is typically port 80, where normal HTTP data is served, the other is an administrative interface, which is what you want to specify with -T. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp48o51KyH9R.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish 2.0.4 backend errors
On Tue, Jul 14, 2009 at 11:46:58AM +0200, Lazy wrote: 2009/7/14 Kristian Lyngstol krist...@redpill-linpro.com: 6000 threads is too much. Since it's per pool, it'll cause up to 12 000 threads to start. That's not likely to go over all that well. If you have that sort of traffic, you need to scale out. Also, 10 thread minimum is pretty low. I typically recommend setting the minimum thread count to what you expect your normal traffic to be at peak hours. It's probably a dedicated machines, and idle threads have barely any overhead, while creating new threads can take some time. at first i had 3000 threads set and varnish ocassionly droped connections, so I doubled it so what whould be a recomended values ? will -w 1024,1024 -p thread_pools=6 whould be ok ? 6 thread pools is overkill. And the number of threads is multiplied with the thread pools, so in this case you're essentially writing -w 6k,6k... I'd advice something like -w 200,1200 -p thread_pools=2 to begin with. Just watch the overflows in varnishstat (it'll increase a good bit during startup since it takes a little time to create the 400 threads). It should stay fairly static after startup. the site is usually not so busy, but it has sometimes spikes of static traffic (about 50Mbps) that's why i upped the thread limit, 3000 was to low I seriously doubt 3k was too low. More likely, the min threads was hurting you. Three thousand threads is quite a bit. Remember that these are actual requests being handled, not keep-alive connections and the like. is it safe to change thread_pools on runtime ? Safe; I'd assume so. But I don't know if it actually takes effect. I've yet to see any good reason to change it from the default. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpGG7y6WlKUh.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: best varnishd config for static files
On Fri, Apr 24, 2009 at 03:28:22PM +0200, Kain Sascha wrote: Thanks a lot for the great Hints. First of all, varnishtop -i TxURL hangs on my system, it shows 3-5 Entries, then freez and uses nearly 100%cpu on one of 8. Have to kill the pid. varnishtop with no args works. Ok, that's not good... I've seen varnishtop hang like that before, but not with CPU usage. I've yet to be able to properly debug this as I can't reproduce it myself... What GNU+Linux distribution are you using? If your miss-rate is low enough, you may be able to get some sense out of varnishlog -b instead of varnishtop, but varnishtop really is the tool you want... At any rate, with static content, varnishstat should show you a hitrate in the 0.99-range (99% hit-rate, for static content, I'd say 0.999 is well within reach), unless you have a huge dataset with very even distribution of hits. Here is our Answer-Header, so we provide the max-age. Cache-Control max-age=2592000 Vary Accept-Encoding Expires Wed, 29 Apr 2009 21:14:47 GMT Content-Type image/jpeg Etag 857407780 Last-Modified Mon, 30 Mar 2009 20:35:52 GMT Server lighttpd/1.4.20 Content-Length 1392 Date Fri, 24 Apr 2009 13:19:59 GMT X-Varnish 623951818 451344757 Age 2131515 Via 1.1 varnish Connection keep-alive Looks good. Age: confirms that this object is 592 hours old too. On long-duration caching like this, keep in mind that these headers affect client-cache too. If that doesn't bother you, you can leave them as is, but if you want varnish to cache for, say 720 hours, but clients should check back every 24 hours (in case you force purges), you can modify the max-age header before sending it out. Sometimes im getting a 503 error when the static server has high load. Also sometimes i am waiting a long time for some files that has to be on RAM allready. Can you check if the X-Varnish header has one or two numbers on these errors/slow pages? The Age: header can tell you some of the same information, but on slow gets, it's not uncommon to see a positive age on cache misses. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpFnJDwg7mTn.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: best varnishd config for static files
On Fri, Apr 24, 2009 at 01:28:36PM +0200, Kain Sascha wrote: /usr/local/sbin/varnishd -a :80 -b 89.250.xx.xx:80 -s malloc,15360M You may want to supply a -w. The default is really whimpy which could create problems during spikes of traffic. Try something like -w 400,800. Varnishstat n_wrk_limited and n_wrk_drop are values you don't want to increase after the initial startup. n_wrk_limited can increase a little every once in a while, but it should only happen on significant load spikes. I didnt use a config file, because i think i dont need it for static files only. is this correct? The default TTL is 120s if I am not much mistaken. You probably want it to be something closer to 500h or some other really long period of time. This has to be done in VCL if your backends don't deliver s-maxage/max-age/Expires: headers. For instance: vcl_fetch: set obj.ttl = 500h; Is this a good solution? Yes, though you should make sure varnish caches as much as you want it to and watch the hitrate in varnishstat and check what is actually going to the backends most frequently with varnishtop -i TxURL. With varnishtop -i TxURL, you really shouldn't see many values above 1-3 for static content. Sometimes im getting a 503 error when the static server has high load. Also sometimes i am waiting a long time for some files that has to be on RAM allready. The X-Varnish HTTP header can tell you if it is a hit or not. It gives you one or two numbers: One is a transaction id, and the last identifies the original transaction that fetched the object. One number: MISS; two numbers: HIT. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpb2SjRqOexX.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Internal redirecition with dynamic backend (fetch 302 and return content)
On Wed, Apr 22, 2009 at 08:41:58AM +0200, Augustin Amann wrote:
1) RP do GET to the backend
2) backend reply a 302 to another server
3) RP do a GET to this new location
4) RP return content and store it.
Maybe i can use restart, like [1].
I have no idea where the new location (gived by 302) is, this could be a
problem for a reverse proxy !
So I think it's not realy the varnish way to deal with dynamic
backend, but vcl is powerfull,
and if I can define backend host and url on the fly, it could save my day !
In vcl_fetch:
if (obj.status == 302) {
set req.url = obj.http.Location;
restart;
}
The normal routines in vcl_recv will have to determine which backend to use
based on the new req.url. This will have to be pre-defined backends, so no
random redirects I'm afraid.
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
pgp1tWetzNmtQ.pgp
Description: PGP signature
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: multiple domains in one backend 2 varnish servers
On Tue, Apr 21, 2009 at 12:46:15PM +0200, Andreas Kaiser - Atela CC wrote: should be served by Europe server and others by the usa server I think I need to configure each varnish install with a specific vcl config file. Why? They'll be working against the same backend server, and the Host-header differentiates between each domain. What I don't have clear is if I need to use as backend only the IP of the CMS server or multiple backends for every domain, also all domains would have the same configuration related with cookies, compression, etc: You only need to define one backend if you only have one backend. My experience with varnish is related with one webserver with one domain that also has varnish installed... If I'm not wrong, varnish would create a cache for each hostname/domain but I'm lost how varnish and cmsserver should be configured for showing the specific content for each domain if the servers are different ones... No configuration needed. Your webserver reads the Host: header which is part of HTTP 1.1, and Varnish, by default, uses this header combined with the URL as part of the cache hash. In essence: There's no difference in configuration from just having one domain and having multiple domains. There are some differences when you start talking purging, since the old purge.url doesn't supply a hostname, but with http-based PURGE in vcl_hit, this isn't an issue. And with purge.regexp or the new purge system, it's not an issue either. The reason of this configuration is that the cms-server will be only accessible from a private network and the 2 varnish server (which will be used as public frontend servers) Would varnish be a good choice in this situation? From the information you've given, it sounds like a reasonable choice, yes. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpr4j64k75hk.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Frequent Varnish 2.0.4 Panics
On Fri, Apr 17, 2009 at 02:54:25PM -0700, Igor Minar wrote: Hi there, I see frequent (~every 30 seconds) Varnish panics when running a light load tests on our webapp which is proxied by varnish. What sort of load are you testing with? Has anyone else experienced the same issue? Let me know if you need more info. By the looks of it, the child is killed of because it's unresponsive, and it seems to me that it assert-error in the child is a result of this. Have you tried simply increasing the client timeout? (-p cli_timeout). The default is very agressive, and I've seen it kill off healthy childs that are bussy delivering content even with a cli_timeout as high as 21 seconds. Definitely something to test at least. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpxaUC0K8dbN.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Cookie handling
On Mon, Apr 20, 2009 at 02:54:33PM +0100, Sam Crawford wrote: The issue arises when we write our custom definitions of vcl_fetch. If we find a static URL we'd do some like remove req.http.cookie and then lookup. However, if the content was not found in memory then it goes to the backend. But by this point the incoming SSO cookie of the user has been removed, so their request to the backend is denied. So, is there a way to lookup a query in the hashtable *without* cookies, but then if the request has to go to the backend *keep* the cookies? This is actually the default behavior. By default, only the url and hostname is part of the hash. However, the default vcl also explicitly passes any request with a cookie, which is why stripping the cookie is necessary. If you replicate the behavior you want from the default vcl in your own VCL, then end vcl_recv with lookup with a cookie, the object will be fetched using the cookie, then stored (as long as vcl_fetch allows it to be stored). Keep in mind, however, that any protection is lost, as cached data won't have their cookie confirmed. But I guess you're already aware of that. Hope this can point you in the right direction. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpC7VABrMY9l.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish killing off the child process after a few minutes
On Tue, Apr 07, 2009 at 01:01:51PM -0700, Tung Nguyen wrote: Hi guys, We're on varnish 2.0.3 It looks like varnish restarts the child process for us every so often, causing 503s :(. Was wondering if this is a known issue. Can you check the syslog for any more information? -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp9e6JrQxoi6.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish 2.0.4 and new config changes
On Tue, Apr 07, 2009 at 05:29:23PM -0700, Jauder Ho wrote: On Tue, Apr 7, 2009 at 4:46 AM, Kristian Lyngstol krist...@redpill-linpro.com wrote: On Tue, Apr 07, 2009 at 04:11:04AM -0700, Jauder Ho wrote: FYI, beresp.ttl does not work on varnish 2.0.4. It does work on -trunk though. It would seem I was mistaken. It doesn't look like the obj to beresp commits made it into 2.0.4 after a quick check. So that's only relevant to trunk. That could be a problem for some people as obj.ttl is not available in vcl_fetch in 2.0.4 and beresp.ttl is not available until trunk. That's not correct. Obj.ttl is available in hit, fetch, discard, timeout and error in 2.0.4. It's only available in hit, discard, timeout and error in trunk as beresp.ttl replaces it in fetch. Right now, only nginx1 sets X-Forwarded-For; varnish does not set X-Forwarded-For; nginx2 is a fastcgi frontend and does not set X-Forwarded-For. With this config (and using -trunk), nginx2 still logs the gateway IP so something strange is going on here. I'm not familiar with how nginx logs, but I'd start by verifying: 1. That the X-Forwarded-For does indeed reach the relevant server. 2. That the entity that logs honors X-Forwarded-For. Looking at varnishlog (which covers the incoming request from nginx1 and response from nginx2. It looks like X-Forwarded-For is properly set on nginx1. However on the response, it looks like there are 2(?) X-Forwarded-For headers instead of being chained as in http://en.wikipedia.org/wiki/X-Forwarded-For#Format Varnish doesn't touch these headers unless you tell it to, so it looks like nginx is confused. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpySywXr6W0T.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish 2.0.4 and new config changes
On Tue, Apr 07, 2009 at 04:11:04AM -0700, Jauder Ho wrote:
On Tue, Apr 7, 2009 at 3:19 AM, Kristian Lyngstol
krist...@redpill-linpro.com wrote:
On Tue, Apr 07, 2009 at 02:55:13AM -0700, Jauder Ho wrote:
I just downloaded the new 2.0.4 release and noticed that a couple of
things had changed causing my existing config to stop working.
Therefore, I had a couple of questions that I'm hoping that you can
help answer.
a) Is beresp.ttl basically replacing obj.ttl? If so, I should be able
to do the following right in vcl_fetch?
if (beresp.ttl 300s) {
set beresp.ttl = 300s;
}
Yes, for all intents and purposes, beresp is obj. The name change is
reflecting some underlying changes that doesn't really affect how you write
VCL - yet.
FYI, beresp.ttl does not work on varnish 2.0.4. It does work on -trunk
though.
It would seem I was mistaken. It doesn't look like the obj to beresp
commits made it into 2.0.4 after a quick check. So that's only relevant to
trunk.
c) Lastly, I have a config of user nginx1 varnish nginx2
php-fpm and noticed that the IP being logged on nginx2 is the
internal IP. nginx1 has X-Forwarded-For set.
(...)
If your first nginx sets X-Forwarded-For, then setting
req.http.X-Forwarded-For = client.ip; will overwrite it with nginx1's IP.
So don't set X-Forwarded-For in vcl_recv and it will pass straight through
Varnish. 2.0.4 does not affect this compared to 2.0.3.
Right now, only nginx1 sets X-Forwarded-For; varnish does not set
X-Forwarded-For; nginx2 is a fastcgi frontend and does not set
X-Forwarded-For.
With this config (and using -trunk), nginx2 still logs the gateway IP so
something strange is going on here.
I'm not familiar with how nginx logs, but I'd start by verifying:
1. That the X-Forwarded-For does indeed reach the relevant server.
2. That the entity that logs honors X-Forwarded-For.
--
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
pgpaJ2gJHx0nH.pgp
Description: PGP signature
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnishd runtime parameters
On Mon, Mar 23, 2009 at 05:58:58PM -0700, Tung Nguyen wrote: Hi guys, So, Im reading over an archive email thread about twitters configuration. http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html It looks like they had to adjust a lot of parameters... and Im not finding all the parameters definitions in the varnishd man pages. Im wondering if for most cases running varnish with the defaults is fine? Any caveats here which run time parameters should I focus on. You can mostly run it with the defaults, yes. This depends on what sort of usage you have though. One notable exception is that I strongly recommend that you bring thread_pool_min up to a decent level (reflecting how many users you actually have. Numbers in the hundreds is normal). You'll also want to adjust the cache size to your system, but that goes without saying. If you expect extremely high load, you might have to increase cli_timeout too. I've run tests where even setting it to 15 seconds is insufficient and causes childs to be killed off. Though for most production sites, I'd guess 5 seconds could work and 10 seconds would definitely work. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgp5aO5D7HUgO.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
