Re: Weird log entries
Morning. Same behaviour, same lines, same nulls over here, with the same amd64 arch. I just discarded those lines in the filter i use to split varnishncsa logs. Not really a big problem, but something else to take into account apart of crashes :-) David (Kitai) Cruz E-Comm Factory 2009/4/7 Alecs Henry alecshe...@gmail.com: So, after a while digging through other stuff I realized what the (null) entries are. They are cache misses. For (apparently) every miss there's an entry on varnishncsa for that miss, and it's logged as a bunch of nulls. I haven't figured out what the other entries are though. I still see some entries (not as many as the null ones) with the date field set to [00/Jan/1900:00:00:00 +], with all the other information valid (URL, Referer, BackendName, etc..) except for the object size, which is set to (null) and ther server response code (also (null)). There are also entries that the only field set is the server response (200 or 404 -- these have the date field set correctly) and some entries with the host (or where the backend name should go) set to 127.0.0.1 and the other fields set correctly. One thing though, in none of the entries (correct, broken or plain weird) the object size is set, all I see are -, except for one entry that had it set (strangely enough, favicon.ico!) and even then, only once. If anybody else has seen this stuff or know anything that might help explain, I'd greatly appreciate some help. I can also send you any information you might deem necessary in order to figure this stuff out. This is the varnish package for debian (2.0.3-2_amd64) all with HTTP/1.1 requests. Thanks! Alecs On Mon, Feb 9, 2009 at 5:29 PM, Alecs Henry alecshe...@gmail.com wrote: Hi Nick, I forgot to say... It's varnish-trunk. But I saw it on both varnish 2.0.1 and 2.0.2. Alecs On Mon, Feb 9, 2009 at 6:20 PM, Nick Loman n...@loman.net wrote: Alecs Henry wrote: EDITED TO INCLUDE THE ORIGINAL MESSAGE... Hi guys, I was wondering if any of you has seen the following log entries when using varnishncsa: 127.0.0.1 - - [00/Jan/1900:00:00:00 +] (null) (null) (null) (null) - - - Those are coupled with: 127.0.0.1 - - [09/Feb/2009:19:39:46 +] (null) (null) (null) 200 39678 - - I can see an object in the page that has that size (image) -- through firebug, but the object didn't load into the browser until I hit reload. And there are also many entries like the following: 127.0.0.1 - - [09/Feb/2009:19:39:52 +] GET http://www.example.com/img/mod.gif HTTP/1.1 200 951 http://www.example.com/style.css; Mozilla/5.0 (X11; U; Linux x86_64; en; rv:1.9.0.5) Gecko/2008121623 Ubuntu/8.10 (intrepid) Firefox/3.0.5 These entries are from communication with the backend server I expected it to show the backend name instead of 127.0.0.1. Is there an explanation? Thanks! Alecs I've seen log entries like this for HTTP 1.0 requests (using Varnish 1.2), as that version of Varnish did not have support for parsing HTTP 1.0 headers. Cheers, Nick. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: 503 Service Unavailable + Stop Responding
Search in the /var/log/messages log if varnish it's restarting. After every crash, varnish starts answering with a 503 to every request. DAvid (Kitai) Cruz 2009/4/7 Sascha Kain s.k...@eraffe-media.de: Hi im getting folloing error when accessing a Picture (jpg), delivered by Varnish-Cache. Error 503 Service Unavailable Service Unavailable Guru Meditation: XID: 502211958 Varnish http://www.varnish-cache.org/ This happens sporadically. When i access the picture from the Backend directly, its working. My varnishd is running on a Debian Server proxycache2:~# uname -a Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008 x86_64 GNU/Linux proxycache2:~# varnishd -V varnishd (varnish-2.0.3) Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M It also happens, that the daemon just stop answering on Port 80, i have to manually restart it! proxycache2:~# varnishstat -1 uptime 691288 . Child uptime client_conn 13757134 19.90 Client connections accepted client_req 62501336 90.41 Client requests received cache_hit 59008654 85.36 Cache hits cache_hitpass 35 0.00 Cache hits for pass cache_miss 3454784 5.00 Cache misses backend_conn 3491596 5.05 Backend connections success backend_unhealthy 0 0.00 Backend connections not attempted backend_busy 0 0.00 Backend connections too many backend_fail 1094 0.00 Backend connections failures backend_reuse 3190621 4.62 Backend connections reuses backend_recycle 3300087 4.77 Backend connections recycles backend_unused 0 0.00 Backend connections unused n_srcaddr 1082 . N struct srcaddr n_srcaddr_act 69 . N active struct srcaddr n_sess_mem 4061 . N struct sess_mem n_sess 325 . N struct sess n_object 727166 . N struct object n_objecthead 412871 . N struct objecthead n_smf 0 . N struct smf n_smf_frag 0 . N small free smf n_smf_large 0 . N large free smf n_vbe_conn 25 . N struct vbe_conn n_bereq 173 . N struct bereq n_wrk 46 . N worker threads n_wrk_create 6590 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_queue 0 0.00 N queued work requests n_wrk_overflow 40391 0.06 N overflowed work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 1 . N backends n_expired 498 . N expired objects n_lru_nuked 2717782 . N LRU nuked objects n_lru_saved 0 . N LRU saved objects n_lru_moved 50761824 . N LRU moved objects n_deathrow 0 . N objects on deathrow losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 43664646 63.16 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 13757122 19.90 Total Sessions s_req 62501369 90.41 Total Requests s_pipe 10 0.00 Total pipe s_pass 37897 0.05 Total pass s_fetch 3477236 5.03 Total fetch s_hdrbytes 19737092290 28551.19 Total header bytes s_bodybytes 681892000484 986407.98 Total body bytes sess_closed 544021 0.79 Session Closed sess_pipeline 164402 0.24 Session Pipeline sess_readahead 65588 0.09 Session Read Ahead sess_linger 0 0.00 Session Linger sess_herd 61809876 89.41 Session herd shm_records 2591397419 3748.65 SHM records shm_writes 170178899 246.18 SHM writes shm_flushes 65 0.00 SHM flushes due to overflow shm_cont 5567 0.01 SHM MTX contention shm_cycles 931 0.00 SHM cycles through buffer sm_nreq 0 0.00 allocator requests sm_nobj 0 . outstanding allocations sm_balloc 0 . bytes allocated sm_bfree 0 . bytes free sma_nreq
Re: Varnish and virtual hosting
3. There doesn't seem to be a way to process live varnishncsa data
via a pipe. If this were possible, then I think I would find this whole
process a lot easier. Instead, it seems there are two options: to
historically dump log data via an option, or run as a daemon and send
output to a file (which cannot be via a pipe).
Wrong. Both varnishlog and varnishncsa will, by default, process live
data and print it to stdout, where you can pipe it to whatever you like.
Agree.
You can post-process varnishncsa data to any post-processing program you want.
Or, for example, using Perl ..you may launch the program and take the
output directly to the files you want.
#!/usr/bin/perl
open (COMMAND, varnishncsa | ) or die Could not open varnishncsa...\n;
while (COMMAND) {
# bla bla bla
print $_; # Print actual line
}
close COMMAND;
Every request is logged with full DN in the URL.
62.151.2.42 - - [29/Jan/2009:16:56:16 +0100] GET
http://www.FULLDOMAIN.info/static HTTP/1.0 503 466 referer
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR
1.1.4322)
So it's not difficult to make a program to separate logs depending on
your needs.
Btw, a program in C made by varnish programmers to accomplish this may
be greatly appreciated by the community ;-)
Kitai
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Logging for multiple sites
Maybe create a simple script to separate logs Every URL is logged using http://domain/uri structure. Really simple script to program. Kitai 2008/11/26 Marcus Smith [EMAIL PROTECTED]: Dear list, I currently have a machine running Varnish (2.0.2) set up in front of several different websites. I would like to be able to collect access logs for each of the different sites separately. Obviously the Apache logs for these sites will be incomplete, as varnishd will serve cache hits without passing them on. With a single instance of varnishd handling all the sites, varnishncsa outputs to a single log file with logs for all the sites together. I would like each site to be logged to a separate file, in the same way that Apache's virtual hosts can have their own log files. What is the recommended way to achieve this with Varnish? At first I considered running a separate named instance of varnishd and varnishncsa for each site, thus: varnishd -n example1 -a www.example1.com example1.com -f /path/to/vcl/example1.vcl varnishd -n example2 -a www.example2.com example2.com -f /path/to/vcl/example2.vcl ... varnishncsa -n example1 -D -a -w /path/to/logs/example1.log varnishncsa -n example2 -D -a -w /path/to/logs/example2.log ... ...but of course this will not work because once the first instance of varnish for example1.com has bound itself to port 80, the second instance will not start unless it's on a different port. So I'm stuck. How should I be doing this? Many thanks in advance, Marcus -- Marcus Smith Information Officer The Council for British Archaeology ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Logging for multiple sites
So, if i've got 900 domains, do i have to start 900 varnishlog processes? Interesting:;-) Kitai 2008/11/26 Marcus Smith [EMAIL PROTECTED]: Dag-Erling Smørgrav wrote: Actually varnishncsa uses the same log filtering / selection code as varnishlog, which *can* select requests based on URL. Extending varnishncsa to do the same should not be too hard. Other issues with varnishncsa / varnishlog: varnishlog doesn't allow -o and -w to be used at the same time. There is no reason why it shouldn't. If it did, you could play tricks like this: varnishlog -w /dev/stdout -c -o RxURL foo | varnishncsa -r /dev/stdin Ah, I see! Hmmm. Well in that case, is there any reason why I shouldn't simply do something like: varnishlog -c -o RxHeader Host: (www\.)?example1\.com /path/to/logs/example1.log varnishlog -c -o RxHeader Host: (www\.)?example2\.com /path/to/logs/example2.log ...etc for each site, logging each to a separate varnish log file? I could then use varnishncsa's '-r' option to convert them into NCSA format once the logs are rotated out. It seems like that would do pretty much what I want. Many thanks, Marcus -- Marcus Smith Information Officer The Council for British Archaeology ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
