Re: Varnish and virtual hosting

2009-02-13 Thread Kitai

 3. There doesn't seem to be a way to process live varnishncsa data
 via a pipe. If this were possible, then I think I would find this whole
 process a lot easier. Instead, it seems there are two options: to
 historically dump log data via an option, or run as a daemon and send
 output to a file (which cannot be via a pipe).

 Wrong.  Both varnishlog and varnishncsa will, by default, process live
 data and print it to stdout, where you can pipe it to whatever you like.


You can post-process varnishncsa data to any post-processing program you want.

Or, for example, using Perl may launch the program and take the
output directly to the files you want.


open (COMMAND, varnishncsa | ) or die Could not open varnishncsa...\n;

while (COMMAND) {

# bla bla bla
print $_; # Print actual line


close COMMAND;

Every request is logged with full DN in the URL. - - [29/Jan/2009:16:56:16 +0100] GET HTTP/1.0 503 466 referer
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CLR

So it's not difficult to make a program to separate logs depending on
your needs.

Btw, a program in C made by varnish programmers to accomplish this may
be greatly appreciated by the community ;-)

varnish-misc mailing list

Re: Weird log entries

2009-04-07 Thread Kitai

Same behaviour, same lines, same nulls over here, with the same amd64 arch.

I just discarded those lines in the filter i use to split varnishncsa
logs. Not really a big problem, but something else to take into
account apart  of crashes :-)

David (Kitai) Cruz
E-Comm Factory

2009/4/7 Alecs Henry
 So, after a while digging through other stuff I realized what the (null)
 entries are.
 They are cache misses.
 For (apparently) every miss there's an entry on varnishncsa for that miss,
 and it's logged as a bunch of nulls.
 I haven't figured out what the other entries are though.
 I still see some entries (not as many as the null ones) with the date field
 set to [00/Jan/1900:00:00:00 +], with all the other information valid
 (URL, Referer, BackendName, etc..) except for the object size, which is set
 to (null) and ther server response code (also (null)).
 There are also entries that the only field set is the server response (200
 or 404 -- these have the date field set correctly) and some entries with the
 host (or where the backend name should go) set to and the other
 fields set correctly.

 One thing though, in none of the entries (correct, broken or plain weird)
 the object size is set, all I see are -, except for one entry that had it
 set (strangely enough, favicon.ico!) and even then, only once.

 If anybody else has seen this stuff or know anything that might help
 explain, I'd greatly appreciate some help.
 I can also send you any information you might deem necessary in order to
 figure this stuff out.

 This is the varnish package for debian (2.0.3-2_amd64) all with HTTP/1.1



 On Mon, Feb 9, 2009 at 5:29 PM, Alecs Henry wrote:

 Hi Nick,

 I forgot to say... It's varnish-trunk.
 But I saw it on both varnish 2.0.1 and 2.0.2.


 On Mon, Feb 9, 2009 at 6:20 PM, Nick Loman wrote:

 Alecs Henry wrote:

 Hi guys,

 I was wondering if any of you has seen the following log entries when
 using varnishncsa: - - [00/Jan/1900:00:00:00 +] (null) (null) (null) (null)
 - - -

 Those are coupled with: - - [09/Feb/2009:19:39:46 +] (null) (null) (null) 200
 39678 - -
 I can see an object in the page that has that size (image) -- through
 firebug, but the object didn't load into the browser until I hit reload.

 And there are also many entries like the following: - - [09/Feb/2009:19:39:52 +] GET HTTP/1.1 200 951; Mozilla/5.0 (X11; U; Linux x86_64; en;
 rv: Gecko/2008121623 Ubuntu/8.10 (intrepid) Firefox/3.0.5

 These entries are from communication with the backend server

 I expected it to show the backend name instead of

 Is there an explanation?



 I've seen log entries like this for HTTP 1.0 requests (using Varnish
 1.2), as that version of Varnish did not have support for parsing HTTP 1.0



 varnish-misc mailing list

varnish-misc mailing list

Re: 503 Service Unavailable + Stop Responding

2009-04-07 Thread Kitai
Search in the /var/log/messages log if varnish it's restarting.

After every crash, varnish starts answering with a 503 to every request.

DAvid (Kitai) Cruz

2009/4/7 Sascha Kain
 im getting folloing error when accessing a Picture (jpg), delivered by


  Error 503 Service Unavailable

 Service Unavailable

      Guru Meditation:

 XID: 502211958



 This happens sporadically.
 When i access the picture from the Backend directly, its working.

 My varnishd is running on a Debian Server
 proxycache2:~# uname -a
 Linux proxycache2 2.6.18-6-amd64 #1 SMP Mon Jun 16 22:30:01 UTC 2008
 x86_64 GNU/Linux
 proxycache2:~# varnishd -V
 varnishd (varnish-2.0.3)
 Copyright (c) 2006-2008 Linpro AS / Verdens Gang AS

 varnishd -a :80 -b xx.xx.xx.40:80 -s malloc,15360M

 It also happens, that the daemon just stop answering on Port 80, i have
 to manually restart it!

 proxycache2:~# varnishstat -1
 uptime                 691288          .   Child uptime
 client_conn          13757134        19.90 Client connections accepted
 client_req           62501336        90.41 Client requests received
 cache_hit            59008654        85.36 Cache hits
 cache_hitpass              35         0.00 Cache hits for pass
 cache_miss            3454784         5.00 Cache misses
 backend_conn          3491596         5.05 Backend connections success
 backend_unhealthy            0         0.00 Backend connections not attempted
 backend_busy                0         0.00 Backend connections too many
 backend_fail             1094         0.00 Backend connections failures
 backend_reuse         3190621         4.62 Backend connections reuses
 backend_recycle       3300087         4.77 Backend connections recycles
 backend_unused              0         0.00 Backend connections unused
 n_srcaddr                1082          .   N struct srcaddr
 n_srcaddr_act              69          .   N active struct srcaddr
 n_sess_mem               4061          .   N struct sess_mem
 n_sess                    325          .   N struct sess
 n_object               727166          .   N struct object
 n_objecthead           412871          .   N struct objecthead
 n_smf                       0          .   N struct smf
 n_smf_frag                  0          .   N small free smf
 n_smf_large                 0          .   N large free smf
 n_vbe_conn                 25          .   N struct vbe_conn
 n_bereq                   173          .   N struct bereq
 n_wrk                      46          .   N worker threads
 n_wrk_create             6590         0.01 N worker threads created
 n_wrk_failed                0         0.00 N worker threads not created
 n_wrk_max                   0         0.00 N worker threads limited
 n_wrk_queue                 0         0.00 N queued work requests
 n_wrk_overflow          40391         0.06 N overflowed work requests
 n_wrk_drop                  0         0.00 N dropped work requests
 n_backend                   1          .   N backends
 n_expired                 498          .   N expired objects
 n_lru_nuked           2717782          .   N LRU nuked objects
 n_lru_saved                 0          .   N LRU saved objects
 n_lru_moved          50761824          .   N LRU moved objects
 n_deathrow                  0          .   N objects on deathrow
 losthdr                     0         0.00 HTTP header overflows
 n_objsendfile               0         0.00 Objects sent with sendfile
 n_objwrite           43664646        63.16 Objects sent with write
 n_objoverflow               0         0.00 Objects overflowing workspace
 s_sess               13757122        19.90 Total Sessions
 s_req                62501369        90.41 Total Requests
 s_pipe                     10         0.00 Total pipe
 s_pass                  37897         0.05 Total pass
 s_fetch               3477236         5.03 Total fetch
 s_hdrbytes        19737092290     28551.19 Total header bytes
 s_bodybytes      681892000484    986407.98 Total body bytes
 sess_closed            544021         0.79 Session Closed
 sess_pipeline          164402         0.24 Session Pipeline
 sess_readahead          65588         0.09 Session Read Ahead
 sess_linger                 0         0.00 Session Linger
 sess_herd            61809876        89.41 Session herd
 shm_records        2591397419      3748.65 SHM records
 shm_writes          170178899       246.18 SHM writes
 shm_flushes                65         0.00 SHM flushes due to overflow
 shm_cont                 5567         0.01 SHM MTX contention
 shm_cycles                931         0.00 SHM cycles through buffer
 sm_nreq                     0         0.00 allocator requests
 sm_nobj                     0          .   outstanding allocations
 sm_balloc                   0          .   bytes allocated
 sm_bfree                    0          .   bytes free

Re: Logging for multiple sites

2008-11-26 Thread David (Kitai) Cruz
Maybe create a simple script to separate logs
Every URL is logged using http://domain/uri structure.

Really simple script to program.


2008/11/26 Marcus Smith [EMAIL PROTECTED]:
 Dear list,

 I currently have a machine running Varnish (2.0.2) set up in front of
 several different websites.  I would like to be able to collect access
 logs for each of the different sites separately.

 Obviously the Apache logs for these sites will be incomplete, as
 varnishd will serve cache hits without passing them on.  With a single
 instance of varnishd handling all the sites, varnishncsa outputs to a
 single log file with logs for all the sites together.  I would like each
 site to be logged to a separate file, in the same way that Apache's
 virtual hosts can have their own log files.  What is the recommended way
 to achieve this with Varnish?

 At first I considered running a separate named instance of varnishd and
 varnishncsa for each site, thus:

  varnishd -n example1 -a -f
  varnishd -n example2 -a -f
  varnishncsa -n example1 -D -a -w /path/to/logs/example1.log
  varnishncsa -n example2 -D -a -w /path/to/logs/example2.log

 ...but of course this will not work because once the first instance of
 varnish for has bound itself to port 80, the second
 instance will not start unless it's on a different port.

 So I'm stuck.  How should I be doing this?

 Many thanks in advance,

 Marcus Smith
 Information Officer
 The Council for British Archaeology
 varnish-misc mailing list

varnish-misc mailing list

Re: Logging for multiple sites

2008-11-26 Thread David (Kitai) Cruz
So, if i've got 900 domains, do i have to start 900 varnishlog processes?



2008/11/26 Marcus Smith [EMAIL PROTECTED]:
 Dag-Erling Smørgrav wrote:
 Actually varnishncsa uses the same log filtering / selection code as
 varnishlog, which *can* select requests based on URL.  Extending
 varnishncsa to do the same should not be too hard.

 Other issues with varnishncsa / varnishlog: varnishlog doesn't allow -o
 and -w to be used at the same time.  There is no reason why it
 shouldn't.  If it did, you could play tricks like this:

 varnishlog -w /dev/stdout -c -o RxURL foo | varnishncsa -r /dev/stdin

 Ah, I see!  Hmmm.

 Well in that case, is there any reason why I shouldn't simply do
 something like:

  varnishlog -c -o RxHeader Host: (www\.)?example1\.com 

  varnishlog -c -o RxHeader Host: (www\.)?example2\.com 

 ...etc for each site, logging each to a separate varnish log file?

 I could then use varnishncsa's '-r' option to convert them into NCSA
 format once the logs are rotated out.  It seems like that would do
 pretty much what I want.

 Many thanks,

 Marcus Smith
 Information Officer
 The Council for British Archaeology
 varnish-misc mailing list

varnish-misc mailing list