Re: log size handling

2008-07-31 Thread z.szalbot

Hello,

 The restart process with apache is that all the child processes are
 shutdown -- either straight away (for a plain restart) or once they've
 finished their current bit of work (fora graceful restart). As the child
 process shuts down, it closes all the file descriptors used for logging.

I put two entries in newsyslog.conf. One for rotating all logs at the start
of the month, and the other one specifically for the site which gets many
hits to rotate its log once it grows more than 25MB in size.

/var/log/*-access.log   644  12*$M1D0 GZC 
/var/run/httpd.pid   30
/var/log/example.com-access.log644  10025000* ZC 
/var/run/httpd.pid   30

The log was indeed rotated when it grew larger than the specified size but
it was not rotated at the start of the month. Any idea why? Is it so that
the first entry effectively eliminates the second? If so, how can I make
sure the log does not grow too large and is rotated at the start of the
month?

Many thanks!

Zbigniew Szalbot

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-19 Thread Mel
On Thursday 17 July 2008 13:31:39 Zbigniew Szalbot wrote:

 Yes, I am compressing them. Load is not an issue yet but I want to
 optimize things that can be optimized. So I guess, rotate more often? I
 like the idea of having one monthly log because it is then processed by
 web stat software so it is easier to submit one than a couple of them.

Even though the awstats code is a bit kludgy perl (as most perl is I suppose), 
it is able to track the last line that it processed and so doesn't care if it 
gets half a log file or 'full' ones.

I haven't looked at the more recent logprocessors in a while, so there may be 
others. In short: you may want to look around for logprocessors that can 
handle intermediate logfiles and still correctly create periodical stats.

In these cases it's actually beneficial to do frequent rotations, because the 
arrays the software keeps in memory, are smaller by definition.
-- 
Mel

Problem with today's modular software: they start with the modules
and never get to the software part.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-18 Thread Zbigniew Szalbot

Hello,

Matthew Seaman:

Do I have to rotate them myself via a script in crontab?


There are several ways to do this.  Here's three in addition to the
script that someone else just posted:

 * Use the 'G' option to newsyslog.  'G' says that the filename
   field of newsyslog.conf actually contains a shell globbing pattern
   (wildcard) and files matching that should be rotated and compressed.


So would something like this work?

/var/log/site-*.log 644  12*$M1D0 GZC /var/run/httpd.pid

Many thanks in advance!

--
Zbigniew Szalbot
www.LCWords.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-18 Thread Matthew Seaman

Zbigniew Szalbot wrote:

Hello,

Matthew Seaman:

Do I have to rotate them myself via a script in crontab?


There are several ways to do this.  Here's three in addition to the
script that someone else just posted:

 * Use the 'G' option to newsyslog.  'G' says that the filename
   field of newsyslog.conf actually contains a shell globbing pattern
   (wildcard) and files matching that should be rotated and compressed.


So would something like this work?

/var/log/site-*.log 644  12*$M1D0 GZC /var/run/httpd.pid

Many thanks in advance!


Correct.  Although you may want to add '30' as the 8th field -- that means
'send signal 30 (SIGUSR1) to apache instead of SIGHUP' -- SIGUSR1 causes
apache to do a graceful restart rather than abruptly killing and restarting
everything: http://httpd.apache.org/docs/2.2/stopping.html

You'll need to experiment though -- if your user HTTP connections are 
long-lived, you can end up with apache appending data to a rotated logfile

that it still has an open file descriptor on.  The file will be unlinked
once the gzip compression has run.  Now, writing to an unlinked file is
allowed under Unix, but once that apache child process terminates and 
releases the descriptor the  data will disappear into the ether,  so you

can lose log entries.

Cheers,

Matthew

--
Dr Matthew J Seaman MA, D.Phil.   7 Priory Courtyard
 Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
 Kent, CT11 9PW



signature.asc
Description: OpenPGP digital signature


Re: log size handling

2008-07-18 Thread Zbigniew Szalbot

Hello,

Matthew Seaman:


Correct.  Although you may want to add '30' as the 8th field -- that means
'send signal 30 (SIGUSR1) to apache instead of SIGHUP' -- SIGUSR1 causes
apache to do a graceful restart rather than abruptly killing and restarting
everything: http://httpd.apache.org/docs/2.2/stopping.html

You'll need to experiment though -- if your user HTTP connections are 
long-lived, you can end up with apache appending data to a rotated logfile

that it still has an open file descriptor on.  The file will be unlinked
once the gzip compression has run.  Now, writing to an unlinked file is
allowed under Unix, but once that apache child process terminates and 
releases the descriptor the  data will disappear into the ether,  so you

can lose log entries.


Thank you very much Matthew! Just one final question. When the apache 
child process finally terminates, will apache start logging to a new 
log? Or will it be writing into the ether until it gets restarted?


Thanks again! I appreciate your help.

--
Zbigniew Szalbot
www.LCWords.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-18 Thread Matthew Seaman

Zbigniew Szalbot wrote:

Hello,

Matthew Seaman:

Correct.  Although you may want to add '30' as the 8th field -- that 
means

'send signal 30 (SIGUSR1) to apache instead of SIGHUP' -- SIGUSR1 causes
apache to do a graceful restart rather than abruptly killing and 
restarting

everything: http://httpd.apache.org/docs/2.2/stopping.html

You'll need to experiment though -- if your user HTTP connections are 
long-lived, you can end up with apache appending data to a rotated 
logfile

that it still has an open file descriptor on.  The file will be unlinked
once the gzip compression has run.  Now, writing to an unlinked file is
allowed under Unix, but once that apache child process terminates and 
releases the descriptor the  data will disappear into the ether,  so you

can lose log entries.


Thank you very much Matthew! Just one final question. When the apache 
child process finally terminates, will apache start logging to a new 
log? Or will it be writing into the ether until it gets restarted?


Thanks again! I appreciate your help.



The restart process with apache is that all the child processes are
shutdown -- either straight away (for a plain restart) or once they've
finished their current bit of work (fora graceful restart). As the child
process shuts down, it closes all the file descriptors used for logging.

Ultimately the master process will respawn itself -- in the event of a graceful 
restart this may occur before all of the old child processes
have closed down.  I'm not sure if the master process actually logs anything 
much, but even if it doesn't this will similarly close and
then reopen all file descriptors.

The renewed master process then starts up a whole new set of child
processes, each of which will open up new logging connections.

Now, if logging direct to a file, apache will open the file by name.
As newsyslog has just moved away the old file and created a new file
with the appropriate name, this means that all log output from the
new apache processes will go into the new log file.

Which is what you want.

Cheers,

Matthew


--
Dr Matthew J Seaman MA, D.Phil.   7 Priory Courtyard
 Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
 Kent, CT11 9PW



signature.asc
Description: OpenPGP digital signature


Re: log size handling

2008-07-17 Thread Bill Moran
In response to Zbigniew Szalbot [EMAIL PROTECTED]:

 Hello,
 
 I am just wondering if for the software like Apache the log size matters 
 at all. I rotate httpd logs monthly and each domain has its own log. One 
 is over 145 MB in size. Just curious if I can keep it like that or 
 should rotate more often? I have enough space on HD so the log's size is 
 of no concern for me. I guess I am just asking for best practices 
 advice... Thanks!

There are potential performance issues.  I believe that appending to a
file becomes more expensive when a file gets very large (which is
obviously the scenario with log files)

Also, at rotation time, if you're compressing those logs it can be
a pretty big load to compress 145M.  If you're not compressing and
not moving the files to another partition when you rotate them, then
this part probably isn't an issue.

-- 
Bill Moran
http://www.potentialtech.com
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-17 Thread Zbigniew Szalbot

Hi,

Bill Moran:

In response to Zbigniew Szalbot [EMAIL PROTECTED]:


Hello,

I am just wondering if for the software like Apache the log size
matters at all. I rotate httpd logs monthly and each domain has its
own log. One is over 145 MB in size. Just curious if I can keep it
like that or should rotate more often? I have enough space on HD so
the log's size is of no concern for me. I guess I am just asking
for best practices advice... Thanks!


There are potential performance issues.  I believe that appending to
a file becomes more expensive when a file gets very large (which is 
obviously the scenario with log files)


Thank you Bill - this is what I had in mind - whether appending to a
large file will not be an issue? It is currently about 145MB. By the end
of month it will surely be around 300MB.


Also, at rotation time, if you're compressing those logs it can be a
pretty big load to compress 145M.  If you're not compressing and not
moving the files to another partition when you rotate them, then this
part probably isn't an issue.


Yes, I am compressing them. Load is not an issue yet but I want to
optimize things that can be optimized. So I guess, rotate more often? I 
like the idea of having one monthly log because it is then processed by 
web stat software so it is easier to submit one than a couple of them.


Thanks!

--
Zbigniew Szalbot
www.LCWords.com


smime.p7s
Description: S/MIME Cryptographic Signature


Re: log size handling

2008-07-17 Thread Mario Lobo
On Thursday 17 July 2008, Zbigniew Szalbot wrote:
 Hi,

 Bill Moran:
  In response to Zbigniew Szalbot [EMAIL PROTECTED]:
  Hello,
 
  I am just wondering if for the software like Apache the log size
  matters at all. I rotate httpd logs monthly and each domain has its
  own log. One is over 145 MB in size. Just curious if I can keep it
  like that or should rotate more often? I have enough space on HD so
  the log's size is of no concern for me. I guess I am just asking
  for best practices advice... Thanks!
 
  There are potential performance issues.  I believe that appending to
  a file becomes more expensive when a file gets very large (which is
  obviously the scenario with log files)

 Thank you Bill - this is what I had in mind - whether appending to a
 large file will not be an issue? It is currently about 145MB. By the end
 of month it will surely be around 300MB.

  Also, at rotation time, if you're compressing those logs it can be a
  pretty big load to compress 145M.  If you're not compressing and not
  moving the files to another partition when you rotate them, then this
  part probably isn't an issue.

 Yes, I am compressing them. Load is not an issue yet but I want to
 optimize things that can be optimized. So I guess, rotate more often? I
 like the idea of having one monthly log because it is then processed by
 web stat software so it is easier to submit one than a couple of them.

 Thanks!

What would be the proper way to set the rotation of apache logs in 
newsyslog.conf when there are separate log files for each virtual host?

Do I have to rotate them myself via a script in crontab?

Thanks

-- 
Mario Lobo
Segurança de Redes - Desenvolvimento e Análise
IPAD - Instituto de Pesquisa e Apoio ao Desenvolvimento Tecnológico e 
Científico


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-17 Thread Volodymyr Kostyrko

Mario Lobo wrote:

On Thursday 17 July 2008, Zbigniew Szalbot wrote:


What would be the proper way to set the rotation of apache logs in 
newsyslog.conf when there are separate log files for each virtual host?


/var/log/httpd/*.log www:wheel 644 7 102400 * JG /var/run/httpd.pid 30


Do I have to rotate them myself via a script in crontab?


If you want it.

ls /var/log/httpd/*.bz2 | sed 's|\.\([0-9]*\)\.|,\1,|' | awk 
'BEGIN{FS=,}{printmv $1.$2.$3 $1.$2+1.$3}' | sh

ls /var/log/httpd/*.log | xargs -n1 -Ifoo mv foo foo.0
kill -30 `cat /var/run/httpd.pid`
sleep 1
bzip2 -9 /var/log/httpd/*.0
find /var/log/httpd/ -type -f -mtime +7 -delete

--
Sphinx of black quartz judge my vow.

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: log size handling

2008-07-17 Thread Matthew Seaman

Mario Lobo wrote:

What would be the proper way to set the rotation of apache logs in 
newsyslog.conf when there are separate log files for each virtual host?


Do I have to rotate them myself via a script in crontab?


There are several ways to do this.  Here's three in addition to the
script that someone else just posted:

 * Use the 'G' option to newsyslog.  'G' says that the filename
   field of newsyslog.conf actually contains a shell globbing pattern
   (wildcard) and files matching that should be rotated and compressed.

 * Use the rotatelogs program that comes with Apache.  In this case,
   you replace the logging configuration statements in https.conf
   eg. instead of:

CustomLog /var/log/httpd-access.log combine

   you have:

CustomLog |/usr/local/sbin/rotatelogs /var/log/httpd-access.log 86400

   which will create a new log file every 86400 seconds (= 1 day) and
   label each one with the unix time it was created appended to the
   name. eg:

   /var/log/httpd-access.log.1216252800

   That will change files at midnight UTC each day, which is fine
   if your server lives in the UK but not quite as convenient if your
   server is in Australia.

 * Use the conceptually very similar program 'cronolog' (in ports as
   sysutils/cronolog) This works like 'rotatelogs' except that you give
   it a date-time pattern using the %-escapes as in strftime(3) -- so
   
   CustomLog |/usr/local/sbin/cronolog /var/log/%Y/%m/%d/httpd-access.log


   will create /var/log/2008/07/17/httpd-access.log  for today's
   traffic.  Timestamps are based on the TZ setting the server uses
   I believe, so would likely be the same as local wall-clock time.

With the last two, there's no facility to compress or delete old log
files: however it is pretty simple to write a very small shell script to
do either of those things.  The last two have the subtle advantage that
you don't need to signal apache at file rotation time, which means there
is absolutely no effect on performance when it switches to a new log file.  
You could switch log files every hour on a really busy site and not

notice the overhead.

Cheers,

Matthew

--
Dr Matthew J Seaman MA, D.Phil.   7 Priory Courtyard
 Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
 Kent, CT11 9PW



signature.asc
Description: OpenPGP digital signature


Re: log size handling

2008-07-17 Thread David Newman

On 7/17/08 7:10 AM, Matthew Seaman wrote:


 * Use the rotatelogs program that comes with Apache.  In this case,
   you replace the logging configuration statements in https.conf
   eg. instead of:

CustomLog /var/log/httpd-access.log combine

   you have:

CustomLog |/usr/local/sbin/rotatelogs /var/log/httpd-access.log 
86400


   which will create a new log file every 86400 seconds (= 1 day) and
   label each one with the unix time it was created appended to the
   name. eg:

   /var/log/httpd-access.log.1216252800

   That will change files at midnight UTC each day, which is fine
   if your server lives in the UK but not quite as convenient if your
   server is in Australia.


Yes. Apache's rotatelogs is a better choice than newsyslog. Under heavy 
load, the latter can corrupt Apache logs.


rotatelogs also can do changes based on log size, like this:

CustomLog |/usr/local/sbin/rotatelogs/var/log/httpd-access.log 5M \
combined

and if you don't like deciphering Unix epoch time, you can embed times 
in your log filenames like this:


CustomLog |/usr/local/sbin/rotatelogs \
/var/log/httpd-access.log.%Y-%m-%d-%H_%M_%S 86400 combined

(n.b. I added the backslashes in the examples here; you want everything 
on a single line in your config file.)


dn


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]