Re: log size handling
Hello, > The restart process with apache is that all the child processes are > shutdown -- either straight away (for a plain restart) or once they've > finished their current bit of work (fora graceful restart). As the child > process shuts down, it closes all the file descriptors used for logging. I put two entries in newsyslog.conf. One for rotating all logs at the start of the month, and the other one specifically for the site which gets many hits to rotate its log once it grows more than 25MB in size. /var/log/*-access.log 644 12*$M1D0 GZC /var/run/httpd.pid 30 /var/log/example.com-access.log644 10025000* ZC /var/run/httpd.pid 30 The log was indeed rotated when it grew larger than the specified size but it was not rotated at the start of the month. Any idea why? Is it so that the first entry effectively eliminates the second? If so, how can I make sure the log does not grow too large and is rotated at the start of the month? Many thanks! Zbigniew Szalbot ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
On Thursday 17 July 2008 13:31:39 Zbigniew Szalbot wrote: > Yes, I am compressing them. Load is not an issue yet but I want to > optimize things that can be optimized. So I guess, rotate more often? I > like the idea of having one monthly log because it is then processed by > web stat software so it is easier to submit one than a couple of them. Even though the awstats code is a bit kludgy perl (as most perl is I suppose), it is able to track the last line that it processed and so doesn't care if it gets half a log file or 'full' ones. I haven't looked at the more recent logprocessors in a while, so there may be others. In short: you may want to look around for logprocessors that can handle intermediate logfiles and still correctly create periodical stats. In these cases it's actually beneficial to do frequent rotations, because the arrays the software keeps in memory, are smaller by definition. -- Mel Problem with today's modular software: they start with the modules and never get to the software part. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
Zbigniew Szalbot wrote: Hello, Matthew Seaman: Correct. Although you may want to add '30' as the 8th field -- that means 'send signal 30 (SIGUSR1) to apache instead of SIGHUP' -- SIGUSR1 causes apache to do a graceful restart rather than abruptly killing and restarting everything: http://httpd.apache.org/docs/2.2/stopping.html You'll need to experiment though -- if your user HTTP connections are long-lived, you can end up with apache appending data to a rotated logfile that it still has an open file descriptor on. The file will be unlinked once the gzip compression has run. Now, writing to an unlinked file is allowed under Unix, but once that apache child process terminates and releases the descriptor the data will disappear into the ether, so you can lose log entries. Thank you very much Matthew! Just one final question. When the apache child process finally terminates, will apache start logging to a new log? Or will it be writing into the ether until it gets restarted? Thanks again! I appreciate your help. The restart process with apache is that all the child processes are shutdown -- either straight away (for a plain restart) or once they've finished their current bit of work (fora graceful restart). As the child process shuts down, it closes all the file descriptors used for logging. Ultimately the master process will respawn itself -- in the event of a graceful restart this may occur before all of the old child processes have closed down. I'm not sure if the master process actually logs anything much, but even if it doesn't this will similarly close and then reopen all file descriptors. The renewed master process then starts up a whole new set of child processes, each of which will open up new logging connections. Now, if logging direct to a file, apache will open the file by name. As newsyslog has just moved away the old file and created a new file with the appropriate name, this means that all log output from the new apache processes will go into the new log file. Which is what you want. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: log size handling
Hello, Matthew Seaman: Correct. Although you may want to add '30' as the 8th field -- that means 'send signal 30 (SIGUSR1) to apache instead of SIGHUP' -- SIGUSR1 causes apache to do a graceful restart rather than abruptly killing and restarting everything: http://httpd.apache.org/docs/2.2/stopping.html You'll need to experiment though -- if your user HTTP connections are long-lived, you can end up with apache appending data to a rotated logfile that it still has an open file descriptor on. The file will be unlinked once the gzip compression has run. Now, writing to an unlinked file is allowed under Unix, but once that apache child process terminates and releases the descriptor the data will disappear into the ether, so you can lose log entries. Thank you very much Matthew! Just one final question. When the apache child process finally terminates, will apache start logging to a new log? Or will it be writing into the ether until it gets restarted? Thanks again! I appreciate your help. -- Zbigniew Szalbot www.LCWords.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
Zbigniew Szalbot wrote: Hello, Matthew Seaman: Do I have to rotate them myself via a script in crontab? There are several ways to do this. Here's three in addition to the script that someone else just posted: * Use the 'G' option to newsyslog. 'G' says that the filename field of newsyslog.conf actually contains a shell globbing pattern (wildcard) and files matching that should be rotated and compressed. So would something like this work? /var/log/site-*.log 644 12*$M1D0 GZC /var/run/httpd.pid Many thanks in advance! Correct. Although you may want to add '30' as the 8th field -- that means 'send signal 30 (SIGUSR1) to apache instead of SIGHUP' -- SIGUSR1 causes apache to do a graceful restart rather than abruptly killing and restarting everything: http://httpd.apache.org/docs/2.2/stopping.html You'll need to experiment though -- if your user HTTP connections are long-lived, you can end up with apache appending data to a rotated logfile that it still has an open file descriptor on. The file will be unlinked once the gzip compression has run. Now, writing to an unlinked file is allowed under Unix, but once that apache child process terminates and releases the descriptor the data will disappear into the ether, so you can lose log entries. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: log size handling
Hello, Matthew Seaman: Do I have to rotate them myself via a script in crontab? There are several ways to do this. Here's three in addition to the script that someone else just posted: * Use the 'G' option to newsyslog. 'G' says that the filename field of newsyslog.conf actually contains a shell globbing pattern (wildcard) and files matching that should be rotated and compressed. So would something like this work? /var/log/site-*.log 644 12*$M1D0 GZC /var/run/httpd.pid Many thanks in advance! -- Zbigniew Szalbot www.LCWords.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
On 7/17/08 7:10 AM, Matthew Seaman wrote: * Use the rotatelogs program that comes with Apache. In this case, you replace the logging configuration statements in https.conf eg. instead of: CustomLog "/var/log/httpd-access.log" combine you have: CustomLog "|/usr/local/sbin/rotatelogs /var/log/httpd-access.log 86400" which will create a new log file every 86400 seconds (= 1 day) and label each one with the unix time it was created appended to the name. eg: /var/log/httpd-access.log.1216252800 That will change files at midnight UTC each day, which is fine if your server lives in the UK but not quite as convenient if your server is in Australia. Yes. Apache's rotatelogs is a better choice than newsyslog. Under heavy load, the latter can corrupt Apache logs. rotatelogs also can do changes based on log size, like this: CustomLog "|/usr/local/sbin/rotatelogs/var/log/httpd-access.log 5M" \ combined and if you don't like deciphering Unix epoch time, you can embed times in your log filenames like this: CustomLog "|/usr/local/sbin/rotatelogs \ /var/log/httpd-access.log.%Y-%m-%d-%H_%M_%S 86400" combined (n.b. I added the backslashes in the examples here; you want everything on a single line in your config file.) dn ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
Mario Lobo wrote: What would be the proper way to set the rotation of apache logs in newsyslog.conf when there are separate log files for each virtual host? Do I have to rotate them myself via a script in crontab? There are several ways to do this. Here's three in addition to the script that someone else just posted: * Use the 'G' option to newsyslog. 'G' says that the filename field of newsyslog.conf actually contains a shell globbing pattern (wildcard) and files matching that should be rotated and compressed. * Use the rotatelogs program that comes with Apache. In this case, you replace the logging configuration statements in https.conf eg. instead of: CustomLog "/var/log/httpd-access.log" combine you have: CustomLog "|/usr/local/sbin/rotatelogs /var/log/httpd-access.log 86400" which will create a new log file every 86400 seconds (= 1 day) and label each one with the unix time it was created appended to the name. eg: /var/log/httpd-access.log.1216252800 That will change files at midnight UTC each day, which is fine if your server lives in the UK but not quite as convenient if your server is in Australia. * Use the conceptually very similar program 'cronolog' (in ports as sysutils/cronolog) This works like 'rotatelogs' except that you give it a date-time pattern using the %-escapes as in strftime(3) -- so CustomLog "|/usr/local/sbin/cronolog /var/log/%Y/%m/%d/httpd-access.log" will create /var/log/2008/07/17/httpd-access.log for today's traffic. Timestamps are based on the TZ setting the server uses I believe, so would likely be the same as local wall-clock time. With the last two, there's no facility to compress or delete old log files: however it is pretty simple to write a very small shell script to do either of those things. The last two have the subtle advantage that you don't need to signal apache at file rotation time, which means there is absolutely no effect on performance when it switches to a new log file. You could switch log files every hour on a really busy site and not notice the overhead. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW signature.asc Description: OpenPGP digital signature
Re: log size handling
Mario Lobo wrote: On Thursday 17 July 2008, Zbigniew Szalbot wrote: What would be the proper way to set the rotation of apache logs in newsyslog.conf when there are separate log files for each virtual host? /var/log/httpd/*.log www:wheel 644 7 102400 * JG /var/run/httpd.pid 30 Do I have to rotate them myself via a script in crontab? If you want it. ls /var/log/httpd/*.bz2 | sed 's|\.\([0-9]*\)\.|,\1,|' | awk 'BEGIN{FS=","}{print"mv "$1"."$2"."$3" "$1"."$2+1"."$3}' | sh ls /var/log/httpd/*.log | xargs -n1 -Ifoo mv foo foo.0 kill -30 `cat /var/run/httpd.pid` sleep 1 bzip2 -9 /var/log/httpd/*.0 find /var/log/httpd/ -type -f -mtime +7 -delete -- Sphinx of black quartz judge my vow. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
On Thursday 17 July 2008, Zbigniew Szalbot wrote: > Hi, > > Bill Moran: > > In response to Zbigniew Szalbot <[EMAIL PROTECTED]>: > >> Hello, > >> > >> I am just wondering if for the software like Apache the log size > >> matters at all. I rotate httpd logs monthly and each domain has its > >> own log. One is over 145 MB in size. Just curious if I can keep it > >> like that or should rotate more often? I have enough space on HD so > >> the log's size is of no concern for me. I guess I am just asking > >> for best practices advice... Thanks! > > > > There are potential performance issues. I believe that appending to > > a file becomes more expensive when a file gets very large (which is > > obviously the scenario with log files) > > Thank you Bill - this is what I had in mind - whether appending to a > large file will not be an issue? It is currently about 145MB. By the end > of month it will surely be around 300MB. > > > Also, at rotation time, if you're compressing those logs it can be a > > pretty big load to compress 145M. If you're not compressing and not > > moving the files to another partition when you rotate them, then this > > part probably isn't an issue. > > Yes, I am compressing them. Load is not an issue yet but I want to > optimize things that can be optimized. So I guess, rotate more often? I > like the idea of having one monthly log because it is then processed by > web stat software so it is easier to submit one than a couple of them. > > Thanks! What would be the proper way to set the rotation of apache logs in newsyslog.conf when there are separate log files for each virtual host? Do I have to rotate them myself via a script in crontab? Thanks -- Mario Lobo Segurança de Redes - Desenvolvimento e Análise IPAD - Instituto de Pesquisa e Apoio ao Desenvolvimento Tecnológico e Científico ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: log size handling
Hi, Bill Moran: In response to Zbigniew Szalbot <[EMAIL PROTECTED]>: Hello, I am just wondering if for the software like Apache the log size matters at all. I rotate httpd logs monthly and each domain has its own log. One is over 145 MB in size. Just curious if I can keep it like that or should rotate more often? I have enough space on HD so the log's size is of no concern for me. I guess I am just asking for best practices advice... Thanks! There are potential performance issues. I believe that appending to a file becomes more expensive when a file gets very large (which is obviously the scenario with log files) Thank you Bill - this is what I had in mind - whether appending to a large file will not be an issue? It is currently about 145MB. By the end of month it will surely be around 300MB. Also, at rotation time, if you're compressing those logs it can be a pretty big load to compress 145M. If you're not compressing and not moving the files to another partition when you rotate them, then this part probably isn't an issue. Yes, I am compressing them. Load is not an issue yet but I want to optimize things that can be optimized. So I guess, rotate more often? I like the idea of having one monthly log because it is then processed by web stat software so it is easier to submit one than a couple of them. Thanks! -- Zbigniew Szalbot www.LCWords.com smime.p7s Description: S/MIME Cryptographic Signature
Re: log size handling
In response to Zbigniew Szalbot <[EMAIL PROTECTED]>: > Hello, > > I am just wondering if for the software like Apache the log size matters > at all. I rotate httpd logs monthly and each domain has its own log. One > is over 145 MB in size. Just curious if I can keep it like that or > should rotate more often? I have enough space on HD so the log's size is > of no concern for me. I guess I am just asking for best practices > advice... Thanks! There are potential performance issues. I believe that appending to a file becomes more expensive when a file gets very large (which is obviously the scenario with log files) Also, at rotation time, if you're compressing those logs it can be a pretty big load to compress 145M. If you're not compressing and not moving the files to another partition when you rotate them, then this part probably isn't an issue. -- Bill Moran http://www.potentialtech.com ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "[EMAIL PROTECTED]"
log size handling
Hello, I am just wondering if for the software like Apache the log size matters at all. I rotate httpd logs monthly and each domain has its own log. One is over 145 MB in size. Just curious if I can keep it like that or should rotate more often? I have enough space on HD so the log's size is of no concern for me. I guess I am just asking for best practices advice... Thanks! -- Zbigniew Szalbot www.LCWords.com smime.p7s Description: S/MIME Cryptographic Signature