Re: Apache Rotate Logs and Log Rotate.
On 3/3/07, Peter Pluta [EMAIL PROTECTED] wrote: I see, thanks. Does the shell script you use automatically delete the original logs after verbalizer or awstats makes it's own? I imagine the ones those programs use are smaller in size? No, the shell script does not delete any logs. Log rotation and compression is the job of newsyslog. Alright, after some more RTFM on Apache logs, here's what your newsyslog.conf(5) configuration should look like. /var/log/httpd/access.log640 5 1048576 * B /var/run/httpd.pid 30 /var/log/httpd/error.log640 5 1048576 * B /var/run/httpd.pid 30 Of course, you should taylor this to suit your own needs (like the size, ownership and number logs kept on disk, etc.) But keep the B flag for Binary which will prevent newsyslog from adding a line in your logs which says it was rotated. It _may_ confuse some log analyser (depends on your log analyser software). Also make sure to add the 30 at the end of each line. This is the kill(1) number for signal -USR1 which gracefully restarts Apache. Now the reason I removed the Z flags, which eliminates compression, is to make sure all of your children httpd processes have enough time to write their logs into the log file. If a request on your site is rather long, them this is the best way to go. Of couse, that means you will need a little bit more disk space. But not that much depending on how much logs you keep (i.e. 5 in the example above). HTH, David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
Thanks, David. I had already configured it like that the first time around after reading up on it a bit. Most articles/tips I have read say to wait 10 minutes or so and then compress the logs with a shell script in order to be sure Apache finished logging to the files. Another thing, just to be sure. If I had 30 vhosts on my server and each had logs in their home directory, I would still use newsyslog to rotate and delete them, correct? I assume one needs tons of disk space to do that if the sites are rather large. David Robillard wrote: On 3/3/07, Peter Pluta [EMAIL PROTECTED] wrote: I see, thanks. Does the shell script you use automatically delete the original logs after verbalizer or awstats makes it's own? I imagine the ones those programs use are smaller in size? No, the shell script does not delete any logs. Log rotation and compression is the job of newsyslog. Alright, after some more RTFM on Apache logs, here's what your newsyslog.conf(5) configuration should look like. /var/log/httpd/access.log640 5 1048576 * B /var/run/httpd.pid 30 /var/log/httpd/error.log640 5 1048576 * B /var/run/httpd.pid 30 Of course, you should taylor this to suit your own needs (like the size, ownership and number logs kept on disk, etc.) But keep the B flag for Binary which will prevent newsyslog from adding a line in your logs which says it was rotated. It _may_ confuse some log analyser (depends on your log analyser software). Also make sure to add the 30 at the end of each line. This is the kill(1) number for signal -USR1 which gracefully restarts Apache. Now the reason I removed the Z flags, which eliminates compression, is to make sure all of your children httpd processes have enough time to write their logs into the log file. If a request on your site is rather long, them this is the best way to go. Of couse, that means you will need a little bit more disk space. But not that much depending on how much logs you keep (i.e. 5 in the example above). HTH, David ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
On 3/5/07, Peter Pluta [EMAIL PROTECTED] wrote: Thanks, David. I had already configured it like that the first time around after reading up on it a bit. Most articles/tips I have read say to wait 10 minutes or so and then compress the logs with a shell script in order to be sure Apache finished logging to the files. Another thing, just to be sure. If I had 30 vhosts on my server and each had logs in their home directory, I would still use newsyslog to rotate and delete them, correct? I assume one needs tons of disk space to do that if the sites are rather large. Well, if you do use newsyslog to rotate Apache log files, then it's just a matter of setting the number of files you whish to keep. From newsyslog.conf(5) count Specify the maximum number of archive files which may exist. This does not consider the current log file. Let's say you rotate your files once they reach 2Mb for example and that you've configured 10 in your newsyslog,conf count field. Then that means a maximum of 10 x 2Mb = 20Mb will be kept for one VirtualHost. Now if you have 100 virtual hosts all configured this way, then you will need 100 x 20Mb = 2000Mb or 2Gb for all your Apache logs. Considering today's disk drive sizes are well beyond the 300Gb, I don't think this is a problem at all. Of course, YMMV so check your own needs and do the math. Cheers, David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
Gotcha, do you use a script to compress the logs after the SIGUSR1 and after waiting for a bit for apache to clear it's logging buffer (to not have missing logs)? David Robillard wrote: On 3/5/07, Peter Pluta [EMAIL PROTECTED] wrote: Thanks, David. I had already configured it like that the first time around after reading up on it a bit. Most articles/tips I have read say to wait 10 minutes or so and then compress the logs with a shell script in order to be sure Apache finished logging to the files. Another thing, just to be sure. If I had 30 vhosts on my server and each had logs in their home directory, I would still use newsyslog to rotate and delete them, correct? I assume one needs tons of disk space to do that if the sites are rather large. Well, if you do use newsyslog to rotate Apache log files, then it's just a matter of setting the number of files you whish to keep. From newsyslog.conf(5) count Specify the maximum number of archive files which may exist. This does not consider the current log file. Let's say you rotate your files once they reach 2Mb for example and that you've configured 10 in your newsyslog,conf count field. Then that means a maximum of 10 x 2Mb = 20Mb will be kept for one VirtualHost. Now if you have 100 virtual hosts all configured this way, then you will need 100 x 20Mb = 2000Mb or 2Gb for all your Apache logs. Considering today's disk drive sizes are well beyond the 300Gb, I don't think this is a problem at all. Of course, YMMV so check your own needs and do the math. Cheers, David ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
On 3/5/07, Peter Pluta [EMAIL PROTECTED] wrote: Gotcha, do you use a script to compress the logs after the SIGUSR1 and after waiting for a bit for apache to clear it's logging buffer (to not have missing logs)? No I don't. I don't even see why one would want to do this? Newsyslog deletes extra logs. So if our disk space is enough to hold the amount of logs we require (see math below), then there's no need to compress any Apache logs at all. Right!?!! If we come back to my example of 100 VirtualHost with log files of 2Mb each and we keep only 10 of them. Using USR1 as the kill signal, For an httpd children to miss any log entry would mean that this children writes more than 10 times 2Mb of logs in a very short period of time. Check your VirtualHost load and determine the average response time for each httpd children. If it's 2min (which is HUGE for an httpd children) That would mean that you'd need to have more than 20Mb of logs generated in less than 2min. In ASCII, that's a whole lot of logs. I'd say your best bet would be to switch your LogLevel from debug to info in your httpd.conf and restart Apache... ;) Or you run a really busy website. Or your web application code/architecture may need a revision. Have fun! David Well, if you do use newsyslog to rotate Apache log files, then it's just a matter of setting the number of files you whish to keep. From newsyslog.conf(5) count Specify the maximum number of archive files which may exist. This does not consider the current log file. Let's say you rotate your files once they reach 2Mb for example and that you've configured 10 in your newsyslog,conf count field. Then that means a maximum of 10 x 2Mb = 20Mb will be kept for one VirtualHost. Now if you have 100 virtual hosts all configured this way, then you will need 100 x 20Mb = 2000Mb or 2Gb for all your Apache logs. Considering today's disk drive sizes are well beyond the 300Gb, I don't think this is a problem at all. Of course, YMMV so check your own needs and do the math. Cheers, David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
I made a cron entry to compress the logs. We keep them for 10 days. Each log on this server is 940mb for the access and 430mb for the error. This site is very busy and I like having logs for various reasons (dump users uploading illegal content is one reason). The memory buffer clears out in around 10 minutes because the new log started 10 minutes after the old ended. I added 306 * * bzip2 -1q /var/log/http-*.log.* to cron. I think this is a pretty good setup. Our log error level is set to info as well. The web application (php driven) is not spitting up that many errors. Most of them are missing images that were deleted and just broken old links. Thanks for all you're replies David. I really learned a lot and i know understand much more. I started carbon copying the emails to the mailing list in case anyone is interested in this as well. David Robillard wrote: On 3/5/07, Peter Pluta [EMAIL PROTECTED] wrote: Gotcha, do you use a script to compress the logs after the SIGUSR1 and after waiting for a bit for apache to clear it's logging buffer (to not have missing logs)? No I don't. I don't even see why one would want to do this? Newsyslog deletes extra logs. So if our disk space is enough to hold the amount of logs we require (see math below), then there's no need to compress any Apache logs at all. Right!?!! If we come back to my example of 100 VirtualHost with log files of 2Mb each and we keep only 10 of them. Using USR1 as the kill signal, For an httpd children to miss any log entry would mean that this children writes more than 10 times 2Mb of logs in a very short period of time. Check your VirtualHost load and determine the average response time for each httpd children. If it's 2min (which is HUGE for an httpd children) That would mean that you'd need to have more than 20Mb of logs generated in less than 2min. In ASCII, that's a whole lot of logs. I'd say your best bet would be to switch your LogLevel from debug to info in your httpd.conf and restart Apache... ;) Or you run a really busy website. Or your web application code/architecture may need a revision. Have fun! David Well, if you do use newsyslog to rotate Apache log files, then it's just a matter of setting the number of files you whish to keep. From newsyslog.conf(5) count Specify the maximum number of archive files which may exist. This does not consider the current log file. Let's say you rotate your files once they reach 2Mb for example and that you've configured 10 in your newsyslog,conf count field. Then that means a maximum of 10 x 2Mb = 20Mb will be kept for one VirtualHost. Now if you have 100 virtual hosts all configured this way, then you will need 100 x 20Mb = 2000Mb or 2Gb for all your Apache logs. Considering today's disk drive sizes are well beyond the 300Gb, I don't think this is a problem at all. Of course, YMMV so check your own needs and do the math. Cheers, David ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
On 3/3/07, Peter Pluta [EMAIL PROTECTED] wrote: I see, thanks. Does the shell script you use automatically delete the original logs after verbalizer or awstats makes it's own? I imagine the ones those programs use are smaller in size? No, the shell script does not delete any logs. Log rotation and compression is the job of newsyslog. Webalizer creates and maintains his own files which grow slowly over time. How fast they grow depends on how busy your site is and how much data you need to extract from the logs. Try it on one VirtualHost and you'll see. If you like it, then extend your configuration to your other VirtualHosts. Talking about logs, you might want to send them to syslog. Here's a quick article on this topic: http://www.oreillynet.com/pub/a/sysadmin/2006/10/12/httpd-syslog.html Cheers, David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
On 3/1/07, Peter Pluta [EMAIL PROTECTED] wrote: What I did was made a new log format to include the %v (it includes the vhost name in the logs). Lowered my error log to just info. I also got rid of the errorlog and customlog in my vhost brackets and setup newsyslog to rotate the http-access.log and http-error.log after 24 hours. This is what I pretty much wanted. I have more space in /home/ now since there are no log files in there and I also have 1 main log that I can rotate and view or separate if needed. It makes it a lot easier. I have a quick question though. Say I am hosting a few sites for customers and they want to run their own statistics programs that rely on log files. How would I deal with the logs if they were in each users home directory? Those logs add up after a week or so; not to mention if someone had a larger site that generated larger logs. What exactly could be done in that situation to allow stats and still have a functional web server? Hi Peter, What I do with stats is use webalizer which is available from the ports directory as www/webalizer. Webalizer keeps the history of your logs, so you don't have to keep the old ones around. I run webalizer from cron once and a while to generate stats. I've wraped it in a simple shell script to check all my virtual sites listed in a custom config file in /usr/local/etc and dump the stats file into /path/to/virtual/host/stats. I then setup a /stats Alias in httpd.conf for each virtual site and protect it with a simple .htpasswd. Easy. BTW, may I suggest you also include the freebsd-questions list in Cc when you write back? Some people might be interested by what we're talking about. In fact, ideally we should only 'talk' via the list, but that's ok with me. Cheers, David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
On 2/28/07, Peter Pluta [EMAIL PROTECTED] wrote: Hey David, quick question. I found this while doing a bit of reading. Is it safe for Syslogd to send a kill -HUP to apache? This site is extremely high traffic and I wouldn't want it cutting off users during the HUP to rotate the logs. I'm running Apache 2.2.4 and FreeBSD 6.2 http://www.freebsddiary.org/startstop.php It looks like Apachectl graceful is the only safe way to restart apache. Hi Peter, The article you're refering to is for Apache 1.3.x and you seem to be running 2.2.x Should you want, you can get more detailed information on how Apache 1.3.x handles kill signals here: http://httpd.apache.org/docs/1.3/stopping.html It's basically the same for Apache 2.2.x which is covered here: http://httpd.apache.org/docs/2.2/stopping.html Having said that, if your site is really busy, then consider changing the kill signal in newsyslog.conf from -HUP to -USR1 which will gracefully ask running httpd processes to restart once they have finished talking to their user. As the article says: ''The USR1 signal causes the parent process to advise the children to exit after their current request (or to exit immediately if they're not serving anything). The parent re-reads its configuration files and re-opens its log files. As each child dies off the parent replaces it with a child from the new generation of the configuration, which begins serving new requests immediately.'' Check the man page for newsyslog.conf(5) at http://www.freebsd.org/cgi/man.cgi?query=newsyslog.confapropos=0sektion=0manpath=FreeBSD+6.2-RELEASEformat=html The last field in newsyslog.conf is where you setup which signal is used. Here's what the man page says: signal_number This optional field specifies the signal number that will be sent to the daemon process (or to all processes in a process group, if the U flag was specified). If this field is not present, then a SIGHUP signal will be sent. Cheers, David David Robillard wrote: Hi Peter, Someone told me that I need to gracefully restart apache for it to make a new log; and then wait till Apache's memory buffer is emptied to disk before gziping or bziping the files. Well, I've never had to do this. Newsyslog send a `kill -HUP` to apache's master PID. Which causes Apache to reopen it's log files. For me anyway, the newsyslog configuration I gave you never caused me any problem at all. Keep in mind that you do have to send Apache a -HUP signal, otherwise you'll lose logs when newsyslog rotates them. Also, is it wise to have logs for each user in their home directory? Someone told me this is a serious security issue; but I can't see why it would be. It is a security issue if the user has the rights to login to you machine. If he dosen't, then you shouldn't be worried. But I just don't take that chance and make all of my Apache log files under /usr/local/www/virtalhost1/logs which is not accessible from Apache itself because I setup my DocumentRoot under /usr/local/www/virtalhost1/public_html. This way, I know for sure that everything for virtualhost1 is under a single directory, but that my logs can't be seen by anyone via Apache. David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate
Peter Pluta wrote: I have Apache making separate log files for each of my virtual hosts and putting them in /home/vhostname/log. Rotate logs makes a new log every 24 hours, but the logs quickly add up and since the sites are fairly busy the logs are at times over 5gigs. Is there any way to make rotate logs delete the log files after two days? Someone recommended me Log Rotate (from the ports tree), but this program does basically what Rotate logs does; except it makes things more complicated because it needs to restart apache and such. Is there a easy way to just have Apache's rotatelogs rotate the logs and then delete them after two days? Any feedback, suggestions, or comments would be greatly appreciated. rotatelogs doesn't do any sort of deletion stuff. It just doesn't have that capability. On the other hand a simple cronjob that deletes all but the N newest files in the directory is just a small matter of scripting. Assuming you want to keep at least 30 of the latest log files, you can generate a list of files to delete by something like: ls -1t /home/vhostname/log.* | sed -n -e '30,$p' Alternatively you can abuse the daily_clean_tmps periodic job to delete any files from that directory over a certain age. Cheers, Matthew -- Dr Matthew J Seaman MA, D.Phil. Flat 3 7 Priory Courtyard PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate Kent, CT11 9PW, UK signature.asc Description: OpenPGP digital signature
Re: Apache Rotate Logs and Log Rotate
Hi Peter, Peter Pluta wrote: Any feedback, suggestions, or comments would be greatly appreciated. you can use newsyslog for this, see man newsyslog.conf for more details. I use e.g. the following line: /home/http/*/logs/*.log 664 72*$M1D0 JG /var/run/httpd.pid Best regards, Matthias -- Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to produce bigger and better idiots. So far, the universe is winning. -- Rich Cook ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Apache Rotate Logs and Log Rotate.
I have Apache making separate log files for each of my virtual hosts and putting them in /home/vhostname/log. Rotate logs makes a new log every 24 hours, but the logs quickly add up and since the sites are fairly busy the logs are at times over 5gigs. Is there any way to make rotate logs delete the log files after two days? Someone recommended me Log Rotate (from the ports tree), but this program does basically what Rotate logs does; except it makes things more complicated because it needs to restart apache and such. Is there a easy way to just have Apache's rotatelogs rotate the logs and then delete them after two days? Any feedback, suggestions, or comments would be greatly appreciated. Hi Peter, I personaly don't use neither Log Rotate nor Rotate Logs, but configure newsyslog.conf(5) to handle the job of Apache log rotation and clean-up. The newsyslog software is part of FreeBSD's base system, so you don't need to install anything. Just configure /etc/newsyslog.conf and that's it. No need to restart anything because newsyslog is already active in FreeBSD's base system via /etc/crontab. It can rotate the logs, compress them with either gzip(1) or bzip2(1) and remove the old ones to preserve disk space. For example, let's say you have two virtual host's logs into /home/vhostname1/log and /home/vhostname2/log, you can configure newsyslog to: a) Keep only 10 log files. Remove the older ones as they grow. (i.e. 10 in the config below) b) Create files with chmod 640 and owner root:www (i.e. root:www and 640) c) Rotate the files when they reach 1Mb in size. (i.e. 1048576) d) Compress the files with gzip(1) to preserve compatibility with webalizer. (i.e. Z) # logfilename [owner:group]mode count size when flags [/pid_file] [sig_num] # Host vhostname1. # /home/vhostname1/log/access.log root:www640 10 1048576 * Z /var/run/httpd.pid /home/vhostname1/log/error.log root:www640 10 1048576 * Z /var/run/httpd.pid # Host vhostname2. # /home/vhostname2/log/access.log root:www640 10 1048576 * Z /var/run/httpd.pid /home/vhostname2/log/error.log root:www640 10 1048576 * Z /var/run/httpd.pid Check the man pages for newsyslog(8) and newsyslog.conf(8) for more information. I've been using this for more then two years now and it works like a charm. HTH, David -- David Robillard UNIX systems administrator Oracle DBA CISSP, RHCE Sun Certified Security Administrator Montreal: +1 514 966 0122 ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Apache Rotate Logs and Log Rotate
I have Apache making separate log files for each of my virtual hosts and putting them in /home/vhostname/log. Rotate logs makes a new log every 24 hours, but the logs quickly add up and since the sites are fairly busy the logs are at times over 5gigs. Is there any way to make rotate logs delete the log files after two days? Someone recommended me Log Rotate (from the ports tree), but this program does basically what Rotate logs does; except it makes things more complicated because it needs to restart apache and such. Is there a easy way to just have Apache's rotatelogs rotate the logs and then delete them after two days? Any feedback, suggestions, or comments would be greatly appreciated. Thanks, Peter ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
