Re: Virtual Host Logging Perl Script (mod_macro note)
On Wed, Jul 16, 2003 at 12:40:10PM -0700, Marc M. Adkins wrote: > > > Right now it seems a bit silly having a separate ErrorLog line in each > > > of the apache virtual host stubs, but as far as I am aware there isn't > > > an easier way is there? > > > > You could look at mod_macro. > mod_macro (http://www.coelho.net/mod_macro) works great for me. I found > that I had to make a few changes to get it to build for Apache 2.0.46. To My main motivation for running apache2 would be to make use of the PerChild MPM which from what I can glean is still not working :( This directive would save me a lot of hassles wrt running cgi scripts (php in particular) with the euid of the script owner. All this as I'm tussling with mod_suphp... -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
On Wed, Jul 16, 2003 at 07:37:21PM +0100, Ged Haywood wrote: > Hi Jez, > > On Wed, 16 Jul 2003, Jez Hancock wrote: > > > [snip] We started looking at mod_log_sql: [snip] > > but had trouble getting it to work on FreeBSD unfortunately. > > I'd have thought something a bit lighter might do for this. > > > Right now it seems a bit silly having a separate ErrorLog line in each > > of the apache virtual host stubs, but as far as I am aware there isn't > > an easier way is there? > > You could look at mod_macro. This is what I was looking for when I 'found' mod_perl (I was after something to make repetitive vhost entries simpler to build and the ability to do this with mp seemed great)! Cheers. -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
On Wed, Jul 16, 2003 at 12:40:06PM +0200, Guillaume Fougnies wrote: > Wed, Jul 16, 2003 at 11:07:47AM +0100: Jez Hancock wrote: > > I would do this but we wanted to give our users 'live' logfiles, rather > > than making them wait until log rotation before being able to view them > > (or did I misunderstand you?). > > During my work on the ENodes project, i've developed a tiny threaded logger > in c to manage internal logfiles (by webmaster/website/version). > (you can rewrite it in perl with a 5.8.x built with thread...) > > Perl handlers send log lines through a UNIX socket to the logger which > is opening logfiles when needed and keeping them opened. > It avoids the overhead of opening and locking each time the file and > provides as many online debug logfile as you ask. /me nods. This would save a lot of overhead. The only benefit of opening a new pipe each time I suppose is that you don't need to worry about rotation. > The logger is launched in a 'PerlRequire' file. > The client have a persistant connection to the logger. > > If it fits your needs, you can get sources here: http://www.enodes.org > Server : utils/enodes_logger/enodes_logger.c > Client module: enodes_core/modules/ENodes/Core/ApLog.pm Many thanks, I'll investigate now. Well ENodes sounds very interesting, particularly the part about webmasters being able to test new versioning without applying changes and the reduced httpd restart feature. Very interesting. Will certainly have a look at this, we're currently considering something called webcp, a PHP project which has much potential but is sitll unfortunately very much in beta. Thanks! -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
Hi Ged, On Wed, Jul 16, 2003 at 07:06:43AM +0100, Ged Haywood wrote: > On Tue, 15 Jul 2003, Jez Hancock wrote: > > Does anyone how one could log errorlog entries in a similar manner to > > the script above - ie pipe the errorlog to a script which appends one > > copy of the error entry to a main error logfile and another copy to the > > virtual host's error logfile? > It's possible, but I don't think you really want to do it. You would > be asking the server to do more than is necessary while handling each > request (you probably already are:). This was my reason for considering using mod_perl :) > Would it not be better for example to rotate your logs frequently, and > to process them afterwards, off-line? You would then have a choice of > doing things with the various log file analysis tools too. If you are > a little bit creative in what you log, it can be simple to extract the > information you need for each vhost from one file. I would do this but we wanted to give our users 'live' logfiles, rather than making them wait until log rotation before being able to view them (or did I misunderstand you?). > Have you considered using something other than flat files for logging? We started looking at mod_log_sql: http://www.grubbybaby.com/mod_log_sql/ but had trouble getting it to work on FreeBSD unfortunately. The plan was to log everything to a central MySQL (or pgsql) db table, a copy to the vhost's logging table, same with errors (gawd knows how intense this would be on the system). Then when a user wanted to view their logfiles, they'd login to their control panel and request to view only the files they were interested in. At this point a static copy of the log they requested would be created in the filesystem and (optionally) a compressed copy transferred to them in the browser. To be honest I don't know about the above. First of all it requires a lot of DB transactions which would probably mean using pgsql to be safe. Secondly there's the CGI coding that would need to be done and I don't know how efficient or practical such a CGI script would really be. Finally I prefer the idea of user's automatically just getting a set of logfiles created directly to the filesystem. Right now it seems a bit silly having a separate ErrorLog line in each of the apache virtual host stubs, but as far as I am aware there isn't an easier way is there? -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
On Tue, Jul 15, 2003 at 02:39:18PM -0400, Perrin Harkins wrote: > On Tue, 2003-07-15 at 14:27, Jez Hancock wrote: > > Much obliged, that does look to be something I could use. Reminds me I > > need to be locking the logfile as well ;) > > Actually, if you are just printing one short line I think you'll be fine > without locking on unix systems. See the FAQ: > http://www.perldoc.com/perl5.8.0/pod/perlfaq5.html#All-I-want-to-do-is-append-a-small-amount-of-text-to-the-end-of-a-file.--Do-I-still-have-to-use-locking- Ok thanks Perrin. The script is actually running nicely as it is right now on a server hosting 50 or so domains, although admittedly it's not taking a massive number of hits (10,000 or so per day). Does anyone how one could log errorlog entries in a similar manner to the script above - ie pipe the errorlog to a script which appends one copy of the error entry to a main error logfile and another copy to the virtual host's error logfile? -- Jez http://www.munk.nu/
Re: Virtual Host Logging Perl Script
On Tue, Jul 15, 2003 at 01:32:11PM +0300, Stas Bekman wrote: > Jez Hancock wrote: > >My question is: would it be possible to use mod_perl in some way to > >perform the function of the script? In testing the speed of the script > >seems reasonable enough, is there a better way to do what I'm doing > >below? > > Take a look at: > http://perl.apache.org/docs/2.0/user/handlers/http.html#PerlLogHandler > a similar code will work for mp1 if you don't use 2.0. Much obliged, that does look to be something I could use. Reminds me I need to be locking the logfile as well ;) Thanks Stas, -- Jez http://www.munk.nu/
Virtual Host Logging Perl Script
Hi,
I've just written a short perl script to perform logging for our virtual
hosts. The code has plenty of comments so I'll paste it below.
My question is: would it be possible to use mod_perl in some way to
perform the function of the script? In testing the speed of the script
seems reasonable enough, is there a better way to do what I'm doing
below?
On a related note, I'd ideally like to be able to do similar for Apache
Error logging - ie log apache error log entries once into a main
errorlog file and once into a vhost errorlog file. Is this possible
with ErrorLog lines?
My initial understanding is this isn't possible using the framework
outlined in the code below.
Code follows:
#!/usr/bin/perl
# Script to pipe apache log entries to virtually hosted log files
# Assumes httpd.conf has the following:
# LogFormat "%v %h %l %u %t \"%r\" %>s %b" commonvhost
# and that ONLY the following logging line is used in the httpd.conf:
# CustomLog "| /path/to/logger.pl" commonvhost
# DO NOT configure the CustomLog directive in the vhost stubs for vhosts
# or this will not work.
# Script logs commonvhost entries to a logfile with a template of:
# /var/log/httpd/virtual.domain/$year/$month/$day
use strict;
my $logEntry = <>;
# get the vhost from this log entry:
$logEntry=~/(.*?) /;
my $vhost = $1;
my ($year, $month, $day) = (
(localtime)[5]+1900,
sprintf("%02d", (localtime)[4]+1),
sprintf("%02d", (localtime)[3])
);
# Name of access logfiles:
my $accessLogName = "httpd-access.log";
=comment
$logdir:
Location to put all logfiles
This will log everything into:
$logdir/all/$year/$month/$day/httpd-access.log
and put individual vhost logfiles into:
$logdir/$vhost/$year/$month/$day/httpd-access.log
=cut
my $logDir = "/var/log/httpd";
my $allLogDir = "$logDir/all/$year/$month/$day";
my $vhostLogDir = "$logDir/$vhost/$year/$month/$day";
writeLog($allLogDir, "all");
writeLog($vhostLogDir, "vhost");
# write a log entry to a file
sub writeLog(){
my $logDir = shift @_;
my $type = shift @_;
if( ! -d $logDir ) {
`mkdir -p $logDir`;
}
open(FD, ">>$logDir/$accessLogName");
# if type is vhost, strip off the vhost data:
if($type eq "vhost"){
$logEntry =~s/.*? //;
}
print FD $logEntry;
close FD;
}
--
Jez
http://www.munk.nu/
Virtual Host Logging Perl Script
Hi,
I've just written a short perl script to perform logging for our virtual
hosts. The code has plenty of comments so I'll paste it below.
My question is: would it be possible to use mod_perl in some way to
perform the function of the script? In testing the speed of the script
seems reasonable enough, is there a better way to do what I'm doing
below?
On a related note, I'd ideally like to be able to do similar for Apache
Error logging - ie log apache error log entries once into a main
errorlog file and once into a vhost errorlog file. Is this possible
with ErrorLog lines?
My initial understanding is this isn't possible using the framework
outlined in the code below.
Code follows:
#!/usr/bin/perl
# Script to pipe apache log entries to virtually hosted log files
# Assumes httpd.conf has the following:
# LogFormat "%v %h %l %u %t \"%r\" %>s %b" commonvhost
# and that ONLY the following logging line is used in the httpd.conf:
# CustomLog "| /path/to/logger.pl" commonvhost
# DO NOT configure the CustomLog directive in the vhost stubs for vhosts
# or this will not work.
# Script logs commonvhost entries to a logfile with a template of:
# /var/log/httpd/virtual.domain/$year/$month/$day
use strict;
my $logEntry = <>;
# get the vhost from this log entry:
$logEntry=~/(.*?) /;
my $vhost = $1;
my ($year, $month, $day) = (
(localtime)[5]+1900,
sprintf("%02d", (localtime)[4]+1),
sprintf("%02d", (localtime)[3])
);
# Name of access logfiles:
my $accessLogName = "httpd-access.log";
=comment
$logdir:
Location to put all logfiles
This will log everything into:
$logdir/all/$year/$month/$day/httpd-access.log
and put individual vhost logfiles into:
$logdir/$vhost/$year/$month/$day/httpd-access.log
=cut
my $logDir = "/var/log/httpd";
my $allLogDir = "$logDir/all/$year/$month/$day";
my $vhostLogDir = "$logDir/$vhost/$year/$month/$day";
writeLog($allLogDir, "all");
writeLog($vhostLogDir, "vhost");
# write a log entry to a file
sub writeLog(){
my $logDir = shift @_;
my $type = shift @_;
if( ! -d $logDir ) {
`mkdir -p $logDir`;
}
open(FD, ">>$logDir/$accessLogName");
# if type is vhost, strip off the vhost data:
if($type eq "vhost"){
$logEntry =~s/.*? //;
}
print FD $logEntry;
close FD;
}
--
Jez
http://www.munk.nu/
