2010/11/28 Peter Wolfenden <pwolfen...@qualys.com>:
> If you want to make absolutely sure to process all the log lines *and* you
> are in a position to control how your backup application writes to its log
> files, then it may be worth considering using "multilog" to send one copy of
> the data to an automatically rotated log file and another copy of the data
> to SEC via named pipe (so that you can bounce SEC without losing data).
> Multilog is described here, and has proven to be very, very reliable (as
> have the other components of DJB's daemontools package):
>
> http://cr.yp.to/daemontools/multilog.html
>
...and if by chance the log files with timestamps are generated by
syslog-ng, it is probably worthwhile to create another log{} statement
in syslog-ng conf which creates the log file with the same content
without timestamps in file name.
One important note, though - if you go with the named pipe approach,
make sure that the writer (the log producer, that is) opens the pipe
for writing in blocking mode, or uses some sort of reliable buffering
scheme if non-blocking write fails. If that's not the case, data
transmission without losses can not be guaranteed.
Finally, a minor side note - if SEC switches to new log file, it does
not jump to the end of file, but always processes it from the very
beginning. So there is no danger of losing data in the beginning of
the newly created log file.
kind regards,
risto
> Cheers,
>
> Peter Wolfenden
> p...@qualys.com
>
> On Fri, Nov 26, 2010 at 9:38 AM, Tim Peiffer <peif...@umn.edu> wrote:
>>
>> I have a backup application that writes logs to file names that are
>> dated.. e.g. logfile.MMDDYY , and I am looking for the best way of detecting
>> the existence of the file, and then open the file from the beginning under
>> SEC. The backup application can fire off nearly any time of day, so the
>> file is not guaranteed to be there because of a variable backup window. The
>> directory that the logfile is written to is also read-only to my correlator
>> process. What are some methods of elegantly handling these logs?
>>
>> The -input=logfile* glob is ok, but is only useful on start/restart, with
>> the new logs being effectively invisible until the next restart. I have
>> tried dropping symbolic links in a directory that I have write access to
>> linking the fixed target with the monitored file name logfile.MMDDYY. When
>> the file is created by the application, SEC handles the new monitored file
>> name as a log file shuffle. What I am thinking is that at midnight, I
>> remove the existing link, and then re-link to the monitored file name. I
>> wrote the below example, and it works as expected, but I am interested how
>> SEC will handle the hours in between the time of symbolic link creation, and
>> the time of the monitored file creation.
>>
>> type=Calendar
>> time=* * * * *
>> desc=create new logfile sym link
>> action=spawn ( \
>> cd /Users/peiffer/sec-2.5.3/glob ; \
>> rm logfile ; \
>> ln -s logfile.`date +%m%d%y` logfile \
>> )
>>
>> type=Single
>> ptype=RegExp
>> pattern=(.*)
>> desc=new event
>> action=logonly %s $1
>>
>> Tim
>>
>> --
>> Tim Peiffer
>> Network Support Engineer
>> Office of Information Technology
>> University of Minnesota/NorthernLights GigaPOP
>>
>> +1 612 626-7884 (desk)
>>
>>
>> ------------------------------------------------------------------------------
>> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
>> Tap into the largest installed PC base & get more eyes on your game by
>> optimizing for Intel(R) Graphics Technology. Get started today with the
>> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
>> http://p.sf.net/sfu/intelisp-dev2dev
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
>
> ------------------------------------------------------------------------------
> Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
> Tap into the largest installed PC base & get more eyes on your game by
> optimizing for Intel(R) Graphics Technology. Get started today with the
> Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
> http://p.sf.net/sfu/intelisp-dev2dev
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
>
------------------------------------------------------------------------------
Increase Visibility of Your 3D Game App & Earn a Chance To Win $500!
Tap into the largest installed PC base & get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
