On Thu, 22 Dec 2016, mostolog--- via rsyslog wrote:
well, if logs are redirected to the host syslog, you don't have the logs in
files for imfile to have to read, you have them in rsyslog (or the systemd
journal, but that's your problem if you do :-)
if you have logs that are written to disk, then you have two choices
1. have something tail the logs as they are written, processing the logs in
near-real-time
2. wait for a log rotation point and at that time, take the rotated log
(after some delay to give the app writing to it a chance to actually close
it), and process them.
processing the completed log files can be done in a couple of different
ways, but fundamentally you are running a script to do the processing, so
it can do anything
2a. move/copy the files someplace for processing (which could even be a
different machine)
2b. run a process to read the file and do something with it.
2b can be to run logger against the file and then rm the file after logger
completes.
That sounds overcomplicated to me, maybe I'm missing something.
Considering each application logs to /appName-YYYY-MM-DD-HH.log, /we could
run a cron job (eg:hourly) which compares state file "reading position" with
file size for older logs (eg: 2 hour ago), deleting if they are equal.
most of the time logs are written to /appname and then a log rotation script
moves them to a timestamped filename.
Shouldn't that work if we're sending with RELP?
yes, your approach should work for RELP or anything else.
By spec, the write to /dev/log will not complete until rsyslog has the
message in it's queue. So your script can feed the log to rsyslog and
remove it immediatly after it's completed.
I'm not sure if I understood this. Could you elaborate?
The reason that sending machines can stall and not even let you login when
sending to a remote machine stops is because the syslog spec says that a
write to /dev/log is not supposed to return until the log message is safe
on disk (including fsync) and would survive a reboot.
Didn't know that. Do you have some literature about this?
I'd have to do some digging, a quick look isn't showing this, although
rfc-3164 does talk about blocking a bit
AFAIK "my logs" are written to /var/log/messages, not using /dev/log. I guess
these doesn't behave that way (not waiting for fsync) and that's much faster,
isnt it? Otherwise, why aren't logs written on /dev/log?
apps deliver logs to rsyslog by writing to /dev/log, which *syslog* reads and
then writes to various locations, including /var/log/messages
Rsyslog relaxes this so that it lets the write return as soon as the log
message is in the rsyslog queue. But if there isn't room in the queue
(because output has stopped), the write to /dev/log is unable to complete.
And, in the event of an error, queue messages that weren't written on
/dev/log are lost, right?
if rsyslog crashed, messages in the queue are lost. If the queue is disk
assisted and save-on-shutdown is turned on, a graceful shutdown of rsyslog will
save the queue to disk to send after it restarts.
you could configure rsyslog for utter reliablity, (disk only queues, checkpoint
interval of 1, syncs enabled), but the performance suffers by >1000x
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
