a corner case to be aware of, if rsyslog gets behind in writing the logs, it
could continue writing after the service IP moves over.
Also, unless you have configured STONITH, there are failure modes of
Corosync/Pacemaker that can leave the IP active on both boxes at the same time
(at which point things get 'interesting' when you look at network traffic)
The vast majority of the time, things will work just fine, it's the odd corener
cases when things go very wrong that you could run into grief. Unfortunantly
that's frequently also when you most need the logs
to be bulletproof, I always write the files to a local disk first, and then copy
the data to the shared filesystem with unique names.
David Lang
On Fri, 22 Nov 2013, Walid Moghrabi
wrote:
Old thread but I personnaly write on a GlusterFS shared storage for my central log clustered servers but .. not both at the same time.
In fact, I have 2 nodes and Rsyslog is configured to store local logs locally and remote logs (logs coming from remote servers) onto the DFS.
Both Rsyslog servers are active but, only one receive the log stream at a time thanks to a service IP that is handled by Corosync/Pacemaker.
So, as soon as one of log nod is failing, the service IP is moved to the second node which can then continue to write in the log file.
This works pretty well, I also enable/disable a a logrotate script so that it run on one node at a time too.
To conclude, I also write log data in a MySQL database for using with Loganalyzer, this database is stored locally on each nodes and it is working in master/master replication betwwen each other so this is active/active setup.
I'm quite happy with this setup, just struggling with mmnormalize on some things but on the overall, this is quite efficient, I might write some "howto" one day about this setup for those interested.
Regards.
----- Mail original -----
De: "David Lang" <[email protected]>
À: "rsyslog-users" <[email protected]>
Envoyé: Jeudi 28 Mars 2013 19:38:45
Objet: Re: [rsyslog] Rsyslog & Distributed File Systems??
The issue I would be concerned about with these races is that rsyslog can issue
large writes to the OS. The OS then can split the writes on block size boundries
when it sends them on to the filesystem. Since the block boundries won't match
up with line boundries, you will have parts of lines from different systems
combined into one line in the resulting file.
This is what typically happens on a local filesystem if you have two processes
writing to the same file without explicit locking between them. Having the disk
(or in this case, GlusterFS) try to guess what that locking should be between
two processes just doesn't work well.
you can fake it without explicit locking if you make sure that all of your
writes are smaller than block size, you disable buffering in the language and
you make sure that the application never writes part of a line. But rsyslog can
handle lines larger than block size (if message size is set large enough), and
it can output multiple messages at once, so I would not expect that rsyslog
would work well in these conditions.
David Lang
On Thu, 28 Mar 2013, Gregory Patmore wrote:
seems glusterfs 'should' handle the races with its locking translator, but like every distributed <add noun here>, they have their throughput limits.
so I guess it would depend on the load of your use case.
A quick google search turned up some stories of race condition drama with GlusterFS ( https://www.google.com/search?q=glusterfs+race+conditions ), but I'm sure it would be just as easy to find similar for any file system.
Greg
Sent from my iPhone
On Mar 28, 2013, at 5:03 AM, David Lang <[email protected]> wrote:
On Wed, 27 Mar 2013, Jiann-Ming Su wrote:
Is there a version of rsyslog that can properly use distributed filesystems like GlusterFS? For example, I have two nodes each running rsyslog but also sharing a GlusterFS filesystem. Can those independent rsyslog processes running on different servers write to the same file on the GlusterFS?
rsyslog doesn't have any code in it that cares what filesystem it uses.
What would it have to do to "properly use distributed filesystems like GlusterFS"?
I would expect that different distributed filesystems have different behavior if multiple clients are writing to the same file at the same time.
If nothing else, without explicit locking, I would expect that you will end up with race conditions between the different writers, which can cause writes from the different writers to be intermingled (and intermingled in chunks that make sense to the OS layer, not to the application layer)
I know that some distributed filesystems 'handle' this problem by only allowing
one machine/process to have a given file open for writing at a time, other
attempts to open the file either fail or block. What does GlusterFS do?
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
