Old thread but I personnaly write on a GlusterFS shared storage for my central log clustered servers but .. not both at the same time. In fact, I have 2 nodes and Rsyslog is configured to store local logs locally and remote logs (logs coming from remote servers) onto the DFS. Both Rsyslog servers are active but, only one receive the log stream at a time thanks to a service IP that is handled by Corosync/Pacemaker. So, as soon as one of log nod is failing, the service IP is moved to the second node which can then continue to write in the log file. This works pretty well, I also enable/disable a a logrotate script so that it run on one node at a time too.
To conclude, I also write log data in a MySQL database for using with Loganalyzer, this database is stored locally on each nodes and it is working in master/master replication betwwen each other so this is active/active setup. I'm quite happy with this setup, just struggling with mmnormalize on some things but on the overall, this is quite efficient, I might write some "howto" one day about this setup for those interested. Regards. ----- Mail original ----- De: "David Lang" <[email protected]> À: "rsyslog-users" <[email protected]> Envoyé: Jeudi 28 Mars 2013 19:38:45 Objet: Re: [rsyslog] Rsyslog & Distributed File Systems?? The issue I would be concerned about with these races is that rsyslog can issue large writes to the OS. The OS then can split the writes on block size boundries when it sends them on to the filesystem. Since the block boundries won't match up with line boundries, you will have parts of lines from different systems combined into one line in the resulting file. This is what typically happens on a local filesystem if you have two processes writing to the same file without explicit locking between them. Having the disk (or in this case, GlusterFS) try to guess what that locking should be between two processes just doesn't work well. you can fake it without explicit locking if you make sure that all of your writes are smaller than block size, you disable buffering in the language and you make sure that the application never writes part of a line. But rsyslog can handle lines larger than block size (if message size is set large enough), and it can output multiple messages at once, so I would not expect that rsyslog would work well in these conditions. David Lang On Thu, 28 Mar 2013, Gregory Patmore wrote: > seems glusterfs 'should' handle the races with its locking translator, but > like every distributed <add noun here>, they have their throughput limits. > > so I guess it would depend on the load of your use case. > > A quick google search turned up some stories of race condition drama with > GlusterFS ( https://www.google.com/search?q=glusterfs+race+conditions ), but > I'm sure it would be just as easy to find similar for any file system. > > Greg > > Sent from my iPhone > > On Mar 28, 2013, at 5:03 AM, David Lang <[email protected]> wrote: > >> On Wed, 27 Mar 2013, Jiann-Ming Su wrote: >> >>> Is there a version of rsyslog that can properly use distributed filesystems >>> like GlusterFS? For example, I have two nodes each running rsyslog but also >>> sharing a GlusterFS filesystem. Can those independent rsyslog processes >>> running on different servers write to the same file on the GlusterFS? >> >> rsyslog doesn't have any code in it that cares what filesystem it uses. >> >> What would it have to do to "properly use distributed filesystems like >> GlusterFS"? >> >> I would expect that different distributed filesystems have different >> behavior if multiple clients are writing to the same file at the same time. >> >> If nothing else, without explicit locking, I would expect that you will end >> up with race conditions between the different writers, which can cause >> writes from the different writers to be intermingled (and intermingled in >> chunks that make sense to the OS layer, not to the application layer) >> >> I know that some distributed filesystems 'handle' this problem by only >> allowing one machine/process to have a given file open for writing at a >> time, other attempts to open the file either fail or block. What does >> GlusterFS do? >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
