On Wed, 14 Dec 2016, mostolog--- via rsyslog wrote:
we are using RELP.
according to some comments in
https://github.com/rsyslog/rsyslog-doc/pull/281, seems rulesets should have a
queue to avoid "creating a temporary queue"
no such thing as a temporary queue
as we are looking for 0% loss...
under all conditions? including a fire in the server room or a metor hitting the
building?
0% loss is a very strong statement, and really needs to be clarified. If you
really do mean 0% loss under all conditions (includeing the building being
destroyed), then rsyslog may not actually be the right tool for you.
Due to the performance impact of trying to comply with such a requirement, the
cost of implementing such a system would also be extremely high
But almost nobody really means 0% loss under all conditions, so the first thing
you really need to do is to think really hard about what the real requirements
are. Are there conditions where you would be willing to have data lost if the
building is destroyed, or some server looses power?
1. imrelp -> ruleset+main_queue(disk) is there any chance a message
being lost between RELP reception and queueing?
no, RELP does not consider the data accepted (and send it's ack) unless the data
is in the queue.
but main_queue being disk does not neccessarily mean that the log will survive
under all conditions (if you allow the OS to buffer writes to disk, it could be
lost in a crash, if you only write to one disk it could be lost if the disk
fails, if you only write to one system, a fire in that system could destroy all
disks, if you only write to systems in one datacenter, a bomb could destroy all
the systems ....)
2. ruleset+main_queue(disk) -> mmjsonparse -> mmnormalize -> omfile is
there any chance a message being lost anywhere?
that depends on how you configure omfile. If you allow the OS to buffer writes
to disk... (see discussion above)
message processing (including mm modules) cannot cause data loss, but it does
take time that the message spends in the queues, and if those queues are in
ram...)
3. imfile -> omelasticsearch is there any change a message being lost?
for imfile -> rsyslog, that depends on the queues (see above)
for omeleasticsearch, that depends on the ES configuration. There are many ways
that ES could loose a message if things crash at just the wrong time...
PS: using RELP, when there are connection issues we noticed some problems,
bottlenecks and so. hence, we decided to use a file between RELP and elastic
let's talk about this more in the other thread. omfile/imfile is NOT the correct
way to implement a buffer between rsyslog and ES.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
