On Thu, 18 Dec 2014, singh.janmejay wrote:
Usually load-balancers don't issue RST for unknown sessions. This is done
in order to avoid additional overhead of processing packets even for
unknown sessions.
I haven't worked with ELB, but doesn't it have a config to enable issuing
RST (instead of simply dropping packets for unknown sessions)? Usually
load-balancers have a way to turn on RST for this case.
Once turned on, regardless of it being RELP or not, it should work just
fine. Write to socket on rsyslog side will fail with the same error codes
as it would in case receiver dropped connection (crash or whatever).
without relp, rsyslog considers the log delivered to the remote system when it
sends it to the local TCP stack (because there is no mechanism to know anything
beyond this), so you can have a bunch of messages 'sent' on a connection that is
in the process of being closed, and these messages are going to be lost. RELP
adds application layer acks so that rsyslog can know that the message really got
to the remote system.
As far as needing a separate queue for sending RELP, it depends on what else is
going on in the system. The separate queue can fill up, and when it does the
main queue blocks anyway. So you have to have enough space in the queues to
handle the backlog. Once you have enough space, you won't stop accepting new
messages. The difference between having the space in the main queue and in a
separate RELP queue is that if you have other destinations you are delivering
to, having a separate queue allows you to deliver to those other destinations
while RELP is blocked.
David Lang
On Thu, Dec 18, 2014 at 10:17 AM, Erik Steffl <[email protected]> wrote:
On 12/17/2014 07:44 PM, Otis Gospodnetic wrote:
Hi,
I remember reading something about not being able to use ELB with multiple
rsyslog servers behind it. I can't find this information any more.... so
I'm wondering if anyone here knows if there are any issues with running
multiple rsyslog instances behind ELB?
might have been my emails,
based on my experience:
- both ends: definitely use RELP
- sender: if possible try to use keep alive (it wasn't available for
RELP but I think it's available now)
- sender: use action.resumeRetryCount and action.resumeInterval
(otherwise rsyslog does not reconnect or at least it takes a loooong time)
- sender: make sure you have a separate queue for RELP, otherwise
blocked RELP blocks main queue (you want rsyslog to read incoming messages
even if it has temporary problems sending them out)
example action for RELP (sender):
action(
type="omrelp"
target="elb.hostname"
port="5140"
template="json"
# see http://www.rsyslog.com/doc/node32.html
# disk used if forwarding blocked
queue.type="LinkedList"
queue.spoolDirectory="/some/path/"
queue.filename="json"
queue.maxdiskspace="75161927680" # 70GB (valuable data)
# these might help with ELB disconnecting after 60s of inactivity
action.resumeRetryCount="-1"
action.resumeInterval="5"
)
where:
- elb.hostname is the ELB hostname
- /some/path/ is spool directory
the main problem with ELB is that rsyslog opens bunch of connections
(one per thread) and these connections might or might not be used
(depending on traffic). If the connection is not used to some time ELB
closes it but rsyslog has hard time recognizing that. Connection can also
go dead because of backend problem (instance behind ELB has a problem).
Rsyslog is not able to figure out that these connections are dead and
continues to send data. At least in some scenarios it's because ELB simply
pretends everything is fine, in some other scenarios I am not entirely sure
what happens.
The end result is that there is no data logged, in some cases rsyslog
keeps retrying, in some cases it keeps sending data to nowhere etc. but it
does not recover.
The above config solves the problem but you still might get into
situation where the data is not being send out for 10-15 minutes. That's
why it's important to not block the main queue.
hope this helps... there's lots of details and examples in my previous
emails, just search for my email address or name on rsyslog mailing list...
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
