looked at it and would like to use it (for better load balancing)
unfortunately we are using RELP which makes rsyslog segfault when this
option is used with omrelp (there's a recent thread on this).
BTW for us RELP is a must, with Amazon ELB messages go into black hole
if there's any problem behind the load balancer (e.g. one server behind
ELB goes down, ELB happily keeps the connection open but all data send
to that connection is lost, Amazon support essentially said that's what
it is and higher level protocols should be used to work around this
problem). Maybe other load balancer works better and TCP works well
enough (you can still lose some logs when using TCP but it might not be
a huge problem).
not sure if it would help in our case since the problem is the timeout
while rebind works with number of messages, i.e. rebind wouldn't apply
in our case,
thanks for the suggestion,
erik
On 12/03/2013 01:30 PM, Dan Finn wrote:
> Hey Erik,
>
> Have you checked out the $ActionSendTCPRebindInterval option? When I was
> researching load balancing rsyslog I found that. A previous admin before
> me had tried to load balance rsyslog and failed and it sounds like it was
> because he was missing that option. We haven’t had any issues yet load
> balancing with the use of that.
>
> I’m confident that this is not an issue with our load balancer because it
> was only a few months ago that I implemented the VIP. We used to use a
> single “destination” rsyslog server and we would see the same issue there
> with the postgres rsyslog clients.
>
> Thanks,
> Dan
>
> On 12/3/13, 2:24 PM, "Erik Steffl" <[email protected]> wrote:
>
>> we have sort of similar problem, in our case it's Amazon Elastic Load
>> Balancer (ELB) that somehow causes the connection go "bad" if there is
>> no traffic for 5 min (not sure what the exact time is, 1 minute is ok, 5
>> minutes is not).
>>
>> not sure what going "bad" actually means (still investigating) but
>> the data is not going through, rsyslog sends data but there is no
>> response... it recovers eventually but not sure what exactly triggers
>> the recovery (sending more messages is what triggers it but how exactly
>> is not clear).
>>
>> It's not the same case but maybe you can look into VIP and
>> connections and see what happens there, maybe use strace to see what are
>> the responses when rsyslogd sends data to destination...
>>
>> erik
>>
>> On 12/03/2013 01:12 PM, Dan Finn wrote:
>>> I had kind of wondered about that as well but I have a few reasons that
>>> make it seem like that is not the case.
>>>
>>> The ³central server² is actually a VIP on our F5 load balancer with 4
>>> rsyslog destination servers behind it. We have about 200 servers in our
>>> environment and during these busy times the only servers that ever seem
>>> to
>>> log locally are the postgres servers. The volume of logs being written
>>> on
>>> these servers is certainly much higher than anywhere else. My theory is
>>> that the rsyslog ³client² is not keeping up with the sheer volume on
>>> these
>>> servers during the busy times but until I can find some concrete info
>>> that
>>> is just a theory.
>>>
>>> We are looking gat upgrading to v7 but unfortunately that¹s not going to
>>> be a quick fix. I was hoping maybe there was an issue in my config or
>>> something that could be tweaked but it sounds like maybe that is not the
>>> case?
>>>
>>> I did capture some debug output while this was happening. Unfortunately
>>> it was pretty large so I don¹t know if I can share the whole thing but
>>> is
>>> there anything in particular I would be looking for in there? I see
>>> that
>>> it says it¹s writing the files locally but I didn¹t see where it says
>>> why.
>>>
>>> Thanks,
>>> Dan
>>>
>>> On 12/3/13, 3:03 AM, "David Lang" <[email protected]> wrote:
>>>
>>>> you are sending the logs via TCP, which means that if the system you
>>>> are
>>>> sending
>>>> logs to gets backed up, logs will queue on the sending system, spilling
>>>> to disk
>>>> as needed.
>>>>
>>>> the bottleneck is probably on the central server, but we have no info
>>>> about what
>>>> it's doing.
>>>>
>>>> The go-to tool for diagnosting this sort of problem is the impstats
>>>> module, but
>>>> I don't think that existed back in the 4.x days, and tracking down the
>>>> bottleneck without it is significantly harder. Is there any way you can
>>>> upgrade
>>>> to a current version?
>>>>
>>>> David Lang
>>>>
>>>> On Mon, 2 Dec 2013, Dan Finn wrote:
>>>>
>>>>> Date: Mon, 2 Dec 2013 20:53:54 +0000
>>>>> From: Dan Finn <[email protected]>
>>>>> Reply-To: rsyslog-users <[email protected]>
>>>>> To: "[email protected]" <[email protected]>
>>>>> Subject: [rsyslog] rsyslog frequently queuing to disk when it should
>>>>> be
>>>>> sending over the network
>>>>>
>>>>> Hello,
>>>>>
>>>>> I¹m trying to get some insight into an issue that we have been seeing
>>>>> quite a bit. We have some postgres servers that are quite verbose.
>>>>> When the servers get busy we have an issue where they queue their logs
>>>>> locally instead of sending over the network however I can¹t find any
>>>>> reason why that would be, at least not from a OS resource standpoint.
>>>>> We are running rsyslog4-4.8.0-1.ius.el5. This is my config from the
>>>>> client that was having issues : http://pastebin.com/n3XpRdMm.
>>>>>
>>>>> I watched it queue about 10k files under /var/spool/rsyslog before I
>>>>> finally had to manually delete them out because disk was filling up.
>>>>>
>>>>> What¹s the best way to get some insight into why this might be
>>>>> happening? Is there a way I can enable some debug logging for the
>>>>> rsyslog process itself? Any settings in our config that could be
>>>>> tweaked?
>>>>>
>>>>> Thanks,
>>>>> Dan
>>>>>
>>>>>
>>>>>
>>>>> DAN FINN
>>>>>
>>>>> Linux System Administrator
>>>>>
>>>>>
>>>>>
>>>>> [email protected]<mailto:[email protected]>
>>>>>
>>>>>
>>>>>
>>>>> Backcountry.com<http://www.backcountry.com/>
>>>>>
>>>>> Competitive Cyclist<http://www.competitivecyclist.com/>
>>>>>
>>>>> RealCyclist.com<http://www.realcyclist.com/>
>>>>>
>>>>> Dogfunk.com<http://www.dogfunk.com/>
>>>>>
>>>>> SteepandCheap.com<http://www.steepandcheap.com/>
>>>>>
>>>>> Chainlove.com<http://www.chainlove.com/>
>>>>>
>>>>> WhiskeyMilitia.com<http://www.whiskeymilitia.com/>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>>> if you DON'T LIKE THAT.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>> if you DON'T LIKE THAT.
>>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
