Actually i found the parameter and i changed my configuration.
input(type="imtcp" port="514" ruleset="forward"
supportOctetCountedFraming="off")
But it still doesn't work as expected.
Indeed inside my ruleset i have this condition :
if $programname startswith 'Myapp.' then {
action( type="omrelp"
...
When i disable octet-counting it seems that this condition is not reached.
My log format look like that :
20150115003549 server Myapp.sometag
{"response":{"status":206,"duration":1,"size":311557},"some_other": 4242}
And i use loggen (from syslog-ng) for my benchs.
Note: when i change the condition to
if $fromhost-ip == '10.x.x.x' then {
...
it seems that the condition is reached and i still have a segfault
thanks,
Smana
----- Mail original -----
De: "Rainer Gerhards" <[email protected]>
À: "David Lang" <[email protected]>
Cc: "rsyslog-users" <[email protected]>
Envoyé: Jeudi 23 Juillet 2015 15:13:24
Objet: Re: [rsyslog] Load balancing issue
2015-07-23 15:12 GMT+02:00 David Lang <[email protected]>:
> On Thu, 23 Jul 2015, [email protected] wrote:
>
>> 1- it solved my issue regarding the disk queue not created. i'll update
>> the github issue.
>
>
> good.
>
>> 2- i have a lot of errors like "rsyslogd: Framing Error in received TCP
>> message: invalid octet count -1871509715. [v8.10.0]"
>
>
> this means that you are getting malformed data sent to you. Rsyslog
> implements an extension to the syslog protocol where instead of each log
> message being a string of text followed by a newline, the sending system can
> send a number at the beginning (instead of <PRI> where PRI is the combined
> facility/severity data) and rsyslog will then read that number of bytes as
> the message. This allows a message to contain embedded newlines.
>
> What's happening is that you have something sending you digits at the
> beginning of the message, rsyslog is trying to interpret this, but it's
> garbage data. I don't know if there is a way to disable octet counted mode
> on the reciever or not.
There is a parameter to do that, but I don't remember the name out of my head.
Rainer
> similarly, a message starting with 'z' is
> interpreted as a compressed message.
>
> The 'best' answer is to figure out which system is generating the invalid
> messages and fix it there. If you can do so.
>
> David Lang
>
>> 3- As soon as i enable the "rebindInterval" option, rsyslog segfaults
>> [Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
>> 00007f3a64efa624 sp 00007f3a5b1f5bc8 error 4 in
>> librelp.so.0.1.0[7f3a64eee000+11000]
>>
>> Please find the startup debug logs here :
>> https://gist.github.com/Smana/21f1add821b91f1a0bc1
>>
>> Regards,
>> Smana
>>
>>
>>
>> ----- Mail original -----
>> De: "Brian Knox" <[email protected]>
>> À: "rsyslog-users" <[email protected]>
>> Envoyé: Jeudi 23 Juillet 2015 14:17:05
>> Objet: Re: [rsyslog] Load balancing issue
>>
>> Aha! David - to summarize, is the problem then that:
>>
>> a) the parameter did not exist previously, and
>> b) was only added for the new style configs?
>>
>> Brian
>>
>> On Thu, Jul 23, 2015 at 7:59 AM, David Lang <[email protected]> wrote:
>>
>>> On Thu, 23 Jul 2015, Brian Knox wrote:
>>>
>>> From your diagram, it looks like you are trying to load balance RELP. As
>>>>
>>>> far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
>>>> this has been discussed on the mailing list:
>>>>
>>>> http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
>>>>
>>>> Unless something has changed, you need to use the omfwd module if you
>>>> want
>>>> to use tcp rebinding. This isn't a bug - this is documented behavior.
>>>> The
>>>> rebind interval parameter is documented as a parameter for omfwd. RELP
>>>> uses omrelp, which has no such paramater. See:
>>>> http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
>>>> and
>>>> http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
>>>>
>>>
>>> with the new style config it does in the current git branch. It looks
>>> like
>>> it was added in 7.3.15
>>>
>>> /* tables for interfacing with the v6 config system */
>>> /* action (instance) parameters */
>>> static struct cnfparamdescr actpdescr[] = {
>>> { "target", eCmdHdlrGetWord, 1 },
>>> { "tls", eCmdHdlrBinary, 0 },
>>> { "tls.compression", eCmdHdlrBinary, 0 },
>>> { "tls.prioritystring", eCmdHdlrString, 0 },
>>> { "tls.cacert", eCmdHdlrString, 0 },
>>> { "tls.mycert", eCmdHdlrString, 0 },
>>> { "tls.myprivkey", eCmdHdlrString, 0 },
>>> { "tls.authmode", eCmdHdlrString, 0 },
>>> { "tls.permittedpeer", eCmdHdlrArray, 0 },
>>> { "port", eCmdHdlrGetWord, 0 },
>>> { "rebindinterval", eCmdHdlrInt, 0 },
>>> { "windowsize", eCmdHdlrInt, 0 },
>>> { "timeout", eCmdHdlrInt, 0 },
>>> { "localclientip", eCmdHdlrGetWord, 0 },
>>> { "template", eCmdHdlrGetWord, 0 }
>>> };
>>>
>>>
>>>
>>>
>>> I use ActionTCPRebindInterval with haproxy with plain TCP. It works
>>> very
>>>>
>>>> well.
>>>>
>>>> Cheers,
>>>> Brian
>>>>
>>>> On Thu, Jul 23, 2015 at 7:03 AM, <[email protected]> wrote:
>>>>
>>>> With the architecture enclosed.
>>>>>
>>>>>
>>>>> ----- Mail original -----
>>>>> De: [email protected]
>>>>> À: "rsyslog-users" <[email protected]>
>>>>> Envoyé: Jeudi 23 Juillet 2015 11:59:35
>>>>> Objet: [rsyslog] Load balancing issue
>>>>>
>>>>> Hello all,
>>>>>
>>>>> I'm currently trying to load balance the log traffic accross several
>>>>> servers.
>>>>> I thought my configuration with "ActionSendTCPRebindInterval" option
>>>>> was
>>>>> working properly, unfortunately my recent benchs show that the log flow
>>>>> is
>>>>> not well load balanced.
>>>>>
>>>>> Please find below a part of the architecture :
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> My problem is located on the log aggregators : the rsyslog send its
>>>>> traffic to haproxy on localhost using relp protocol.
>>>>> I monitored the tcp sessions and i can see that haproxy doesn't change
>>>>> the
>>>>> destination servers.
>>>>>
>>>>> watch 'ss -lap -o state established \( dport = :20514 \)'
>>>>> Recv-Q Send-Q Local Address:Port Peer Address:Port
>>>>> 1716 0 127.0.0.1:43652 127.0.0.1:20514 users:(("rsyslogd",8409,88))
>>>>> 0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
>>>>> users:(("haproxy",3922
>>>>> ,2))
>>>>> 1760 0 127.0.0.1:43650 127.0.0.1:20514 users:(("rsyslogd",8409,22))
>>>>> 0 0 10.17.252.4:55583 10.19.12.6:20514 users:(("haproxy",3922,10))
>>>>>
>>>>> Please find enclosed my configuration.
>>>>>
>>>>> NB :
>>>>> - the source pid (rsyslog) never change as it is expected with
>>>>> "ActionSendTCPRebindInterval"
>>>>> - i mixed legacy and new syntaxe because of the following bug
>>>>> https://github.com/rsyslog/rsyslog/issues/96
>>>>> This bug is annoying and i didn't receive any update since about 4
>>>>> months
>>>>>
>>>>> Could you please help me ?
>>>>>
>>>>> OS : debian7
>>>>> rsyslog version : 8.10
>>>>>
>>>>> Regards ,
>>>>> Smana
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>>> _______________________________________________
>>>>
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>
>>>
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
