On Mon, Dec 16, 2013 at 8:27 AM, Erik Steffl <[email protected]> wrote:
> On 12/15/2013 10:44 PM, Rainer Gerhards wrote: > >> On Mon, Dec 16, 2013 at 4:05 AM, Erik Steffl <[email protected]> wrote: >> >> changed config so now the MARK messages are sent (and received). BTW >>> they use kern.info facility, not sure why it's not the same as what >>> Rainer found in source (syslog.info). >>> >>> >>> That's actually a bug - and even though this may break some older >> things, >> one I really need to fix. We should never emit messages under kern.... >> > > ok, I guess we should filter on both kern.info and syslog.info to make > our config forward compatible. > > any idea why would messages from /dev/log be treated differently then > MARK messages? See also my other email about MARK messages being resent and > grouped together... Think I might be missing something obvious about immark > or how the messages are processed or something... > > quite honestly, as there is a solution (keepalive) and I am now extremely short on time, I guess I won't look deeper into that before Jan, 10th. Rainer > thanks! > > erik > > > >> Rainer >> >> >> This solves the simple test case in which I only have sender, load >>> balancer (ELB) and receiver. I strace both sender and receiver, see the >>> MARK messages and the connection is fine, i.e. I send a log message using >>> e.g. logger, then wait 5 min, then send another one and it works. Without >>> MARK the second message never arrives. >>> >>> However in a more complex scenario this does not help at all. Complex >>> scenario looks like this (ascii arrows are flows of syslog messages): >>> >>> - 6 machines -> ELB-prod -> collector-prod >>> >>> - 1 machine -> ELB-test -> collector-test >>> >>> - every 5 minutes: collector-test -> ELB-prod -> collector-prod >>> >>> - every 5 minutes: collector-prod -> ELB-prod -> collector-prod (yes, >>> program on collector-prod sends message to rsyslog on collector-prod over >>> ELB) >>> >>> that 5 minute pause cause the connection to go stale somehow which >>> results in periods of silence. I configured both collector-prod and >>> collector-test to send MARK messages (to collector-prod since that's >>> where >>> they send regular messages) but I still see the periods of silence (on >>> both >>> collectors). Used strace to verify that MARK messages arrive (I guess >>> it's >>> possible that I confused the MARK messages from collector-prod and >>> collector-test, will continue investigating on that front) >>> >>> However adding cron entry to send few log messages every minute DOES >>> solve the problem (there is no silence anymore). >>> >>> Any ideas why that would be? Is it possible that MARK messages are >>> being >>> sent through different connections than other messages? >>> >>> As far as I can tell the only difference between MARK messages and the >>> cron'd messages is that the MARK messages are generated by immark and use >>> kern.info facilty and the cron'd messages arrive via /dev/log and use >>> local0.info facility. >>> >>> Any ideas why would the simpler scenario behave differently than the >>> complex scenario? Or why MARK messages do not solve the problem in >>> complex >>> scenario while cron'd messages do? >>> >>> thanks! >>> >>> erik >>> >>> >>> On 12/12/2013 02:39 PM, David Lang wrote: >>> >>> On Thu, 12 Dec 2013, Erik Steffl wrote: >>>> >>>> I will try to test librelp (with keepalive) but I need some >>>> >>>>> workaround in the meantime (sort of right now). >>>>> >>>>> Already tested that cron-ing logger once per minute keeps the >>>>> connection alive so that's my backup workaround. >>>>> >>>>> immark would be better cause then I only need to install rsyslog >>>>> config (easier deployment) plus it would be more efficient, do you >>>>> think that what David suggested is the best option? >>>>> >>>>> if I understood David's comment something like this is what I am >>>>> looking for: >>>>> >>>>> if >>>>> prifilt("local0.*") or >>>>> prifilt("local1.*") or >>>>> prifilt("local2.*") or >>>>> prifilt("local3.*") or >>>>> prifilt("local4.*") or >>>>> prifilt("local5.*") or >>>>> prifilt("local6.*") or >>>>> prifilt("local7.*") or >>>>> ( prifilt("syslog.info") and ... message is --MARK--) >>>>> >>>>> >>>> pretty much, I would do $msg == '--MARK--' as the second test >>>> >>>> David Lang >>>> >>>> then { >>>> >>>>> action(type="mmjsonparse") >>>>> if $parsesuccess == "OK" then { >>>>> action( >>>>> type="omrelp" >>>>> target="someHost" >>>>> port="5140" >>>>> template="json" >>>>> # see http://www.rsyslog.com/doc/node32.html >>>>> # disk used if forwarding blocked >>>>> queue.filename="json" >>>>> queue.maxdiskspace="75161927680" # 70GB (valuable data) >>>>> action.writeAllMarkMessages="on" >>>>> ) >>>>> } else { >>>>> ... >>>>> } >>>>> >>>>> reasonable? Can be improved? >>>>> >>>>> thanks! >>>>> >>>>> erik >>>>> >>>>> On 12/12/2013 12:44 PM, Rainer Gerhards wrote: >>>>> >>>>> On Thu, Dec 12, 2013 at 9:17 PM, David Lang <[email protected]> wrote: >>>>>> >>>>>> On Thu, 12 Dec 2013, Erik Steffl wrote: >>>>>> >>>>>>> >>>>>>> On 12/12/2013 08:29 AM, David Lang wrote: >>>>>>> >>>>>>> >>>>>>>> what facility and severity do the immark messages show up as? >>>>>>>> >>>>>>>>> >>>>>>>>> immark just generates messages, normal filtering rules determine >>>>>>>>> where >>>>>>>>> theya re sent, and the transport used (in this case RELP) has no >>>>>>>>> effect >>>>>>>>> on if they are sent or not, it's all in the filters. >>>>>>>>> >>>>>>>>> >>>>>>>>> thanks, that makes my question a lot more specific. How do I >>>>>>>> configure >>>>>>>> immark to use a specific facility? >>>>>>>> >>>>>>>> >>>>>>>> I don't think you do. I think they are using the syslog or kernel >>>>>>> facility, but I'd have to setup a quick test to check. I'll try to >>>>>>> do it >>>>>>> tonight if I can, but since you are seeing the messages locally, log >>>>>>> with >>>>>>> RSYSLOG_DebugFormat for a couple of minutes and look at what they are >>>>>>> logged as. >>>>>>> >>>>>>> >>>>>>> its syslog.=info: >>>>>>> >>>>>> >>>>>> http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/ >>>>>> immark/immark.c;h=0e946c0b92d555174b38de42dd236a >>>>>> c4432b98e7;hb=HEAD#l196 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> All I found when searching is this: >>>>>>> >>>>>>> >>>>>>>> $ModLoad immark.so >>>>>>>> $MarkMessagePeriod 60 >>>>>>>> >>>>>>>> which is what I have in my config. >>>>>>>> >>>>>>>> Given that I see the --MARK-- messages in /var/log/syslog and >>>>>>>> /var/log/kern.log I guess they are going to kern facility. Given >>>>>>>> the config >>>>>>>> below I need to use e.g. local0 facility. >>>>>>>> >>>>>>>> >>>>>>>> no, you need to change your filtering config to send these >>>>>>> messages, >>>>>>> not >>>>>>> try to change the messages to match your current config. >>>>>>> >>>>>>> >>>>>>> you actually can't. I considered mark a legacy feature and have not >>>>>>> >>>>>> enhanced it since 8 yrs. >>>>>> >>>>>> Keepalive is the better option. librelp is not yet build due to the >>>>>> current >>>>>> workload. The code actually right now is at github only, as I have >>>>>> some >>>>>> problems with the Adiscon repo. Easy to clone from here >>>>>> >>>>>> https://github.com/rgerhards/librelp >>>>>> >>>>>> >>>>>> >>>>>> messages have the facility that they have, you don't change the >>>>>> >>>>>>> facility >>>>>>> any more than you re-write the message to say something different. >>>>>>> >>>>>>> >>>>>> >>>>>> actually, in this case a config option would make sense. But again, I >>>>>> thought this is just legacy... >>>>>> >>>>>> Rainer >>>>>> >>>>>> >>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> >>>>>>> Unfortunately can't find anything related to --MARK-- and >>>>>>> facilities (or >>>>>>> >>>>>>> anything else other than the two settings above). >>>>>>>> >>>>>>>> Any ideas/pointers? Or if not possible to configure immark can I >>>>>>>> catch >>>>>>>> the --MARK-- message and change its facility? Or catch the --MARK-- >>>>>>>> message >>>>>>>> and have action that uses omrelp and same target (would that use >>>>>>>> same TCP >>>>>>>> connection)? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> erik >>>>>>>> >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, 12 Dec 2013, Erik Steffl wrote: >>>>>>>>> >>>>>>>>> Date: Thu, 12 Dec 2013 02:30:52 -0800 >>>>>>>>> >>>>>>>>> From: Erik Steffl <[email protected]> >>>>>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>>>>> To: rsyslog-users <[email protected]> >>>>>>>>>> Subject: [rsyslog] immark - how to use with action(...) >>>>>>>>>> >>>>>>>>>> How would I use immark to send mark messages for defined >>>>>>>>>> actions that >>>>>>>>>> use omrelp? >>>>>>>>>> >>>>>>>>>> I have tried something like this: >>>>>>>>>> >>>>>>>>>> $ModLoad immark.so >>>>>>>>>> $MarkMessagePeriod 60 >>>>>>>>>> >>>>>>>>>> if(..) >>>>>>>>>> if >>>>>>>>>> prifilt("local0.*") or >>>>>>>>>> prifilt("local1.*") or >>>>>>>>>> prifilt("local2.*") or >>>>>>>>>> prifilt("local3.*") or >>>>>>>>>> prifilt("local4.*") or >>>>>>>>>> prifilt("local5.*") or >>>>>>>>>> prifilt("local6.*") or >>>>>>>>>> prifilt("local7.*") >>>>>>>>>> then { >>>>>>>>>> action(type="mmjsonparse") >>>>>>>>>> if $parsesuccess == "OK" then { >>>>>>>>>> action( >>>>>>>>>> type="omrelp" >>>>>>>>>> target="someHost" >>>>>>>>>> port="5140" >>>>>>>>>> template="json" >>>>>>>>>> # see http://www.rsyslog.com/doc/node32.html >>>>>>>>>> # disk used if forwarding blocked >>>>>>>>>> queue.filename="json" >>>>>>>>>> queue.maxdiskspace="75161927680" # 70GB (valuable data) >>>>>>>>>> action.writeAllMarkMessages="on" >>>>>>>>>> ) >>>>>>>>>> } else { >>>>>>>>>> ... >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> I see --MARK-- messages in /var/log/syslog and /var/log/kern.log >>>>>>>>>> but >>>>>>>>>> they are not send by omrelp action (the action works fine, normal >>>>>>>>>> messages are going through). >>>>>>>>>> >>>>>>>>>> Verified where the --MARK-- messages are going using strace so >>>>>>>>>> pretty >>>>>>>>>> sure they are only going to those two local files, nothing goes >>>>>>>>>> over >>>>>>>>>> RELP. Also checked on the receiving side of RELP, no incoming >>>>>>>>>> messages >>>>>>>>>> have --MARK-- in them. And the connection goes down which is also >>>>>>>>>> very >>>>>>>>>> strong indicator that there are no --MARK-- messages. >>>>>>>>>> >>>>>>>>>> How do I configure it so that the --MARK-- messages are send over >>>>>>>>>> RELP >>>>>>>>>> protocol to someHost (over same TCP connection that the given >>>>>>>>>> action >>>>>>>>>> uses, it's for purpose to keep alive the connection since RELP >>>>>>>>>> does >>>>>>>>>> not support KeepAlive (yet, Rainer just added it to master, >>>>>>>>>> thanks!)) >>>>>>>>>> >>>>>>>>>> This is on Ubuntu 13.10 using rsyslog 7.5.6, librelp 1.2.0 from >>>>>>>>>> adiscon repo. >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> erik >>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>> POST >>>>>>>>>> if you DON'T LIKE THAT. >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> >>>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> myriad >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> you >>>>>>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> myriad >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>> you >>>>>>>> DON'T LIKE THAT. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> >>>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> you >>>>>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>> POST if you DON'T LIKE THAT. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>> if you DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>>> >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
