Yes, if you build off master, that problem should go away (if it was due to lookup-table).
On Thu, Oct 1, 2015, 7:00 PM Rainer Gerhards <[email protected]> wrote: > 2015-10-01 15:14 GMT+02:00 singh.janmejay <[email protected]>: > > If you can share output of all thread backtrace we can confirm if this > > is the cause. > > let's first double-check: Christopher, did you use yesterday evening's > master branch? Because that contains a patch from Janmejay that I > think causes the problem for you. Or am I wrong, Janmejay? > > Rainer > > > > On Thu, Oct 1, 2015 at 2:30 PM, <[email protected]> wrote: > >> Hi, > >> Ups I was not detailed enough. > >> The problem with rsyslog-die does not always occur. But sometimes > unexpectedly. > >> In my environments the files grow or reduce sometimes, so maybe this > has something do with it (or the processing delay). > >> > >> regards > >> Chris > >> > >> > >> -----Ursprüngliche Nachricht----- > >> Gesendet: Donnerstag, 01 Oktober 2015 um 10:57:37 Uhr > >> Von: [email protected] > >> An: "singh.janmejay" <[email protected]>,rsyslog-users < > [email protected]> > >> Betreff: Re: [rsyslog] Separation of actions based on log source - with > good performance > >> Hi, > >> For my opinion it is really good to support looku-tables official. > >> Thanks for the work on the implementation David & Rainer. > >> > >> I have some experiences using lookup-Tables with > 2500 Entries. > >> > >> There are 2 open issues: > >> > >> 1. There is a bug when sending SIGHUP and reprocessing big lists, which > leads to die of rsyslogd. > >> I spend some time to identify this bug, unfortunately I'm still not > able to find the exact reason. > >> The problem seems to occur not directly after sending SIGHUP, but > later. Maybe this has something to do with Queues. > >> > >> 2. The "default" Value is not implemented. This should be mentioned in > the documentation or implemented. > >> I guess its quite less work, but I'm not sure how soon I find the time > to do all the things arround the pure developement... ;) > >> > >> > >> > >> regards > >> Chris > >> > >> -----Ursprüngliche Nachricht----- > >> Gesendet: Donnerstag, 01 Oktober 2015 um 09:41:26 Uhr > >> Von: "singh.janmejay" <[email protected]> > >> An: rsyslog-users <[email protected]> > >> Betreff: Re: [rsyslog] Separation of actions based on log source - with > good performance > >> OK, allow me a few days, I'll add one more test for multiple tables. > Will > >> make the doc change after that. > >> > >> -- > >> Regards, > >> Janmejay > >> > >> PS: Please blame the typos in this mail on my phone's uncivilized soft > >> keyboard sporting it's not-so-smart-assist technology. > >> > >> On Oct 1, 2015 12:29 PM, "Rainer Gerhards" <[email protected]> > wrote: > >> > >>> 2015-09-29 20:58 GMT+02:00 singh.janmejay <[email protected]>: > >>> > Sweet, plan on playing with it tomorrow. > >>> > >>> If you have verified that the current functionality works fine after > >>> your patch, I wouldn't object if you modify the doc to tell the world > >>> that this part of lookup tables is now officially supported. we could > >>> release with 8.14. I think what currently exists is already pretty > >>> useful and if we feel confident enough it works, we should release it. > >>> > >>> Rainer > >>> > > >>> > -- > >>> > Regards, > >>> > Janmejay > >>> > > >>> > PS: Please blame the typos in this mail on my phone's uncivilized > soft > >>> > keyboard sporting it's not-so-smart-assist technology. > >>> > > >>> > On Sep 30, 2015 12:16 AM, "Rainer Gerhards" < > [email protected]> > >>> > wrote: > >>> > > >>> >> It's a long time since I implemented what currently is there. It > should > >>> be > >>> >> relatively solid with probably some minor glitches. It provides the > code > >>> >> functionality as far as I remember. > >>> >> > >>> >> Rainer > >>> >> > >>> >> Sent from phone, thus brief. > >>> >> Am 29.09.2015 20:07 schrieb "singh.janmejay" < > [email protected] > >>> >: > >>> >> > >>> >> > Rainer/David, > >>> >> > > >>> >> > Exactly how much of lookup_table functionality is implemented? > >>> >> > > >>> >> > What can I not do with it? (you mentioned something about single > table > >>> >> > in this thread, can you please elaborate?). > >>> >> > > >>> >> > On Tue, Mar 31, 2015 at 7:23 PM, Rainer Gerhards > >>> >> > <[email protected]> wrote: > >>> >> > > 2015-03-31 15:46 GMT+02:00 <[email protected]>: > >>> >> > >> Hi, > >>> >> > >> Do you have some experience how large Lookup-tables can be > until > >>> there > >>> >> > are "negative" effects? > >>> >> > >> 2400 entries seems to work fine :) > >>> >> > > > >>> >> > > IIRC the current partial implementation is O(log n), so no > problem. > >>> >> > > > >>> >> > >> > >>> >> > >> And another question, do I loose events, when doing a kill -HUP > >>> (for > >>> >> > update of lookup-table)? > >>> >> > >> (e.g. client threads are hard "terminated"...) > >>> >> > > > >>> >> > > *should* not cause any issues. > >>> >> > > > >>> >> > > Rainer > >>> >> > >> > >>> >> > >> best regards > >>> >> > >> Chris > >>> >> > >> > >>> >> > >> > >>> >> > >> > >>> >> > >> Gesendet: Mittwoch, 25. März 2015 um 19:28 Uhr > >>> >> > >> Von: "David Lang" <[email protected]> > >>> >> > >> An: rsyslog-users <[email protected]> > >>> >> > >> Betreff: Re: [rsyslog] Separation of actions based on log > source - > >>> >> with > >>> >> > good performance > >>> >> > >> On Wed, 25 Mar 2015, [email protected] wrote: > Hi, > > I was > >>> >> > doing some experiments with the lookup-table. > Looks really nice > and > >>> the > >>> >> > performance is promising. > (Unfortunately the evaluation of > "nomatch" > >>> >> > attribute is currently not implemented...) > > Never the less: > > My > >>> plan > >>> >> > is, to do diffent actions based on the type of host, mapped in the > >>> >> > lookup-list. > For testing purposes, I use alway omfile. > > > >>> >> Unfortunately > >>> >> > it does not work, to change the ruleset based on the variable. > > Is > >>> there > >>> >> > any other option or is there any mistake? for omfile you can use > the > >>> >> > dynafile approach to use the return variable, for remote things > you > >>> would > >>> >> > need to do an if then else approach for performance reasons many > of > >>> the > >>> >> > fields in rsyslog do not accept variables. This allows them to be > >>> >> > computed/parsed once at startup rather than having to be > evaluated for > >>> >> each > >>> >> > log message. It's a bit of a hassle when you do want to do > something > >>> >> > dynamic, but even in cases where you have some dynamic things, you > >>> tend > >>> >> to > >>> >> > have other static things that benefit from the speedup. David > Lang > > >>> *** > >>> >> > syslog.conf *** > lookup_table(name="lookuptable" > >>> >> > file="/etc/rsyslog.lookup") > set $!dst = lookup("lookuptable", > >>> >> > $fromhost-ip); > ruleset(name="typea"){ > action(type="omfile" > >>> >> > file="/var/log/file_typea.log") > } > ruleset(name="typea"){ > > >>> >> > action(type="omfile" file="/var/log/file_typeb.log") > } > > # > Change > >>> set > >>> >> > default ruleset, based on sourceip > $DefaultRuleset $!dst > > > >>> >> > module(load="imtcp" KeepAlive="on" KeepAlive.Probes="1" > >>> >> > KeepAlive.Interval="2" KeepAlive.Time="20") > input(type="imtcp" > >>> >> > port="7714") > > *** lookup-table *** > { "version":1, > >>> "nomatch":"unk", > >>> >> > "type":"string", > "table":[ {"index":"10.3.5.4", "value":"typea" > }, > > >>> >> > {"index":"10.2.2.1", "value":"typea" }, > {"index":"10.0.2.2", > >>> >> > "value":"typeb" }, > {"index":"10.2.2.3", "value":"typeb" } > ] > > } > > >>> > > > >>> >> > best regards > Chris > > > > Gesendet: Dienstag, 24. März > 2015 um > >>> >> 17:14 > >>> >> > Uhr > Von: [email protected] > An: > [email protected] > > >>> >> > Betreff: Re: [rsyslog] Separation of actions based on log source - > >>> with > >>> >> > good performance > Hi David, > > Thanks sounds great, I will try > this > >>> in > >>> >> > the next days :) > > Chris > > > > Gesendet: Montag, 23. März > >>> 2015 um > >>> >> > 17:44 Uhr > Von: "David Lang" > An: rsyslog-users > Betreff: Re: > >>> >> [rsyslog] > >>> >> > Separation of actions based on log source - with good performance > > > >>> This > >>> >> is > >>> >> > the sort of thing that the table lookup functionality was designed > >>> for. > > >>> >> > It wasn't fully implemented to the design (funding fell through), > but > >>> I > >>> >> > think it works for a single table. > you could use it to do the > >>> mapping > >>> >> > from your many hosts to a couple of values and then have your > test be > >>> >> based > >>> >> > on the resulting value. > > David Lang On Mon, 23 Mar 2015 > > [...] > > >>> >> > >> > >>> >> > >> _______________________________________________ > >>> >> > >> rsyslog mailing list > >>> >> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> >> > >> http://www.rsyslog.com/professional-services/ > >>> >> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> >> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > >>> >> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>> POST if > >>> >> > you DON'T LIKE THAT. > >>> >> > > _______________________________________________ > >>> >> > > rsyslog mailing list > >>> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> >> > > http://www.rsyslog.com/professional-services/ > >>> >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >>> >> myriad > >>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you > >>> >> > DON'T LIKE THAT. > >>> >> > > >>> >> > > >>> >> > > >>> >> > -- > >>> >> > Regards, > >>> >> > Janmejay > >>> >> > http://codehunk.wordpress.com > >>> >> > _______________________________________________ > >>> >> > rsyslog mailing list > >>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> >> > http://www.rsyslog.com/professional-services/ > >>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> myriad > >>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you > >>> >> > DON'T LIKE THAT. > >>> >> _______________________________________________ > >>> >> rsyslog mailing list > >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> >> http://www.rsyslog.com/professional-services/ > >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>> >> DON'T LIKE THAT. > >>> > _______________________________________________ > >>> > rsyslog mailing list > >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> > http://www.rsyslog.com/professional-services/ > >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > > > > > -- > > Regards, > > Janmejay > > http://codehunk.wordpress.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
