Hi, For my opinion it is really good to support looku-tables official. Thanks for the work on the implementation David & Rainer.
I have some experiences using lookup-Tables with > 2500 Entries. There are 2 open issues: 1. There is a bug when sending SIGHUP and reprocessing big lists, which leads to die of rsyslogd. I spend some time to identify this bug, unfortunately I'm still not able to find the exact reason. The problem seems to occur not directly after sending SIGHUP, but later. Maybe this has something to do with Queues. 2. The "default" Value is not implemented. This should be mentioned in the documentation or implemented. I guess its quite less work, but I'm not sure how soon I find the time to do all the things arround the pure developement... ;) regards Chris -----Ursprüngliche Nachricht----- Gesendet: Donnerstag, 01 Oktober 2015 um 09:41:26 Uhr Von: "singh.janmejay" <[email protected]> An: rsyslog-users <[email protected]> Betreff: Re: [rsyslog] Separation of actions based on log source - with good performance OK, allow me a few days, I'll add one more test for multiple tables. Will make the doc change after that. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Oct 1, 2015 12:29 PM, "Rainer Gerhards" <[email protected]> wrote: > 2015-09-29 20:58 GMT+02:00 singh.janmejay <[email protected]>: > > Sweet, plan on playing with it tomorrow. > > If you have verified that the current functionality works fine after > your patch, I wouldn't object if you modify the doc to tell the world > that this part of lookup tables is now officially supported. we could > release with 8.14. I think what currently exists is already pretty > useful and if we feel confident enough it works, we should release it. > > Rainer > > > > -- > > Regards, > > Janmejay > > > > PS: Please blame the typos in this mail on my phone's uncivilized soft > > keyboard sporting it's not-so-smart-assist technology. > > > > On Sep 30, 2015 12:16 AM, "Rainer Gerhards" <[email protected]> > > wrote: > > > >> It's a long time since I implemented what currently is there. It should > be > >> relatively solid with probably some minor glitches. It provides the code > >> functionality as far as I remember. > >> > >> Rainer > >> > >> Sent from phone, thus brief. > >> Am 29.09.2015 20:07 schrieb "singh.janmejay" <[email protected] > >: > >> > >> > Rainer/David, > >> > > >> > Exactly how much of lookup_table functionality is implemented? > >> > > >> > What can I not do with it? (you mentioned something about single table > >> > in this thread, can you please elaborate?). > >> > > >> > On Tue, Mar 31, 2015 at 7:23 PM, Rainer Gerhards > >> > <[email protected]> wrote: > >> > > 2015-03-31 15:46 GMT+02:00 <[email protected]>: > >> > >> Hi, > >> > >> Do you have some experience how large Lookup-tables can be until > there > >> > are "negative" effects? > >> > >> 2400 entries seems to work fine :) > >> > > > >> > > IIRC the current partial implementation is O(log n), so no problem. > >> > > > >> > >> > >> > >> And another question, do I loose events, when doing a kill -HUP > (for > >> > update of lookup-table)? > >> > >> (e.g. client threads are hard "terminated"...) > >> > > > >> > > *should* not cause any issues. > >> > > > >> > > Rainer > >> > >> > >> > >> best regards > >> > >> Chris > >> > >> > >> > >> > >> > >> > >> > >> Gesendet: Mittwoch, 25. März 2015 um 19:28 Uhr > >> > >> Von: "David Lang" <[email protected]> > >> > >> An: rsyslog-users <[email protected]> > >> > >> Betreff: Re: [rsyslog] Separation of actions based on log source - > >> with > >> > good performance > >> > >> On Wed, 25 Mar 2015, [email protected] wrote: > Hi, > I was > >> > doing some experiments with the lookup-table. > Looks really nice and > the > >> > performance is promising. > (Unfortunately the evaluation of "nomatch" > >> > attribute is currently not implemented...) > > Never the less: > My > plan > >> > is, to do diffent actions based on the type of host, mapped in the > >> > lookup-list. > For testing purposes, I use alway omfile. > > > >> Unfortunately > >> > it does not work, to change the ruleset based on the variable. > Is > there > >> > any other option or is there any mistake? for omfile you can use the > >> > dynafile approach to use the return variable, for remote things you > would > >> > need to do an if then else approach for performance reasons many of > the > >> > fields in rsyslog do not accept variables. This allows them to be > >> > computed/parsed once at startup rather than having to be evaluated for > >> each > >> > log message. It's a bit of a hassle when you do want to do something > >> > dynamic, but even in cases where you have some dynamic things, you > tend > >> to > >> > have other static things that benefit from the speedup. David Lang > > *** > >> > syslog.conf *** > lookup_table(name="lookuptable" > >> > file="/etc/rsyslog.lookup") > set $!dst = lookup("lookuptable", > >> > $fromhost-ip); > ruleset(name="typea"){ > action(type="omfile" > >> > file="/var/log/file_typea.log") > } > ruleset(name="typea"){ > > >> > action(type="omfile" file="/var/log/file_typeb.log") > } > > # Change > set > >> > default ruleset, based on sourceip > $DefaultRuleset $!dst > > > >> > module(load="imtcp" KeepAlive="on" KeepAlive.Probes="1" > >> > KeepAlive.Interval="2" KeepAlive.Time="20") > input(type="imtcp" > >> > port="7714") > > *** lookup-table *** > { "version":1, > "nomatch":"unk", > >> > "type":"string", > "table":[ {"index":"10.3.5.4", "value":"typea" }, > > >> > {"index":"10.2.2.1", "value":"typea" }, > {"index":"10.0.2.2", > >> > "value":"typeb" }, > {"index":"10.2.2.3", "value":"typeb" } > ] > } > > > > > >> > best regards > Chris > > > > Gesendet: Dienstag, 24. März 2015 um > >> 17:14 > >> > Uhr > Von: [email protected] > An: [email protected] > > >> > Betreff: Re: [rsyslog] Separation of actions based on log source - > with > >> > good performance > Hi David, > > Thanks sounds great, I will try this > in > >> > the next days :) > > Chris > > > > Gesendet: Montag, 23. März > 2015 um > >> > 17:44 Uhr > Von: "David Lang" > An: rsyslog-users > Betreff: Re: > >> [rsyslog] > >> > Separation of actions based on log source - with good performance > > This > >> is > >> > the sort of thing that the table lookup functionality was designed > for. > > >> > It wasn't fully implemented to the design (funding fell through), but > I > >> > think it works for a single table. > you could use it to do the > mapping > >> > from your many hosts to a couple of values and then have your test be > >> based > >> > on the resulting value. > > David Lang On Mon, 23 Mar 2015 > [...] > > >> > >> > >> > >> _______________________________________________ > >> > >> rsyslog mailing list > >> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > >> http://www.rsyslog.com/professional-services/ > >> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if > >> > you DON'T LIKE THAT. > >> > > _______________________________________________ > >> > > rsyslog mailing list > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > http://www.rsyslog.com/professional-services/ > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> > > >> > > >> > > >> > -- > >> > Regards, > >> > Janmejay > >> > http://codehunk.wordpress.com > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
