Re: [rsyslog] Separation of actions based on log source - with good performance

Mon, 05 Oct 2015 22:37:48 -0700 

a CVE for something that requires manually enabling an experimental feature???
it would be one thing if a default config had the problem, or if it was something entirely dependent on remote data.
I would be very tempted to respond to the CVE with "don't enable this incomplete feature" as the solution. It's very common for incomplete features to be included in released versions 

grumble, we have enough real bugs to worry about.

David Lang

On Tue, 6 Oct 2015, Rainer Gerhards wrote:


Date: Tue, 6 Oct 2015 07:15:31 +0200
From: Rainer Gerhards <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Separation of actions based on log source - with good
    performance

Sorry, folks, good intent always seems to find someone who turns it
into negative. I was yesterday contacted by a distro maintainer who
wants to turn this bug in the officially non-existant lookup table
feature into a CVE and insists that it is a vuln even after the
argument that the feature never oficially existed.

It looks like it was a bad idea to merge potentially useful yet
incomplete code into the main branch (and documenting it to be not
present). It looks like I need to re-think my stance on experimental
features.

Anyhow, I really don't want to support the argument that something
non-existing can be a CVE. As such, I will create a new
master-insecure branch, which will be a clone of the current master
branch. Then I'll remove the lookup table code, so that the code base
matches the documentation. I really don't want to create a general
principle here that we need to create CVEs (and patched) for something
that was just added as a convenience for a handful of folks who were
ready to take a risk.

If there is sufficient interest, we can consider officially adding
this feature to the January 8.15 release iff it is ready by then.
@janmejay: please let me know if you would like to continue with your
work on lookup tables under this new situation.

As soon as I have time, I'll check what else needs to be removed. Not
sure about the ./contributed branch, because the project cannot
guarantee at all this is bug-free. It's documented to be so, but if
that is not sufficient, it should probably live only in the
master-insecure branch.

Rainer

2015-10-02 17:29 GMT+02:00 singh.janmejay <[email protected]>:

As of now it returns empty string for no-match. I guess we should go ahead
with it in its current form. We can add default value any time later
without breaking compatibility(it'd default to "").

I'll add a test for multiple tables on Monday.

On Fri, Oct 2, 2015, 7:16 PM  <[email protected]> wrote:


Hi,
No, I didn't. I even didn't realize the patch.

It seems to be exactly the related issue. So I don't expect any further
issues.
I will use the new version on 2 systems. If there is any other issue, I
will let you know.
Release data for next rsyslog version is quite far so enough time to
test... ;)

The missing implementation of "nomatch" (default) entry as described at
 http://www.rsyslog.com/doc/lookup_tables.html
would from my opinion require changes:

Arround line 132 of lookup.c file (save of value)
Arround line 243 of lookup.c file (search in lookuptable fails, so return
nomatch value.


regards
Chris


> Gesendet: Donnerstag, 01. Oktober 2015 um 16:57 Uhr
> Von: "singh.janmejay" <[email protected]>
> An: rsyslog-users <[email protected]>
> Betreff: Re: [rsyslog] Separation of actions based on log source - with
good performance
>
> Yes, if you build off master, that problem should go away (if it was due
to
> lookup-table).
>
> On Thu, Oct 1, 2015, 7:00 PM Rainer Gerhards <[email protected]>
> wrote:
>
> > 2015-10-01 15:14 GMT+02:00 singh.janmejay <[email protected]>:
> > > If you can share output of all thread backtrace we can confirm if
this
> > > is the cause.
> >
> > let's first double-check: Christopher, did you use yesterday evening's
> > master branch? Because that contains a patch from Janmejay that I
> > think causes the problem for you. Or am I wrong, Janmejay?
> >
> > Rainer
> > >
> > > On Thu, Oct 1, 2015 at 2:30 PM,  <[email protected]> wrote:
> > >> Hi,
> > >> Ups I was not detailed enough.
> > >> The problem with rsyslog-die does not always occur. But sometimes
> > unexpectedly.
> > >> In my environments the files grow or reduce sometimes, so maybe this
> > has something do with it (or the processing delay).
> > >>
> > >> regards
> > >> Chris
> > >>
> > >>
> > >> -----Ursprüngliche Nachricht-----
> > >> Gesendet: Donnerstag, 01 Oktober 2015 um 10:57:37 Uhr
> > >> Von: [email protected]
> > >> An: "singh.janmejay" <[email protected]>,rsyslog-users <
> > [email protected]>
> > >> Betreff: Re: [rsyslog] Separation of actions based on log source -
with
> > good performance
> > >> Hi,
> > >> For my opinion it is really good to support looku-tables official.
> > >> Thanks for the work on the implementation David & Rainer.
> > >>
> > >> I have some experiences using lookup-Tables with > 2500 Entries.
> > >>
> > >> There are 2 open issues:
> > >>
> > >> 1. There is a bug when sending SIGHUP and reprocessing big lists,
which
> > leads to die of rsyslogd.
> > >> I spend some time to identify this bug, unfortunately I'm still not
> > able to find the exact reason.
> > >> The problem seems to occur not directly after sending SIGHUP, but
> > later. Maybe this has something to do with Queues.
> > >>
> > >> 2. The "default" Value is not implemented. This should be mentioned
in
> > the documentation or implemented.
> > >> I guess its quite less work, but I'm not sure how soon I find the
time
> > to do all the things arround the pure developement... ;)
> > >>
> > >>
> > >>
> > >> regards
> > >> Chris
> > >>
> > >> -----Ursprüngliche Nachricht-----
> > >> Gesendet: Donnerstag, 01 Oktober 2015 um 09:41:26 Uhr
> > >> Von: "singh.janmejay" <[email protected]>
> > >> An: rsyslog-users <[email protected]>
> > >> Betreff: Re: [rsyslog] Separation of actions based on log source -
with
> > good performance
> > >> OK, allow me a few days, I'll add one more test for multiple tables.
> > Will
> > >> make the doc change after that.
> > >>
> > >> --
> > >> Regards,
> > >> Janmejay
> > >>
> > >> PS: Please blame the typos in this mail on my phone's uncivilized
soft
> > >> keyboard sporting it's not-so-smart-assist technology.
> > >>
> > >> On Oct 1, 2015 12:29 PM, "Rainer Gerhards" <
[email protected]>
> > wrote:
> > >>
> > >>> 2015-09-29 20:58 GMT+02:00 singh.janmejay <
[email protected]>:
> > >>> > Sweet, plan on playing with it tomorrow.
> > >>>
> > >>> If you have verified that the current functionality works fine
after
> > >>> your patch, I wouldn't object if you modify the doc to tell the
world
> > >>> that this part of lookup tables is now officially supported. we
could
> > >>> release with 8.14. I think what currently exists is already pretty
> > >>> useful and if we feel confident enough it works, we should release
it.
> > >>>
> > >>> Rainer
> > >>> >
> > >>> > --
> > >>> > Regards,
> > >>> > Janmejay
> > >>> >
> > >>> > PS: Please blame the typos in this mail on my phone's uncivilized
> > soft
> > >>> > keyboard sporting it's not-so-smart-assist technology.
> > >>> >
> > >>> > On Sep 30, 2015 12:16 AM, "Rainer Gerhards" <
> > [email protected]>
> > >>> > wrote:
> > >>> >
> > >>> >> It's a long time since I implemented what currently is there. It
> > should
> > >>> be
> > >>> >> relatively solid with probably some minor glitches. It provides
the
> > code
> > >>> >> functionality as far as I remember.
> > >>> >>
> > >>> >> Rainer
> > >>> >>
> > >>> >> Sent from phone, thus brief.
> > >>> >> Am 29.09.2015 20:07 schrieb "singh.janmejay" <
> > [email protected]
> > >>> >:
> > >>> >>
> > >>> >> > Rainer/David,
> > >>> >> >
> > >>> >> > Exactly how much of lookup_table functionality is implemented?
> > >>> >> >
> > >>> >> > What can I not do with it? (you mentioned something about
single
> > table
> > >>> >> > in this thread, can you please elaborate?).
> > >>> >> >
> > >>> >> > On Tue, Mar 31, 2015 at 7:23 PM, Rainer Gerhards
> > >>> >> > <[email protected]> wrote:
> > >>> >> > > 2015-03-31 15:46 GMT+02:00  <[email protected]>:
> > >>> >> > >> Hi,
> > >>> >> > >> Do you have some experience how large Lookup-tables can be
> > until
> > >>> there
> > >>> >> > are "negative" effects?
> > >>> >> > >> 2400 entries seems to work fine :)
> > >>> >> > >
> > >>> >> > > IIRC the current partial implementation is O(log n), so no
> > problem.
> > >>> >> > >
> > >>> >> > >>
> > >>> >> > >> And another question, do I loose events, when doing a kill
-HUP
> > >>> (for
> > >>> >> > update of lookup-table)?
> > >>> >> > >> (e.g. client threads are hard "terminated"...)
> > >>> >> > >
> > >>> >> > > *should* not cause any issues.
> > >>> >> > >
> > >>> >> > > Rainer
> > >>> >> > >>
> > >>> >> > >> best regards
> > >>> >> > >> Chris
> > >>> >> > >>
> > >>> >> > >>
> > >>> >> > >>
> > >>> >> > >> Gesendet: Mittwoch, 25. März 2015 um 19:28 Uhr
> > >>> >> > >> Von: "David Lang" <[email protected]>
> > >>> >> > >> An: rsyslog-users <[email protected]>
> > >>> >> > >> Betreff: Re: [rsyslog] Separation of actions based on log
> > source -
> > >>> >> with
> > >>> >> > good performance
> > >>> >> > >> On Wed, 25 Mar 2015, [email protected] wrote: >
Hi, >
> > I was
> > >>> >> > doing some experiments with the lookup-table. > Looks really
nice
> > and
> > >>> the
> > >>> >> > performance is promising. > (Unfortunately the evaluation of
> > "nomatch"
> > >>> >> > attribute is currently not implemented...) > > Never the
less: >
> > My
> > >>> plan
> > >>> >> > is, to do diffent actions based on the type of host, mapped
in the
> > >>> >> > lookup-list. > For testing purposes, I use alway omfile. > >
> > >>> >> Unfortunately
> > >>> >> > it does not work, to change the ruleset based on the
variable. >
> > Is
> > >>> there
> > >>> >> > any other option or is there any mistake? for omfile you can
use
> > the
> > >>> >> > dynafile approach to use the return variable, for remote
things
> > you
> > >>> would
> > >>> >> > need to do an if then else approach for performance reasons
many
> > of
> > >>> the
> > >>> >> > fields in rsyslog do not accept variables. This allows them
to be
> > >>> >> > computed/parsed once at startup rather than having to be
> > evaluated for
> > >>> >> each
> > >>> >> > log message. It's a bit of a hassle when you do want to do
> > something
> > >>> >> > dynamic, but even in cases where you have some dynamic
things, you
> > >>> tend
> > >>> >> to
> > >>> >> > have other static things that benefit from the speedup. David
> > Lang >
> > >>> ***
> > >>> >> > syslog.conf *** > lookup_table(name="lookuptable"
> > >>> >> > file="/etc/rsyslog.lookup") > set $!dst =
lookup("lookuptable",
> > >>> >> > $fromhost-ip); > ruleset(name="typea"){ > action(type="omfile"
> > >>> >> > file="/var/log/file_typea.log") > } > ruleset(name="typea"){ >
> > >>> >> > action(type="omfile" file="/var/log/file_typeb.log") > } > > #
> > Change
> > >>> set
> > >>> >> > default ruleset, based on sourceip > $DefaultRuleset $!dst > >
> > >>> >> > module(load="imtcp" KeepAlive="on" KeepAlive.Probes="1"
> > >>> >> > KeepAlive.Interval="2" KeepAlive.Time="20") >
input(type="imtcp"
> > >>> >> > port="7714") > > *** lookup-table *** > { "version":1,
> > >>> "nomatch":"unk",
> > >>> >> > "type":"string", > "table":[ {"index":"10.3.5.4",
"value":"typea"
> > }, >
> > >>> >> > {"index":"10.2.2.1", "value":"typea" }, > {"index":"10.0.2.2",
> > >>> >> > "value":"typeb" }, > {"index":"10.2.2.3", "value":"typeb" } >
] >
> > } >
> > >>> > >
> > >>> >> > best regards > Chris >   >   > > Gesendet: Dienstag, 24. März
> > 2015 um
> > >>> >> 17:14
> > >>> >> > Uhr > Von: [email protected] > An:
> > [email protected] >
> > >>> >> > Betreff: Re: [rsyslog] Separation of actions based on log
source -
> > >>> with
> > >>> >> > good performance > Hi David, > > Thanks sounds great, I will
try
> > this
> > >>> in
> > >>> >> > the next days :) > > Chris >   >   > > Gesendet: Montag, 23.
März
> > >>> 2015 um
> > >>> >> > 17:44 Uhr > Von: "David Lang" > An: rsyslog-users > Betreff:
Re:
> > >>> >> [rsyslog]
> > >>> >> > Separation of actions based on log source - with good
performance
> > >
> > >>> This
> > >>> >> is
> > >>> >> > the sort of thing that the table lookup functionality was
designed
> > >>> for. >
> > >>> >> > It wasn't fully implemented to the design (funding fell
through),
> > but
> > >>> I
> > >>> >> > think it works for a single table. > you could use it to do
the
> > >>> mapping
> > >>> >> > from your many hosts to a couple of values and then have your
> > test be
> > >>> >> based
> > >>> >> > on the resulting value. > > David Lang On Mon, 23 Mar 2015 >
> > [...] >
> > >>> >> > >>
> > >>> >> > >> _______________________________________________
> > >>> >> > >> rsyslog mailing list
> > >>> >> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> > >> http://www.rsyslog.com/professional-services/
> > >>> >> > >> What's up with rsyslog? Follow
https://twitter.com/rgerhards
> > >>> >> > >> NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED
> > by a
> > >>> >> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
NOT
> > >>> POST if
> > >>> >> > you DON'T LIKE THAT.
> > >>> >> > > _______________________________________________
> > >>> >> > > rsyslog mailing list
> > >>> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> > > http://www.rsyslog.com/professional-services/
> > >>> >> > > What's up with rsyslog? Follow
https://twitter.com/rgerhards
> > >>> >> > > NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED by
> > a
> > >>> >> myriad
> > >>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
> > if you
> > >>> >> > DON'T LIKE THAT.
> > >>> >> >
> > >>> >> >
> > >>> >> >
> > >>> >> > --
> > >>> >> > Regards,
> > >>> >> > Janmejay
> > >>> >> > http://codehunk.wordpress.com
> > >>> >> > _______________________________________________
> > >>> >> > rsyslog mailing list
> > >>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> > http://www.rsyslog.com/professional-services/
> > >>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
> > >>> myriad
> > >>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
> > if you
> > >>> >> > DON'T LIKE THAT.
> > >>> >> _______________________________________________
> > >>> >> rsyslog mailing list
> > >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> http://www.rsyslog.com/professional-services/
> > >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
> > myriad
> > >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
> > you
> > >>> >> DON'T LIKE THAT.
> > >>> > _______________________________________________
> > >>> > rsyslog mailing list
> > >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> > http://www.rsyslog.com/professional-services/
> > >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
> > >>> DON'T LIKE THAT.
> > >>> _______________________________________________
> > >>> rsyslog mailing list
> > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> http://www.rsyslog.com/professional-services/
> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
> > >>> DON'T LIKE THAT.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
> > you DON'T LIKE THAT.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
> > you DON'T LIKE THAT.
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Janmejay
> > > http://codehunk.wordpress.com
> > > _______________________________________________
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to