Re: [rsyslog] FrosCon 2018

[Brian Knox via rsyslog]

Hello Rainer! That sounds like it would be great.. I'm already traveling in
October and November this year, and I probably couldn't squeeze in another
trip in 2018. I usually need quite a bit of notice to figure out my travel
agenda - I'd be totally up for getting together with people in 2019
sometime, hah!

Cheers,
Brian

On Mon, Jul 30, 2018 at 7:57 AM Rainer Gerhards 
wrote:

> Hi folks,
>
> I had proposed to do some (local) meetup at FrosCon 2018 in Bonn (St.
> Augustin to be precise). I received little feedback and I am
> considering to cancel the meetup based on that feedback.
>
> As such, I would appreciate if those that  are really interested and
> likely to attend please let me know.
>
> Thanks,
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch - failed operation handling

[Brian Knox via rsyslog]

To my knowledge, Rich is correct. This also would explain a case we hit
maybe every couple of months, where rsyslog very quickly duplicates some
messages it is sending to elasticsearch. I would assume this would be a
case where a batch is submitted, only some of the messages are rejected,
and rsyslog then duplicates messages trying to send the batch over and over
again.

On Thu, May 17, 2018 at 12:08 AM David Lang  wrote:

> On Wed, 16 May 2018, Rich Megginson wrote:
>
> > On 05/16/2018 05:58 PM, David Lang wrote:
> >> there's no need to add this extra complexity (multiple rulesets and
> queues)
> >>
> >> What should be happening (on any output module) is:
> >>
> >> submit a batch.
> >>If rejected with a soft error, retry/suspend the output
> >
> > retry of the entire batch?  see below
> >
> >> if batch-size=1 and a hard error, send to errorfile
> >>if rejected with a hard error resubmit half of the batch
> >
> > But what if 90% of the batch was successfully added?  Then you are
> needlessly
> > resubmitting many of the records in the batch.
>
> when submitting batches, you get a success/fail for the batch as a whole
> (for
> 99% of things that actually allow you to insert in batches), so you don't
> know
> what message failed. This is a database transaction (again, in most
> cases), so
> if a batch fails, all you can do is bisect to figure out what message
> fails. If
> the endpoint is inserting some of the messages from a batch that fails,
> that's
> usually a bad thing.
>
> now, if ES batch mode isn't an ACID transaction and it accepts some
> messages and
> then tells you which ones failed, then you can mark the ones accepted as
> done
> and just retry the ones that fail. But there's still no need for a
> separate
> ruleset and queue. In Rsyslog, if an output cannot accept a message and
> there's
> reason to think that it will in the future, then you suspend that output
> and try
> again later. If you have reason to believe that the message is never going
> to be
> able to be delivered, then you need to fail the message or you will be
> stuck
> forever. This is what the error output was made for.
>
> > If using the "index" (default) bulk type, this causes duplicate records
> to be
> > added.
> > If using the "create" type (and you have assigned a unique _id), you
> will get
> > back many 409 Duplicate errors.
> > This causes problems - we know because this is how the fluentd plugin
> used to
> > work, which is why we had to change it.
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_monitoring_individual_nodes.html#_threadpool_section
> > "Bulk Rejections"
> > "It is much better to handle queuing in your application by gracefully
> > handling the back pressure from a full queue. When you receive bulk
> > rejections, you should take these steps:
> >
> > Pause the import thread for 3–5 seconds.
> > Extract the rejected actions from the bulk response, since it is
> probable
> > that many of the actions were successful. The bulk response will tell
> you
> > which succeeded and which were rejected.
> > Send a new bulk request with just the rejected actions.
> > Repeat from step 1 if rejections are encountered again.
> >
> > Using this procedure, your code naturally adapts to the load of your
> cluster
> > and naturally backs off.
> > "
>
> Does it really accept some and reject some in a random manner? or is it a
> matter
> of accepting the first X and rejecting any after that point? The first is
> easier
> to deal with.
>
> Batch mode was created to be able to more efficiently process messages
> that are
> inserted into databases, we then found that the reduced queue congestion
> was a
> significant advantage in itself.
>
> But unless you have a queue just for the ES action, doing queue
> manipulation
> isn't possible, all you can do is succeed or fail, and if you fail, the
> retry
> logic will kick in.
>
> Rainer is going to need to comment on this.
>
> David Lang
>
> >
> >> repeat
> >>
> >> all that should be needed is to add tests into omelasticsearch to
> detect
> >> the soft errors and turn them into retries (or suspend the output as
> >> appropriate)
> >>
> >> David Lang
> >
> >
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you 

Re: [rsyslog] rsyslog 8.28.0 (v8-stable) released

[Brian Knox via rsyslog]

Just finished upgrading us to 8.27 *shakes fist* haha. Thanks for the
update Florian! Definitely interested in the kafka output changes.

Cheers,
Brian

On Tue, Jun 27, 2017 at 10:54 AM Florian Riedl  wrote:

> Hi all,
>
> We have released rsyslog 8.28.0.
>
> This release features a lot of changes. Again, the most notable change
> is a way more robust, yet still experimental, support for Kafka output
> and input. In addition to this, there is a new build requirement for
> librelp 1.2.14 du to API requirements in imrelp and many changes/fixes
> for omfwd, imfile, mmdblookup, imtcp and many more.
>
> Please note that Kafka Support requires the librdkafka library as
> dependency, which itself has some new dependencies.
>
> For a complete list of changes, fixes and enhancements, please visit
> the ChangeLog.
>
> The packages will follow when they are finished.
>
> ChangeLog:
>
> https://github.com/rsyslog/rsyslog/blob/v8-stable/ChangeLog
>
> Download:
>
> http://www.rsyslog.com/downloads/download-v8-stable/
>
> As always, feedback is appreciated.
>
> Best regards,
> Florian Riedl
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] configure / make flags for directory prefix

[Brian Knox via rsyslog]

Thank you so much Michael! Sometimes you just need another pair of eyes on
something.

On Sun, May 7, 2017 at 10:39 PM Michael Biebl <mbi...@gmail.com> wrote:

> 2017-05-05 16:18 GMT+02:00 Brian Knox via rsyslog <
> rsyslog@lists.adiscon.com>:
> > I'm working on an fpm-cookery recipe (
> https://github.com/bernd/fpm-cookery )
> > for  building rsyslog, and running  into issues getting rsyslog to honor
> > prefix variables. I'm stuck in that any combination of --prefix vars I
> use,
> > rsyslog is trying to place rsyslog.service in
> /lib/systemd/rsyslog.service.
> >
> > Is  there a way to get this file to go into another location, or disable
> > installing the systemd service file altogether?
> >  --with-systemdsystemeunitir= doesn't seem to be respected.
>
> Looks like a typo.
> The correct configure switch is --with-systemdsystemunitdir= and this
> works fine:
>
> --with-systemdsystemunitdir=no disables the installation of the .service
> file
> --with-systemdsystemunitdir=/foo/bar/baz/ installs the service file as
> /foo/bar/baz/rsyslog.service
>
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] configure / make flags for directory prefix

[Brian Knox via rsyslog]

I'm working on an fpm-cookery recipe ( https://github.com/bernd/fpm-cookery )
for  building rsyslog, and running  into issues getting rsyslog to honor
prefix variables. I'm stuck in that any combination of --prefix vars I use,
rsyslog is trying to place rsyslog.service in /lib/systemd/rsyslog.service.

Is  there a way to get this file to go into another location, or disable
installing the systemd service file altogether?
 --with-systemdsystemeunitir= doesn't seem to be respected.

Cheers,
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm 2.0.3 released

[Brian Knox via rsyslog]

The load rulebase from a string is nice! Will add that to my wrapper I'm
using in normz ( https://github.com/taotetek/normz ).

On Thu, Mar 23, 2017 at 1:04 PM Florian Riedl  wrote:

> Hi all,
>
> We have just released liblognorm 2.0.3. This new version provides some
> fixes for the the annotate function and adds a test for it. A few
> different issues have also been fixed. See the Changelog for details.
>
> Changelog:Version 2.0.3, 2017-03-22
>
> - add ability to load rulebase from a string
> introduces new API:
> int ln_loadSamplesFromString(ln_ctx ctx, const char *string);
> closes https://github.com/rsyslog/liblognorm/issues/239
> - bugfix: string parser did not correctly parse word at end of line
> - bugfix: literal parser does not always store value if name is specified
> if
> rule=:%{"type":"literal", "text":"a", "name":"var"}%
> is used and matching message is provided, variable var ist not persisted.
> see also
> http://lists.adiscon.net/pipermail/rsyslog/2016-December/043985.html
>
> Download:
> http://www.liblognorm.com/download/liblognorm-2-0-3/
>
> As always, feedback is appreciated.
>
> Best regards,
> Florian Riedl
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.24.0 (v8-stable) released

[Brian Knox via rsyslog]

Yay! Thanks Florian! It will probably be a week or two before I get this
out into production, will provide feedback when it happens.

On Tue, Jan 10, 2017 at 10:03 AM Florian Riedl  wrote:

> Hi everybody,
>
> we have released rsyslog 8.24.0.
>
> This first release for 2017 brings a lot of changes. Most are detail
> enhancements for different modules. The biggest change is probably,
> that rsyslog now builds on the AIX platform.  Also 8.24.0 has two new
> message modification modules, which might be worth checking out. And a
> very useful addition is the ability to bind imudp and omfwd to a
> specific device.
>
> For a complete list of changes, fixes and enhancements, please visit
> the ChangeLog.
>
>
> ChangeLog:
>
> https://github.com/rsyslog/rsyslog/blob/v8-stable/ChangeLog
>
> Download:
>
> http://www.rsyslog.com/downloads/download-v8-stable/
>
> As always, feedback is appreciated.
>
> Best regards,
> Florian Riedl
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] omelasticsearch outstanding pr

[Brian Knox via rsyslog]

Wanted to double check on thoughts concerning this:

https://github.com/rsyslog/rsyslog/pull/1331

It would be super useful to me if i could get it in the next release but I
didn't want to just merge it without checking.

Cheers,
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Merge to Master?

[Brian Knox via rsyslog]

I have a PR for omczmq and for omelasticsearch for the next release - I
just wanted to double check that we merge to master now in the absence of
master-candidate.

Happy New Year!
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Brian Knox via rsyslog]

I noticed looking through the code that it looks like the error file
routine in omelasticsearch is not tied into the stats system - we use
impstats to monitor our rsyslog pipelines, and having a counter for write
errors would be super useful.

 I've submitted a PR to add the counter:
https://github.com/rsyslog/rsyslog/pull/1331

Cheers,
Brian

On Thu, Dec 15, 2016 at 7:58 AM mostolog--- via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Solved using json template (code blindness).
>
> Is there any way to set fields and use them (like @timestamp) but not
> indexing them on elastic? (hidden fields)
>
> Just tried with @timestamp, but it's being indexed :(
>
>
>
> El 15/12/16 a las 12:32, mosto...@gmail.com escribió:
> > Hi
> >
> >
> > At this moment we are frowarding RELP messages to Elasticsearch using
> > omelasticsearch plugin, but sadly message appears as json instead of
> > storing each properties. eg: message is { "app": "app1"... instead of
> > indexed document having a app property.
> >
> > Should we specify an especial param on rsyslog, a setting on elastic...?
> >
> > Regards
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] New omczmq features

[Brian Knox via rsyslog]

Just a heads up that I'm working on new features for the omczmq zeromq
output. The WIP PR is here: https://github.com/rsyslog/rsyslog/pull/1325

The README updates cover the gist of it:
https://github.com/taotetek/rsyslog/blob/020184ea2553e287f4e977d9245f9abe154567a2/contrib/omczmq/README

Summary: adding the ability to publish each log message on multiple zeromq
topics, and to allow the topics to be constructed using templates.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] REK stack

[Brian Knox]

Hahaha. I agree. "erk" sounds like the noise I make when my elasticsearch
cluster goes red on friday night when I was just headed out the door. ;)

On Fri, Dec 2, 2016 at 5:57 AM Bob Gregory  wrote:

> Big +1, because "erk" sounds like the noise you make when somebody stands
> on your toe at a formal social event; Rek Project makes us sound like
> dangerous anarchists, or possibly a dub-techno outfit.
>
> On Fri, 2 Dec 2016 at 10:49 Rainer Gerhards 
> wrote:
>
> Hi all,
>
> I start a new thread as the other one has a million of different topics now
> ;-)
>
> Just a short note: I think we should finally call this projekt "REK
> stack" vs. ERK and other ideas. This seems to be consensus, is logical
> (rsyslog-ES-Kibna, in right order) and as Brian pointed out there
> already is prior art ;-).
>
> Violent objections please here. I have updated the rsyslog github REK
> project:
>
> https://github.com/rsyslog/rsyslog/projects/1
>
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[Brian Knox]

Seeing people referring to "REK" made me smile this morning. We've been
referring to our setup as REK for quit awhile -
http://www.meetup.com/RVA-Data-Hackers/events/214996202/

I don't really see a reason for redis in the equation unless it's already
part of your infrastructure though.  imkafka is potentially interesting
however. I wrote imczmq for rsyslog so I have a little experience writing
input plugins - if there are others with a need for a such a thing that
would  be a fun collaboration.

Brian

On Wed, Nov 23, 2016 at 6:52 AM Bob Gregory  wrote:

> There've been a few discussions over the last few days that are all
> pointing in the same direction:
>
> * Is it better to use Rsyslog's omelasticsearch rather than pushing to
> logstash?
> * Should we have a minimal log shipper component as distinct from rsyslog's
> processing capabilities?
> * Ought we to have an imhiredis module?
>
> Really what we're talking about is replacing Logstash (and the various
> beats) with rsyslog. I'm perfectly happy with that, Logstash is a
> resource-expensive and fickle beast that spoils my otherwise pristine log
> pipeline, but I do think the community ought to think about whether this is
> the direction they want to take.
>
> For my part, I'm quite happy to help build an imhiredis (and imkafka?)
> module but only if I can actually dogfood it, which means replacing
> Logstash in our own environment.
>
> For that, I'd like to see better support for GeoIP tagging, a Riemann
> output plugin, some better guidance on "failed message queues", etc. etc.
> etc.
>
> Are we jointly interested in building the REK stack and, if so, can we
> start to work out the feature set we're missing, and the documentation we'd
> need for this to work? I'm a little concerned that if we tackle the usecase
> piece-meal, we'll end up with lots of disjointed parts that don't really
> solve the problem: logstash is not an adequate logstash.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok

[Brian Knox]

Getting some ideas from reading this. Thank you!

On Tue, Oct 18, 2016 at 3:22 AM Radu Gheorghe 
wrote:

> It look very very very very nice, Rainer! Thanks for publishing!
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Oct 17, 2016 at 4:53 PM, Rainer Gerhards
>  wrote:
> > It took a while, but finally the thesis is online:
> >
> >
> https://www.fernuni-hagen.de/imperia/md/content/rechnerarchitektur/rainer_gerhards.pdf
> >
> > Rainer
> >
> > 2016-10-06 11:32 GMT+02:00 Rainer Gerhards :
> >> 2016-10-06 11:23 GMT+02:00 mosto...@gmail.com :
> >>>
> >>>
> > Totally agree...(actually, liblognorm is giving me segfaults :P)
> 
>  I'll try to check next week when my current task is done.
> >>>
> >>> I know you're busy...trying to contribute as much as I can with
> everything
> >>> I deal with on my daily work.
> >>>
> 
>  Liblognorm is based on work from my MSc Thesis. The thesis paper is
>  currently being processed for upload, I expect it to be available next
>  week. If you'd like to dig down to the details and an explanation why
> it
>  is
>  faster, the thesis will have it in great detail. I can post a link
> once
>  it
>  is online.
> >>>
> >>> Cool
> >>> I guess it should be similar to what a firewall does when it "compiles"
> >>> the rules.
> >>
> >>
> >> Ah, not really, as here we have text detection, which is a different
> story
> >> for a firewall...
> >>
> >> Rainer
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Fwd: Re: rsyslog kills entire system => force reboot

[Brian Knox]

For what it is worth, I am running rsyslog 8.21 on around 12,000 servers
and have not run into any issues with it.

Cheers,
Brian

On Wed, Sep 28, 2016 at 3:20 AM Raffael Sahli 
wrote:

>
>
> On 09/27/2016 01:02 PM, Andre Lorbach wrote:
> > So far it seems to be very difficult to reproduce this problem.
> > Are you still able to reproduce the problem with 8.21?
> As you can imagine its quite difficult for me to reproduce it as well
> and at the moment I won't upgrade my production systems to a later version.
>
> > If yes could you send me the configuration you are using and the output
> > of: ldd /sbin/rsyslogd
> >
> > I am interested to see against which libfastjson library rsyslog is
> using,
> > it should be libfastjson.so.4
> Yes it's libfastjson.so.4
>
>
> But I had further problems with syslog, last friday nearly every server
> got a problem and again it was syslog
> Im not sure if it was the same problem since it was nearly on every
> system. What I found out so far is
> that nscd can block the system and go up 100%CPU and this problem is
> also related to syslog.
> (short story i've removed nscd from all systems since its not really
> required.)
>
>
> What I really need is a configuration which does work and drop messages
> even though messages can not be stored somewhere or whatever problem it is.
> CALL syslog() must not block the entire system. I know its not as
> specified in the RFC but
>
>
> Cheers
> Raffi
>
>
>
> > Best regards,
> > Andre Lorbach
> >
> >> -Original Message-
> >> From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-
> >> boun...@lists.adiscon.com] On Behalf Of singh.janmejay
> >> Sent: Friday, September 16, 2016 10:46 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] Fwd: Re: rsyslog kills entire system => force
> > reboot
> >> How long does it take to go thru one cycle of verifying the problem
> > exists?
> >> I was wondering if bisecting would be viable?
> >>
> >> May not be required though, stats, entire config and all thread
> > backtrace will
> >> likely give you/us enough clues.
> >>
> >> On Sep 16, 2016 12:30 PM, "Raffael Sahli" 
> > wrote:
> >>> yep, I can confirm that the problem is gone.
> >>> Downgrade back to 8.20 solved the problem.
> >>>
> >>> Anybody with the same problem?
> >>>
> >>>
> >>>  Forwarded Message 
> >>> Subject: Re: [rsyslog] rsyslog kills entire system => force reboot
> >>> Date: Mon, 12 Sep 2016 11:03:58 +0200
> >>> From: Raffael Sahli 
> >>> To: rsyslog@lists.adiscon.com
> >>>
> >>> fyi since the downgrade to 8.20 (from 8.21), we didn't notice any
> > problems.
> >>>
> >>>
> >>> On 09.09.2016 15:48, Raffael Sahli wrote:
> >>>
>  On 09.09.2016 15:09, David Lang wrote:
>   > On Fri, 9 Sep 2016, Raffael Sahli wrote:
> 
>   >>
>   >> Actually I tried $ActionResumeRetryCount with a value 10, @see
>  2nd  >> configuration. But faced the same problem.
>   >>
>   >>
>   >> Strange thing is, I deployed new rsyslog configs without the
>  remote  >> forwarding, but this morning one server was unresponsive
>  again, same  >> problem.
>   >>
>   >> Does anybody know, can this also happen without remote
> >> forwarding?
>   >
>   > where are your local logs being written? is there any chance that
>  it's  > running out of space or otherwise falling behind (think of a
>  slow NFS  > server)  >  > remember that even with retries = 10
>  rsyslog won't stop completely, but  > it will slow things down
>  drastically so that it appears to be dead.
> 
>  No, just the local filesystem.
>  And the fs and disk i/o is fine.
> 
> 
>   >
>   >> Maybe this more a general syslog problem, as far as I know the
>  RFC,  >> since syslog should never loose any messages by default.
>   >> I just like to know what rsyslog config I should use with remote
> >> forwarding, but without any timeout for syslog services if syslog
>  is  >> somehow unresponsive.
>   >
>   > per the syslog spec it should block forever if it can't deliver
>  the  > message.
> 
>  Yeah thats the point, I don't get that
> 
>   >
>   > But to really see what's going on, configure impstats and have it
>  write  > to a local file, that will let you see what's going on when
>  it appears  > to stalls.
> 
>  Mhm will try it out, or/and try downgrade to an earlier version since
>  I did not have such problems before.
> 
> 
> 
> 
> 
> >>> --
> >>> Raffael Sahli
> >>>
> >>>
> >>> ___
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> >>> sites 

Re: [rsyslog] rsyslog dying if redis output fails

[Brian Knox]

Hello! I can try to reproduce and take a look in a day or too, thanks for
the bug report!

Brian

On Wed, Aug 31, 2016 at 6:40 AM Angel L. Mateo  wrote:

> Hello,
>
> I'm trying to configure my rsyslog server (8.21.0) to send logs to
> a
> redis server.
>
> I'm using the configuration at
> https://sematext.com/blog/2015/09/28/recipe-rsyslog-redis-logstash/
>
> My problem is that when rsyslog starts everything is fine, but if
> for
> some reason (I'm having problems with my redis server configuration) a
> redis message fails, then rsyslog daemon dies.
>
> To output redis messages I'm using:
>
> if  then {
>  action(type="omhiredis" template="json_lines"
> server="redis.mydomain.com" serverport="6379" mode="queue" key="key")
> }
>
> where json_lines templates is the same defined at above url.
>
> Any help?
>
> --
> Angel L. Mateo Martínez
> Sección de Telemática
> Área de Tecnologías de la Información
> y las Comunicaciones Aplicadas (ATICA)
> http://www.um.es/atica
> Tfo: 868889150
> Fax: 86337
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

I am very much looking forward to the custom data type support!  Safe
travels Rainer!

Brian

On Fri, Jun 24, 2016 at 2:07 AM Rainer Gerhards 
wrote:

> Thanks all for the great discussion and effort going forward! I am in
> preparation for a trip next week and so unfortunately had limited time
> to contribute (and will be unable next week), but I am more than
> interested in helping to move this forward.
>
> Note that we currently have some rulebases inside liblognorm's git:
> https://github.com/rsyslog/liblognorm/tree/master/rulebases This might
> be the place where we can begin to actually gather a full set ... or
> we could create a new git repo. The latter might be a better idea, as
> the folks who primarily maintain it are probably quite different.
>
> Again, I am excited to see all this new activity. Also keep in mind
> that with v2 (finally to be released next month), we can have custom
> data types just like in grok, so building rules is also much easier.
> IMHO it would make sense to first build a set of custom data types
> (like we did in lognorm with the cisco address representation), and
> then base rules on those extended set of base types. This is a sample
> from the testbench of how custom types are defined:
>
> https://github.com/rsyslog/liblognorm/blob/master/tests/usrdef_twotypes.sh
>
> Also, the doc has good information on that topic:
> https://github.com/rsyslog/liblognorm/blob/master/doc/configuration.rst
>
> As I said, I will unfortunately be mostly silent up unitl begin of
> june - please don't treat this as sign of desinterest! Again, I think
> this is an extremely valuable approach.
>
> Rainer
>
> 2016-06-23 19:25 GMT+02:00 David Lang :
> > On Thu, 23 Jun 2016, Champ Clark III wrote:
> >
> >> I assist with a project that pretty heavily depends on liblognorm called
> >> "Sagan" (http://sagan.io).
> >>
> >> While we have other "normalization" methods, we prefer liblognorm.  Our
> >> community rulebase file is at:
> >>
> >> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase
> >>
> >> I agree with David, we don't want 10 different ways to normalize a Cisco
> >> log. At the same time, Cisco logs sometimes differ just enough that you
> >> _might_ need multiple ways to normalize them.
> >
> >
> > as an example of what I'm talking about.
> >
> > take the log example %ASA-6-302014 (end of TCP session)
> >
> > a few variations of which are:
> >
> >  %ASA-6-302014:Teardown TCP connection 42095195 for outside:2.2.9.2/5721
> to
> > inside:192.168.1.1/54151 duration 0:00:30 bytes 0 SYN Timeout
> >
> >  %ASA-6-302014: Teardown TCP connection 43363071 for
> > outside:192.168.2.5\/58949(LOCAL\\D.A) to
> > outside:192.168.2.3\/3283(LOCAL\\CP-G-SEP) duration
> 0:00:00
> > bytes 0 TCP Reset-O (D.A)
> >  %ASA-6-302014: Teardown TCP connection 51708532 for outside:
> 10.1.5.5/54853
> > to backup:192.168.2.1/4784(LOCALCP-G-SEPC999) duration
> 0:00:00
> > bytes 0 
> >
> > some people will parse it so that they have the variables sourceif,
> > sourceip, sourceport, destif, destip, destport etc
> >
> > I do source:{interface,ip,port} dest:{interface,ip,port}
> >
> > this is making use of the v2 ciscointerface type
> >
> > prefix=%timestamp:date-rfc3164% %hostname:word%
> >
> > rule=cisco,disconnect: \x25ASA-6-302014\x3a Teardown %proto:word%
> connection
> > %connection-id:number% for %source:cisco-interface-spec% to
> > %dest:cisco-interface-spec% duration %duration:char-to: % bytes
> > %bytes:number% %reason:rest%
> >
> > So we will need to agree of if we are going to use nesting or not (I
> think
> > we should), and if we do it with Cisco, we need to do it across the board
> >
> > by the way, this also brings up the issue of tags for the message
> >
> >> We have talked about "market place" for rule normalization for years
> now.
> >> It was always my impression that this would be part of the rsyslog team
> >> efforts. It sounds like you have enough on your plate, keeping track for
> >> rulebase isn't high on priority.  I understand this.  With Sagan, we are
> >> doing this "anyways".  That is, we are creating rulebases for different
> >> types of logs either way.  We commit them to the Sagan repo right now.
> >>
> >> I'd like to suggest the following for response:
> >>
> >> 1.  Split off the "normalization.rules" base from Sagan and great a new,
> >> separate github repo for it.
> >> 2.  If someone would like to add some rulebase "rules",  they can do a
> >> "pull" request.
> >> 3.  All rulebase "rules" need to have an example,  anonymized log
> sample.
> >> Used for testing.
> >> 4.  If the rules look good,  then they can be merged.
> >
> >
> > besides the pull request mechansim, I think we also need a way for people
> > who have rulesets to send them out for others to convert to pull
> requests. I
> > think that there is going to be a lot of tweaking/corrections to the
> > proposed rules, and a pull request 

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

David - checked with the powers that be and everything is good. I'm going
to create a normalization-toolkit repo on our public github and will link
it here once a few things are in place.  I don't yet have strong opinions
on what goes in it - I'm going to start with the dockerized setup we're
working on for integration tests (we'll just move development of this into
the open).

People can start contributing rulebases if they want, and I figure we can
organize things and consolidate things as it makes sense. I think
collaboration with PRs and github issues is better than collaboration on
mailing  lists so would prefer to just get something moving and we can take
it from there.

Cheers,
Brian

On Thu, Jun 23, 2016 at 1:09 PM David Lang <da...@lang.hm> wrote:

> On Thu, 23 Jun 2016, Brian Knox wrote:
>
> > David - I'm sure I could get some time to devote to shepherding this,
> and I
> > could get some time and resources from our community team to write some
> > articles / tutorials about rsyslog + mmnormalize and generate some
> > publicity for the project.  Additionally I have access to a decently
> large
> > sampling of logs from a reasonably scaled environment for testing.
>
> I can also do some article writing.
>
> > If this is something people are interested in and the only blocker is
> time
> > and resources let me talk to a couple of people today and I'll update the
> > list.
>
> given the number of times this has come up, I'm sure there is some
> interest.
>
> Thanks for volunteering on this.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

Hi Champ! My toes are fine and the more the merrier.  Well - the more
collaborators, not toes.

I've used Sagan's rulebase as a reference before - great stuff!

It comes to mind that a coworker and I are currently working on a
dockerized rsyslog + elasticsearch environment for doing rsyslog
integration testing,  that we were planning on tossing up github.  Via
docker compose it starts up a very small ES cluster ( 2 indexers in
containers + 1 master, client, and kibana ) along with an rsyslog instance
configured to receive over TCP and UDP and forward to ES.

I'm currently imagining a system where people could check in mmnormalize
rules + add to a sample rulebase and log(s), and travis CI could fire off,
run the sample log for the rule through the rulebase, then verify the
results.  Such an environment could also be spun up locally for testing
while developing new rules.

If such a project is something that others would find useful, I could
definitely get my employer to sponsor my time on it.

I was thinking we could use a dev process we use in the ZeroMQ community
that is designed for low friction and high amounts of collaboration without
a lot of up front coordination ( http://rfc.zeromq.org/spec:42/C4/ ) so
that people with good ideas who want to contribute can just jump in.
People who contribute under this process are promoted to maintainers
without any fuss, so no one has to worry about central ownership.

Let me poke a couple of people - I can probably get what we have as far as
the test environment up on github by the end of the week - the more the
merrier.  We were going to release at least the test environment regardless.

Cheers,
Brian

On Thu, Jun 23, 2016 at 9:25 AM Champ Clark III <ccl...@quadrantsec.com>
wrote:

> I assist with a project that pretty heavily depends on liblognorm called
> "Sagan" (http://sagan.io).
>
> While we have other "normalization" methods,  we prefer liblognorm.  Our
> community rulebase file is at:
>
> https://github.com/beave/sagan-rules/blob/master/normalization.rulebase
>
> I agree with David,  we don't want 10 different ways to normalize a Cisco
> log.   At the same time,  Cisco logs sometimes differ just enough that you
> _might_ need multiple ways to normalize them.
>
> We have talked about "market place" for rule normalization for years now.
>  It was always my impression that this would be part of the rsyslog team
> efforts.  It sounds like you have enough on your plate,  keeping track for
> rulebase isn't high on priority.   I understand this.   With Sagan,  we are
> doing this "anyways".  That is,  we are creating rulebases for different
> types of logs either way.   We commit them to the Sagan repo right now.
>
> I'd like to suggest the following for response:
>
> 1.  Split off the "normalization.rules" base from Sagan and great a new,
> separate github repo for it.
> 2.  If someone would like to add some rulebase "rules",  they can do a
> "pull" request.
> 3.  All rulebase "rules" need to have an example,  anonymized log sample.
> Used for testing.
> 4.  If the rules look good,  then they can be merged.
>
> I'm certainly not trying to step on Brian's or anyone elses toe's.
>  IMHO,  Sagan will benefit from a project like this.  Obviously, rsyslog
> will as well.   This would likely bring other people outside rsyslog to the
> project as well).
>
> Let me know your thoughts and thank you.
>
>
>
> - Original Message -
> From: "Ryan Ward" <ryan.w...@gliacelltechnologies.com>
> To: "rsyslog-users" <rsyslog@lists.adiscon.com>
> Sent: Thursday, June 23, 2016 8:51:48 AM
> Subject: Re: [rsyslog] mmnormalize rule database Re: mmgrok packages
>
> All as a newbie to rsyslog I think this is a great idea and would find a
> marketplace for rulebases and examples very beneficial.
>
>
>
> On Thu, Jun 23, 2016 at 7:06 AM, Brian Knox <bk...@digitalocean.com>
> wrote:
>
> > David - I'm sure I could get some time to devote to shepherding this,
> and I
> > could get some time and resources from our community team to write some
> > articles / tutorials about rsyslog + mmnormalize and generate some
> > publicity for the project.  Additionally I have access to a decently
> large
> > sampling of logs from a reasonably scaled environment for testing.
> >
> > If this is something people are interested in and the only blocker is
> time
> > and resources let me talk to a couple of people today and I'll update the
> > list.
> >
> > Cheers,
> > Brian
> >
> > On Wed, Jun 22, 2016 at 7:24 PM David Lang <da...@lang.hm> wrote:
> >
> > > On Wed, 22 Jun 2016, Joe Blow wrote:
> > >
> &

Re: [rsyslog] mmnormalize rule database Re: mmgrok packages

[Brian Knox]

David - I'm sure I could get some time to devote to shepherding this, and I
could get some time and resources from our community team to write some
articles / tutorials about rsyslog + mmnormalize and generate some
publicity for the project.  Additionally I have access to a decently large
sampling of logs from a reasonably scaled environment for testing.

If this is something people are interested in and the only blocker is time
and resources let me talk to a couple of people today and I'll update the
list.

Cheers,
Brian

On Wed, Jun 22, 2016 at 7:24 PM David Lang  wrote:

> On Wed, 22 Jun 2016, Joe Blow wrote:
>
> > What about soliciting people to start sharing their mmnormalize rules?
> > I've already shared my checkpoint rules, I could see about sharing my
> Cisco
> > rules as well.  I avoid regex engines like the plague (for obvious
> > reasons), but would also like to see larger log source parsers adopted
> and
> > open sourced.
> >
> > Thoughts?  Should we try and start a larger repository for parsing well
> > adopted log sources via liblognorm?
>
> This thought keeps getting raised. Yes this shoudl be done. The problem is
> that
> nobody has stepped up to organize this.
>
> We don't want to have 50 different ways to handle the same Cisco message,
> but
> how do we pick which of the many different versions we are going to use?
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with rsyslog Version > 8.16.0

[Brian Knox]

I'm in them middle of upgrading some high load servers to rsyslog 8.18 -
and have some fairly complex configs  on them. I'll keep an eye on on
things today and report back how it goes.



On Wed, Apr 20, 2016 at 12:31 PM David Lang  wrote:

> If you can get a coredump, a gdb analysis of the core file is useful
>
> cat gdb.commands
> bt
> bt full
> info threads
> thread apply all bt
> thread apply all bt full
> quit
>
>
>
> gdb rsyslogd corefile 
> send the report directly to Rainer rather than the list (as it can contain
> sensitive stuff). Looking at the output of these commands may give you a
> clue as
> to what functions are triggering the segfault.
>
> I did see a patch for libfastjson that did not make it into a release yet
> that
> could cause a segfault in cases where a variable doesn't exist.
>
> moving to 8.18 should move you from json-c to libfastjson, and that fixes
> quite
> a number of known segfaulting issues.
>
> David Lang
>
>
> On Wed, 20 Apr 2016, Christopher Racky wrote:
>
> > Date: Wed, 20 Apr 2016 14:18:25 +0200
> > From: Christopher Racky 
> > Reply-To: rsyslog-users 
> > To: rsyslog-users 
> > Subject: [rsyslog] segfault with rsyslog Version > 8.16.0
> >
> > Hello,
> >
> > Thanks for all your great Support in the past!
> >
> > Today I upgraded from rsyslog 8.16 over 8.17 to 8.18.
> > (As I noticed several Memory leaks in my Environment with arroung 10k
> logs/sec)
> > I use a Red-Hat 6.7 Server with all latest patches installed.
> >
> > Unfortunately during start of rsyslog I get the following message in
> > kernel ring-buffer.
> > Apr 20 13:15:09 demosrv kernel: rsyslogd[16640]: segfault at
> > 7f3b276229d0 ip 7f3b2cecb213 sp 7fff3099bd30 error 4 in
> > libpthread-2.12.so[7f3b2cec3000+17000]
> >
> > This message did not appear with rsyslog 8.16. Buf after an upgrade to
> > 8.17 and with the latest version 8.18 I get this error.
> > Everythings seems to work fine, but whenever I do a restart, I get the
> > mentioned kernel error message.
> >
> >
> > Do you have any idear / hint?
> > How can I debug the issue?
> >
> >
> > My config is quite complex, so I just an excerpt about modules loaded:
> >
> -
> > Module (load="imuxsock") # provides support for local system logging
> > (e.g. via logger command)
> > Module (load="imklog")   # provides kernel logging support (previously
> > done by rklogd)
> > Module (load="impstats" interval="120" severity="7" log.syslog="on"
> > ruleset="pstats")
> > Module (load="imtcp" KeepAlive="on" KeepAlive.Probes="1"
> > KeepAlive.Interval="2" KeepAlive.Time="20" MaxSessions="5000")
> > Module (load="imudp")
> > Module (load="ommysql")
> > Module (load="omudpspoof")
> > Module (load="mmnormalize")
> >
> > # Message Parsers
> > $modload pmlastmsg
> > $modload pmaixforwardedfrom
> > $rulesetparser rsyslog.lastline
> > $rulesetparser rsyslog.aixforwardedfrom
> > $rulesetparser rsyslog.rfc5424
> > $rulesetparser rsyslog.rfc3164
> >
> -
> >
> >
> > [root@demosrv syslog]# cat /etc/redhat-release
> > Red Hat Enterprise Linux Server release 6.7 (Santiago)
> > [root@demosrv syslog]# yum update
> > Loaded plugins: product-id, refresh-packagekit, security,
> subscription-manager
> > This system is not registered to Red Hat Subscription Management. You
> > can use subscription-manager to register.
> > Setting up Update Process
> > No Packages marked for Update
> > [root@demosrv syslog]# tail /var/log/messages
> > Apr 20 13:15:01 demosrv rsyslogd: [origin software="rsyslogd"
> > swVersion="8.17.0" x-pid="16640" x-info="http://www.rsyslog.com;]
> > exiting on signal 15.
> > Apr 20 13:15:09 demosrv kernel: rsyslogd[16640]: segfault at
> > 7f3b276229d0 ip 7f3b2cecb213 sp 7fff3099bd30 error 4 in
> > libpthread-2.12.so[7f3b2cec3000+17000]
> > Apr 20 13:15:09 demosrv rsyslogd: [origin software="rsyslogd"
> > swVersion="8.17.0" x-pid="28237" x-info="http://www.rsyslog.com;]
> > start
> > Apr 20 14:01:32 demosrv yum[31727]: Updated: rsyslog-8.18.0-1.el6.x86_64
> > Apr 20 14:01:32 demosrv yum[31727]: Updated:
> > rsyslog-udpspoof-8.18.0-1.el6.x86_64
> > Apr 20 14:01:32 demosrv yum[31727]: Updated:
> rsyslog-mysql-8.18.0-1.el6.x86_64
> > Apr 20 14:01:32 demosrv yum[31727]: Updated:
> > rsyslog-mmnormalize-8.18.0-1.el6.x86_64
> > Apr 20 14:01:32 demosrv yum[31727]: Updated:
> > rsyslog-pmaixforwardedfrom-8.18.0-1.el6.x86_64
> > Apr 20 14:01:33 demosrv rsyslogd: [origin software="rsyslogd"
> > swVersion="8.17.0" x-pid="28237" x-info="http://www.rsyslog.com;]
> > exiting on signal 15.
> > Apr 20 14:01:36 demosrv kernel: rsyslogd[28237]: segfault at
> > 7f943a72f9d0 ip 7f943ffd8213 sp 7fff20878110 error 4 in
> > libpthread-2.12.so[7f943ffd+17000]
> > Apr 

Re: [rsyslog] pre-releease of rsyslog 08.18.0 available

[Brian Knox]

Yay!  I'll try doing a local build.

Brian

On Mon, Apr 18, 2016 at 8:59 AM, Rainer Gerhards 
wrote:

> Hi all,
>
> I have just placed pre-release tarballs for tomorrow's upcoming
> rsyslog 8.18.0 release on the server.
>
> * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.18.0.tar.gz
>
> That tarbal is most probably the final one.
>
> It would be appreciated if a build could be tried and any problems be
> reported.
>
> Note that I originally intended to do a new release (0.99.3) of
> libfastjson at the same time. However, there are some spots I am
> ironing out, and so I have postponed that. The 8.18.0 codebase works
> with 0.99.2 and will most probably be able to support 0.99.3, when
> available, without any changes to rsyslog.
>
> Thanks,
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Timeline for 8.18 release

[Brian Knox]

that reminds me I need to get in a small patch!  will do it this morning if
it's not too late!

On Wed, Apr 13, 2016 at 2:44 AM, Rainer Gerhards 
wrote:

> 2016-04-12 18:31 GMT+02:00 Alec Swan :
> > Thanks, Rainer, I was mostly looking at the github repo and missed the
> > version field on http://www.rsyslog.com/
>
> just FYJ: I use release milestones on github, so you can see it there,
> as well. But note that release milestones are a bit earlier (usually
> fridays vs. tuesdays) as we need time for final checks and package
> building.
>
> Rainer
>
> >
> > Thanks,
> >
> > Alec
> >
> > On Tue, Apr 12, 2016 at 12:38 AM, Rainer Gerhards <
> rgerha...@hq.adiscon.com>
> > wrote:
> >
> >> 2016-04-12 3:33 GMT+02:00 David Lang :
> >> > I believe it's due out this week.
> >>
> >> It's always stated in the version box on the rsyslog home page (top
> >> right): Apr, 19th is for 8.18.
> >>
> >> Rainer
> >> >
> >> > David Lang
> >> >
> >> > On Mon, 11 Apr 2016, Alec Swan wrote:
> >> >
> >> >> Is there any approximate date when version 8.18 is expected to go
> out?
> >> >
> >> > ___
> >> > rsyslog mailing list
> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> >> of
> >> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T
> >> > LIKE THAT.
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.17.0 (v8-stable) released

[Brian Knox]

Thanks Thomas!

On Thu, Mar 10, 2016 at 10:32 AM, Thomas D. <whi...@whissi.de> wrote:

> Hi,
>
> Brian Knox wrote:
> > Will this fix be appearing in 8.18 in this case?  I just need to know so
> I
> > can plan on patching my local builds of 8.17 as impstats is pretty
> critical
> > for us and I'd rather have the fix than block the malformed stats lines.
> > Not a huge deal either way, just wondering.
>
> Rainer merged the fix today. So it will be in rsyslog-8.18.
>
>
> -Thomas
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.17.0 (v8-stable) released

[Brian Knox]

Will this fix be appearing in 8.18 in this case?  I just need to know so I
can plan on patching my local builds of 8.17 as impstats is pretty critical
for us and I'd rather have the fix than block the malformed stats lines.
Not a huge deal either way, just wondering.

Cheers,
Brian

On Wed, Mar 9, 2016 at 3:00 PM, Andrew Davidoff  wrote:

> On Wed, Mar 9, 2016 at 11:59 AM, singh.janmejay
>  wrote:
> > This is a bug, here is the fix:
> https://github.com/rsyslog/rsyslog/pull/863
> >
> > Workaround(in the meanwhile): Since you are not using dynstats, its
> > safe to ignore that line (everything it reports is related to dynstats
> > buckets).
>
> Thanks for the quick response.
>
> Andy
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.17.0 (v8-stable) released

[Brian Knox]

Andrew - I'm setting up to do an 8.17 build today - will see if I can
replicate.

On Wed, Mar 9, 2016 at 11:04 AM, Andrew Davidoff  wrote:

> On Tue, Mar 8, 2016 at 9:54 AM, Florian Riedl  wrote:
> > Hi all,
> >
> > We have released rsyslog 8.17.0.
>
> I just started testing this release and with the same config I was
> using for 8.13 (which may be the issue) I'm seeing the following
> incomplete JSON being dropped in my rsyslog stats log for "global".
> I'm including the lines before and after for context.
>
> 2016-03-09T15:49:29.427521+00:00 01.syslog.dev.lax1 rsyslogd-pstats:
>
> {"name":"imudp(w0)","origin":"imudp","called.recvmmsg":4,"called.recvmsg":0,"msgs.received":2}
> 2016-03-09T15:50:30.150578+00:00 01.syslog.dev.lax1 rsyslogd-pstats:
> {"name":"global","origin":"dynstats",
> 2016-03-09T15:50:30.150732+00:00 01.syslog.dev.lax1 rsyslogd-pstats:
> {"name":"action
>
> 0","origin":"core.action","processed":0,"failed":0,"suspended":0,"suspended.duration":0,"resumed":0}
>
> I'm not sure if this indicates a configuration issue or a bug, but
> wanted to pass it along.
>
> My pstats config looks like this (looking back over the docs, I am not
> sure anymore why I broke out the file handler into a ruleset instead
> of using log.file):
>
> module(
> load="impstats"
> interval="60"
> format="json"
> ruleset="pstats")
>
> ruleset(
> name="pstats"
> queue.type="FixedArray") {
>
> action(
> type="omfile"
> file="/var/log/rsyslog.stats.log")
> }
>
> I also tried this format (no ruleset) with the same results:
>
> module(
> load="impstats"
> interval="60"
> format="json"
> log.file="/var/log/rsyslog.stats.log")
>
>
>
> Thanks.
> Andy
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog 8.17 release

[Brian Knox]

First: Yay! Can't wait to get this into production :)

Second: There was a feature added to allow adding @cee tags to pure JSON
files that are being consumed by imfile that I think was missed in the
change log.

Cheers,
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] pre-releease of rsyslog 08.17.0 available

[Brian Knox]

Rainer - question - is it possible to build rsyslog without libjson-c now?
I've built libfastjson - but I notice that liblognorm's latest stable
release is still 1.1.2, which I believe still requires libjson-c.

Brian

On Mon, Mar 7, 2016 at 2:54 AM, Rainer Gerhards 
wrote:

> The doc tarball is now also online:
>
> http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-doc-8.17.0.tar.gz
>
> Rainer
>
> 2016-03-07 7:47 GMT+01:00 Rainer Gerhards :
> > Hi all,
> >
> > I have just placed pre-release tarballs for tomorrow's upcoming
> > rsyslog 8.17.0 release on the server. This includes a tarball for
> > libfastjson as prequisite.
> >
> > *
> http://www.rsyslog.com/files/download/rsyslog/rc/libfastjson-0.99.2.tar.gz
> > * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.17.0.tar.gz
> >
> > These tarballs are most probably the final ones. We still have one
> > issue that we would like to get into the release, so the final tarball
> > may change slightly. It addresses an issue with KSI signature
> > integration, a feature only few users use and that usually is not
> > enabled by default in distributions. So this change, if it happens,
> > will most probably not affect anything for almost all users.
> >
> > It would be appreciated if a build could be tried and any problems be
> reported.
> >
> > Thanks,
> > Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] pre-releease of rsyslog 08.17.0 available

[Brian Knox]

I'll try our standard build with this today, thanks Rainer!

On Mon, Mar 7, 2016 at 2:54 AM, Rainer Gerhards 
wrote:

> The doc tarball is now also online:
>
> http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-doc-8.17.0.tar.gz
>
> Rainer
>
> 2016-03-07 7:47 GMT+01:00 Rainer Gerhards :
> > Hi all,
> >
> > I have just placed pre-release tarballs for tomorrow's upcoming
> > rsyslog 8.17.0 release on the server. This includes a tarball for
> > libfastjson as prequisite.
> >
> > *
> http://www.rsyslog.com/files/download/rsyslog/rc/libfastjson-0.99.2.tar.gz
> > * http://www.rsyslog.com/files/download/rsyslog/rc/rsyslog-8.17.0.tar.gz
> >
> > These tarballs are most probably the final ones. We still have one
> > issue that we would like to get into the release, so the final tarball
> > may change slightly. It addresses an issue with KSI signature
> > integration, a feature only few users use and that usually is not
> > enabled by default in distributions. So this change, if it happens,
> > will most probably not affect anything for almost all users.
> >
> > It would be appreciated if a build could be tried and any problems be
> reported.
> >
> > Thanks,
> > Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] imfile buffer overflow master-candidate

[Brian Knox]

Here we go - https://github.com/rsyslog/rsyslog/pull/840

On Thu, Mar 3, 2016 at 9:15 AM, Brian Knox <bk...@digitalocean.com> wrote:

>
> https://github.com/rsyslog/rsyslog/blob/b5649a98107a8e6b7042e103f17bb16e907504f2/plugins/imfile/imfile.c#L686
>
> Looks like getBasename should perhaps return a  -1 if it doesn't find a
> slash - and then we can "do the right thing" based on that.  I'll see if I
> can sneak in time for a fix today or tomorrow.
>
> Cheers,
> Brian
>
> On Thu, Mar 3, 2016 at 9:04 AM, Brian Knox <bk...@digitalocean.com> wrote:
>
>> line 727 in imfile.c :
>>
>> memcpy(dirn, inst->pszFileName, i); /* do not copy slash */
>>
>>
>> On Thu, Mar 3, 2016 at 8:53 AM, Brian Knox <bk...@digitalocean.com>
>> wrote:
>>
>>> I've found a buffer overflow in imfile in the master-candidate branch.
>>> To reproduce, make an imfile config that uses a relative path rather than
>>> absolute to a file:
>>>
>>> ```
>>> module(load="imfile" PollingInterval="10")
>>>
>>> input(
>>> type="imfile"
>>> tag="crash"
>>> File="crashme"
>>> )
>>>
>>> *.* /var/log/syslog
>>> ```
>>>
>>> This results in:
>>>
>>> ```
>>> 3146.392981790:main thread: deletestateonfiledelete: (unset)
>>> 3146.392987727:main thread: addmetadata: (unset)
>>> 3146.392993638:main thread: addceetag: (unset)
>>> 3146.392999527:main thread: statefile: (unset)
>>> *** buffer overflow detected ***: rsyslogd terminated
>>> === Backtrace: =
>>> /lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7f286982b38f]
>>> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f28698c2c9c]
>>> /lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7f28698c1b60]
>>> /usr/local/lib/rsyslog/imfile.so(+0x22cd)[0x7f286919f2cd]
>>> /usr/local/lib/rsyslog/imfile.so(+0x254d)[0x7f286919f54d]
>>> rsyslogd(inputProcessCnf+0x99)[0x4147a9]
>>> rsyslogd(cnfDoObj+0x90)[0x414ba0]
>>> rsyslogd(yyparse+0xbae)[0x45435e]
>>> rsyslogd(load+0xc35)[0x414145]
>>> rsyslogd(initAll+0x5ef)[0x448e2f]
>>> rsyslogd(main+0x30)[0x40dfe0]
>>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f28697d9ec5]
>>> rsyslogd[0x40e35a]
>>> ```
>>>
>>> I don't have time to dig into it today but wanted to go ahead and report
>>> it.  If I correctly use an absolute path to the file (I used a relative by
>>> mistake when testing and found this), things work as expected.
>>>
>>> If I get some time tomorrow to dig into it I will!
>>>
>>> Cheers,
>>> Brian
>>>
>>>
>>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] imfile buffer overflow master-candidate

[Brian Knox]

https://github.com/rsyslog/rsyslog/blob/b5649a98107a8e6b7042e103f17bb16e907504f2/plugins/imfile/imfile.c#L686

Looks like getBasename should perhaps return a  -1 if it doesn't find a
slash - and then we can "do the right thing" based on that.  I'll see if I
can sneak in time for a fix today or tomorrow.

Cheers,
Brian

On Thu, Mar 3, 2016 at 9:04 AM, Brian Knox <bk...@digitalocean.com> wrote:

> line 727 in imfile.c :
>
> memcpy(dirn, inst->pszFileName, i); /* do not copy slash */
>
>
> On Thu, Mar 3, 2016 at 8:53 AM, Brian Knox <bk...@digitalocean.com> wrote:
>
>> I've found a buffer overflow in imfile in the master-candidate branch.
>> To reproduce, make an imfile config that uses a relative path rather than
>> absolute to a file:
>>
>> ```
>> module(load="imfile" PollingInterval="10")
>>
>> input(
>> type="imfile"
>> tag="crash"
>> File="crashme"
>> )
>>
>> *.* /var/log/syslog
>> ```
>>
>> This results in:
>>
>> ```
>> 3146.392981790:main thread: deletestateonfiledelete: (unset)
>> 3146.392987727:main thread: addmetadata: (unset)
>> 3146.392993638:main thread: addceetag: (unset)
>> 3146.392999527:main thread: statefile: (unset)
>> *** buffer overflow detected ***: rsyslogd terminated
>> === Backtrace: =
>> /lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7f286982b38f]
>> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f28698c2c9c]
>> /lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7f28698c1b60]
>> /usr/local/lib/rsyslog/imfile.so(+0x22cd)[0x7f286919f2cd]
>> /usr/local/lib/rsyslog/imfile.so(+0x254d)[0x7f286919f54d]
>> rsyslogd(inputProcessCnf+0x99)[0x4147a9]
>> rsyslogd(cnfDoObj+0x90)[0x414ba0]
>> rsyslogd(yyparse+0xbae)[0x45435e]
>> rsyslogd(load+0xc35)[0x414145]
>> rsyslogd(initAll+0x5ef)[0x448e2f]
>> rsyslogd(main+0x30)[0x40dfe0]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f28697d9ec5]
>> rsyslogd[0x40e35a]
>> ```
>>
>> I don't have time to dig into it today but wanted to go ahead and report
>> it.  If I correctly use an absolute path to the file (I used a relative by
>> mistake when testing and found this), things work as expected.
>>
>> If I get some time tomorrow to dig into it I will!
>>
>> Cheers,
>> Brian
>>
>>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] imfile buffer overflow master-candidate

[Brian Knox]

line 727 in imfile.c :

memcpy(dirn, inst->pszFileName, i); /* do not copy slash */


On Thu, Mar 3, 2016 at 8:53 AM, Brian Knox <bk...@digitalocean.com> wrote:

> I've found a buffer overflow in imfile in the master-candidate branch.  To
> reproduce, make an imfile config that uses a relative path rather than
> absolute to a file:
>
> ```
> module(load="imfile" PollingInterval="10")
>
> input(
> type="imfile"
> tag="crash"
> File="crashme"
> )
>
> *.* /var/log/syslog
> ```
>
> This results in:
>
> ```
> 3146.392981790:main thread: deletestateonfiledelete: (unset)
> 3146.392987727:main thread: addmetadata: (unset)
> 3146.392993638:main thread: addceetag: (unset)
> 3146.392999527:main thread: statefile: (unset)
> *** buffer overflow detected ***: rsyslogd terminated
> === Backtrace: =
> /lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7f286982b38f]
> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f28698c2c9c]
> /lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7f28698c1b60]
> /usr/local/lib/rsyslog/imfile.so(+0x22cd)[0x7f286919f2cd]
> /usr/local/lib/rsyslog/imfile.so(+0x254d)[0x7f286919f54d]
> rsyslogd(inputProcessCnf+0x99)[0x4147a9]
> rsyslogd(cnfDoObj+0x90)[0x414ba0]
> rsyslogd(yyparse+0xbae)[0x45435e]
> rsyslogd(load+0xc35)[0x414145]
> rsyslogd(initAll+0x5ef)[0x448e2f]
> rsyslogd(main+0x30)[0x40dfe0]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f28697d9ec5]
> rsyslogd[0x40e35a]
> ```
>
> I don't have time to dig into it today but wanted to go ahead and report
> it.  If I correctly use an absolute path to the file (I used a relative by
> mistake when testing and found this), things work as expected.
>
> If I get some time tomorrow to dig into it I will!
>
> Cheers,
> Brian
>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] imfile buffer overflow master-candidate

[Brian Knox]

A little more info:

Program received signal SIGABRT, Aborted.
0x769efcc9 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace
#0  0x769efcc9 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x769f30d8 in __GI_abort () at abort.c:89
#2  0x76a2c394 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x76b3852b "*** %s ***: %s terminated\n") at
../sysdeps/posix/libc_fatal.c:175
#3  0x76ac3c9c in __GI___fortify_fail (msg=,
msg@entry=0x76b384c2 "buffer overflow detected") at
fortify_fail.c:37
#4  0x76ac2b60 in __GI___chk_fail () at chk_fail.c:28
#5  0x763a02cd in memcpy (__len=18446744073709551615,
__src=, __dest=0x7fffd040)
at /usr/include/x86_64-linux-gnu/bits/string3.h:51
#6  checkInstance (inst=0x6b0210) at imfile.c:727
#7  0x763a054d in newInpInst (lst=) at imfile.c:1066
#8  0x004147a9 in inputProcessCnf (o=o@entry=0x6adc60) at
rsconf.c:354
#9  0x00414ba0 in cnfDoObj (o=0x6adc60) at rsconf.c:427
#10 0x0045435e in yyparse () at grammar.y:129
#11 0x00414145 in load (cnf=0x695cd0 , confFile=0x470309
"/etc/rsyslog.conf") at rsconf.c:1286
#12 0x00448e2f in initAll (argc=argc@entry=1,
argv=argv@entry=0x7fffe688)
at rsyslogd.c:1252
#13 0x0040dfe0 in main (argc=1, argv=0x7fffe688) at
rsyslogd.c:1640
(gdb) frame 13
#13 0x0040dfe0 in main (argc=1, argv=0x7fffe688) at
rsyslogd.c:1640
1640initAll(argc, argv);
(gdb) print argc
$1 = 1
(gdb) print argv
$2 = (char **) 0x7fffe688


On Thu, Mar 3, 2016 at 8:53 AM, Brian Knox <bk...@digitalocean.com> wrote:

> I've found a buffer overflow in imfile in the master-candidate branch.  To
> reproduce, make an imfile config that uses a relative path rather than
> absolute to a file:
>
> ```
> module(load="imfile" PollingInterval="10")
>
> input(
> type="imfile"
> tag="crash"
> File="crashme"
> )
>
> *.* /var/log/syslog
> ```
>
> This results in:
>
> ```
> 3146.392981790:main thread: deletestateonfiledelete: (unset)
> 3146.392987727:main thread: addmetadata: (unset)
> 3146.392993638:main thread: addceetag: (unset)
> 3146.392999527:main thread: statefile: (unset)
> *** buffer overflow detected ***: rsyslogd terminated
> === Backtrace: =
> /lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7f286982b38f]
> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f28698c2c9c]
> /lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7f28698c1b60]
> /usr/local/lib/rsyslog/imfile.so(+0x22cd)[0x7f286919f2cd]
> /usr/local/lib/rsyslog/imfile.so(+0x254d)[0x7f286919f54d]
> rsyslogd(inputProcessCnf+0x99)[0x4147a9]
> rsyslogd(cnfDoObj+0x90)[0x414ba0]
> rsyslogd(yyparse+0xbae)[0x45435e]
> rsyslogd(load+0xc35)[0x414145]
> rsyslogd(initAll+0x5ef)[0x448e2f]
> rsyslogd(main+0x30)[0x40dfe0]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f28697d9ec5]
> rsyslogd[0x40e35a]
> ```
>
> I don't have time to dig into it today but wanted to go ahead and report
> it.  If I correctly use an absolute path to the file (I used a relative by
> mistake when testing and found this), things work as expected.
>
> If I get some time tomorrow to dig into it I will!
>
> Cheers,
> Brian
>
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] imfile buffer overflow master-candidate

[Brian Knox]

I've found a buffer overflow in imfile in the master-candidate branch.  To
reproduce, make an imfile config that uses a relative path rather than
absolute to a file:

```
module(load="imfile" PollingInterval="10")

input(
type="imfile"
tag="crash"
File="crashme"
)

*.* /var/log/syslog
```

This results in:

```
3146.392981790:main thread: deletestateonfiledelete: (unset)
3146.392987727:main thread: addmetadata: (unset)
3146.392993638:main thread: addceetag: (unset)
3146.392999527:main thread: statefile: (unset)
*** buffer overflow detected ***: rsyslogd terminated
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7f286982b38f]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f28698c2c9c]
/lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7f28698c1b60]
/usr/local/lib/rsyslog/imfile.so(+0x22cd)[0x7f286919f2cd]
/usr/local/lib/rsyslog/imfile.so(+0x254d)[0x7f286919f54d]
rsyslogd(inputProcessCnf+0x99)[0x4147a9]
rsyslogd(cnfDoObj+0x90)[0x414ba0]
rsyslogd(yyparse+0xbae)[0x45435e]
rsyslogd(load+0xc35)[0x414145]
rsyslogd(initAll+0x5ef)[0x448e2f]
rsyslogd(main+0x30)[0x40dfe0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f28697d9ec5]
rsyslogd[0x40e35a]
```

I don't have time to dig into it today but wanted to go ahead and report
it.  If I correctly use an absolute path to the file (I used a relative by
mistake when testing and found this), things work as expected.

If I get some time tomorrow to dig into it I will!

Cheers,
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] feedback request on format-changing bugfix

[Brian Knox]

None from me.
On Feb 18, 2016 3:28 AM, "Rainer Gerhards"  wrote:

> Hi all,
>
> I received a bug fix for impstats that changes the json-based formats
> slighly. Please see:
>
> https://github.com/rsyslog/rsyslog/pull/798
>
> Any concerns?
>
> Thanks,
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Transaction support for omkafka

[Brian Knox]

Hello Ralph! I'm the initial author of the omhiredis output. I haven't
looked at the omkafka source code yet so I can't speak to it's complexity -
but I can say that writing the transaction support for omhiredis was fairly
simple.  It was just implementing the beginTransaction / endTransaction
macro blocks in addition to the doAction block.

If knowing at the rsyslog level what is going on with your queues is
important, I'd lean towards taking a crack at transaction support in
omkafka.

I'm currently using omelasticsearch heavily - and being able to look in my
imstats logs to see what's happening with my queues and actions definitely
makes it simple to tell what is going on at a glance.

Just my thoughts!
Cheers,
Brian

On Fri, Jan 15, 2016 at 12:31 AM, Ralph Caraveo  wrote:

> Hello,
>
> (I apologize if this got sent twice but I may have sent it too soon before
> the mailing list registration process was completed.)
>
> I'm hoping the group can provide some guidance around a requirement we
> have to have transactional support around having an Rsyslog OM module that
> writes to Kafka.
>
> What we'd like to do, is leverage consuming data from Rsyslog and posting
> to Kafka however it looks like the OMKafka module doesn't currently support
> transactions when posting to Kafka in the event of an error.  So, if we
> write to Kafka, and an error occurs, it looks like we lose the log-line for
> that particular Action item unless we write it to a fallback log file.
>
> Additionally, we've looked at writing our own Kafka Producer using the
> OMProg style where we consume off of STDIN and then connect to kafka and
> produce data.  Unfortunately this approach also doesn't allow us to
> communicate back to Rsyslog that a failure has happened in the event of an
> error.
>
> We tried to deal with this by writing to a fallback text file, and this
> works great when there are errors with Kafka, but if the process dies
> between receiving from STDIN and before writing to Kafka, we can still
> potentially lose messages.
>
> Additionally, I noticed that the OMHIREDIS client does support
> transactions, so it sounds like we want the design of OMHIREDIS (where it
> utilizes transactions) but with that support in OMKafka.
>
> I'm just looking for a recommendation on a way forward from the group.  If
> it makes more sense to enhance OMKafka to have transaction support or if we
> try to go down the path of adding some type of transaction support to the
> Omprog module which would allow us to continue using our custom Kafka
> producer of which is written in Go.
>
> Thanks for anyone's time around this!
>
> Ralph
> @deckarep
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] elasticsearch 2.0 and field names

[Brian Knox]

As a short term solution I'm working on a small service (in golang) that
accepts logs over tcp, can replace characters in JSON field names in a @cee
syslog line, and then forward the line to another syslog destination.  In
tests on my laptop it handles modifying ~ 50,000 reasonably sized log lines
a second per connection.  It gracefully handles tcp connection issues and
I'll test it under adverse circumstances to make sure it's reasonably
robust.  I personally find this preferable to deploying logstash just to
substitute one character.  I'll release it open source this week in case
any one else needs an immediate solution to this problem like I do.

It's less than ideal - ideally elasticsearch would support JSON rather than
a subset of characters JSON allows - but it solves the immediate problem
for us.

Cheers,
Brian



On Sun, Dec 6, 2015 at 2:51 PM, David Lang  wrote:

> On Sat, 5 Dec 2015, Peter Portante wrote:
>
> On Sat, Dec 5, 2015 at 5:03 PM, David Lang  wrote:
>>
>> we really need mmscrubnames or similar
>>>
>>> 1. change all names to lower case
>>> 2. replace characters that rsyslog doesn't allow in names with something
>>> 3. allow other characters to be added to the list to be replaced
>>> 4. change names that are foo!bar into multi-layer structures
>>> 5. handle the case where these changes create nultiple objects with the
>>> same name (probably by appending a string until there are no longer
>>> conflicts)
>>>
>>> #1 may be able to go away in a decade or so if we allow case sensitive
>>> names as an option
>>>
>>>
>> Don't we need to make this go away sooner than later?  If rsyslog is the
>> link in the chain that prevents someone from getting the key names they
>> expect into ES, won't they find something else to replace that link?
>>
>> I have made available RPMs for EPEL 7 (which should work on RHEL 7 and
>> CentOS 7)P, and Fedora 21, 22, and 23.  Why not make the effort to find
>> out
>> what breaks, and put in a switch so that folks can opt-in to
>> case-sensitive
>> names in config files?  I'd be happy to implement the switch, but would
>> need help verifying existing configurations work.
>>
>
> this will break some existing configs, won't it? If someone has something
> that's assuming everything is squished to lower case, and it becomes case
> sensitive, won't that break?
>
> We can add the new case sensitivity as an option quickly, but can't make
> it the default for quite a while (a cycle or two of the enterprise distros)
>
> #2 needs to be done on the actual variable names, not just on the ES
>>> output so that the variables can be accessed and manipulated in rsyslog
>>>
>>>
>> Why do we need to do this?  Is this because we need to reference them in
>> the configuration files?  If so, why not provide an escape syntax for the
>> configuration file?
>>
>> Do we really want rsyslog in the position where it adds restrictions to
>> the
>> data handling pipeline because of how it operates?  I think we all agree
>> that an mmscrubnames module would be good to help put rsyslog in the
>> position of transforming data from one source to another in the overall
>> pipeline.
>>
>
> AFAIK, JSON imposes no limits of field names, so any strange character (or
> unicode character, or even control character) could be part of a field
> name. And even if the JSON spec imposes some limits, do the libraries
> impose such limits in practice?
>
> I don't think it makes sense to support all of this in rsyslog, I think
> it's reasonable to impose something sane. Other log handling software does
> this (for example, logstash doesn't allow '.' in the name, but also is case
> insensitive :-)
>
> and finally, #4 is needed to allow the work-around for problems like ES
>>> has.
>>>
>>>
>> I am not sure I follow why this allows us to work-around problems like ES
>> has.
>>
>> The dots in field names are confusing and ambiguous in ES because you can
>> reference a hierarchical set of objects in the json objects indexed.  So
>> if
>> one has a field name with dots in it in one document and another document
>> in the index has a hierarchy with sub objects, then it is ambiguous which
>> we are dealing with, if I understand the problem correctly.
>>
>
> Ok, that explains why this is an issue, it makes sense. We have the same
> problem with '!'. It's a problem in ES because it's a new requirement,
> breaking existing input.
>
> But #4 would let us say that '.' is an illegal character, along with
> control characters, anything above plain ASCII, and other punctuation
> characters we don't allow and get them replaced by something we do allow.
>
> Folks can stay with ES 1.7 if they need the dots in names.
>>
>
> not long term.
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This 

Re: [rsyslog] elasticsearch 2.0 and field names

[Brian Knox]

David - yes, that exactly describes the situation that I'm in. If I can't
find a short term solution with existing capabilities, I may look into
providing a load balanced pool of sanitization workers that I connect to
over the zeromq plugins I've been working on as a more near term solution.
Ideally, I'd like to be able to handle the sanitization within rsyslog
itself.

For a quick hack, a template on my output from my aggregators replacing "."
characters with "_" might work and I'll give that a spin.  I still have an
elasticsearch 1.5 cluster that is our production cluster in parallel with
the new 2.1 cluster, so I have some room to experiment.

As an aside - does anyone have a link to a config example using a regex
replace on a property using the new v8 template format?

Peter - I'd be very interested if you have an approach to this problem that
works with existing syslog capability.

Cheers,
Brian




On Fri, Dec 4, 2015 at 3:28 PM, Peter Portante <peter.a.porta...@gmail.com>
wrote:

> On Fri, Dec 4, 2015 at 3:00 PM, David Lang <da...@lang.hm> wrote:
>
> > On Fri, 4 Dec 2015, Peter Portante wrote:
> >
> > On Fri, Dec 4, 2015 at 12:40 PM, Brian Knox <bk...@digitalocean.com>
> >> wrote:
> >>
> >> In my case, I have "flat" ( 1 level deep ) CEE JSON logs with field
> names
> >>> that are dot delimited  (  @cee { "resp.duration_ms" : 1,
> >>> "resp.code" :
> >>> 200 }  ).
> >>>
> >>>
> >> So if you have a "flat" namespace where the fields include dots in them,
> >> then if you move to a hierarchical namespace then won't the field name
> >> references still work?
> >>
> >
> > the problem he's having is the the field names in his incoming logs are
> > not hierarchical. He's not hand-crafting the structure the way you are,
> > he's parsing incoming logs and then outputting $! to ES (or something
> > similar)
> >
> > As such, he's pretty much stuck with the names on the incoming data.
> >
>
> We are using rsyslog to normalize the data.  I'll post an example config
> file for what we are doing shortly (prolly on github).
>
> -peter
>
>
> >
> > Rsyslog hasn't had a requirement before now to change/sanitize the field
> > names, so there's nothing setup to do this.
> >
> > the work-around that I can think of basically involved re-parsing the
> > message after manipulating it.
> >
> > you could use omexternal to pass the json data to an external script that
> > can muck with the names and pass them back. unfortunantly this interface
> > can't delete fields, just alter or add them, so you would want to do
> > something along the lines of moving everything down a level so instead of
> > $!blah you have $!fixed!blah (or in json instead of { 'blah': 'value',
> > 'foo': 'value' } you would have { "fixed": { "blah": "value", "foo":
> > "value" } }
> >
> > another possibility would be to do something in rsyslog where you use a
> > template to replace all '.' with some other character, and then parse the
> > result with mmnormalize, but this is ugly as well.
> >
> > We've got a few cases where field names just don't work (case sensitivity
> > , () in field names, etc), so it may be a good idea for someone to write
> a
> > mm (message modification) module that goes through all the field names
> and
> > sanitizes them, with several options as to what to do (and especially
> what
> > to do if the sanitized version already exists, overwrite, try a different
> > name, ??)
> >
> > David Lang
> >
> >
> >
> >> GIven my lack of control over the incoming logs, I think the simplest
> >>> solution to this issue would be a way to change the attribute names
> >>> themselves  ( "resp_duration_ms", "resp_code" ).
> >>> Given that I don't know the total space of all possible keys, I'd like
> >>> this
> >>> to work with the $!all-json property.
> >>>
> >>> If there's not already a way to do this that I'm missing, I think given
> >>> the
> >>> change in elasticsearch and that the suggested solution to this problem
> >>> is
> >>> "use logstash", I'd like to look at the possibility of adding a
> property
> >>> formatter that could handle this sanitization.
> >>>
> >>>
> >>> On Fri, Dec 4, 2015 at 11:37 AM, Peter Portante <
> >>> peter.a.porta...@gmail.com>
> 

[rsyslog] elasticsearch 2.0 and field names

[Brian Knox]

I found out today that elasticsearch 2.x does not allow field names to have
the period character in them.  This is making my life interesting as I use
elasticsearch with rsyslog end to end (no logstash), and a lot of our field
names have "." as a delimiter in them.

In a perfect world, I'd like an "elasticsearch" property formatter that
could look for and replace "." in field names with "_", that would also
work with the all-json property, something like:

property(name="$!all-json" format="elasticsearch")

Or, if this is to ES specific for rsyslog core, perhaps we could add this
functionality to the omelasticsearch output itself (I'll look over the code
today).

I'd like to not have to introduce logstash to my environment just to regex
a character in field names.  I'm open to other ideas as well, just wanted
to start the conversation.

Cheers,
BRian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.14.0 (v8-stable) released

[Brian Knox]

Thanks for the heads up Florian!

On Tue, Nov 3, 2015 at 11:16 AM, Florian Riedl  wrote:

> Hi all,
>
> We have released rsyslog 8.14.0.
>
> This is primarily a bug-fixing release with a couple of fixes for imfile
> and Rainerscript. Also the property engine has now a new property:
> rawmsg-after-pri.
> For more details, please take a look at the Changelog.
>
> ChangeLog:
>
> http://www.rsyslog.com/changelog-for-8-14-0-v8-stable/
>
> Download:
>
> http://www.rsyslog.com/downloads/download-v8-stable/
>
> As always, feedback is appreciated.
>
> Best regards,
> Florian Riedl
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] RFC: dynamic-stats support

[Brian Knox]

I personally would argue that stats around the actual content of syslog
messages is outside of the domain that rsyslog should be responsible for.

impstats makes sense to me as it provides statistics around rsyslogs
operation itself.  Once I start wanting stats and counters around message
content, I would rather delegate that to a different system entirely.

On Tue, Oct 6, 2015 at 12:04 PM, singh.janmejay 
wrote:

> Rainer,
>
> I see this as something completely outside the scope of  variables.
> Building stats collector over variables is possible, but then we are
> then talking about a general purpose language which allows building
> such complex things. This increases the scope of Rainerscript and with
> larger scope comes complexity. I feel this is in-line with the other
> Lua discussion where you emphasized that Rainerscript should not
> become a fully-general-purpose language?
>
> Eg. creating an atomic-increment function for variable requires that
> we educate users about what can and can't be done if atomic-increment
> function is used anywhere on a variable. What relationship they can
> expect it to have with other atomic-incrementing variables (which gets
> into memory model).
>
>
>
> On Tue, Oct 6, 2015 at 8:49 PM, Rainer Gerhards
>  wrote:
> > I can't fully dig into this, but I think we must *very carefully*
> > evaluate the overall design. Some time ago we introduced arrays for
> > the limited liblognorm use case, and it hurts us every now and then
> > when folks want to use arrays for other use cases. It may probably
> > make sense to re-think how the variable engine etc behaves before
> > adding more functionality. And make sure that everything works smooth
> > in all use cases. While anything else may take care for some use
> > cases, I fear we may get too fragmented. At least this is what I
> > learned in the past months discussions.
> >
> > Anyone else?
> >
> > Rainer
> >
> > 2015-10-06 17:10 GMT+02:00 singh.janmejay :
> >> It is possible to use global-variables (it'll require some
> >> enhancements, table-support etc), but it'll be very inefficient
> >> compared to this approach. For instance, choice of data-structure etc
> >> allows making the solution a lot more efficient.
> >>
> >> Here its possible to locklessly increment counters in most cases, so
> >> its overhead is a lot lesser than global-variables.
> >>
> >> Recycle is precisely to allow this lockless mechanism to work. Its
> >> basically saying, it'll track metric-names he has seen in last 1 hour.
> >> If we kill tracking of it as soon as we don't see an increment
> >> (between 2 reporting runs of impstats), it'll lead to unnecessary
> >> churn when low-values are common or load is not uniform in time.
> >>
> >> Implementing it on top of global-variables is not only has very high
> >> performance-penalty(it'll be prohibitive for high-throughput
> >> scenarios), it also exposes too much complexity to the user (where
> >> user has to worry about reset etc).
> >>
> >> I don't plan to have a scheduler in this implementation.
> >> GetAllStatsLines call will purge the tree instead of reset at that
> >> interval. Its basically a balance between freeing-up memory occupied
> >> by stale-metric-names vs. performance (lockless handling of
> >> increment). So it will be governed by impstat schedule. May be I
> >> should change name to better name (equivalent of
> >> purge_known_keys_after_they_have_been_reported_N_times).
> >>
> >>
> >> On Tue, Oct 6, 2015 at 4:30 PM, David Lang  wrote:
> >>> On Tue, 6 Oct 2015, singh.janmejay wrote:
> >>>
>  Hi,
> 
>  I am working on support for stats with dynamic-name. This comes handy
>  in situations where metric-name is dependent upon value of a certain
>  attribute of the message.
> 
>  Say, for a central log-aggregation service, its valuable to know what
>  is inbound message-count distribution across application-clusters that
>  send logs to it, or for a shared-server, its valuable to know what is
>  the log-volume generation across users etc.
> 
>  Im thinking of using functions-like interface to support this. It may
>  look similar to this:
> 
>  
>  dyn_stats("user_msg_count")
> 
>  ...
> 
>  ruleset(...) {
>  ...
>  dyn_inc("user_msg_count", $.user)
>  ...
>  }
>  
> 
>  dyn_stats signature looks like:
>  dyn_stats(, ,   default=10k>, )
> 
>  dyn_inc signature looks like:
>  dyn_inc(, )
> 
> 
>  Reporting would work similar to static-metric via impstats. Mapping:
>  statsobj_s.name = name_space
>  statsobj_s.origin = "dyn"
>  ctr_s.name = "foo" (say $.user had value foo)
> 
> 
>  Thoughts / suggestions?
> >>>
> >>>
> >>> how is this different/better than global variables? (although we may
> need to
> 

Re: [rsyslog] Log Shipping Format Poll Results + Docker Logging Webinar

[Brian Knox]

Hi Otis!  Nice work on gathering the data!  I tried to attend the webinar
but the video conferencing software didn't work with my microphone or
camera on linux :/

Brian

On Wed, Sep 30, 2015 at 10:25 AM, Otis Gospodnetić <
otis.gospodne...@gmail.com> wrote:

> Hi,
>
> Thanks to anyone who voted!
>
> The results are in, if you're curious:
> http://blog.sematext.com/2015/09/30/log-shipping-poll-results/
>
> Somewhat related, if anyone's interested, this is happening today:
>
> http://blog.sematext.com/2015/08/11/docker-logging-webinar/
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Sep 21, 2015 at 5:04 PM, Otis Gospodnetić <
> otis.gospodne...@gmail.com> wrote:
>
> > Hi,
> >
> > Some of us here at Sematext debated the adoption of RFC 5424.  So instead
> > of guessing, we thought we'd conduct a 1-question poll :)
> >
> > http://blog.sematext.com/2015/09/21/poll-how-do-you-ship-your-logs/
> >
> > Oh, and try to guess before peeking at the results! :)
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.13.0 (v8-stable) released

[Brian Knox]

Yay!  Thanks to everyone involved!

On Tue, Sep 22, 2015 at 11:58 AM, Florian Riedl  wrote:

> Hi all,
>
> We have released rsyslog 8.13.0.
>
> This release sports a big number of changes. While most are bugfixes, there
> are also some additions to existing functionality, most notably the
> enhancements for ZeroMQ and Redis modules.
> For more details, please take a look at the Changelog.
>
> ChangeLog:
>
> http://www.rsyslog.com/changelog-for-8-13-0-v8-stable/
>
> Download:
>
> http://www.rsyslog.com/downloads/download-v8-stable/
>
> As always, feedback is appreciated.
>
> Best regards,
> Florian Riedl
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] last minute omhiredis fix

[Brian Knox]

Of course I found a bug in omhiredis after the merge - here's a fix!

https://github.com/rsyslog/rsyslog/pull/531
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

[Brian Knox]

Limiting the number of batches within a given interval is conceptually
similar to the index refresh rate setting in elasticsearch itself.  At
first blush I like this idea, as it is simple to understand the impact on
when a given log line will be available within elasticsearch if I know this
interval and know my index refresh interval.



On Sat, Aug 29, 2015 at 12:21 PM, David Lang da...@lang.hm wrote:

 On Mon, 24 Aug 2015, Rainer Gerhards wrote:

 I am mostly with Radu on this topic. I think there are some use cases
 where it really would be advantageous to submit a larger batch, even
 if this means waiting. True, these use cases were very seldom in the
 early days of rsyslog and may still be, but I think it's something one
 might validly want.


 The thought hit me that we are loking at this wrong.

 The problem is overloading the receiver with too many small batches.

 rather than trying to define batch size, isn't what we really want to have
 is a limit on how many batches we send in a given timeframe? possibly with
 a 'escape clause' that says tht if we are sending maxbatch size messages
 for the entire timeframe we do something (spawn a new sending thread,
 temporarily allow higher sending rates, or just let the backlog accumulate
 are all valid choices under different conditions)

 thoughts?

 David Lang

 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

[Brian Knox]

I'll see if I can find spare capacity to set up a test cluster.  We use
omelasticsearch heavily.
On Aug 21, 2015 7:44 AM, Radu Gheorghe radu.gheor...@sematext.com wrote:

 On Fri, Aug 21, 2015 at 1:22 PM, Rainer Gerhards rgerha...@hq.adiscon.com
 
 wrote:

  2015-08-21 12:19 GMT+02:00 Otis Gospodnetić otis.gospodne...@gmail.com
 :
   Hi,
  
   This sounds like something that should be om-specific.  What Radu is
   suggesting would definitely help with ES, but may not be relevant for
  other
   output targets.
   What I think is overlooked here is the ES side - more specifically ES
 and
   searches that ES has to handle.  If we don't care about maxing out ES
 and
   just pushing data in it as fast as it arrives, then how
   rsyslog/omelasticsearch works today  makes sense.  But this approach if
   focused on ingestion and ignores how this can hurt ES's ability to
 handle
   queries in a timely manner.  Exposing controls Radu suggested would
 help
   people avoid this problem.  I know David would like to see numbers :)
 I
   love numbers, too, but I'm not sure if we'll have the time to provide
  them
   :(  That said, we work with ES 24/7 and have been doing that for years
   (many hundreds of ES deployments under our belt by now), so I am hoping
   somebody will trust us this option would be great to have in
   omelasticsearch. :)
 
  Not reading the full thread, I, too, think this makes sense. It would
  need to go into the queue engine, as this is the only place where it
  can decently be done. Done properly, it should not hurt performance
  for other cases. But it needs careful implementation. I suggest to
  open a gitub issue tracker, so that I can remember when I have time
  later this year (probably November+).
 

 Thanks, Rainer! I've opened an issue here:
 https://github.com/rsyslog/rsyslog/issues/495

 Best regards,
 Radu
 --
 Performance Monitoring * Log Analytics * Search Analytics
 Solr  Elasticsearch Support * http://sematext.com/
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Filter on if json property exists

[Brian Knox]

Thanks David - that looks reasonable.  I'll give it a try!

On Thu, Aug 13, 2015 at 4:54 PM, David Lang da...@lang.hm wrote:

 On Thu, 13 Aug 2015, Brian Knox wrote:

 Hello!  I was working on an rsyslog configuration today and realized I've
 never run into a situation where I wanted to filter based on whether or
 not
 a property derived from a call to mmjsonparse exists or not.  E.g., some
 logs will have myprop in their json, I want to route all logs that do
 one
 way, and all logs that do not another.


 We don't have an exists() function (and it would be useful), but I've
 found that I can usually get away with $!var != ''.

 David Lang
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Filter on if json property exists

[Brian Knox]

Hello!  I was working on an rsyslog configuration today and realized I've
never run into a situation where I wanted to filter based on whether or not
a property derived from a call to mmjsonparse exists or not.  E.g., some
logs will have myprop in their json, I want to route all logs that do one
way, and all logs that do not another.

Cheers,
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Load balancing issue

[Brian Knox]

From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval.  I believe
this has been discussed on the mailing list:

http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html

Unless something has changed, you need to use the omfwd module if you want
to use tcp rebinding.  This isn't a bug - this is documented behavior.  The
rebind interval parameter is documented as a parameter for omfwd.  RELP
uses omrelp, which has no such paramater.  See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html

I use ActionTCPRebindInterval with haproxy with plain TCP.  It works very
well.

Cheers,
Brian

On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:

 With the architecture enclosed.

 - Mail original -
 De: smain...@free.fr
 À: rsyslog-users rsyslog@lists.adiscon.com
 Envoyé: Jeudi 23 Juillet 2015 11:59:35
 Objet: [rsyslog] Load balancing issue

 Hello all,

 I'm currently trying to load balance the log traffic accross several
 servers.
 I thought my configuration with ActionSendTCPRebindInterval option was
 working properly, unfortunately my recent benchs show that the log flow is
 not well load balanced.

 Please find below a part of the architecture :




 My problem is located on the log aggregators : the rsyslog send its
 traffic to haproxy on localhost using relp protocol.
 I monitored the tcp sessions and i can see that haproxy doesn't change the
 destination servers.

 watch 'ss -lap -o state established \( dport = :20514 \)'
 Recv-Q Send-Q Local Address:Port Peer Address:Port
 1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
 0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
 users:((haproxy,3922
 ,2))
 1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
 0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))

 Please find enclosed my configuration.

 NB :
 - the source pid (rsyslog) never change as it is expected with
 ActionSendTCPRebindInterval
 - i mixed legacy and new syntaxe because of the following bug
 https://github.com/rsyslog/rsyslog/issues/96
 This bug is annoying and i didn't receive any update since about 4 months

 Could you please help me ?

 OS : debian7
 rsyslog version : 8.10

 Regards ,
 Smana


 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Load balancing issue

[Brian Knox]

Aha! David - to summarize, is the problem then that:

a) the parameter did not exist previously, and
b) was only added for the new style configs?

Brian

On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:

 On Thu, 23 Jul 2015, Brian Knox wrote:

  From your diagram, it looks like you are trying to load balance RELP. As
 far as I know, RELP does not suppot ActionTCPRebindInterval.  I believe
 this has been discussed on the mailing list:

 http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html

 Unless something has changed, you need to use the omfwd module if you want
 to use tcp rebinding.  This isn't a bug - this is documented behavior.
 The
 rebind interval parameter is documented as a parameter for omfwd.  RELP
 uses omrelp, which has no such paramater.  See:
 http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
 http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html


 with the new style config it does in the current git branch. It looks like
 it was added in 7.3.15

 /* tables for interfacing with the v6 config system */
 /* action (instance) parameters */
 static struct cnfparamdescr actpdescr[] = {
 { target, eCmdHdlrGetWord, 1 },
 { tls, eCmdHdlrBinary, 0 },
 { tls.compression, eCmdHdlrBinary, 0 },
 { tls.prioritystring, eCmdHdlrString, 0 },
 { tls.cacert, eCmdHdlrString, 0 },
 { tls.mycert, eCmdHdlrString, 0 },
 { tls.myprivkey, eCmdHdlrString, 0 },
 { tls.authmode, eCmdHdlrString, 0 },
 { tls.permittedpeer, eCmdHdlrArray, 0 },
 { port, eCmdHdlrGetWord, 0 },
 { rebindinterval, eCmdHdlrInt, 0 },
 { windowsize, eCmdHdlrInt, 0 },
 { timeout, eCmdHdlrInt, 0 },
 { localclientip, eCmdHdlrGetWord, 0 },
 { template, eCmdHdlrGetWord, 0 }
 };




  I use ActionTCPRebindInterval with haproxy with plain TCP.  It works very
 well.

 Cheers,
 Brian

 On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:

  With the architecture enclosed.

 - Mail original -
 De: smain...@free.fr
 À: rsyslog-users rsyslog@lists.adiscon.com
 Envoyé: Jeudi 23 Juillet 2015 11:59:35
 Objet: [rsyslog] Load balancing issue

 Hello all,

 I'm currently trying to load balance the log traffic accross several
 servers.
 I thought my configuration with ActionSendTCPRebindInterval option was
 working properly, unfortunately my recent benchs show that the log flow
 is
 not well load balanced.

 Please find below a part of the architecture :




 My problem is located on the log aggregators : the rsyslog send its
 traffic to haproxy on localhost using relp protocol.
 I monitored the tcp sessions and i can see that haproxy doesn't change
 the
 destination servers.

 watch 'ss -lap -o state established \( dport = :20514 \)'
 Recv-Q Send-Q Local Address:Port Peer Address:Port
 1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
 0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
 users:((haproxy,3922
 ,2))
 1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
 0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))

 Please find enclosed my configuration.

 NB :
 - the source pid (rsyslog) never change as it is expected with
 ActionSendTCPRebindInterval
 - i mixed legacy and new syntaxe because of the following bug
 https://github.com/rsyslog/rsyslog/issues/96
 This bug is annoying and i didn't receive any update since about 4 months

 Could you please help me ?

 OS : debian7
 rsyslog version : 8.10

 Regards ,
 Smana


 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

  ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.


 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com

[rsyslog] field extraction with v8 format syntax

[Brian Knox]

I wish to extract the second part of a hostname as a property.  The field
replacer ( where you set fromChar to F and toChar as the number of the
field you wish to extract) looks to fit the bill for what I want.

I realized I don't know how to specify fromChar / toChar in v8 format in a
list template.  Is this possible?  Or do I need to use a string template?

Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] field extraction with v8 format syntax

[Brian Knox]

Aha found it -

   - field.number - obtain this field match
   - field.delimiter - decimal value of delimiter character for field
   extraction

Thanks anyway!

Brian

On Tue, Jul 7, 2015 at 11:29 AM, Brian Knox bk...@digitalocean.com wrote:

 I wish to extract the second part of a hostname as a property.  The field
 replacer ( where you set fromChar to F and toChar as the number of the
 field you wish to extract) looks to fit the bill for what I want.

 I realized I don't know how to specify fromChar / toChar in v8 format in a
 list template.  Is this possible?  Or do I need to use a string template?

 Brian


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] build system update

[Brian Knox]

Thanks for the update!  Guess it's time to work on some packages :)

On Tue, Jun 23, 2015 at 4:46 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Hi all,

 just a quick update: we have successfully migrated to the new build
 system and are ready for team contributions.

 I plan to do a little bit of restructuring of the system I imported,
 but that's an ongoing project the next time in any case. I'll probably
 switch to the new system later today (the scripts are no longer
 compatible with the old one).

 As we currently do not have a web server on the new system, this will
 mean that daily tarballs will temporarily be unavailable. They will
 come back online when the necessary infrastructure is in place.

 I keep you posted on the progress.

 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] templates with overrides

[Brian Knox]

On the subject of templates - I have cases where being able to use
different templates on the same output action would be advantageous.  If an
override variable as proposed here would accomplish that, it would solve a
problem for me.

It's something I work around in various ways now so it's not critical - it
would just be a nice to have.

On Thu, Jun 18, 2015 at 4:47 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2015-06-18 8:31 GMT+02:00 David Lang da...@lang.hm:
  There are so many cases where someone needs to override something in the
  default template (bad hostname, bad programname, etc) that having to
 create
  a custom tempalte for each thing gets to be a hassle.
 
  I'd like to propose that we have some templates that are exactly the
 same as
  the default ones, but that look (for example) for $.hostname and if it's
  set, use that instead of $hostname. Have this setup for everything in the
  default template set.
 
  I don't know if we dare to do this to the existing templates, or if these
  would need to be new templates. If new, could they be built-in, or should
  they be string modules that are explicitly loaded (and if string modules,
  could they override the default definitions??)

 This doesn't sound too trivial.

 
  I also think that we need to have a couple of JSON based templates:
 
  file and forwarding versions of the base templates that use $! for the
  message body instead of $msg (do we need a version with '@cee:? or can we
  modify mmjsonparse to have an option to no longer require that now that
 cee
  is dead?)

 Either way is possible, and adding such a thing *is* trivial.

 
 
  and finally a template that is just $!\n to use when forwarding to
 other
  tools (nxlog, logstash, etc)

 do we really need this? As of my understanding, they thros the \n away
 in json parsing. Or do you mean as part of TCP syslog framing?

 Rainer
 
  thoughts?
 
  David Lang
 
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of
  sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
  LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] templates with overrides

[Brian Knox]

Yeah - I think a performance decrease would have to be incurred only when
using this feature.  There would need to be a way to still use static
templates as well.  I know I'd personally whether work around things the
way I do now than take a performance hit across all templates.

On Thu, Jun 18, 2015 at 11:05 AM, Tomas Heinrich thein...@redhat.com
wrote:

 TL;DR: Extending the current output-formatting capabilities would be
 great, but, to me, the proposal seems half-hearted. I'd prefer a more
 powerful solution.

 On 06/18/15 08:31, David Lang wrote:

 There are so many cases where someone needs to override something in the
 default template (bad hostname, bad programname, etc) that having to
 create a custom tempalte for each thing gets to be a hassle.


 Agreed; been there and it's not just a hassle, it doesn't work very well.

  I'd like to propose that we have some templates that are exactly the

 ...

 I don't think extending templates this way is the right approach.
 It feels like a kludge, using the instruments at hand, to get this use
 case working. It would be a code that has to be maintained and I'd hope
 that if something in this area gets implemented, it would be a more
 flexible solution.

 A less flexible solution would be to allow overriding properties of
 messages when some condition is met. But that doesn't solve all the needs
 and some property values are interdependent.


 I've recently dealt with a scenario where an omfile template had to be
 tweaked depending on the message content. This doesn't work well for omfile
 because you get two actions with different templates writing into the same
 file.

 So I think there are actually two problems (or limitations):
 * the first is that templates are a static property (can't switch between
 templates dynamically)
 * the second is that they're a property of the output action ONLY instead
 of a message +  output action / output module / ... combinations. (Here I'm
 treading on thin ice...)


 With static templates, what changes are the values of the referenced
 properties - which can't be overridden, AFAIK. If you want to change the
 static bits or use different properties, you have to duplicate the action
 with a different template.

 It would feel natural to have an action that switches/assigns a template
 to a particular message on a particular path. You can construct arbitrarily
 complex conditions in rainerscript for calling these actions.


 The downside is that all this would probably decrease the performance
 quite a bit.

 Tomas


 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] templates with overrides

[Brian Knox]

Interesting.  I didn't know about this technique with exec_template.

On Thu, Jun 18, 2015 at 1:33 PM, David Lang da...@lang.hm wrote:

 On Thu, 18 Jun 2015, Rainer Gerhards wrote:

  2015-06-18 8:31 GMT+02:00 David Lang da...@lang.hm:

 There are so many cases where someone needs to override something in the
 default template (bad hostname, bad programname, etc) that having to
 create
 a custom tempalte for each thing gets to be a hassle.

 I'd like to propose that we have some templates that are exactly the
 same as
 the default ones, but that look (for example) for $.hostname and if it's
 set, use that instead of $hostname. Have this setup for everything in the
 default template set.

 I don't know if we dare to do this to the existing templates, or if these
 would need to be new templates. If new, could they be built-in, or should
 they be string modules that are explicitly loaded (and if string modules,
 could they override the default definitions??)


 This doesn't sound too trivial.


 another approach would be to allow rainerscript to set the variables that
 are currently fixed.

 i.e.

 set $hostname = foo;


 David Lang
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] packaging project

[Brian Knox]

Rainer -

Thanks for the update!  Enjoy your family time!

Brian

On Sun, Jun 14, 2015 at 8:14 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 I have setup a new github repro:

 https://github.com/rsyslog/rsyslog-infrastructure

 As it's readme says, it shall cover all the task that are not specific
 to the packaging project but are used across several infrastructure
 projects (most notable doc generation, packaging, testbench). If it
 turns out this is goving overboard, we can always merge projects
 later.

 I have begun to write some doc (not great markdown as I just notice,
 be sure to use RAW view):


 https://raw.githubusercontent.com/rsyslog/rsyslog-infrastructure/master/doc/README.md

 It contains some information on groups and users used. I think it's
 not overly critical, but please let me know if there are security
 concerns. The current names are preliminary and we can change them if
 desired. However, if we do we should also think about how we document
 that so that it is most easy to setup the environment.

 Note that I am currently working on the initial environment setup.
 Will probably not finish it today (family time now!). I would suggest
 not to make any changes until I have the basic setup done.

 All existing team members (Brian, David, Andre, Florian) should have
 received access information for the machine. If not, let me know. Also
 let me know if you are interested in participating.

 Thanks,
 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages

[Brian Knox]

Rainer - truth be told the best datacenter we currently have is Frankfurt
:)  It just opened recently, and the hypervisors in Germany are on the new
(v2) version of our hardware platform, have 40gbit nics, etc.

If you wanted to do it in NYC instead, I'd suggest NYC3.

Brian



On Fri, Jun 12, 2015 at 8:25 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2015-06-11 23:06 GMT+02:00 Brian Knox bk...@digitalocean.com:
  Florian - plan will probably depend on what we decide we're deploying.
  Andre has set up an account already ( Rainer may have the details as well
  ).  We gave them access to the beta for the team account features - so it
  should already be set up as a team account.   We should make sure any
  infrastructure we set up is provisioned from that account since we (DO)
  have applied quite a bit of credit to it.
 
  If we're just working together on how we'd like to automate package
  building I'd suggest a small server to start with and if we grow out of
 it
  we can bump it up.. maybe the 2GB ram / 2 core / 40GB ssd option to
  prototype on?  Hard to know until we nail down exactly what we're trying
 to
  stand up.

 OK, I think I'll simply provision such a machine (so I finally get my
 hands on your interface as well ;)). Does the NYC datacenter sound
 good given our collaborator base?

 What do you think makes most sense from a file system structure point
 of view? I have something along this on my mind:

 /home/...
 pkgproject - packaging project user (also for cron jobs)
 rainer
 brian
 andre
 florian
 

 under pkgproject, I'd create subdirs for each git repository.

 Once done, I'd see that I can migrate my current daily build
 environment to that machine. That will probably raise questions, it's
 currently tied to the result of testbench runs. Maybe issues to
 discuss... Let's assume this works then we can begin to modify it in a
 way that suits us better.

 It would probably good to describe the environment I use. I could do a
 Hangout on that, but I am not sure if I have sufficient time the next
 days and enough advance notice timeframe to make it fully interactive.

 Comments?

 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] packaging project mission statement

[Brian Knox]

Excellent!  I'll start in on the omczmq and imczmq packages when I get a
little time!

Cheers,
Brian

On Fri, Jun 12, 2015 at 8:17 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Hi all,

 I have amended the README.md with a mission statement plus some more
 details:

 https://github.com/rsyslog/rsyslog-pkg-ubuntu/blob/master/README.md

 Please have a look and act accordingly.

 Thx,
 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages

[Brian Knox]

Florian - plan will probably depend on what we decide we're deploying.
Andre has set up an account already ( Rainer may have the details as well
).  We gave them access to the beta for the team account features - so it
should already be set up as a team account.   We should make sure any
infrastructure we set up is provisioned from that account since we (DO)
have applied quite a bit of credit to it.

If we're just working together on how we'd like to automate package
building I'd suggest a small server to start with and if we grow out of it
we can bump it up.. maybe the 2GB ram / 2 core / 40GB ssd option to
prototype on?  Hard to know until we nail down exactly what we're trying to
stand up.

Cheers!
Brian


On Thu, Jun 11, 2015 at 12:17 PM, Florian Riedl fri...@adiscon.com wrote:

 Hi,

 I am very happy, that this project already received some initial drive and
 I am looking forward to working with you all.

 +1 for the mailing list as communication device

 I guess we could establish some prefix for the subject, so emails regarding
 the packaging project can be easily distinguished from the regular stuff.
 Something like [rpp]?

 +1 for the Digital Ocean hosting

 I guess, using the cloud hosting solution by DO might be the best way to
 set this up, because it is probably the most accessible solution. No need
 to buy Hardware, no need to set up complicated access clutter to internal
 networks. Though, I am not sure where to start in that matter or what plan
 to choose from the available options. Brian, do you have a suggestion what
 could be a good starting point?

 Another question that needs to be discussed is the use of SSH certificates
 for the signing of packages. This is not ideally solved for the current
 packages. I guess there are two possibilities:
 1. All packages are built and signed with a common project key.
 2. All packages are built and signed with a individual key from each
 person.

 Option 1 has some charme, but it is impossible to determine who actually
 built the packages, whereas this is possible with option 2, but there every
 builder needs to have a key. This even applies to logins.

 Opinions and comments anyone?

 Florian

 2015-06-09 19:32 GMT+02:00 Brian Knox bk...@digitalocean.com:

  Rsyslog is very useful and appreciated :)
 
  On Tue, Jun 9, 2015 at 1:28 PM, Rainer Gerhards 
 rgerha...@hq.adiscon.com
  wrote:
 
   Sent from phone, thus brief.
   Am 09.06.2015 19:01 schrieb Brian Knox bk...@digitalocean.com:
   
Coordinating on the mailing list is fine with me.  My employer
(DigitalOcean) has given the rsyslog project a fairly sizeable free
   hosting
budget so I'd be remiss to not advocate for us ;).
  
   I just realize that I slipped something (the Digital Ocean sponsoring)
  that
   I wanted to announce more formally ... Well that will follow but let me
  say
   now that it is very useful and appreciated :)
  
   Rainer
If another option works
out to be better I certainly understand!
   
I totally agree that converging on an agreed upon problem statement
 is
   the
right place to start.
   
Cheers,
Brian
   
On Tue, Jun 9, 2015 at 12:53 PM, David Lang da...@lang.hm wrote:
   
 On Tue, 9 Jun 2015, Rainer Gerhards wrote:

  Hi all,

 chances are extremely well to get to better packaging projects. We
  had
 some discussions internally in Adiscon, and I was able to secure
 the
 help of Florian Riedl for getting this in the best possible shape.

 Our goal is to get

 - better packages
 - more timely support for new distro releases
 - support for a broader set of distros (e.g. Fedora, often
  requested)
 - more ability for the community to steer this previous
 all-Adiscon
 project

 The 0mq discussion that started this thread is a good example of
  what
 this means.

 With the help of more community involvment we can reach the goals.
  And
 in order to make it easier to contribute, we need to streamline
 the
 process of how we build, release, test, and announce packages.

 Thankfully, Brian has offered to become an active team member. It
 would be great if others would also join in.

 I currently think that the right path to success is to start with
 small but sufficiently large project part. So I would propose that
  we
 focus on Ubuntu initially, get that part organized, learn a couple
  of
 things and apply the gained experience later to a final project
  that
 covers other distros as well (as mentioned by darix, the use of
 OBS
 sounds very appealing to me).

 In order to get going, I would like to see some ideas float on:

 - how should we communicate?
  (rsyslog mailing list, dedicated mailing list, github issue
 trackers, IRC, ...)


 I'd say the rsyslog mailing list, failing that a dedicated mailing
   list.
 But I think that the issues we will be working through

Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages

[Brian Knox]

Coordinating on the mailing list is fine with me.  My employer
(DigitalOcean) has given the rsyslog project a fairly sizeable free hosting
budget so I'd be remiss to not advocate for us ;).  If another option works
out to be better I certainly understand!

I totally agree that converging on an agreed upon problem statement is the
right place to start.

Cheers,
Brian

On Tue, Jun 9, 2015 at 12:53 PM, David Lang da...@lang.hm wrote:

 On Tue, 9 Jun 2015, Rainer Gerhards wrote:

  Hi all,

 chances are extremely well to get to better packaging projects. We had
 some discussions internally in Adiscon, and I was able to secure the
 help of Florian Riedl for getting this in the best possible shape.

 Our goal is to get

 - better packages
 - more timely support for new distro releases
 - support for a broader set of distros (e.g. Fedora, often requested)
 - more ability for the community to steer this previous all-Adiscon
 project

 The 0mq discussion that started this thread is a good example of what
 this means.

 With the help of more community involvment we can reach the goals. And
 in order to make it easier to contribute, we need to streamline the
 process of how we build, release, test, and announce packages.

 Thankfully, Brian has offered to become an active team member. It
 would be great if others would also join in.

 I currently think that the right path to success is to start with
 small but sufficiently large project part. So I would propose that we
 focus on Ubuntu initially, get that part organized, learn a couple of
 things and apply the gained experience later to a final project that
 covers other distros as well (as mentioned by darix, the use of OBS
 sounds very appealing to me).

 In order to get going, I would like to see some ideas float on:

 - how should we communicate?
  (rsyslog mailing list, dedicated mailing list, github issue
 trackers, IRC, ...)


 I'd say the rsyslog mailing list, failing that a dedicated mailing list.
 But I think that the issues we will be working through are useful to people
 who need to roll their own version (to test something from git, or to
 enable specific features).

  - where do we track issues?
  (I  have a strong preferrence for the github issue trackers)
 - what about doc?
 - where should we place the build platform
  (cloud I would guess, could we use Digital Ocean sponsorship for this)?


 I don't know the details of using it, but the Suse Open Build Platform is
 already setup to support a whole bunch of target distros. How close does it
 come to covering everything we need?

 Can it be used for all the different uses we want from this (distro
 release builds, nightly builds, other)

 If we have to roll our own infrastructure, some sort of cloud system is
 right. Google donates time on their cloud system to opensource projects, I
 don't know if it would be enough or not. Sponsorship from whoever is good
 :-)

  - when do we start ;)


 Clarification of the problem statement and where we are starting from :-)

 right now there is the rsyslog-pkg-* repos on github that have the scripts
 that adiscon uses internally. As I found when I went to use them, there are
 a few oddities and too much hard-coded for adiscon internal use. But there
 is also a lot of useful stuff there as well.

 As we look at the build options, let's see how much of the existing stuff
 we can re-use.

 Also, let's try to make this be something that people can use when
 building from git.

 David Lang


  - ... whatever else I haven't yet thought about.

 Please take a moment to voice your preferrences!

 Thanks,
 Rainer


 2015-06-03 21:08 GMT+02:00 David Lang da...@lang.hm:

 take a look at

 https://github.com/rsyslog/rsyslog-pkg-ubuntu

 to build locally without using the PPA infrastructure I apply the
 attached
 patch (remove the sections for disabling usertools, that's a debugging
 thing
 I have in place at the moment)

 do pbuilder --create to create the compile environment, then I use the
 following script to pull the latest updates and compile test packages

 find . -name .git |sed s/.git// |while read file
 do
   echo $file
   cd $file
   /usr/bin/git fetch
   /usr/bin/git pull
   /usr/bin/git fetch --tags
 #  /usr/bin/git gc -q --aggressive
   autoreconf -fi
   ./configure -q
   rm *master*
   make dist --quiet
   cd -
 done
 echo finished making source packages
 cd rsyslog-pkg-ubuntu
 rm */LAST_VERSION.*
 for i in libestr liblogging liblognorm librelp rsyslog
 do
   cd $i
   rm ${i}_*
   cp ../../$i/*master* .
   ../scripts/auto_daily_project.sh trusty v8-devel master
   (
 echo '1'
 echo '1'
 echo '1'
 echo '1'
   ) |../scripts/build.sh
   echo finished making $i
   cd -
 done

 This should help get you started :-)

 David Lang

 On Wed, 3 Jun 2015, Brian Knox wrote:

  Date: Wed, 3 Jun 2015 11:48:20 -0400
 From: Brian Knox bk...@digitalocean.com
 Reply-To: rsyslog-users rsyslog@lists.adiscon.com
 To: rsyslog-users rsyslog@lists.adiscon.com
 Subject

Re: [rsyslog] rsyslog packaging project - was: rsyslog adiscon packages

[Brian Knox]

Rsyslog is very useful and appreciated :)

On Tue, Jun 9, 2015 at 1:28 PM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Sent from phone, thus brief.
 Am 09.06.2015 19:01 schrieb Brian Knox bk...@digitalocean.com:
 
  Coordinating on the mailing list is fine with me.  My employer
  (DigitalOcean) has given the rsyslog project a fairly sizeable free
 hosting
  budget so I'd be remiss to not advocate for us ;).

 I just realize that I slipped something (the Digital Ocean sponsoring) that
 I wanted to announce more formally ... Well that will follow but let me say
 now that it is very useful and appreciated :)

 Rainer
  If another option works
  out to be better I certainly understand!
 
  I totally agree that converging on an agreed upon problem statement is
 the
  right place to start.
 
  Cheers,
  Brian
 
  On Tue, Jun 9, 2015 at 12:53 PM, David Lang da...@lang.hm wrote:
 
   On Tue, 9 Jun 2015, Rainer Gerhards wrote:
  
Hi all,
  
   chances are extremely well to get to better packaging projects. We had
   some discussions internally in Adiscon, and I was able to secure the
   help of Florian Riedl for getting this in the best possible shape.
  
   Our goal is to get
  
   - better packages
   - more timely support for new distro releases
   - support for a broader set of distros (e.g. Fedora, often requested)
   - more ability for the community to steer this previous all-Adiscon
   project
  
   The 0mq discussion that started this thread is a good example of what
   this means.
  
   With the help of more community involvment we can reach the goals. And
   in order to make it easier to contribute, we need to streamline the
   process of how we build, release, test, and announce packages.
  
   Thankfully, Brian has offered to become an active team member. It
   would be great if others would also join in.
  
   I currently think that the right path to success is to start with
   small but sufficiently large project part. So I would propose that we
   focus on Ubuntu initially, get that part organized, learn a couple of
   things and apply the gained experience later to a final project that
   covers other distros as well (as mentioned by darix, the use of OBS
   sounds very appealing to me).
  
   In order to get going, I would like to see some ideas float on:
  
   - how should we communicate?
(rsyslog mailing list, dedicated mailing list, github issue
   trackers, IRC, ...)
  
  
   I'd say the rsyslog mailing list, failing that a dedicated mailing
 list.
   But I think that the issues we will be working through are useful to
 people
   who need to roll their own version (to test something from git, or to
   enable specific features).
  
- where do we track issues?
(I  have a strong preferrence for the github issue trackers)
   - what about doc?
   - where should we place the build platform
(cloud I would guess, could we use Digital Ocean sponsorship for
 this)?
  
  
   I don't know the details of using it, but the Suse Open Build Platform
 is
   already setup to support a whole bunch of target distros. How close
 does it
   come to covering everything we need?
  
   Can it be used for all the different uses we want from this (distro
   release builds, nightly builds, other)
  
   If we have to roll our own infrastructure, some sort of cloud system is
   right. Google donates time on their cloud system to opensource
 projects, I
   don't know if it would be enough or not. Sponsorship from whoever is
 good
   :-)
  
- when do we start ;)
  
  
   Clarification of the problem statement and where we are starting from
 :-)
  
   right now there is the rsyslog-pkg-* repos on github that have the
 scripts
   that adiscon uses internally. As I found when I went to use them, there
 are
   a few oddities and too much hard-coded for adiscon internal use. But
 there
   is also a lot of useful stuff there as well.
  
   As we look at the build options, let's see how much of the existing
 stuff
   we can re-use.
  
   Also, let's try to make this be something that people can use when
   building from git.
  
   David Lang
  
  
- ... whatever else I haven't yet thought about.
  
   Please take a moment to voice your preferrences!
  
   Thanks,
   Rainer
  
  
   2015-06-03 21:08 GMT+02:00 David Lang da...@lang.hm:
  
   take a look at
  
   https://github.com/rsyslog/rsyslog-pkg-ubuntu
  
   to build locally without using the PPA infrastructure I apply the
   attached
   patch (remove the sections for disabling usertools, that's a
 debugging
   thing
   I have in place at the moment)
  
   do pbuilder --create to create the compile environment, then I use
 the
   following script to pull the latest updates and compile test packages
  
   find . -name .git |sed s/.git// |while read file
   do
 echo $file
 cd $file
 /usr/bin/git fetch
 /usr/bin/git pull
 /usr/bin/git fetch --tags
   #  /usr/bin/git gc -q --aggressive
 autoreconf -fi
 ./configure -q

Re: [rsyslog] Using Buildbot for rsyslog?

[Brian Knox]

The i3 window manager developers have a pretty good article explaining how
they use buildbot that might be a good reference -
http://i3wm.org/docs/buildbot.html

I have no personal experience with buildbot, but I'd be glad to start
reading and join in the fun.

Brian

On Mon, Jun 8, 2015 at 12:08 PM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Hi all,

 as you may have seen, we are going to form a release team and will
 setup some resources for them (all to be discussed). Before really
 starting that discussion, I wonder if there would be value in using
 Buildbot[1].

 I sounds rather interesting not only for the release cycles but (TBH
 much more) for the test machine farm.

 Does anyone has experience with buildbot or at least an opinion? Does
 anyone know how much effort it is to get a system going? And would
 anyone be interested in joining an effort that actually makes this
 happen?

 Thanks,
 Rainer

 [1] http://buildbot.net/
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[Brian Knox]

David -

I agree that RELP would be the right place for it.  For TCP load balancing
with rsyslog currently, I find using an external load balancer such as
haproxy works nicely.

Brian

On Thu, Jun 4, 2015 at 1:40 PM, David Lang da...@lang.hm wrote:

 If we do decide to do this, it would be better to base the work on relp
 than tcp (if the purpose is reliable delivery under failure conditions)

 The thing is that failover and load balancing can be a rather complex
 problem with many different solutions (different ones are better in
 different conditions). Trying to implement the best options of everything
 inside rsyslog is a lot of work, and I'd prefer the time being spent on
 improving the things that can't be done with exiting tools :-)

 Rsyslog already has better support for load balancing than logstash and
 nxlog (I haven't looked at syslog-ng)

 One question, if an action is configured to go to a name, when it
 reconnects does it do another name lookup? or is it cached?


 On Thu, 4 Jun 2015, singh.janmejay wrote:

  Yes L4 load-balancing will work to significant scale. L7
 load-balancing will do even better in terms of even load, but not sure
 if syslog protocol is widely supported in load-balancers.


 The syslog protocol is not supported by load balancers at L7.

 However, this is still one of the places where existing load balancing
 solutions would do better than your proposed solution. Having each client
 connect randomly would result in more even load balancing only if they are
 all generating the same amount of traffic. Since they aren't, it's going to
 be uneven, and the clients cannot know what the right thing to do is.

 Doing L2 load balancing at the destination, the load balancer can see all
 the traffic and make descisions on it.

  DNS scaling and propagation delay are sometimes not acceptable, but
 BGP anycast is something that'd work at data-center scale with very
 large PODs.


 DNS and BGP failovers within your own network are as fast as you configure
 them to be :-). I'm not even saying BGP anycast, just normal BGP failover
 for when a set of IPs becomes unavailable, route them to a different
 destination.

  This is an alternative to that. It has fewer moving parts (just
 producer and consumer), no LB and it doesn't require the complexity of
 anycast.


 on the other hand, it requires much more complex configuration on every
 client. Every time there is a change on the number of systems in the
 cluster, every single client must be updated, or they will only deliver to
 a subset of the available systems. From a sysadmin point of view, this is a
 horrible thing to maintain. It's possible if you have a centralized config
 management system, but that's a lot more complexity.

  It trades-off engineering complexity of load-balancer and anycast with
 smarter-clients and servers (increasing the complexity of clients and
 servers a little, but also simplifying the deployment topology
 significantly).


 I see this as being a significantly more complex deployment topology :-)

  I think all three are valid approaches and choice of one over the
 other(best fit) will vary across deployments.


 The question I have is if the value of adding this option in rsyslog is
 greater than the features that would be added instead.

 David Lang

 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[Brian Knox]

You might want to take a look at the omczmq and imczmq (the new ZeroMQ
input and output plugins I've been working on).  See -
https://github.com/rsyslog/rsyslog/tree/master/contrib/omczmq  and
https://github.com/rsyslog/rsyslog/tree/master/contrib/imczmq

Out of the box they currently support fan out / fan in  and  pub-sub (
note: pub sub does not apply backpressure - it's meant to be lossy in
zeromq). topologies.  I recently started adding support for zbeacon (
http://czmq.zeromq.org/manual:zbeacon ), a UDP based discovery service.
While UDP multicast is not the best solution everywhere, it works for some
cases.

My short to medium term plans going forward with these plugins include:

* Support for non encrypted connections (curvezmq encryption was my first
priority, the plugins don't actually have options for non encrypted
connections yet).
* credit based flow control
* malamute (an embedded broker - see https://github.com/zeromq/malamute )

I've been pondering other discovery support.  So far, I've been wary of
introducing additional dependencies.  There's so many discovery services
that are en vogue right now (zookeeper, etcd, consul...) and every
additional protocol supported is more overhead from a support and
maintenance standpoint.  So at the moment I've been focused solely on ZMTP
( the protocol libzmq implements - http://rfc.zeromq.org/spec:23 )

Brian









On Thu, Jun 4, 2015 at 4:46 AM, singh.janmejay singh.janme...@gmail.com
wrote:

 It won't be a very large change really if we develop it in an external
 library.

 In rsyslog codebase, its a fairly small change, limited to input and
 output modules that we pick. It'll be small parts of plugin code
 (where new connection is established that will call this library
 function conditionally, thats about it).



 On Thu, Jun 4, 2015 at 2:09 PM, Rainer Gerhards
 rgerha...@hq.adiscon.com wrote:
  Sorry if this sounds discouraging: I currently have such a large
  backlog that I can't engage in that effort and I think I am also
  unable to merge any change of this magnitude any time before the
  backlog has become shorter (Q4+ 2015 I guess).
 
  Sorry I have no better answer, but you see yourself what all is going
  on and I really need to make sure I can follow at least the bare
  essentials.
 
  Rainer
 
  2015-06-04 5:53 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
  Yes L4 load-balancing will work to significant scale. L7
  load-balancing will do even better in terms of even load, but not sure
  if syslog protocol is widely supported in load-balancers.
 
  DNS scaling and propagation delay are sometimes not acceptable, but
  BGP anycast is something that'd work at data-center scale with very
  large PODs.
 
  This is an alternative to that. It has fewer moving parts (just
  producer and consumer), no LB and it doesn't require the complexity of
  anycast.
 
  It trades-off engineering complexity of load-balancer and anycast with
  smarter-clients and servers (increasing the complexity of clients and
  servers a little, but also simplifying the deployment topology
  significantly).
 
  I think all three are valid approaches and choice of one over the
  other(best fit) will vary across deployments.
 
 
  On Thu, Jun 4, 2015 at 8:45 AM, David Lang da...@lang.hm wrote:
  I don't see the advantage of adding all this complexity as opposed to
 using
  existing load balancing approaches. With existing tools we can deliver
 the
  log stream to a cluster of systems, and deal with them failing. Yes,
 the
  easy approaches to doing this are limited to the throughput of a single
  wire, but since that single wire is commonly 10Gb/sec (and easily
 40Gb/sec)
  with off-the-shelf technology, and the fact that the log stream can be
  compressed, this isn't likely to be an issue for much of anyone below
 Google
  scale.
 
  There is a lot of advantages to keeping the failover logic and config
  contained to as small an area of the network and as few devices as
 possible.
  The systems accepting the ogs _must- participate in the process
 (responding
  to health checks if nothing else), it only takes a couple other boxes
 (if
  any) to perform TCP load balancing. And having everything local
 increases
  the accuracy of the detection and speed of recovery.
 
  If you want to deal with larger failures (datacenter scale), then
 existing
  DNS/BGP failover tools can come into play.
 
  What advantage do we gain by pushing the configuration and failover
 logic to
  the senders?
 
  David Lang
 
 
  On Thu, 4 Jun 2015, singh.janmejay wrote:
 
  Hi,
 
  This is proposal towards first-class support for notion of a 'cluster'
  as a log-forwarding destination. It talks about a
  technology-independent service-discovery-support implementation.
 
  Scenario / Context:
  
  Say an environment is supposed to relay all logs to a logical
  destination for aggregation/archival purpose. Such a setup at large
  scale would have a several log-producers 

Re: [rsyslog] rsyslog adiscon packages

[Brian Knox]

I'm a member of the zeromq team :)  What would I need to do?

Brian

On Wed, Jun 3, 2015 at 9:56 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2015-06-03 14:50 GMT+02:00 Brian Knox bk...@digitalocean.com:
  I've been working on the new zeromq plugins ( contrib/omczmq and
  contrib/imczmq) for a bit, and have been using them in production myself
  for quite awhile.
 
  Yesterday, the zeromq organization released zeromq 4.1 (
  http://zeromq.org/intro:get-the-software ) and czmq 3.0 (
  http://czmq.zeromq.org/page:get-the-software ) stable releases, which
 are
  the versions of zeromq and czmq ( a nice C api on top of libzmq ) that
  om/imczmq depend on.
 
  I was wondering if it would be possible to get these added to the
 packages
  now the same way librdkafka / omkafka is set up.
 
  The dependencies are:
 
  * libsodium ( included in most distros now afaik, but latest stable
 source
  is also at
  https://download.libsodium.org/libsodium/releases/libsodium-1.0.3.tar.gz
 )
  * zeromq 4.1 ( http://download.zeromq.org/zeromq-4.1.1.tar.gz )
  * czmq 3.0 ( http://download.zeromq.org/czmq-3.0.1.tar.gz )


 The main problem is that we do not want to be responsible to
 maintain 0mq in regard to checking for patches etc... However, if a
 member of the 0mq team is also a member of our release team and looks
 after that, this problem can be solved ;)

 Maybe we could use a DigitalOcean droplet to run the package creation
 scripts in the future, this would also solve a bit on the
 collaboration front. What do you think?

 If that's a way to go, it's probably not done immediately, as a
 little bit of setup and redesign work is involved, but we could work
 towards that goal...

 Rainer

 
 
  The only option flag needed is --enable-libsodium on the zeromq 4.1
 build.
 
  I'd be happy to work with whoever works on the packages.  Getting this
 easy
  to install would be helpful, and the more people attempting to use the
  plugins the more feedback I can get.
 
  Cheers,
  Brian
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog adiscon packages

[Brian Knox]

I've been working on the new zeromq plugins ( contrib/omczmq and
contrib/imczmq) for a bit, and have been using them in production myself
for quite awhile.

Yesterday, the zeromq organization released zeromq 4.1 (
http://zeromq.org/intro:get-the-software ) and czmq 3.0 (
http://czmq.zeromq.org/page:get-the-software ) stable releases, which are
the versions of zeromq and czmq ( a nice C api on top of libzmq ) that
om/imczmq depend on.

I was wondering if it would be possible to get these added to the packages
now the same way librdkafka / omkafka is set up.

The dependencies are:

* libsodium ( included in most distros now afaik, but latest stable source
is also at
https://download.libsodium.org/libsodium/releases/libsodium-1.0.3.tar.gz )
* zeromq 4.1 ( http://download.zeromq.org/zeromq-4.1.1.tar.gz )
* czmq 3.0 ( http://download.zeromq.org/czmq-3.0.1.tar.gz )


The only option flag needed is --enable-libsodium on the zeromq 4.1 build.

I'd be happy to work with whoever works on the packages.  Getting this easy
to install would be helpful, and the more people attempting to use the
plugins the more feedback I can get.

Cheers,
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog adiscon packages

[Brian Knox]

Ubuntu LTS is currently what I'm using so that's advantageous.  In
addition, I've built a custom rsyslog package for Ubuntu that includes
omczmq / imczmq along with debs for the dependencies.  However, my rsyslog
package is monolithic and I'm using brew2deb, which is kind of a strange
wrapper around fpm and homebrew that probably isn't the best way to do
things.

Ubuntu and Debian pacakge libzmq (but not czmq).  Their packages are behind
current, but maybe their packages would be a good and hopefully easy place
to start, depending on how packages are being built currently for the
ubuntu repo.

If there's build scripts for the current repo now, I'd be happy to work
through them and do the work.

Brian


On Wed, Jun 3, 2015 at 11:38 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Would it be a good idea to start with Ubuntu?

 Sent from phone, thus brief.
 Am 03.06.2015 16:23 schrieb Brian Knox bk...@digitalocean.com:

  I'm on board!
 
  Cheers,
  Brian
 
  On Wed, Jun 3, 2015 at 10:07 AM, Rainer Gerhards 
 rgerha...@hq.adiscon.com
  
  wrote:
 
   Sent from phone, thus brief.
   Am 03.06.2015 15:58 schrieb Brian Knox bk...@digitalocean.com:
   
I'm a member of the zeromq team :)
  
   I know ;)
  
What would I need to do?
  
   Join the rsyslog release team and keep an eye especially on zmq. As I
  wrote
   is just something we need to newly setup.
   
Brian
   
On Wed, Jun 3, 2015 at 9:56 AM, Rainer Gerhards 
   rgerha...@hq.adiscon.com
wrote:
   
 2015-06-03 14:50 GMT+02:00 Brian Knox bk...@digitalocean.com:
  I've been working on the new zeromq plugins ( contrib/omczmq and
  contrib/imczmq) for a bit, and have been using them in production
   myself
  for quite awhile.
 
  Yesterday, the zeromq organization released zeromq 4.1 (
  http://zeromq.org/intro:get-the-software ) and czmq 3.0 (
  http://czmq.zeromq.org/page:get-the-software ) stable releases,
   which
 are
  the versions of zeromq and czmq ( a nice C api on top of libzmq )
   that
  om/imczmq depend on.
 
  I was wondering if it would be possible to get these added to the
 packages
  now the same way librdkafka / omkafka is set up.
 
  The dependencies are:
 
  * libsodium ( included in most distros now afaik, but latest
 stable
 source
  is also at
 
  
 https://download.libsodium.org/libsodium/releases/libsodium-1.0.3.tar.gz
 )
  * zeromq 4.1 ( http://download.zeromq.org/zeromq-4.1.1.tar.gz )
  * czmq 3.0 ( http://download.zeromq.org/czmq-3.0.1.tar.gz )


 The main problem is that we do not want to be responsible to
 maintain 0mq in regard to checking for patches etc... However, if a
 member of the 0mq team is also a member of our release team and
 looks
 after that, this problem can be solved ;)

 Maybe we could use a DigitalOcean droplet to run the package
 creation
 scripts in the future, this would also solve a bit on the
 collaboration front. What do you think?

 If that's a way to go, it's probably not done immediately, as a
 little bit of setup and redesign work is involved, but we could
 work
 towards that goal...

 Rainer

 
 
  The only option flag needed is --enable-libsodium on the zeromq
 4.1
 build.
 
  I'd be happy to work with whoever works on the packages.  Getting
   this
 easy
  to install would be helpful, and the more people attempting to
 use
   the
  plugins the more feedback I can get.
 
  Cheers,
  Brian
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
   myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
  you
 DON'T LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
   myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
  you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
  myriad
   of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
   DON'T LIKE THAT.
   ___
   rsyslog mailing list
   http://lists.adiscon.net

Re: [rsyslog] looking for an example of using rsyslog-pkg-ubuntu scripts

[Brian Knox]

If it's helpful to know at all - I've been building our rsyslog packages
with a fork of brew2deb - which is a rather odd beast that combines
homebrew, and fpm to build packages.  In my case, I just build one rsyslog
package that contains everything we need since I'm not building it for
general consumption.  My package formulas are here -
https://github.com/taotetek/brew2deb/tree/master/packages

It's probably not the right solution for general use ubuntu packages but
just putting it out there in case anyone might find it useful or get ideas
from it.

Brian



On Wed, Apr 22, 2015 at 4:41 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2015-04-22 10:34 GMT+02:00 David Lang da...@lang.hm:
  will do, work on liblognorm today ;-)
 
  In my digging to try and understand what the scripts are doing, I ran
 across
  the gbp tool, which seems to be the tool to do 'the right thing' for
 making
  packages where the upstream is in git, I've started trying to figure out
 how
  to use it, and it may simplify things compared to these scripts, but it's
  hard to say since i don't understand what they are supposed to be doing
 yet
  :-)

 From my very high level perspective: they should build good enough
 packages as automatically as possible with the least effort required.
 If there is any other tool to change to that's better at that, I am
 all ears ;-)

 Rainer
 
  David Lang
 
  On Wed, 22 Apr 2015, Rainer Gerhards wrote:
 
  Date: Wed, 22 Apr 2015 10:27:21 +0200
 
  From: Rainer Gerhards rgerha...@hq.adiscon.com
  Reply-To: rsyslog-users rsyslog@lists.adiscon.com
  To: rsyslog-users rsyslog@lists.adiscon.com
  Subject: Re: [rsyslog] looking for an example of using
 rsyslog-pkg-ubuntu
  scripts
 
  2015-04-21 19:50 GMT+02:00 David Lang da...@lang.hm:
 
  hmm, it looks like there may be some other stuff in your directories
 that
  it's accidentially depending on.
 
  root@linux-dev01:/usr/src/rsyslog-pkg-ubuntu/rsyslog# cp
  /usr/src/raw/rsyslog/rsyslog-8.10.0.master.tar.gz .
 
  root@linux-dev01:/usr/src/rsyslog-pkg-ubuntu/rsyslog#
  ../scripts/auto_daily_project.sh trusty v8-devel master
 
 
  cat: LAST_VERSION.master.trusty: No such file or directory
  rm: cannot remove '*.changes': No such file or directory
  rm: cannot remove '*.dsc': No such file or directory
  rm: cannot remove '*.build': No such file or directory
  rm: cannot remove '*.debian.tar.gz': No such file or directory
  cp: cannot stat '../trusty/master/debian': No such file or directory
 
 
  These messages are OK. I've just updated my script to do not emit most
  of them. I've also added comments to the script (please review). Along
  that've seen it includes some Adiscon email addreses, you may want to
  change that for now. I'll update that soon, but would like to dig more
  into liblognorm today.
 
  This package has a Debian revision number but there does not seem to be
  an appropriate original tar file or .orig directory in the parent
  directory;
  (expected one of rsyslog_8.10.0.20150421165741.orig.tar.gz,
  rsyslog_8.10.0.20150421165741.orig.tar.bz2,
  rsyslog_8.10.0.20150421165741.orig.tar.lzma,
  rsyslog_8.10.0.20150421165741.orig.tar.xz or
 8.10.0.20150421165741.orig)
  continue anyway? (y/n) y
 
 
  Mhhh, this should have been handled properly. Can you send me a full
  log of the run, including executed comments (set -o xtrace). I would
  need to set up a brand-new VM, and again I'd really like to look at
  liblognorm instead...
 
  I've also asked Florian to look into your other questions and add some
  comments to the package. That'llbe breif, though, as much of it will
  no longer be true in the not so distant future. As I said, we are
  working on changing the process, and it is great to have someone
  outside of Adiscon using the scripts so we can really see what it
  takes to make them more generic.
 
  Rainer
 
   dpkg-buildpackage -rfakeroot -d -us -uc -S -sa
  dpkg-buildpackage: warning: using a gain-root-command while being root
  dpkg-buildpackage: source package rsyslog
  dpkg-buildpackage: source version 8.10.0.20150421165741-0adiscon1trusty
  dpkg-buildpackage: source distribution trusty
  dpkg-buildpackage: source changed by Adiscon package maintainers
  adiscon-pkg-maintain...@adiscon.com
   dpkg-source --before-build 8.10.0.20150421165741
  dpkg-source: info: applying 01-dont_create_db.patch
   fakeroot debian/rules clean
  dh clean --with autoreconf
 dh_testdir
 dh_auto_clean
 dh_autoreconf_clean
 dh_clean
   dpkg-source -b 8.10.0.20150421165741
  dpkg-source: error: can't build with source format '3.0 (quilt)': no
  upstream tarball found at
  ../rsyslog_8.10.0.20150421165741.orig.tar.{bz2,gz,lzma,xz}
  dpkg-buildpackage: error: dpkg-source -b 8.10.0.20150421165741 gave
 error
  exit status 255
  debuild: fatal error at line 1364:
  dpkg-buildpackage -rfakeroot -d -us -uc -S -sa failed
 
 
  On Tue, 21 Apr 2015, Rainer Gerhards wrote:
 
  Date: Tue, 21 Apr 2015 19:05:29 +0200
  From: Rainer 

Re: [rsyslog] Best practice for an application to get structured data to rsyslog

[Brian Knox]

We keep our logs in JSON format and don't find it to be a drawback.  We
have logs searchable in elasticsearch - and for working with logs on disk,
have a small program that logs can be piped through that strips out
everything but the json which makes it very easy to pipe logs to jq (a
command line json processor - see https://stedolan.github.io/jq/ ).



On Wed, Apr 15, 2015 at 12:48 AM, David Lang da...@lang.hm wrote:

 On Wed, 15 Apr 2015, Ezell, Matthew A. wrote:

  Sure, as a system administrator it's pretty clear how best to handle this.
 If there's CEE JSON data coming over the wire, use mmjsonparse.  If it's
 unstructured traditional syslog(3) data, use mmnormalize to try to extract
 relevant fields based on rules I setup.  Write the traditional message
 field to /var/log/messages and send the structured data to ElasticSearch.
 But I'm a system administrator who cares about structured logging, so I
 would have a custom rsyslog setup to handle this seamlessly.

 The question is really from the application developer's point of view.
 How do you log structured data in a way that doesn't change the format of
 /var/log/messages for most users, but provides additional information for
 those system administrators who choose to handle the structured data?

 Imagine going to the developers of OpenSSH and requesting that they start
 logging structured data.  If they simply changed all their syslog(3) calls
 to output CEE JSON instead instead of plain strings, it's going to break
 just about every brute-force login detection system out there.  That's
 unacceptable.  What is the *right* thing for them to do?


 do like ossec does and have a config option that switches to JSON output.

 since they have to have their software work everywhere that it's working
 today, they can't change it's output at all. anything they do will break
 parsers.

 but with a config switch (which a distro could turn on by default), they
 can output a different format, and that format could be JSON with the old
 log text in a msg field (again though, which is the source of truth if they
 differ)

 David Lang

 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omkafka module for rsyslog

[Brian Knox]

If you are interested in RPMs have you checked out the adiscon repo?

http://rpms.adiscon.com/v8-stable/

Brian



On Thu, Mar 12, 2015 at 7:00 AM, Sloot, Hans-Peter 
hans-peter.sl...@atos.net wrote:

 Hi,

 My OS is Redhat EL6.

 When doing the configure it starts complaining.
 First about uuid  so I ran : ./configure --enable-omkafka  --disable-uuid
 But now it complains about

 checking for LIBLOGGING_STDLOG... no
 configure: error: Package requirements (liblogging-stdlog = 1.0.3) were
 not met:

 No package 'liblogging-stdlog' found

 Consider adjusting the PKG_CONFIG_PATH environment variable if you
 installed software in a non-standard prefix.

 Alternatively, you may set the environment variables
 LIBLOGGING_STDLOG_CFLAGS
 and LIBLOGGING_STDLOG_LIBS to avoid the need to call pkg-config.
 See the pkg-config man page for more details.

 I cannot find a liblogging-stdlog rpm in the official yum repositories.

 Regards Hans-Peter
 -Original Message-
 From: rsyslog-boun...@lists.adiscon.com [mailto:
 rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox
 Sent: Thursday, March 12, 2015 11:07 AM
 To: rsyslog-users
 Subject: Re: [rsyslog] omkafka module for rsyslog

 I can't speak to whether the module is packaged or not for your OS as I
 build my own rsyslog packages.  If you are building rsyslog yourself, you
 need to make sure you have this library:

 https://github.com/edenhill/librdkafka

 And then ./configure --enable-omkafka should work fine.

 On Thu, Mar 12, 2015 at 5:46 AM, Sloot, Hans-Peter 
 hans-peter.sl...@atos.netmailto:hans-peter.sl...@atos.net wrote:

  Hi,
 
  I have been trying to get the omkafka plugin working in rsyslog The
  version installed from epel is : rsyslog-8.8.0.ad1-1.el6.x86_64 The
  module omkafka is listed in the list of output modules
  http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.htm
  l
 
  When starting rsyslog  the following is shown in the messages file:
  Mar 12 09:44:33 nlvora154 rsyslogd-2066: could not load module
  '/lib64/rsyslog/omkafka.so', dlopen: /lib64/rsyslog/omkafka.so: cannot
  open shared object file: No such file or directory  [v8.8.0.ad1 try
  http://www.rsyslog.com/e/2066 ]
  Mar 12 09:44:33 nlvora154 rsyslogd-2209: module name 'omkafka' is
  unknown
  [v8.8.0.ad1 try http://www.rsyslog.com/e/2209 ] Mar 12 09:44:33
  nlvora154 rsyslogd-2207: error during parsing file
  /etc/rsyslog.d/kafka.conf, on or before line 60: errors occured in
  file '/etc/rsyslog.d/kafka.conf' around line 60 [v8.8.0.ad1 try
  http://www.rsyslog.com/e/2207 ]
 
  For elasticsearch there is a separate rpm .
  How can I get the omkafka module working?
 
  Regards HansP
 
  This e-mail and the documents attached are confidential and intended
  solely for the addressee; it may also be privileged. If you receive
  this e-mail in error, please notify the sender immediately and destroy
  it. As its integrity cannot be secured on the Internet, Atos’
  liability cannot be triggered for the message content. Although the
  sender endeavours to maintain a computer virus-free network, the
  sender does not warrant that this transmission is virus-free and will
  not be liable for any damages resulting from any virus transmitted. On
  all offers and agreements under which Atos Nederland B.V. supplies
  goods and/or services of whatever nature, the Terms of Delivery from
 Atos Nederland B.V. exclusively apply.
  The Terms of Delivery shall be promptly submitted to you on your request.
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
  WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
  sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
 This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
 beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
 THAT.

 This e-mail and the documents attached are confidential and intended
 solely for the addressee; it may also be privileged. If you receive this
 e-mail in error, please notify the sender immediately and destroy it. As
 its integrity cannot be secured on the Internet, Atos’ liability cannot be
 triggered for the message content. Although the sender endeavours to
 maintain a computer virus-free network, the sender does not warrant that
 this transmission is virus-free and will not be liable for any damages
 resulting from any virus transmitted. On all offers and agreements under
 which Atos Nederland B.V. supplies goods and/or services of whatever
 nature, the Terms of Delivery from Atos

Re: [rsyslog] upcoming log normalization work

[Brian Knox]

Congratulations, Rainer!  This sounds like a great opportunity.

Cheers,
Brian

On Thu, Feb 5, 2015 at 11:39 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Hi all,

 finally, I can dispense some word on upcoming work for liblognorm.

 The short story is that I will have ample time in the next months to
 seriously work on and improve liblognorm, including some new tooling to
 make it easier to use, and make it usuable as a stand-alone tool. This will
 become available to the rsyslog project via the mmnormalize module.

 The full story is a bit longer ;) As some of you may already know, I have
 decided to brush up my academic credentials a bit and I am working on my
 MSc. I have gotten the opportunity to work on the topic of log
 normalization for my thesis. This, of course, is no implementation work,
 but I plan to use liblognorm as a working sample of whatever comes out of
 the thesis and plan to implement and proof ideas as they come up (using
 liblognorm as a testbed like I did with rsyslog during the IETF syslog
 standadization process).

 As such, I will try to develop liblognorm side-by-side with concept
 development, but I may run into some subtle issue of original authorship:
 the thesis of course must contain my own work and any third-party
 suggestions in regard to algorithms must be quoted and can not count
 against thesis work. So in a strange way the more good suggestions I get,
 even for things I already considered, the more I run into trouble with the
 thesis. Pure feedback like this does not work for my environment is no
 problem, but sketches of algorithms are. So this is a bit complicated,
 especially with the regular open source development model on ones mind.
 I'll still try to work on that slippery slope, but may switch to a private
 archive and silence mode if this turns out to become a real problem. In
 any case, once the thesis is done I am more than open to discuss any
 further suggestions.

 What I have on my mind for liblognorm is much more than wiggeling a bit
 with it. What we currently use is actually a proof of concept (a useful
 one, obviously), but there are more than a couple of rough edges. I think
 the core algorithm can be improved, if not replaced, and there is much more
 work needed to aid in developing and maintaining sample bases. I have some
 semi-automatic process for the creation of sample bases on my mind, but
 that's something that really must be investigated first. Also, I think we
 need a different, better, description language, ... and so on.

 I have talked with Adiscon and I will work only part-time during the thesis
 period and the prep work. That means I will be working less actively on
 adding new features to rsyslog, but I am able to look at bug reports and
 other important things. Actually, from a rsyslog PoV, I'll be working on a
 big feature that is even better log normalization capability.

 I need to do some prep work before I can start with the actual thesis work.
 Most importantly, I need a set, hopefully large and diverse, of actual log
 messages. The better this set, the better most probably the end result will
 be (some heuristics will be involved for sure). I hope to receive community
 support in collecting the log set. But I'll detail that in another mail.

 Finally, I need to say that I am super-excited about this ability to
 combine thesis work with something that I had on my mind for quite a while
 but that I probably would realistically never have been able to look at in
 this depth. And the implementation hopefully will be useful for the
 community as well. So it's a win-win-win situation from my PoV.

 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] plans for rsyslog 8.8

[Brian Knox]

Rainer - the pull model is something I want to add to the zeromq plugins as
well.  The idea being, if I have multiple downstream zeromq destinations,
they can then request more logs as they are able to perform work on them -
which of course allows you to load balance across downstream workers that
are ready for more work.

Brian

On Thu, Jan 15, 2015 at 11:17 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Hi folks,

 I thought I share what I will (most probably) be working on the next couple
 of weeks:

 http://blog.gerhards.net/2015/01/whats-next-with-rsyslog.html

 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)

[Brian Knox]

This is great, because I was thinking I should start on an output plugin
for kafka, and now I can be lazy ;)  I'll be glad to test this module
against our kafka infrastructure at work!

Brian

On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 So here it is:

 https://github.com/rsyslog/rsyslog/tree/master-omkafka

 I have deliberately left out the impstats calls as I saw you included them
 in your source, so feel free to add them again ;)

 Note that there is a different approach to parameters: my version has only
 very few hardcoded ones, but instead permits the user to set actual kafka
 parameters via the confparam and topicparam parameters. In my experience,
 this is far better with a rapidly evolving backend, because now any new or
 changed parameter is immediately available.

 I'll write up some bare doc later today. So far this is tested on Ubuntu
 14.04LTS only, and I would *deeply* appreciate if you (and anyone else)
 could test it - especially as I have only very limited Kafka know-how and
 no real use case myself.

 Thanks,
 Rainer

 2014-12-15 0:07 GMT+01:00 Tait Clarridge t...@clarridge.ca:

  Fantastic, that's one area I definitely omitted. I look forward to
  contributing to this great project!
 
  On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards
  rgerha...@hq.adiscon.com wrote:
   excellent! I hope I have the initial commit tomorrow morning. Now
  working a
   bit on error detection...
  
   Rainer
  
   2014-12-14 19:14 GMT+01:00 Tait Clarridge t...@clarridge.ca:
  
   I think your idea works; if you commit yours (which is way more robust
   than mine), I can submit a pull request for the additional items my
   module had that are not present in yours.
  
   The only parts of the original contribution that I re-used were parts
   of the send_message function - I added much more on top of it so that
   should cover all of the items I will/might add.
  
   Tait
  
   On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards
   rgerha...@hq.adiscon.com wrote:
side-note on how to carry on: I have seen check the link to the
  original
contribution (I admit I totally overlooked it at that time). It did
  not
have an explicit license, which is a bit problematic. Some folks
 would
   say
this means it default back to the project license, which then would
  mean
GPLv3, which would cause problems with our move to ASL 2.0.
   
Here is what I have on my mind on how to solve and be able to
 merge: I
   hope
I can commit tomorrow. I could then
   
1. pick from your code what is missing in mine
2. apply it under your your commit --author
3. and finally update to copyright in the header comment to reflect
  both
   us
   
Would that work for you? In step 1 there is a problem if I run into
  code
that stems back to the original contribution, which, if so, I can
 not
   merge
do to licensing. I hope that won't be the case.
   
If you like, you can of course create a PR based on my commit
  tomorrow,
   you
just need to make sure that all code that is added is originally
  written
   by
you or under some license that is OK with ASL 2.0 (like the BSD
  2-clause
from librdkafka).
   
What would work for you?
   
Rainer
   
2014-12-14 17:26 GMT+01:00 Rainer Gerhards 
 rgerha...@hq.adiscon.com
  :
   
just sent it your way :-)
   
2014-12-14 17:24 GMT+01:00 Tait Clarridge t...@clarridge.ca:
   
 just leave it as is for now. I need to merge manually in any
  case, as
both
 commits will be just a module. Bear a day with me, I don't want
 to
   do a
 hasty commit of my unfinished work. If you like, I can share a
   working
 copy, though. Just let me know.

 Rainer
   
That sounds great, when you get a chance I'd like to take a look.
  I'm
in no rush as well so take your time :)
   
Tait
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
   myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
  you
DON'T LIKE THAT.
   
   
   
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
  myriad
   of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
   DON'T LIKE THAT.
   ___
   rsyslog mailing list
   http://lists.adiscon.net/mailman/listinfo/rsyslog
   http://www.rsyslog.com/professional-services/
   What's up with rsyslog? Follow https://twitter.com/rgerhards
   

Re: [rsyslog] omkafka: Output Module for Kafka (v8 version)

[Brian Knox]

So 8.7 is aiming for Jan 13th?  I'll try to get some zeromq input and
output improvements done for that release then as well.

Cheers, and enjoy your vacation!
Brian

On Mon, Dec 15, 2014 at 6:54 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2014-12-15 12:29 GMT+01:00 Brian Knox bk...@digitalocean.com:

  This is great, because I was thinking I should start on an output plugin
  for kafka, and now I can be lazy ;)  I'll be glad to test this module
  against our kafka infrastructure at work!
 
 
 excellent! Please note that I will leave Friday for a holiday vacation, and
 I probably won't touch the code when away. IAW: early bug reports are
 appreciated. I still hope to get this module in in 8.7.0 on Jan, 13th.

 Rainer


  Brian
 
  On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards 
 rgerha...@hq.adiscon.com
  
  wrote:
  
   So here it is:
  
   https://github.com/rsyslog/rsyslog/tree/master-omkafka
  
   I have deliberately left out the impstats calls as I saw you included
  them
   in your source, so feel free to add them again ;)
  
   Note that there is a different approach to parameters: my version has
  only
   very few hardcoded ones, but instead permits the user to set actual
 kafka
   parameters via the confparam and topicparam parameters. In my
 experience,
   this is far better with a rapidly evolving backend, because now any new
  or
   changed parameter is immediately available.
  
   I'll write up some bare doc later today. So far this is tested on
 Ubuntu
   14.04LTS only, and I would *deeply* appreciate if you (and anyone else)
   could test it - especially as I have only very limited Kafka know-how
 and
   no real use case myself.
  
   Thanks,
   Rainer
  
   2014-12-15 0:07 GMT+01:00 Tait Clarridge t...@clarridge.ca:
  
Fantastic, that's one area I definitely omitted. I look forward to
contributing to this great project!
   
On Sun, Dec 14, 2014 at 1:23 PM, Rainer Gerhards
rgerha...@hq.adiscon.com wrote:
 excellent! I hope I have the initial commit tomorrow morning. Now
working a
 bit on error detection...

 Rainer

 2014-12-14 19:14 GMT+01:00 Tait Clarridge t...@clarridge.ca:

 I think your idea works; if you commit yours (which is way more
  robust
 than mine), I can submit a pull request for the additional items
 my
 module had that are not present in yours.

 The only parts of the original contribution that I re-used were
  parts
 of the send_message function - I added much more on top of it so
  that
 should cover all of the items I will/might add.

 Tait

 On Sun, Dec 14, 2014 at 11:41 AM, Rainer Gerhards
 rgerha...@hq.adiscon.com wrote:
  side-note on how to carry on: I have seen check the link to the
original
  contribution (I admit I totally overlooked it at that time). It
  did
not
  have an explicit license, which is a bit problematic. Some folks
   would
 say
  this means it default back to the project license, which then
  would
mean
  GPLv3, which would cause problems with our move to ASL 2.0.
 
  Here is what I have on my mind on how to solve and be able to
   merge: I
 hope
  I can commit tomorrow. I could then
 
  1. pick from your code what is missing in mine
  2. apply it under your your commit --author
  3. and finally update to copyright in the header comment to
  reflect
both
 us
 
  Would that work for you? In step 1 there is a problem if I run
  into
code
  that stems back to the original contribution, which, if so, I
 can
   not
 merge
  do to licensing. I hope that won't be the case.
 
  If you like, you can of course create a PR based on my commit
tomorrow,
 you
  just need to make sure that all code that is added is originally
written
 by
  you or under some license that is OK with ASL 2.0 (like the BSD
2-clause
  from librdkafka).
 
  What would work for you?
 
  Rainer
 
  2014-12-14 17:26 GMT+01:00 Rainer Gerhards 
   rgerha...@hq.adiscon.com
:
 
  just sent it your way :-)
 
  2014-12-14 17:24 GMT+01:00 Tait Clarridge t...@clarridge.ca:
 
   just leave it as is for now. I need to merge manually in any
case, as
  both
   commits will be just a module. Bear a day with me, I don't
  want
   to
 do a
   hasty commit of my unfinished work. If you like, I can
 share a
 working
   copy, though. Just let me know.
  
   Rainer
 
  That sounds great, when you get a chance I'd like to take a
  look.
I'm
  in no rush as well so take your time :)
 
  Tait
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE

Re: [rsyslog] mmnormalize improvements

[Brian Knox]

Rainer - do you have a link to a quick summary of the changes?  Would love
to know about them and haven't been paying attention due to other work
priorities.

Thanks!
Brian

On Wed, Dec 10, 2014 at 10:01 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Hi all,

 I have now merged Janmejay's awesome mmnormalize improvements. It is
 currently available in the master-mmnormalize branch. It would be great if
 some folks could try out that branch and provide feedack. Note that in
 order to compile, the git master version of liblognorm is required.

 Right now, one of the new testbench tests fails for me. If we can get this
 fixed, and nobody reports bad things, I plan to merge it around Friday into
 the master-candidate branch, from where it is expected to migrate into
 master early next week.

 Thanks,
 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] cee json + mmsequence

[Brian Knox]

So two questions then:

1.  In the case of using mmjsonparse on messages where we do not know ahead
of time what all the. json attributes will be, is there a var for a place
in the tree where we can get all attributes that were in the cee message?

2. If mmsequence is being deprecated, is there a pointer to how to
increment global variables to provide a sequence?

3. This morning I remembered the mmcount module - is this also being
deprecated in favor of global variables?

Brian

On Wed, Dec 3, 2014 at 3:52 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2014-12-02 21:59 GMT+01:00 Brian Knox bk...@digitalocean.com:

  We have a case where we are receiving messages in @cee: format, and would
  like to add a sequence variable generated via mmsequence to the message.
 
  Before I spend too much time digging into this I wanted to ask if, using
  some template hackery and json subtrees, this is something that sounds
  possible - and if maybe someone else was already doing it.
 
 
 David has answered the real question, just let me add that mmsequence was a
 work-around for v7 where global variables were not available. It is now
 deprecated and will probably removed in a couple of month.

 I'll check the doc if it tells this clearly enough.

 Rainer

  Thanks!
  Brian
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] cee json + mmsequence

[Brian Knox]

er, three questions ;)

On Wed, Dec 3, 2014 at 6:04 AM, Brian Knox bk...@digitalocean.com wrote:

 So two questions then:

 1.  In the case of using mmjsonparse on messages where we do not know
 ahead of time what all the. json attributes will be, is there a var for a
 place in the tree where we can get all attributes that were in the cee
 message?

 2. If mmsequence is being deprecated, is there a pointer to how to
 increment global variables to provide a sequence?

 3. This morning I remembered the mmcount module - is this also being
 deprecated in favor of global variables?

 Brian

 On Wed, Dec 3, 2014 at 3:52 AM, Rainer Gerhards rgerha...@hq.adiscon.com
 wrote:

 2014-12-02 21:59 GMT+01:00 Brian Knox bk...@digitalocean.com:

  We have a case where we are receiving messages in @cee: format, and
 would
  like to add a sequence variable generated via mmsequence to the message.
 
  Before I spend too much time digging into this I wanted to ask if, using
  some template hackery and json subtrees, this is something that sounds
  possible - and if maybe someone else was already doing it.
 
 
 David has answered the real question, just let me add that mmsequence was
 a
 work-around for v7 where global variables were not available. It is now
 deprecated and will probably removed in a couple of month.

 I'll check the doc if it tells this clearly enough.

 Rainer

  Thanks!
  Brian
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] cee json + mmsequence

[Brian Knox]

We have a case where we are receiving messages in @cee: format, and would
like to add a sequence variable generated via mmsequence to the message.

Before I spend too much time digging into this I wanted to ask if, using
some template hackery and json subtrees, this is something that sounds
possible - and if maybe someone else was already doing it.

Thanks!
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] cee json + mmsequence

[Brian Knox]

The other option we're discussing is using a template on a ruleset that is
guaranteed to contain only @cee formatted messages, and using some trickery
with a list template and position.relativetoend to chop the message so we
can inject the json attribute via a constant / property pair - but we
wanted to see if there was a cleaner way of doing it first!

Brian

On Tue, Dec 2, 2014 at 3:59 PM, Brian Knox bk...@digitalocean.com wrote:

 We have a case where we are receiving messages in @cee: format, and would
 like to add a sequence variable generated via mmsequence to the message.

 Before I spend too much time digging into this I wanted to ask if, using
 some template hackery and json subtrees, this is something that sounds
 possible - and if maybe someone else was already doing it.

 Thanks!
 Brian

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] ruleset with only stop

[Brian Knox]

Nice!  Thanks Rainer!

Brian

On Thu, Nov 20, 2014 at 12:28 PM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2014-11-20 15:59 GMT+01:00 Rainer Gerhards rgerha...@hq.adiscon.com:

  As it looks, this does the trick:
 
 
 
 https://github.com/rsyslog/rsyslog/commit/4cc10e7434d21f98e5cacd542edb834f894db620
 
  It's currently in master-candidate and will migrate to master when the
  testruns look good (the manual one did).
 
  Totally empty configs are still not tolerated. I'll see if I can add an
  option for that.
 
 
 I have now extended the -N option to support -N3, which can be used for
 checking include files:


 https://github.com/rsyslog/rsyslog/commit/a75eb98c058665d82f168f834b392683b30e001d

 It does not require the existence of actions.

 Rainer

  Rainer
 
  2014-11-19 20:07 GMT+01:00 Brian Knox bk...@digitalocean.com:
 
  Thanks!
 
  On Wed, Nov 19, 2014 at 1:59 PM, Rainer Gerhards 
  rgerha...@hq.adiscon.com
  wrote:
 
   Ok then I need to see how I  can fiddle that into the grammar ;)
  
   Sent from phone, thus brief.
   Am 19.11.2014 19:56 schrieb David Lang da...@lang.hm:
  
On Wed, 19 Nov 2014, Rainer Gerhards wrote:
   
 2014-11-19 16:40 GMT+01:00 Brian Knox bk...@digitalocean.com:
   
 Ok - perhaps we have accidently conflated two problems:
   
1) An empty ruleset
2) A ruleset with only stop
   
this will pass validation:
   

ruleset(name=foo) {
stop
}
*.* /var/log/test
call foo

   
If the ruleset is empty, however, it will not:
   

ruleset(name=foo) {
}
*.* /var/log/test
call foo

   
rsyslogd: version 8.5.0, config validation run (level 1), master
  config
./test.conf
rsyslogd: error during parsing file ./test.conf, on or before line
  2:
syntax error on token '}' [try http://www.rsyslog.com/e/2207 ]
rsyslogd: CONFIG ERROR: could not interpret master config file
'./test.conf'. [try http://www.rsyslog.com/e/2207 ]
rsyslogd: run failed with error -2207 (see rsyslog.h or try
http://www.rsyslog.com/e/2207 to learn what that number means)
   
   
 OK, that's a different question. Is the consensus we need to
  support
this
as well?
   
   
I think so, with a warning reported at startup or something like
 that.
   
David Lang
   
 Rainer
   
   
 Brian
   
   
   
On Wed, Nov 19, 2014 at 10:35 AM, Brian Knox 
  bk...@digitalocean.com
wrote:
   
 For verifying the problem I ran rsyslog -N1 -f against just the
  subset
of
the config, if I recall correctly.  I believe my coworker had the
  same
issue with the full config that definitely had actions in it -
 but
   I'll
   
ask
   
him to reproduce with the full configuration.  Thanks!
   
Brian
   
On Wed, Nov 19, 2014 at 10:13 AM, Rainer Gerhards 
rgerha...@hq.adiscon.com wrote:
   
 Brian,
   
I just revisited this problem report. I have now taken a look at
  the
   
code.
   
The error message actually tells you that there is no action
 inside
   the
*entire config*, not just an empty ruleset. Can you confirm
 there
  was
nothing else in the config? If not, can you send me the config,
 so
   that
   
I
   
can try to see what's going on.
   
I assume we agree that a totally action-less config is an error
 ;)
   
Rainer
   
2014-11-11 22:49 GMT+01:00 Brian Knox bk...@digitalocean.com:
   
 If was able to use an empty ruleset, a warning resulting from
  that
   
wouldn't
   
bother me at all.
   
Brian
   
On Tue, Nov 11, 2014 at 4:25 PM, David Lang da...@lang.hm
  wrote:
   
 On Tue, 11 Nov 2014, Rainer Gerhards wrote:
   
 2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:
   
   
 On Tue, 11 Nov 2014, Brian Knox wrote:
   
   
 Rainer,
   
   
I agree that an empty ruleset is an oddity.  In our case,
 the
   
short
   
answer
is that we are generating configurations from templates
 using
   
chef,
   
and
   
the
ability to have a ruleset that simply discards would make
  part
   of
   
that
   
process much simpler for us.
   
It is admittedly an edge case.
   
   
 It's an edge case, but I think it's one that should be
   supported
   
if
   
possible.
   
automated config generation is very useful, and being able
 to
   
group
   
rules
   
into rulesets and call them with the calling function not
 having
   
any
   
idea
   
of what is going to happen with the logs at that point is very
   
useful,
   
being able to have a high level config split the logs up and
 call
different
rulesets on different logs without having to worry if the
  ruleset
   
does
   
something with the logs or just throws them away is _very_

Re: [rsyslog] ruleset with only stop

[Brian Knox]

For verifying the problem I ran rsyslog -N1 -f against just the subset of
the config, if I recall correctly.  I believe my coworker had the same
issue with the full config that definitely had actions in it - but I'll ask
him to reproduce with the full configuration.  Thanks!

Brian

On Wed, Nov 19, 2014 at 10:13 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Brian,

 I just revisited this problem report. I have now taken a look at the code.
 The error message actually tells you that there is no action inside the
 *entire config*, not just an empty ruleset. Can you confirm there was
 nothing else in the config? If not, can you send me the config, so that I
 can try to see what's going on.

 I assume we agree that a totally action-less config is an error ;)

 Rainer

 2014-11-11 22:49 GMT+01:00 Brian Knox bk...@digitalocean.com:

  If was able to use an empty ruleset, a warning resulting from that
 wouldn't
  bother me at all.
 
  Brian
 
  On Tue, Nov 11, 2014 at 4:25 PM, David Lang da...@lang.hm wrote:
 
   On Tue, 11 Nov 2014, Rainer Gerhards wrote:
  
2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:
  
On Tue, 11 Nov 2014, Brian Knox wrote:
  
Rainer,
  
  
   I agree that an empty ruleset is an oddity.  In our case, the short
   answer
   is that we are generating configurations from templates using chef,
  and
   the
   ability to have a ruleset that simply discards would make part of
 that
   process much simpler for us.
  
   It is admittedly an edge case.
  
  
   It's an edge case, but I think it's one that should be supported if
   possible.
  
   automated config generation is very useful, and being able to group
  rules
   into rulesets and call them with the calling function not having any
  idea
   of what is going to happen with the logs at that point is very
 useful,
   being able to have a high level config split the logs up and call
   different
   rulesets on different logs without having to worry if the ruleset
 does
   something with the logs or just throws them away is _very_ useful.
  
   So it is a corner case, but I think it's a valuable one to support.
  
  
ack
  
  
I would suggest that at config load time, that this is an odd enough
   corner case that it's worth logging a ruleset X can't do anything
 with
   the
   logs message, not just for the case where the only action is to
 throw
  it
   away, but also for the cases where the conditions in a ruleset cannot
   possibly match any log message (if priority = info then *.crit also
   cannot
   match anything for example)
  
  
Let's start with what we have on the table. I think it is best to
 add
  a
   ruleset parameter permitEmpty=on. That way, we don't generate
   error/warning messages when the user is aware of what he is doing. In
  any
   manual case (without config gen), I'd still say that's an error
   indication.
  
  
   I think that this is a sufficently corner case that I'm not sure it's
   worth the extra complexity to silence the warning. I think that people
  who
   do this intentionally can just ignore the log message.
  
On the topic of no filter matches. That's quite complex, as you would
  need
   to evaluate all the filters and possible conditions. Not sure if it
 can
   even reliably done. Am I overlooking something?
  
  
   I am not saying that it should try to catch every possible case, but I
  was
   thinking that the configuration optimization step would optomize away
   some impossible combinations, and that could result in an empty
 ruleset.
  
   David Lang
  
   ___
   rsyslog mailing list
   http://lists.adiscon.net/mailman/listinfo/rsyslog
   http://www.rsyslog.com/professional-services/
   What's up with rsyslog? Follow https://twitter.com/rgerhards
   NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
 myriad
   of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
   DON'T LIKE THAT.
  
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https

Re: [rsyslog] ruleset with only stop

[Brian Knox]

Ok - perhaps we have accidently conflated two problems:

1) An empty ruleset
2) A ruleset with only stop

this will pass validation:


ruleset(name=foo) {
stop
}
*.* /var/log/test
call foo


If the ruleset is empty, however, it will not:


ruleset(name=foo) {
}
*.* /var/log/test
call foo


rsyslogd: version 8.5.0, config validation run (level 1), master config
./test.conf
rsyslogd: error during parsing file ./test.conf, on or before line 2:
syntax error on token '}' [try http://www.rsyslog.com/e/2207 ]
rsyslogd: CONFIG ERROR: could not interpret master config file
'./test.conf'. [try http://www.rsyslog.com/e/2207 ]
rsyslogd: run failed with error -2207 (see rsyslog.h or try
http://www.rsyslog.com/e/2207 to learn what that number means)

Brian



On Wed, Nov 19, 2014 at 10:35 AM, Brian Knox bk...@digitalocean.com wrote:

 For verifying the problem I ran rsyslog -N1 -f against just the subset of
 the config, if I recall correctly.  I believe my coworker had the same
 issue with the full config that definitely had actions in it - but I'll ask
 him to reproduce with the full configuration.  Thanks!

 Brian

 On Wed, Nov 19, 2014 at 10:13 AM, Rainer Gerhards 
 rgerha...@hq.adiscon.com wrote:

 Brian,

 I just revisited this problem report. I have now taken a look at the code.
 The error message actually tells you that there is no action inside the
 *entire config*, not just an empty ruleset. Can you confirm there was
 nothing else in the config? If not, can you send me the config, so that I
 can try to see what's going on.

 I assume we agree that a totally action-less config is an error ;)

 Rainer

 2014-11-11 22:49 GMT+01:00 Brian Knox bk...@digitalocean.com:

  If was able to use an empty ruleset, a warning resulting from that
 wouldn't
  bother me at all.
 
  Brian
 
  On Tue, Nov 11, 2014 at 4:25 PM, David Lang da...@lang.hm wrote:
 
   On Tue, 11 Nov 2014, Rainer Gerhards wrote:
  
2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:
  
On Tue, 11 Nov 2014, Brian Knox wrote:
  
Rainer,
  
  
   I agree that an empty ruleset is an oddity.  In our case, the short
   answer
   is that we are generating configurations from templates using chef,
  and
   the
   ability to have a ruleset that simply discards would make part of
 that
   process much simpler for us.
  
   It is admittedly an edge case.
  
  
   It's an edge case, but I think it's one that should be supported if
   possible.
  
   automated config generation is very useful, and being able to group
  rules
   into rulesets and call them with the calling function not having any
  idea
   of what is going to happen with the logs at that point is very
 useful,
   being able to have a high level config split the logs up and call
   different
   rulesets on different logs without having to worry if the ruleset
 does
   something with the logs or just throws them away is _very_ useful.
  
   So it is a corner case, but I think it's a valuable one to support.
  
  
ack
  
  
I would suggest that at config load time, that this is an odd enough
   corner case that it's worth logging a ruleset X can't do anything
 with
   the
   logs message, not just for the case where the only action is to
 throw
  it
   away, but also for the cases where the conditions in a ruleset
 cannot
   possibly match any log message (if priority = info then *.crit also
   cannot
   match anything for example)
  
  
Let's start with what we have on the table. I think it is best to
 add
  a
   ruleset parameter permitEmpty=on. That way, we don't generate
   error/warning messages when the user is aware of what he is doing. In
  any
   manual case (without config gen), I'd still say that's an error
   indication.
  
  
   I think that this is a sufficently corner case that I'm not sure it's
   worth the extra complexity to silence the warning. I think that people
  who
   do this intentionally can just ignore the log message.
  
On the topic of no filter matches. That's quite complex, as you would
  need
   to evaluate all the filters and possible conditions. Not sure if it
 can
   even reliably done. Am I overlooking something?
  
  
   I am not saying that it should try to catch every possible case, but I
  was
   thinking that the configuration optimization step would optomize
 away
   some impossible combinations, and that could result in an empty
 ruleset.
  
   David Lang
  
   ___
   rsyslog mailing list
   http://lists.adiscon.net/mailman/listinfo/rsyslog
   http://www.rsyslog.com/professional-services/
   What's up with rsyslog? Follow https://twitter.com/rgerhards
   NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
 myriad
   of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
   DON'T LIKE

Re: [rsyslog] ruleset with only stop

[Brian Knox]

Thanks!

On Wed, Nov 19, 2014 at 1:59 PM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Ok then I need to see how I  can fiddle that into the grammar ;)

 Sent from phone, thus brief.
 Am 19.11.2014 19:56 schrieb David Lang da...@lang.hm:

  On Wed, 19 Nov 2014, Rainer Gerhards wrote:
 
   2014-11-19 16:40 GMT+01:00 Brian Knox bk...@digitalocean.com:
 
   Ok - perhaps we have accidently conflated two problems:
 
  1) An empty ruleset
  2) A ruleset with only stop
 
  this will pass validation:
 
  
  ruleset(name=foo) {
  stop
  }
  *.* /var/log/test
  call foo
  
 
  If the ruleset is empty, however, it will not:
 
  
  ruleset(name=foo) {
  }
  *.* /var/log/test
  call foo
  
 
  rsyslogd: version 8.5.0, config validation run (level 1), master config
  ./test.conf
  rsyslogd: error during parsing file ./test.conf, on or before line 2:
  syntax error on token '}' [try http://www.rsyslog.com/e/2207 ]
  rsyslogd: CONFIG ERROR: could not interpret master config file
  './test.conf'. [try http://www.rsyslog.com/e/2207 ]
  rsyslogd: run failed with error -2207 (see rsyslog.h or try
  http://www.rsyslog.com/e/2207 to learn what that number means)
 
 
   OK, that's a different question. Is the consensus we need to support
  this
  as well?
 
 
  I think so, with a warning reported at startup or something like that.
 
  David Lang
 
   Rainer
 
 
   Brian
 
 
 
  On Wed, Nov 19, 2014 at 10:35 AM, Brian Knox bk...@digitalocean.com
  wrote:
 
   For verifying the problem I ran rsyslog -N1 -f against just the subset
  of
  the config, if I recall correctly.  I believe my coworker had the same
  issue with the full config that definitely had actions in it - but
 I'll
 
  ask
 
  him to reproduce with the full configuration.  Thanks!
 
  Brian
 
  On Wed, Nov 19, 2014 at 10:13 AM, Rainer Gerhards 
  rgerha...@hq.adiscon.com wrote:
 
   Brian,
 
  I just revisited this problem report. I have now taken a look at the
 
  code.
 
  The error message actually tells you that there is no action inside
 the
  *entire config*, not just an empty ruleset. Can you confirm there was
  nothing else in the config? If not, can you send me the config, so
 that
 
  I
 
  can try to see what's going on.
 
  I assume we agree that a totally action-less config is an error ;)
 
  Rainer
 
  2014-11-11 22:49 GMT+01:00 Brian Knox bk...@digitalocean.com:
 
   If was able to use an empty ruleset, a warning resulting from that
 
  wouldn't
 
  bother me at all.
 
  Brian
 
  On Tue, Nov 11, 2014 at 4:25 PM, David Lang da...@lang.hm wrote:
 
   On Tue, 11 Nov 2014, Rainer Gerhards wrote:
 
   2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:
 
 
   On Tue, 11 Nov 2014, Brian Knox wrote:
 
 
   Rainer,
 
 
  I agree that an empty ruleset is an oddity.  In our case, the
 
  short
 
  answer
  is that we are generating configurations from templates using
 
  chef,
 
  and
 
  the
  ability to have a ruleset that simply discards would make part
 of
 
  that
 
  process much simpler for us.
 
  It is admittedly an edge case.
 
 
   It's an edge case, but I think it's one that should be
 supported
 
  if
 
  possible.
 
  automated config generation is very useful, and being able to
 
  group
 
  rules
 
  into rulesets and call them with the calling function not having
 
  any
 
  idea
 
  of what is going to happen with the logs at that point is very
 
  useful,
 
  being able to have a high level config split the logs up and call
  different
  rulesets on different logs without having to worry if the ruleset
 
  does
 
  something with the logs or just throws them away is _very_ useful.
 
  So it is a corner case, but I think it's a valuable one to
 
  support.
 
 
 
   ack
 
 
 
   I would suggest that at config load time, that this is an odd
 
  enough
 
  corner case that it's worth logging a ruleset X can't do anything
 
  with
 
  the
  logs message, not just for the case where the only action is to
 
  throw
 
  it
 
  away, but also for the cases where the conditions in a ruleset
 
  cannot
 
  possibly match any log message (if priority = info then *.crit
 
  also
 
  cannot
  match anything for example)
 
 
   Let's start with what we have on the table. I think it is best
 to
 
  add
 
  a
 
  ruleset parameter permitEmpty=on. That way, we don't generate
  error/warning messages when the user is aware of what he is doing.
 
  In
 
  any
 
  manual case (without config gen), I'd still say that's an error
  indication.
 
 
  I think that this is a sufficently corner case that I'm not sure
 
  it's
 
  worth the extra complexity to silence the warning. I think that
 
  people
 
  who
 
  do this intentionally can just ignore the log message.
 
   On the topic of no filter matches. That's quite complex, as you
 
  would
 
  need
 
  to evaluate all the filters and possible conditions

Re: [rsyslog] Monitor rsyslog performance

[Brian Knox]

I have a service I wrote that we use, that accepts impstats output from
rsyslog, calculates various metrics from them and can forward those metrics
to opentsdb, graphite, etc - I could check into open sourcing it if anyone
else might have a use for such a thing.

Brian

On Tue, Nov 18, 2014 at 1:17 PM, Nathan Brown nbrown...@gmail.com wrote:

 Same here, collectd tail and a specific file for the stats output

 rsyslog configuration:

 module(load=impstats interval=10 severity=7 format=json
 log.file=/var/log/rsyslog_stats.log)

 collectd configuration:

 https://gist.github.com/nbrownus/7a8fa65e644d4c371b3b

 We use a specific file to avoid collectd having to tail everything going
 through rsyslog

 On Tue, Nov 18, 2014 at 8:17 AM, singh.janmejay singh.janme...@gmail.com
 wrote:

  Collectd tail plugin has been working well for me with impstats output.
 
  --
  Regards,
  Janmejay
 
  PS: Please blame the typos in this mail on my phone's uncivilized soft
  keyboard sporting it's not-so-smart-assist technology.
 
  On Nov 18, 2014 9:36 PM, Boylan, James james.boy...@orbitz.com
 wrote:
 
   I also have a python application I made for parsing the impstats file
   output and submitting them to graphite.
  
   I'm going to be working on daemonizing the utility and documenting how
   best to configure to use it. Please feel free to look at it and open
  issues
   if you have suggestions of features you'd like to see.
  
   The benefit of this setup is that it allows the parsing of the impstats
   based entirely on the names you have assigned to the various actions,
   queues and rulesets.
  
   https://github.com/Ralnoc/rsyslog-statcollector
  
   -- James
   
   From: rsyslog-boun...@lists.adiscon.com 
  rsyslog-boun...@lists.adiscon.com
   on behalf of Michael Hart michael.h...@arcticwolf.com
   Sent: Tuesday, November 18, 2014 9:28 AM
   To: rsyslog-users; Damian
   Subject: Re: [rsyslog] Monitor rsyslog performance
  
   I have rsyslog configured with impstats, forwarding to statsd/graphite,
  so
   I can graph the metrics and monitor them (I have Nagios pulling metrics
   from graphite).
  
   I keep meaning to put a blog post together to document this properly,
 but
   here’s a config snippet that gets you the basics. The hard part is
   figuring out which queues you want monitored. I’ve only got “main Q”
   showing here for brevity but I have a lot more defined.
  
   snip
   module(load=impstats interval=10 severity=7 format=cee”)
   module(load=mmjsonparse”)
  
   #json format: {name:main
   Q,size:25,enqueued:32,full:0,discarded.full:0,discarded.nf
   :0,ma
   xqsize:25}
   template(name=mainQTemplate type=list) {
   constant(value=rsyslog.myhost_example_com.main_q.size:)
   property(name=$!size)
   constant(value=|g\n)
   constant(value=rsyslog.myhost_example_com.main_q.enqueued:)
   property(name=$!enqueued)
   constant(value=|c|@10\n)
   constant(value=rsyslog.myhost_example_com.main_q.discarded.full:)
   property(name=$!discarded.full)
   constant(value=|c|@10\n)
   constant(value=rsyslog.myhost_example_com.main_q.discarded.nf:)
   property(name=$!discarded.nf)
   constant(value=|c|@10\n)
   constant(value=“rsyslog.myhost_example_com.main_q.maxqsize:)
   property(name=$!maxqsize)
   constant(value=|g\n)
   }
  
  
   if $syslogtag contains rsyslogd-pstats then {
   action(type=mmjsonparse”)
   #write to file here for debugging.
   action(type=“omfile” file=“/var/log/stats.log”)
   if $!name == main Q then {
   action(type=omfwd Target=127.0.0.1 Protocol=udp
 Port=8125
   template=mainQTemplate”)
   }
   stop
   }
  
   /snip
  
   There is still some wonkiness in the enqueued stat as occasionally it
 has
   an absolutely massive unrealistic spike, I have never tracked down why
 it
   does that, but this should give you a start.
  
   Cheers
   mike
  
   --
   Michael Hart
   Arctic Wolf Networks
   M: 226-388-4773
  
  
  
  
  
  
  
  
   On 2014-11-18, 15:14, Dave Caplinger davecaplin...@solutionary.com
   wrote:
  
   Absolutely.  Rsyslog has statistics counters via the impstats module;
  you
   can process the log lines it generates to determine the health of the
   rsyslog instance, including individual queues, drop rates, forwarding
   rates, etc.
   
   See:
   
   http://www.rsyslog.com/rsyslog-statistic-counter/
   http://www.rsyslog.com/how-to-use-impstats/
   
   --
   Dave Caplinger, Director of Architecture | Ph: (402) 361-3063 |
   Solutionary — An NTT Group Security Company
   
On Nov 18, 2014, at 6:46 AM, Damian damoskeet-r...@yahoo.com
 wrote:
   
Hi,
I'm trying to determine whether it's possible to monitor the health
 of
   an rsyslog daemon running as a forwarder.
ie. If I'm running it as a component in a logging service, how do I
   check the event rates, or know it's not losing events or queuing
   incoming data.  Are there any 

Re: [rsyslog] tls + tcp input

[Brian Knox]

It looks like the parameters that control the auth mode are global:

http://www.rsyslog.com/doc/master/configuration/modules/imtcp.html

I could be misunderstanding the documentation.

Brian

On Wed, Nov 12, 2014 at 8:02 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Iirc it is possible. Isn't that an input parameter?

 Sent from phone, thus brief.
 Am 11.11.2014 23:11 schrieb Brian Knox bk...@digitalocean.com:

  Is it possible to specify TLS on a per input basis for the TCP input, or
 is
  it all or nothing?
 
  Brian
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] liblognorm

[Brian Knox]

I'm trying to build the latest (1.0.1) version of liblognorm and getting an
error during configure - I'm wondering if anyone else has run into this.
I'm building on Ubuntu 14.04LTS:

checking for JSON_C... yes
checking for sphinx-build... no
checking for sphinx-build3... no
checking for sphinx-build2... no
./configure: line 421: test: install: integer expression expected
configure: error: it
./configure: line 310: return: install: numeric argument required
./configure: line 320: exit: install: numeric argument required


Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm

[Brian Knox]

Interesting - 1.0.0 builds fine - and 1.0.1 builds fine if I use the
configure script from 1.0.0.

Brian

On Wed, Nov 12, 2014 at 9:06 AM, Brian Knox bk...@digitalocean.com wrote:

 I'm trying to build the latest (1.0.1) version of liblognorm and getting
 an error during configure - I'm wondering if anyone else has run into
 this.  I'm building on Ubuntu 14.04LTS:

 checking for JSON_C... yes
 checking for sphinx-build... no
 checking for sphinx-build3... no
 checking for sphinx-build2... no
 ./configure: line 421: test: install: integer expression expected
 configure: error: it
 ./configure: line 310: return: install: numeric argument required
 ./configure: line 320: exit: install: numeric argument required


 Brian

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] tls + tcp input

[Brian Knox]

Sneaky - that makes sense.  :)

On Wed, Nov 12, 2014 at 9:17 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2014-11-12 14:25 GMT+01:00 Brian Knox bk...@digitalocean.com:

  It looks like the parameters that control the auth mode are global:
 
 
 ah, you are right. I just checked the code. I thought I had changed that.
 Anyways... the trick to do then is to use imtcp for tls and imptcp for
 plain.

 HTH
 Rainer


  http://www.rsyslog.com/doc/master/configuration/modules/imtcp.html
 
  I could be misunderstanding the documentation.
 
  Brian
 
  On Wed, Nov 12, 2014 at 8:02 AM, Rainer Gerhards 
 rgerha...@hq.adiscon.com
  
  wrote:
 
   Iirc it is possible. Isn't that an input parameter?
  
   Sent from phone, thus brief.
   Am 11.11.2014 23:11 schrieb Brian Knox bk...@digitalocean.com:
  
Is it possible to specify TLS on a per input basis for the TCP input,
  or
   is
it all or nothing?
   
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
  myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
 you
DON'T LIKE THAT.
   
   ___
   rsyslog mailing list
   http://lists.adiscon.net/mailman/listinfo/rsyslog
   http://www.rsyslog.com/professional-services/
   What's up with rsyslog? Follow https://twitter.com/rgerhards
   NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
 myriad
   of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
   DON'T LIKE THAT.
  
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] ruleset with only stop

[Brian Knox]

Rainer,

I agree that an empty ruleset is an oddity.  In our case, the short answer
is that we are generating configurations from templates using chef, and the
ability to have a ruleset that simply discards would make part of that
process much simpler for us.

It is admittedly an edge case.

Brian



On Tue, Nov 11, 2014 at 4:06 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:

  Today I noticed a ruleset with only stop as it's action will fail to
  parse with rsyslog 8.4, but the same rule with a ~ will pass.
 
  ruleset(name=testme) {
  *.* ~
  }
 
  bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
  rsyslogd: version 8.5.0, config validation run (level 1), master config
  ./test.conf
  rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
  statement instead [try http://www.rsyslog.com/e/2307 ]
 
 
  Changing to stop :
  ruleset(name=testme) {
  stop
  }
 
  bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
  rsyslogd: version 8.5.0, config validation run (level 1), master config
  ./test.conf
  rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs
 will
  run, but no output whatsoever is created. [try
  http://www.rsyslog.com/e/2103
  ]
  rsyslogd: run failed with error -2103 (see rsyslog.h or try
  http://www.rsyslog.com/e/2103 to learn what that number means)
 
  I have a situation where rules are being generated via templates in chef,
  and having a rule that just discards messages would actually be a useful
  thing to have, and ran into this.
 
  So my question is, should a rule that only calls a discard action be
  valid?  If so, is this a bug in the config parser?
 
 
 It's a little bit complex. The thing is that ~ actually *is* an action,
 whereas stop is a statement. When I wrote that checking code, I never
 envisioned that an empty ruleset could be useful for any case (if there is
 just a stop inside it, it's practically empty, in that it simply does
 nothing). I think in most cases this really is a config error. Maybe I
 could add an permitEmpty parameter to the ruleset, which will then not
 emit that error message.

 To understand the whole picture: why do you need these empty rulesets?

 Rainer

 Brian
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] ruleset with only stop

[Brian Knox]

+1 for permitEmpty=on - it would definitely simplify our lives.

Brian

On Tue, Nov 11, 2014 at 11:40 AM, Boylan, James james.boy...@orbitz.com
wrote:

 I think that the permitEmpty=on field is a reasonable starting place. I
 have a config management app that I use with rsyslog that this field would
 help significantly with.

 -- James
 --- Sent from my mobile phone ---

 - Reply message -
 From: Rainer Gerhards rgerha...@hq.adiscon.com
 To: rsyslog-users rsyslog@lists.adiscon.com
 Subject: [rsyslog] ruleset with only stop
 Date: Tue, Nov 11, 2014 10:29 AM

 2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:

  On Tue, 11 Nov 2014, Brian Knox wrote:
 
   Rainer,
 
  I agree that an empty ruleset is an oddity.  In our case, the short
 answer
  is that we are generating configurations from templates using chef, and
  the
  ability to have a ruleset that simply discards would make part of that
  process much simpler for us.
 
  It is admittedly an edge case.
 
 
  It's an edge case, but I think it's one that should be supported if
  possible.
 
  automated config generation is very useful, and being able to group rules
  into rulesets and call them with the calling function not having any idea
  of what is going to happen with the logs at that point is very useful,
  being able to have a high level config split the logs up and call
 different
  rulesets on different logs without having to worry if the ruleset does
  something with the logs or just throws them away is _very_ useful.
 
  So it is a corner case, but I think it's a valuable one to support.
 
 
 ack


  I would suggest that at config load time, that this is an odd enough
  corner case that it's worth logging a ruleset X can't do anything with
 the
  logs message, not just for the case where the only action is to throw it
  away, but also for the cases where the conditions in a ruleset cannot
  possibly match any log message (if priority = info then *.crit also
 cannot
  match anything for example)
 
 
 Let's start with what we have on the table. I think it is best to add a
 ruleset parameter permitEmpty=on. That way, we don't generate
 error/warning messages when the user is aware of what he is doing. In any
 manual case (without config gen), I'd still say that's an error indication.

 On the topic of no filter matches. That's quite complex, as you would need
 to evaluate all the filters and possible conditions. Not sure if it can
 even reliably done. Am I overlooking something?

 Rainer


  David Lang
 
 
   Brian
 
 
 
  On Tue, Nov 11, 2014 at 4:06 AM, Rainer Gerhards 
  rgerha...@hq.adiscon.com
  wrote:
 
   2014-11-10 16:23 GMT+01:00 Brian Knox bk...@digitalocean.com:
 
   Today I noticed a ruleset with only stop as it's action will fail to
  parse with rsyslog 8.4, but the same rule with a ~ will pass.
 
  ruleset(name=testme) {
  *.* ~
  }
 
  bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
  rsyslogd: version 8.5.0, config validation run (level 1), master
 config
  ./test.conf
  rsyslogd: warning: ~ action is deprecated, consider using the 'stop'
  statement instead [try http://www.rsyslog.com/e/2307 ]
 
 
  Changing to stop :
  ruleset(name=testme) {
  stop
  }
 
  bknox@seriamau:~$ rsyslogd -N1 -f ./test.conf
  rsyslogd: version 8.5.0, config validation run (level 1), master
 config
  ./test.conf
  rsyslogd: CONFIG ERROR: there are no active actions configured. Inputs
 
  will
 
  run, but no output whatsoever is created. [try
  http://www.rsyslog.com/e/2103
  ]
  rsyslogd: run failed with error -2103 (see rsyslog.h or try
  http://www.rsyslog.com/e/2103 to learn what that number means)
 
  I have a situation where rules are being generated via templates in
  chef,
  and having a rule that just discards messages would actually be a
 useful
  thing to have, and ran into this.
 
  So my question is, should a rule that only calls a discard action be
  valid?  If so, is this a bug in the config parser?
 
 
   It's a little bit complex. The thing is that ~ actually *is* an
  action,
  whereas stop is a statement. When I wrote that checking code, I never
  envisioned that an empty ruleset could be useful for any case (if there
  is
  just a stop inside it, it's practically empty, in that it simply does
  nothing). I think in most cases this really is a config error. Maybe I
  could add an permitEmpty parameter to the ruleset, which will then
 not
  emit that error message.
 
  To understand the whole picture: why do you need these empty rulesets?
 
  Rainer
 
  Brian
 
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
 myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
   ___
  rsyslog

Re: [rsyslog] ruleset with only stop

[Brian Knox]

If was able to use an empty ruleset, a warning resulting from that wouldn't
bother me at all.

Brian

On Tue, Nov 11, 2014 at 4:25 PM, David Lang da...@lang.hm wrote:

 On Tue, 11 Nov 2014, Rainer Gerhards wrote:

  2014-11-11 17:22 GMT+01:00 David Lang da...@lang.hm:

  On Tue, 11 Nov 2014, Brian Knox wrote:

  Rainer,


 I agree that an empty ruleset is an oddity.  In our case, the short
 answer
 is that we are generating configurations from templates using chef, and
 the
 ability to have a ruleset that simply discards would make part of that
 process much simpler for us.

 It is admittedly an edge case.


 It's an edge case, but I think it's one that should be supported if
 possible.

 automated config generation is very useful, and being able to group rules
 into rulesets and call them with the calling function not having any idea
 of what is going to happen with the logs at that point is very useful,
 being able to have a high level config split the logs up and call
 different
 rulesets on different logs without having to worry if the ruleset does
 something with the logs or just throws them away is _very_ useful.

 So it is a corner case, but I think it's a valuable one to support.


  ack


  I would suggest that at config load time, that this is an odd enough
 corner case that it's worth logging a ruleset X can't do anything with
 the
 logs message, not just for the case where the only action is to throw it
 away, but also for the cases where the conditions in a ruleset cannot
 possibly match any log message (if priority = info then *.crit also
 cannot
 match anything for example)


  Let's start with what we have on the table. I think it is best to add a
 ruleset parameter permitEmpty=on. That way, we don't generate
 error/warning messages when the user is aware of what he is doing. In any
 manual case (without config gen), I'd still say that's an error
 indication.


 I think that this is a sufficently corner case that I'm not sure it's
 worth the extra complexity to silence the warning. I think that people who
 do this intentionally can just ignore the log message.

  On the topic of no filter matches. That's quite complex, as you would need
 to evaluate all the filters and possible conditions. Not sure if it can
 even reliably done. Am I overlooking something?


 I am not saying that it should try to catch every possible case, but I was
 thinking that the configuration optimization step would optomize away
 some impossible combinations, and that could result in an empty ruleset.

 David Lang

 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] can someone lend me a hand on git procedure

[Brian Knox]

I'm in favor of the process being as simple as possible.  The more work and
thought that has to be put into dealing with pull requests, the more likely
either work will get slowed down, or someone will make a mistake.

I feel like if I want to squash, etc I can just do that on branches on my
own fork, and then send the cleaned up version back to you.  I don't feel
like you should have to deal with squashing my commits.  I also don't have
a problem personally with seeing the merge messages.

On Fri, Nov 7, 2014 at 6:39 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 2014-11-07 12:35 GMT+01:00 singh.janmejay singh.janme...@gmail.com:

  Rainer,
 
  Do we really need to squash? Why not just keep it simple and merge
 changes
  as they come? Its so much better for looking at exactly how/why things
 are
  the way they are.
 
  No rebase, no rewrites of history etc, just the simple commit and merge.
 
 
 well, I don't need all of that overhead. But from the other thread it
 looked like folks wanted it and nobody said anything else...

 Rainer

  --
  Regards,
  Janmejay
 
  PS: Please blame the typos in this mail on my phone's uncivilized soft
  keyboard sporting it's not-so-smart-assist technology.
 
  On Nov 7, 2014 4:57 PM, Rainer Gerhards rgerha...@hq.adiscon.com
  wrote:
 
   Hi all,
  
   based on recent discussion ([1] is a good entry point), it looks like
  there
   is consensus that feature-branch commits shall be squashed before
 merging
   them into master. This is a bit bad for me because in almost all cases
 I
   like the ability to see the interim steps that lead to a feature in
   question (for bisect, but also to better understand what was going
 on). I
   have also discussed this with my peers in Adiscon and they also prefer
  the
   way it currently is.
  
   To satisfy both requirements, we have now setup an internal git for
  Adiscon
   use. Our plan is to have a parallel adiscon-master branch inside that
  repo,
   which will contain every detail. Its master branch will mirror the
 public
   git and contain squashed commits.
  
   We now have contributions from Adiscon (including me) and others. Those
   from Adiscon will be done in feature branches, with detail commits and
 be
   merged into the adiscon-master branch (so that it contains all
 details).
   Then, I will squash the feature branch into a single commit and merge
  that
   into master. So far, so good.
  
   But now we also have non-Adiscon contributions. A current example is
 [2].
   One question is if they must be squashed as well? Let's assume this is
  not
   the case for whatever reason. So I merge them directly into master.
 Then,
   to keep my actual working tree up to date, I need to cherry-pick them
  into
   adiscon-master. This is where I am a bit hesitant, because of the
 manual
   action. I fear that the master and adiscon-master branches may begin to
   diverge, and be it through a simple mistake.
  
   So maybe it is better to merge pull requests into new feature branches,
  and
   then work as usual: merge feature branch into adiscon-master, squash
   feature branch, then merge it as single commit into master.
  
   To sum up: I would like to have two branches, the private one with all
   detail information, the public one minus those commits that are
  considered
   distracting. What is the best way to achieve this goal?
  
   Feedback appreciated,
   Rainer
  
   [1]
 http://lists.adiscon.net/pipermail/rsyslog/2014-November/038883.html
   [2] https://github.com/rsyslog/rsyslog/pull/147
   ___
   rsyslog mailing list
   http://lists.adiscon.net/mailman/listinfo/rsyslog
   http://www.rsyslog.com/professional-services/
   What's up with rsyslog? Follow https://twitter.com/rgerhards
   NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
 myriad
   of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
   DON'T LIKE THAT.
  
  ___
  rsyslog mailing list
  http://lists.adiscon.net/mailman/listinfo/rsyslog
  http://www.rsyslog.com/professional-services/
  What's up with rsyslog? Follow https://twitter.com/rgerhards
  NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
  of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
  DON'T LIKE THAT.
 
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE 

Re: [rsyslog] Feedback Request: do we still need -devel versions?

[Brian Knox]

Thanks for the schedule information!  I'll try to have the new rsyslog
plugins ready for the 2014-12-02 release.  The output plugin should be
finished this week and then I'll be starting in on the new input.

Brian

On Sat, Nov 1, 2014 at 6:20 AM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:

 Just FYI folks. I am now switching to a 6w cycle. Expect blog posts to come
 up.  The next release will be 8.6.0 stable, based on 8.5.0. I've also done
 a little bit of calendar lookup. I don't like to start a new release cycle
 method with an exception, and I don't do releases before the xmas holiday
 period. As such, we will have the following upcoming release dates:

 8.6.0: 2014-12-02
 8.6.1: 2015-01-13

 Just so that you know.

 Adiscon will probably post non-project, adiscon-specific interim releases
 to the Adiscon repositories. This is because we need to deliver
 fixesenhancements to support customers in package form (for the paying
 customers, it's not an option to build from source). Just think of the
 Adiscon packages in the same way now like you do about RH, Debian or Ubuntu
 packages. The Adiscon repositories will remain open to everyone.

 I also continue to setup more testbench machines.

 Rainer

 2014-10-31 12:56 GMT+01:00 singh.janmejay singh.janme...@gmail.com:

  +1 for 6w
 
  --
  Regards,
  Janmejay
 
  PS: Please blame the typos in this mail on my phone's uncivilized soft
  keyboard sporting it's not-so-smart-assist technology.
 
  On Oct 31, 2014 4:07 PM, Rainer Gerhards rgerha...@hq.adiscon.com
  wrote:
 
   2014-10-31 11:01 GMT+01:00 David Lang da...@lang.hm:
  
On Fri, 31 Oct 2014, Rainer Gerhards wrote:
   
 2014-10-31 0:38 GMT+01:00 David Lang da...@lang.hm:
   
 On Thu, 30 Oct 2014, Rainer Gerhards wrote:
   
   
   
+1 for a time-based release approach.
   
   
 I am not sure if David and you talked about the same thing. If I
understood
David correctly (please correct me if I am wrong), he says that we
release
versions (88 to avoid confusion with existing versions), e.g.
  88.6.1,
88.6.2, 88.6.3, 88.6.x whenever they are ready. However, every 6
  month
we
would begin a new series, e.g. 88.7.1. From then on, only 88.7 is
updated.
   
   
I'm actually thinking of the kernel model
   
every X months release 88.7, 88.8, 88.9, etc. If there are bugfixes
   that
need to go out between the X month releases, they become 88.7.1
  88.7.2
etc.
3-6 months seems to work fairly well for individual projects. In
   between
people can just compile from the master. I don't think we have
 enough
testing participation to go the -rcX route.
   
If there is a major (risky) change, it would justify an 89 release,
  but
that would end up being something like a re-write of the queue
 model
  or
other very intrusive (and therefor risky) change, not the ongoing
features,
modules, performance optimizations.
   
   
 mmhh... isn't that -except for the timing- what we do with the
  current
-devel/-stable just in other terms? I agree that terms are important
  but
should we than name the master branch releases as stable and the
  monthly
as
old stable. Also, I have the impression that with the kernel
 almost
everyone uses the bi-annually releases (in our words the -stable)
 and
   not
the master.
   
If I am not wrong, that model would probably result in the same
  problem,
that is I develop new things in master branch, but everyone begins
 to
test them when it is rolled into the bi-annually releases.
   
   
The releases don't need to be bi-annual, there are advantages to
  shorter
cycles.
   
People do need some stability in what's shipped, so they really
 aren't
going to be running things from git. So the question is, how quickly
  can
you release things without annoying people too much?
   
for the kernel, they are making new releases about every 2.5-3
 months.
Firefox is making releases about every 6 weeks. I don't remember what
Chrome's cycle is like, but it's also rapid.
   
   
   So how about every 6 weeks for rsyslog? On that cycle, bug reports
 would
   still hit me with a relatively fresh idea of what I changed.
  
   Rainer
  
  
People are going to start off being afraid of new releases, but they
  seem
to accept them if they don't have frequent regressions. They also
 seem
   far
more afraid of changing major versions than minor versions (and even
   there,
firefox and chrome are getting people to accept that)
   
   
Today we have the master tree, -devel releases, -stable releases, and
bugfix releases.
   
I'm saying that we would have the master tree, -stable releases, and
occasional bugfix releases (the bugfixes would only fix regressions
  that
were missed)
   
   
David Lang
___
rsyslog mailing list

Re: [rsyslog] Feedback Request: do we still need -devel versions?

[Brian Knox]

Rainer - for zeromq we break things up into previous stable releases, and
then the master of the git repo.  We don't allow breaking changes on
master - so I tend to develop against master and even use snapshots of git
master in production projects.  It was a bit anxiety inducing at first but
really, it's caused very few problems and bugs are found almost immediately
unless they are really nasty ones. It's been working well.



On Wed, Oct 29, 2014 at 6:20 AM, Boylan, James james.boy...@orbitz.com
wrote:

 A lot of how this works depends on how many people are contributing. With
 a lot of active contributors a common practice is to have a release branch
 and a development branch. It makes it cleaner from a commit history when
 you can squash many commits into a single one to push into the release
 branch. I have mixed feelings about the pros/cons of that process.

 Another method I've seen is that there is only one branch and when you
 feel that it has been tested thoroughly enough you merely tag the 'release
 commit' and generate your release tarfiles off that.

 Both of these methods have their positive and negative aspects. A lot of
 it depends on you development cycle and what fits best with your team in
 regards to working more efficiently.

 -- James
 --- Sent from my mobile phone ---

 - Reply message -
 From: Rainer Gerhards rgerha...@hq.adiscon.com
 To: rsyslog-users rsyslog@lists.adiscon.com
 Subject: [rsyslog] Feedback Request: do we still need -devel versions?
 Date: Wed, Oct 29, 2014 4:47 AM

 Hi all,

 it may sound strange, but I strongly think about dropping -devel versions
 and instead moving new features directly into the -stable branch.

 The reason is that almost nobody nowadays tries out the -devel versions.
 The past two years, I've always seen the same pattern: when I started a new
 -stable branch, a lot of bug reports immediately appeared - bugs that
 obviously were not detected because nobody used -devel. The really bad
 thing about this is that usually the feature causing the bug was
 implemented some month ago, so I do not have a clear memory what may be the
 root cause. Also, in a new stable branch there are many changes intermixed,
 which makes troubleshooting even harder.

 As such, I consider a policy change where we will support the current and
 previous stable release (right now that would be 8.4.2 and 8.4.1) and
 enhancements going directly into the -stable release. Actually, we would
 drop the -stable, -devel qualifiers, it would just be the rsyslog v8
 release.

 Let's consider the next version: changes would go into 8.4.3, but we would
 still support 8.4.2 in regard to questions. So if someone hits a regression
 with 8.4.3, he would need to go back to 8.4.2 until 8.4.4 is released.

 On the plus side, that would also mean new features would be more readily
 available, in contrast to the 3 to 8 month wait period we currently have
 for those that insist on stable versions.

 I am not sure, however, if we should release new versions more rapidly than
 we did with -stable versions. Technically, it makes sense, but many users
 don't like that (I know from past conversations).

 Comments appreciated.
 Rainer
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.
 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Plugin Questions

[Brian Knox]

Rainer - so we are saying that if the plugin is configured to connect the 
zeromq socket, we will allow multiple instance mode, and if it is configured to 
bind the zeromq socket, we will restrict to single instance mode?  This 
sounds reasonable to me.

This evening (GMT-5 evening) I'll add the configuration parameter for choosing 
between binding and connecting the socket to the new plugin code and get it up 
on github and we can go from there!

BRian

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer Gerhards
Sent: Monday, February 10, 2014 10:00 AM
To: rsyslog-users
Subject: Re: [rsyslog] Plugin Questions

Folks, with Brian's permission I reply to our conversation on the rsyslog 
mailing list. Brian is updating the 0mq plugins to v8 and we are discussing 
some design issues. I think this is probably of interest for some others, so I 
asked that we carry on on the list. Context is provided in the mail text 
below...

On Mon, Feb 10, 2014 at 12:52 PM, Brian Knox taote...@gmail.com wrote:

 In ZeroMQ, the main thing binding on the output plugin would allow is 
 easier load balancing in some cases.

 Let's say you have an omzmq output on tcp://127.0.0.1:, that you 
 bind it, and that it is a ZMQ_PUSH socket.

 If I now connect say.. 4 ZMQ_PULL sockets to the bound output socket, 
 then ZeroMQ will round robin load balance across the four connections.

 So, if you have some process that does some work that has a 
 significant cost to rsyslog output.. you bind an output, and connect 
 multiple workers to it to load balance.

 However from what you are saying, this will now keep us from having 
 multiple worker instances on the output.


OK, I think I finally understand. Let's see if we can nail this down with a 
simple solution. First let me repeat how I see the situation:

If you bind an output, than you expect that the output does not have a 
performance problem. The performance problem is on the receivers. So in this 
case a single instance of omzmq is considered fast enough to push messages into 
the queue. If so, we could effectively work with a single action worker 
instance, as described on slide 10 of this presentation:

http://www.slideshare.net/rainergerhards1/the-rsyslog-v8-engine-developers-view

It would just be that we do it because we want it that way and not because of 
we are unable to do it any other way. Even further, we could have a config 
parameter for the bind port. If given, we'd use a single instance, if not, 
then multiple. IN ANY CASE this will only affect the omzmq code.
Even in single instance mode, the rsyslog core engine would do message 
filtering, formating, etc on multiple workers -- just like it did in v7.
It's even a bit faster than in v7 because we have these optimized interface 
callbacks.

If I am right with my assumption, I'd say that's the way to go. I would be 
happy to help with the plumbing for this single instance mode, if you have 
you code on github or somewhere else I can pull it from, I'd happy to simply 
add it (so that you can review).

Rainer




 It's not a huge deal.. it just means to do that sort of pattern I 
 would need an external zmq proxy that rsyslog connects to, and then 
 the proxy would be responsible for work distribution outside of 
 zeromq.  I could write such a thing and release it open source (I'm 
 the original author and maintainer of the zproxy proxy code in czmq so I'm 
 very familiar with it).

 I have another idea for a solution that would work within the output 
 plugin itself, but it is more complex and I'd rather keep things 
 simple for the initial port to 8.

 My proposal at this point is:

 1) I finish the new omzmq output for 8.  It supports the latest 
 release of czmq.  I will only allow connects to keep things simple.  
 I will add in support for CURVE authentication and encryption ( 
 http://rfc.zeromq.org/spec:26).

 2) I create a new version of imzmq.  The refactor will include code 
 cleanup, and reducing the complexity of the options so that imzmq only 
 supports binds.  I add in support on the input side for CURVE as well.
  This lets use connect together rsyslog instances over zeromq, with 
 certificate based authentication and encryption.

 3) The new plugins will be imzmq and omzmq rather than imzmq3 and
 omzmq3.   We can deprecate the 3 versions of the plugins.

 How's that sound?
 Brian




 On Mon, Feb 10, 2014 at 5:56 AM, Rainer Gerhards 
 rgerha...@hq.adiscon.com
  wrote:


 On Sun, Feb 9, 2014 at 9:23 PM, Brian Knox taote...@gmail.com wrote:

 The reason I ask is - with zeromq, connections are stateless and 
 either side may bind or connect.  A transport address 
 (tcp://127.0.0.1:5556, ipc:///tmp/myipsock, etc) can be -connected- 
 to many times but only -bound- one time.  This effects how things like fan 
 out an fan in patterns work.

 I would like to allow -binds- on the omzmq side.  However, if each 
 thread spins up

Re: [rsyslog] liblognorm documentation

[Brian Knox]

Nice Pavel!  I've been working with the 1.0 release (playing outside of 
rsyslog) getting to know the changes you made.  I haven't done performance 
testing yet, but so far the API changes weren't too disruptive and it was 
pretty easy to figure out.

Brian

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Pavel Levshin
Sent: Monday, December 23, 2013 8:10 AM
To: rsyslog-users; logn...@lists.adiscon.com
Subject: [rsyslog] liblognorm documentation

Hello all.

I am trying to refresh liblognorm documentation, as a part of our documentation 
effort. Current results can be seen in my own GitHub repository; I will push 
them when I'll be sure they are reasonable.

Here is the repository: 
https://github.com/flicker581/liblognorm/tree/master-doc/doc

And here are generated and published docs: 
http://flicker581.github.io/liblognorm/

Feedbacks are welcome. Currently, I'm just trying all this framework (sphinx, 
autoconf/automake, github). Chances are that many things are done in a broken 
way.


--
Pavel Levshin

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is 
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our 
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Modules in other programming languages?

[Brian Knox]

I was thinking that omrest would be a module that made outbound http requests 
to send messages; not as something that waited for incoming http requests for 
messages.  So, in my mind it was something far more similar to the current 
elastic search module, which pushes messages out.

So - more of an outgoing http request to an external rest api, rather than 
something providing a rest api to make calls into.

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 16, 2013 1:57 PM
To: rsyslog-users
Subject: Re: [rsyslog] Modules in other programming languages?

On Mon, 16 Dec 2013, Brian Knox wrote:

 I believe the output module for elastic search might be a good place 
 to start looking for anyone interested in writing an omrest module?  
 If I recall correctly the elastic search output uses libcurl.

not really.

currently every om* module consists of code that is executed by a rsyslog 
worker thread that is passed a list of messages and acts on each message. 
omelasticsearch pushes messages via libcurl

for omrest you would need to change the entire paradigm of how an om* module 
would work.

Instead of being code invoked by a worker thread that's invoking many other
om* code as well on a given message, the omrelp module would need to listen for 
a connection from the outside, and when it receives a request, it would need to 
retrieve messages from the queue, and the worker threads would need to leave 
the messages on the queue.

You should see by now that this is a really ugly thing to talk about 
implementing. It's almost a complete re-write of the rsyslog core to support 
this.


The other approach is to have omrelp maintain it's own queue of messages and 
knowledge of who should be asking for messages, timeouts for messages that 
aren't asked for, etc. At that point, omrelp's interface to rsyslog is 
straightforward, it's just the omrelp queue and interface stuff that gets 
really 'interesting' and people who want omrelp should work on writing some 
code that will perform all the relp functions that you want to support but just 
accepts new messages on stdin (which would let it be driven by omprog for now), 
and then as you get it running and find it useful the input portion could be 
changed to make it into a 'real' rsyslog om* module. Depending on what you do 
for your queue, you may be able to use a different language to handle adding 
things to the queue and pulling things from the queue. This would make it easy 
to have some rsyslog C code that adds things to the queueu.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] People insight?

[Brian Knox]

I think the benefit of git is that there's really no draw back to the adiscon 
repo maintaining status as the official repo, while having an official 
presence on github.

When I was working on rsyslog patches before there was a github copy, I would 
just fork the adiscon repo to github for convenience, merge any changes from 
the adiscon repo to my fork as they came, and create a patch to send back when 
I was ready.

Similarly now if you prefer working entirely on github, you can fork the now 
github available version instead.

I don't see the benefit to anyone (inside or outside of adiscon) to deprecating 
the adiscon repo.

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Otis Gospodnetic
Sent: Tuesday, December 17, 2013 1:47 AM
To: rsyslog-users
Subject: Re: [rsyslog] People insight?

Hi,

On Mon, Dec 16, 2013 at 5:34 PM, David Lang da...@lang.hm wrote:

 The adiscon repo is _the_ authoritative repo.

 The github repo is the office repo on github


Ah, interesting that is not at all how I understood that OFFICIAL rsyslog 
repository on github on https://github.com/rsyslog/rsyslog . But maybe you are 
right, maybe Rainer really phrased it like this in order to say this is THE 
rsyslog repo *on Github*, all other rsyslog repos you see on Github and 
probably forks of this repo.  And the on Github part implies that this is 
the official rsyslog repo *ONLY* on Github and there are actually other 
non-Github repos and if so, I don't quite get why.
 Sure, I get the fear of Github disappearing, but while Github is here, 
wouldn't it be simpler to just think of that repo as THE official repo?
 Please see below.


 If github goes away, it doesn't take _the_ master repo with it.


Right, because the Adiscon one is in sync, or at least that's my understanding.
So if that is the case, isn't this Adiscon repo simply Adiscon's insurance in 
case Github disappears and not something Rsyslog contributors really need to 
know (or be confused by)?

Please read Rainer's blog post on the issue of why he doesn't want to be
 dependent on github or any other third party.


Is there really a dependency is the repo on Github is *the* official repo if 
the Adiscon repo is synced to it?


 It's very possible that there will end up being another official repo 
 on a github competitor if that competitor becomes popular enough.


Another official repo created by Rainer?  Would be a little confusing and 
needless IMHO, but I may be missing something.

Thanks,
Otis


On Mon, 16 Dec 2013, Otis Gospodnetic wrote:

  Date: Mon, 16 Dec 2013 17:24:50 -0500
 From: Otis Gospodnetic otis.gospodne...@gmail.com
 Reply-To: rsyslog-users rsyslog@lists.adiscon.com
 To: rsyslog-users rsyslog@lists.adiscon.com
 Subject: Re: [rsyslog] People insight?

 No need to eliminate.  But if one repo is labeled as official, it 
 makes more sense to point to that repo if having 2 repos is not possible.
  Though
 in my mind a project wants to help contributors contribute more 
 easily and I think in this particular case 3M+ people having Github 
 accounts helps with that.

 Otis
 --
 Performance Monitoring * Log Analytics * Search Analytics Solr  
 Elasticsearch Support * http://sematext.com/


 On Mon, Dec 16, 2013 at 4:48 PM, David Lang da...@lang.hm wrote:

  remember that we are not trying to eliminate the adiscon 
 infrastructure
 or
 the google juice that goes with it.

 At most we would want to add github to Ohloh, so the issue isn't 
 that it points at adiscon, but that the text should get an added 
 pointer to github.

 David Lang

 On Mon, 16 Dec 2013, Otis Gospodnetic wrote:

  The key part is that it doesn't encourage easier PRs or help with juice.


 Otis


 On Mon, Dec 16, 2013 at 4:30 PM, David Lang da...@lang.hm wrote:

  On Mon, 16 Dec 2013, Otis Gospodnetic wrote:


  I forgot about Ohloh.  Thanks - looks good!


 Note they point to git://git.adiscon.com/git/rsyslog


  since the content is the same, that's not a problem.

 David Lang

  Otis

  --
 Performance Monitoring * Log Analytics * Search Analytics Solr  
 Elasticsearch Support * http://sematext.com/


 On Mon, Dec 16, 2013 at 3:12 PM, Rainer Gerhards
 rgerha...@hq.adiscon.comwrote:

  Ohloh has many of these metrics, i think better than on github:


 http://www.ohloh.net/p/rsyslog

 For sone reason, the code size seems to have a problem, though.

 Active *direct* commit access? Me. Hasn't been a problem so far. 
 As i said, i known those that need few to zero review.

 Rainer

 Sent from phone, thus brief.
 Am 16.12.2013 20:59 schrieb David Lang da...@lang.hm:

  On Mon, 16 Dec 2013, Otis Gospodnetic wrote:


  Hi,


  On Mon, Dec 16, 2013 at 2:25 PM, David Lang da...@lang.hm wrote:

  On Mon, 16 Dec 2013, Otis Gospodnetic wrote:


   Hi,


  Btw. where can one see a list of people who

 A) have commit rights


  shouldn't matter, send your pull request/patches to the 
 mailing

  list so
 they can 

Re: [rsyslog] Modules in other programming languages?

[Brian Knox]

I agree that omhttp would be a better name.  Note - I'm not signing up for that 
one quite yet - my first priority is going to be moving the omzmq3 plugins to 
the rsyslog v8 plugin api, and moving them from zeromq3 to zeromq4.

Brian

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Boylan, James
Sent: Tuesday, December 17, 2013 7:09 AM
To: rsyslog-users
Subject: Re: [rsyslog] Modules in other programming languages?

That would probably be better called 'omhttp' as 'omrest' paints the picture of 
a REST interface for accessing into Rsyslog, not outputting http post calls to 
a destination.

-- James

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Brian Knox
Sent: Tuesday, December 17, 2013 4:59 AM
To: rsyslog-users
Subject: Re: [rsyslog] Modules in other programming languages?

I was thinking that omrest would be a module that made outbound http requests 
to send messages; not as something that waited for incoming http requests for 
messages.  So, in my mind it was something far more similar to the current 
elastic search module, which pushes messages out.

So - more of an outgoing http request to an external rest api, rather than 
something providing a rest api to make calls into.

-Original Message-
From: rsyslog-boun...@lists.adiscon.com 
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, December 16, 2013 1:57 PM
To: rsyslog-users
Subject: Re: [rsyslog] Modules in other programming languages?

On Mon, 16 Dec 2013, Brian Knox wrote:

 I believe the output module for elastic search might be a good place 
 to start looking for anyone interested in writing an omrest module?
 If I recall correctly the elastic search output uses libcurl.

not really.

currently every om* module consists of code that is executed by a rsyslog 
worker thread that is passed a list of messages and acts on each message. 
omelasticsearch pushes messages via libcurl

for omrest you would need to change the entire paradigm of how an om* module 
would work.

Instead of being code invoked by a worker thread that's invoking many other
om* code as well on a given message, the omrelp module would need to listen for 
a connection from the outside, and when it receives a request, it would need to 
retrieve messages from the queue, and the worker threads would need to leave 
the messages on the queue.

You should see by now that this is a really ugly thing to talk about 
implementing. It's almost a complete re-write of the rsyslog core to support 
this.


The other approach is to have omrelp maintain it's own queue of messages and 
knowledge of who should be asking for messages, timeouts for messages that 
aren't asked for, etc. At that point, omrelp's interface to rsyslog is 
straightforward, it's just the omrelp queue and interface stuff that gets 
really 'interesting' and people who want omrelp should work on writing some 
code that will perform all the relp functions that you want to support but just 
accepts new messages on stdin (which would let it be driven by omprog for now), 
and then as you get it running and find it useful the input portion could be 
changed to make it into a 'real' rsyslog om* module. Depending on what you do 
for your queue, you may be able to use a different language to handle adding 
things to the queue and pulling things from the queue. This would make it easy 
to have some rsyslog C code that adds things to the queueu.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is 
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our 
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is 
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our 
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is 
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our 
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL