On Thu, Jan 23, 2014 at 6:37 AM, Radu Gheorghe <[email protected]>wrote:
> Hi David,
>
> I had the same thoughts about writing to /dev/log instead of having a new
> improg module. Two comments here, though:
> - omprog does a nice job of babysitting the program (starting it,
> restarting it if it crashes). With /dev/log, you have to manage the daemon
> - logger is typically truncating messages at 1K
>
... and a much bigger problem: with journald around, and always sitting in
the middle, we cannot longer use this as a decent way to gather input
destined purely to the syslogd. It's far too slow, and it spams the journal
logs.
>
> Whatever we do for "inputs in other languages" I think we need to be able
> to simply get the received message as it is (ie: don't require syslog
> format) and parse it later, or add timestamp, or...
>
> And (don't throw sticks or onions!) support multiline messages.
>
>
With JSON that's not a problem. We can assure that everything is on one
line, as the JSON values need to have LF encoded as \n (by RFC
definition!).
If we don't request JSON ("as is"), we are in the guesswork business of
when the message starts and stops. Not sure if we really want that.
Rainer
> 2014/1/22 David Lang <[email protected]>
>
> > I think we should have at least a stub module in each language in the
> main
> > repository. All the stubs could do the same 'write to file', but they
> would
> > give people a place to start
> >
> > Actually, I think we need to do three modules in every language
> >
> > read from file
> > write to file
> > message modification that gets the message and passes it back.
> >
> > while we could just use pipes and stdin/stdout, I've been thinking about
> > this a bit more over the last week or so, and think we should really go
> > ahead and be more formal about this
> >
> > parsing messages is hard, formatting messages to be parsed is error
> prone,
> > and both tend to be slow, and when working with text messages, tend to
> > accumulate a lot of cruft to work around corner cases.
> >
> > Also, as Rainer said above, we need to have the ability to get feedback
> > from these other languages, and I'm thinking we probably need to be able
> to
> > send command-type messages (we just got a HUP, close and reopen your
> > inputs/outputs)
> >
> > I think we should look at creating new immultilang, ommultilang, and
> > mmmultilang modules for rsyslog that use either google protocol buffers
> or
> > apache thrift to do the message passing (ideally both would be available
> > since there are people who prefer both)
> >
> > What I'm currently thinking of is that we should have the ability to
> > include configuration for these external modules in the rsyslog config
> (and
> > the modules can do whatever they want for external configs), when the
> > module is initialized, rsyslog passes the config data to the module and
> > doesn't consider it good until it gets a reply saying that it's ready.
> This
> > could also include handshaking to coordinate capabilities (does this
> module
> > have the ability to process batches of messages, if so, how large?, can
> it
> > do async message processing where messages may be processed out of
> order?,
> > other things) I'm not saying that rsyslog needs to support such advanced
> > features immediately, and some may never be supported, but there needs to
> > be a way to find out what is and isn't supported by a module.
> >
> > Yes, this will be more work initially, and in the meantime we can go
> ahead
> > and use omprog (I'm not sure we need improg since programs can just write
> > to /dev/log or logger), but I think the long-term results will be much
> > better.
> >
> > thoughts?
> >
> > David Lang
> >
> >
> >
> > On Wed, 22 Jan 2014, Boylan, James wrote:
> >
> > Hmm. That's a tougher question. On the one hand I can see the benefit to
> >> it. But on the other hand, is it the best choice for location... One
> could
> >> argue that processing scripts like that should be in their own repo
> since
> >> they are not directly linked to the Rsyslog codebase.
> >>
> >> -- James
> >>
> >>
> >> -----Original Message-----
> >> From: [email protected] [mailto:rsyslog-bounces@lists.
> >> adiscon.com] On Behalf Of Rainer Gerhards
> >> Sent: Wednesday, January 22, 2014 9:31 AM
> >> To: rsyslog-users
> >> Subject: Re: [rsyslog] non-C output plugins
> >>
> >> sorry for being so imprecise...
> >>
> >> What I meant to ask is if we should place such non-C plugins into the
> >> main rsyslog repo. I see lot's of good in this (assuming the
> contributor is
> >> happy with that, of course), but that means that changes can only go
> into
> >> them via PR, patches...
> >>
> >> Rainer
> >>
> >>
> >> On Wed, Jan 22, 2014 at 4:07 PM, Boylan, James <[email protected]
> >
> >> wrote:
> >>
> >> It seems to me that if it's either added as a modification to omprog
> >>> or a new ompipe then having it added to master as a default disabled
> >>> option should work. Less maintenance overhead and is available for use.
> >>>
> >>> -- James
> >>> -- Sent from my mobile --
> >>>
> >>> ----- Reply message -----
> >>> From: "Rainer Gerhards" <[email protected]>
> >>> To: "rsyslog-users" <[email protected]>
> >>> Subject: [rsyslog] non-C output plugins
> >>> Date: Wed, Jan 22, 2014 9:04 AM
> >>>
> >>> On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards
> >>> <[email protected]>wrote:
> >>>
> >>> On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James
> >>>> <[email protected]
> >>>> wrote:
> >>>>
> >>>> I agree with everyone that omprog should be a good solution for this.
> >>>>>
> >>>> I'm
> >>>
> >>>> going to be doing some tests in the near future with omprog so I'll
> >>>>> be looking at Radu's post as well. :)
> >>>>>
> >>>>> Mostly I think using a stdout pipe is a good way of handling this.
> >>>>> It is similar to how Hadoop handles non-Java mapreduce jobs as well.
> >>>>>
> >>>>>
> >>>>> A key thing to note is that a pipe is a quite efficient
> >>>> inter-process communication facility and also includes some built-in
> >>>> flow control (so
> >>>>
> >>> the
> >>>
> >>>> reading end of the pipe can push back the writer). Of course, it
> >>>> requires additional context switches, but that should not be too bad.
> >>>>
> >>>> An interesting thought is to write a simple write-to-file script and
> >>>> measure it's performance vs. omfile. That should be an indication of
> >>>> the actual pipe overhead, especially if /dev/null is written to...
> >>>>
> >>>>
> >>>> Better even: a native C program, so that we don't measure language
> >>> runtime overhead...
> >>>
> >>> Question in this regard: should all of this go into the main repo?
> >>> There are pros and cons...
> >>>
> >>> Rainer
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>> DON'T LIKE THAT.
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >>> DON'T LIKE THAT.
> >>>
> >>> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> >> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> >> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
> >> THAT.
> >> _______________________________________________
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >> _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
