On Wed, Jan 22, 2014 at 5:42 PM, Boylan, James <[email protected]>wrote:
> I definitely agree. Using pipe and stdin is the primary process for Hadoop > non-Java process as I pointed out and it is a widely accepted method for > handling it. I'd like to see what additions can do done with omprog. I need > to start working on testing our utility that uses syslog-ng with it's pipe > output with Rsyslog's omprog. I'll have a better idea on concerns and > improvements once I'm able to dig into that. > > One result I immediately got from my own testing was that we need to (optionally) provide some backchannel, so that rsyslog knows when something went wrong in the program. stderr + defined verbs could be a method to do so... Rainer > -- James > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Rainer Gerhards > Sent: Wednesday, January 22, 2014 10:35 AM > To: rsyslog-users > Subject: Re: [rsyslog] non-C output plugins > > On Wed, Jan 22, 2014 at 4:05 PM, Rainer Gerhards > <[email protected]>wrote: > > > On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards > > <[email protected] > > > wrote: > > > >> On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James <[email protected] > >wrote: > >> > >>> I agree with everyone that omprog should be a good solution for this. > >>> I'm going to be doing some tests in the near future with omprog so > >>> I'll be looking at Radu's post as well. :) > >>> > >>> Mostly I think using a stdout pipe is a good way of handling this. > >>> It is similar to how Hadoop handles non-Java mapreduce jobs as well. > >>> > >>> > >> A key thing to note is that a pipe is a quite efficient inter-process > >> communication facility and also includes some built-in flow control > >> (so the reading end of the pipe can push back the writer). Of course, > >> it requires additional context switches, but that should not be too bad. > >> > >> An interesting thought is to write a simple write-to-file script and > >> measure it's performance vs. omfile. That should be an indication of > >> the actual pipe overhead, especially if /dev/null is written to... > >> > >> > > Better even: a native C program, so that we don't measure language > > runtime overhead... > > > > > I couldn't stand it and gave it a very quick try. Tested on a low-end > laptop with Linux running on bare metal und Fedora 19, with just my > development build. Quick results: > > > https://docs.google.com/spreadsheet/pub?key=0AjQIRIQG1RAEdHNxaGtveGYtNTRyaGZQUFpNZUtzdXc&output=html > > In general, I think the performance loss is quite acceptable -- I guess it > doesn't really matter for those plugins that do real work like HTTP > requests and such. Interesting fact, though, is that writing via the piped > program to /dev/null took longer, but I didn't evaluate the reason. After > all, this was just for a quick feeling of how things work. > > My conclusion is that this is not a bad interface for those kinds of > things and so improving omprog definitely makes sense. > > Comments? > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE > THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
