I definitely agree. Using pipe and stdin is the primary process for Hadoop non-Java process as I pointed out and it is a widely accepted method for handling it. I'd like to see what additions can do done with omprog. I need to start working on testing our utility that uses syslog-ng with it's pipe output with Rsyslog's omprog. I'll have a better idea on concerns and improvements once I'm able to dig into that.
-- James -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Rainer Gerhards Sent: Wednesday, January 22, 2014 10:35 AM To: rsyslog-users Subject: Re: [rsyslog] non-C output plugins On Wed, Jan 22, 2014 at 4:05 PM, Rainer Gerhards <[email protected]>wrote: > On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards > <[email protected] > > wrote: > >> On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James >> <[email protected]>wrote: >> >>> I agree with everyone that omprog should be a good solution for this. >>> I'm going to be doing some tests in the near future with omprog so >>> I'll be looking at Radu's post as well. :) >>> >>> Mostly I think using a stdout pipe is a good way of handling this. >>> It is similar to how Hadoop handles non-Java mapreduce jobs as well. >>> >>> >> A key thing to note is that a pipe is a quite efficient inter-process >> communication facility and also includes some built-in flow control >> (so the reading end of the pipe can push back the writer). Of course, >> it requires additional context switches, but that should not be too bad. >> >> An interesting thought is to write a simple write-to-file script and >> measure it's performance vs. omfile. That should be an indication of >> the actual pipe overhead, especially if /dev/null is written to... >> >> > Better even: a native C program, so that we don't measure language > runtime overhead... > > I couldn't stand it and gave it a very quick try. Tested on a low-end laptop with Linux running on bare metal und Fedora 19, with just my development build. Quick results: https://docs.google.com/spreadsheet/pub?key=0AjQIRIQG1RAEdHNxaGtveGYtNTRyaGZQUFpNZUtzdXc&output=html In general, I think the performance loss is quite acceptable -- I guess it doesn't really matter for those plugins that do real work like HTTP requests and such. Interesting fact, though, is that writing via the piped program to /dev/null took longer, but I didn't evaluate the reason. After all, this was just for a quick feeling of how things work. My conclusion is that this is not a bad interface for those kinds of things and so improving omprog definitely makes sense. Comments? Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
