On Wed, Jan 22, 2014 at 4:05 PM, Rainer Gerhards <[email protected]>wrote:
> On Wed, Jan 22, 2014 at 4:02 PM, Rainer Gerhards <[email protected] > > wrote: > >> On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James >> <[email protected]>wrote: >> >>> I agree with everyone that omprog should be a good solution for this. >>> I'm going to be doing some tests in the near future with omprog so I'll be >>> looking at Radu's post as well. :) >>> >>> Mostly I think using a stdout pipe is a good way of handling this. It is >>> similar to how Hadoop handles non-Java mapreduce jobs as well. >>> >>> >> A key thing to note is that a pipe is a quite efficient inter-process >> communication facility and also includes some built-in flow control (so the >> reading end of the pipe can push back the writer). Of course, it requires >> additional context switches, but that should not be too bad. >> >> An interesting thought is to write a simple write-to-file script and >> measure it's performance vs. omfile. That should be an indication of the >> actual pipe overhead, especially if /dev/null is written to... >> >> > Better even: a native C program, so that we don't measure language runtime > overhead... > > I couldn't stand it and gave it a very quick try. Tested on a low-end laptop with Linux running on bare metal und Fedora 19, with just my development build. Quick results: https://docs.google.com/spreadsheet/pub?key=0AjQIRIQG1RAEdHNxaGtveGYtNTRyaGZQUFpNZUtzdXc&output=html In general, I think the performance loss is quite acceptable -- I guess it doesn't really matter for those plugins that do real work like HTTP requests and such. Interesting fact, though, is that writing via the piped program to /dev/null took longer, but I didn't evaluate the reason. After all, this was just for a quick feeling of how things work. My conclusion is that this is not a bad interface for those kinds of things and so improving omprog definitely makes sense. Comments? Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
