On Wed, Jan 22, 2014 at 3:59 PM, Boylan, James <[email protected]>wrote:
> I agree with everyone that omprog should be a good solution for this. I'm > going to be doing some tests in the near future with omprog so I'll be > looking at Radu's post as well. :) > > Mostly I think using a stdout pipe is a good way of handling this. It is > similar to how Hadoop handles non-Java mapreduce jobs as well. > > A key thing to note is that a pipe is a quite efficient inter-process communication facility and also includes some built-in flow control (so the reading end of the pipe can push back the writer). Of course, it requires additional context switches, but that should not be too bad. An interesting thought is to write a simple write-to-file script and measure it's performance vs. omfile. That should be an indication of the actual pipe overhead, especially if /dev/null is written to... Rainer > -- James > > -----Original Message----- > From: [email protected] [mailto: > [email protected]] On Behalf Of Rainer Gerhards > Sent: Wednesday, January 22, 2014 8:27 AM > To: rsyslog-users > Subject: Re: [rsyslog] non-C output plugins > > On Wed, Jan 22, 2014 at 3:17 PM, Radu Gheorghe <[email protected] > >wrote: > > > Hi Rainer, > > > > I'd like to see an Apache Solr <https://lucene.apache.org/solr/> > > output plugin. > > > > You can see in my presentation > > here< > > http://blog.sematext.com/2013/12/16/video-using-solr-for-logs-with-rsy > > slog-flume-fluentd-and-logstash/ > > >at > > slides 21-26 that this is what I did as a demo with omprog. The actual > > python script that pushes logs to Solr is on slide 26. > > > > > Oh, I think that is an excellent example, especially as it is very brief > and still does useful work. That's one of the core ideas to convey so that > others become interested in doing a similar thing. > > > > The thing works, but: > > - it would only send logs one by one > > - it has no error handling > > > > Though I think it should be easy to add. > > > > Would be great if you could help with that over time, as I neither have > the know how nor environment... > > > > > > I think Solr is a really nice destination for logs, offering similar > > stuff to what Elasticsearch does. Plus some, minus some, and they're > > both based on Apache Lucene <https://lucene.apache.org/>. You can see > > in the same presentation that other tools already send logs to Solr, > > and I think Apache Flume <https://flume.apache.org/> is the most widely > used of them. > > > > > Big question now: which changes in rsyslog are required to make you > happier as an author of such a non-C module? What would you like to see > improved? > > My thinking is that we could (assuming your permission) put this into the > source tree (or somewhere else?) and say "look, this is a sample of how to > do it". Indeed, that alone I think would be useful. But what would be the > next steps? What caused you trouble? etc, etc... > > Rainer > > > > Best regards, > > Radu > > > > > > 2014/1/22 Rainer Gerhards <[email protected]> > > > > > Hi folks, > > > > > > I would like to create (maybe with somone's help) a real non-C > > > output plugin. Ideally this would be something that's not too hard > > > to do so that the code can act as a simple sample. For that reason, > > > it should work on a push model basis. But that's not a hard > requirement. > > > > > > Any suggestions what would be needed? What would you like to see? > > > > > > Once we have suggestions, we can see that we can implement it and I > > > can > > see > > > what in rsyslog's plumbing actually needs to be changed. Your > > collaboration > > > during that effort would be appreciated. > > > > > > Please let me know your suggestions. > > > > > > Thanks, > > > Rainer > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > > > you DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE > THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
