Yes, mainly being able to avoid the network is interesting. Basically it would be like ompipe (assuming it was sock_stream) with a way to ack back.
On Fri, Jan 31, 2014 at 8:47 AM, David Lang <[email protected]> wrote: > On Fri, 31 Jan 2014, Rainer Gerhards wrote: > > On Fri, Jan 31, 2014 at 8:14 AM, Nathan Brown <[email protected]> >> wrote: >> >> What if a plugin wants to reliably handle events? For example: I am >>> currently using a relp protocol implementation to ensure I handle >>> every event reliably and this plugin architecture sounds like an >>> intriguing way to get rid of the tcp overhead and relp window sizing. >>> I would want to be able to replicate the same offerings as relp in >>> this interface though. >>> >>> I can also imagine more exotic scenarios where a plugin is used as >>> part of a ruleset to decide which branch to take. Like selecting which >>> aggregator node to send an event to or whether to forward an event in >>> general. >>> >>> All of these scenarios require a more structured protocol, and the >>> right answer might be that this architecture isn't for that. >>> >>> >>> Well, you could probably build this on top of the interface. But it's >> far >> beyond the current "just make it simple to write a connector" approach. >> This however requires the feedback verbs, which are not yet defined. >> Current focus is on enable people to write simple connectors. Quite >> honestly, I think the target base is people who can write automation >> scripts, but are no real developers. And what you describe would be far >> above their head. >> >> Digging a bit deeper, I wonder what you want to save over RELP? If you >> require it's feature, I think it is very hard to come up with some >> protocol >> that does not look very similar to RELP... >> > > I think that what he's looking for is very similar to RELP, but not over > the network (which would make it a bit simpler). > > I think for a local connection, we can get away with a synchronous > connection as long as we support the rsyslog internal batching to the > external module. but this really is looking for the protocol buffers/thrift > type interface that I was talking about doing for the second version of > this. > > David Lang > > > Rainer >> >> >> On Jan 30, 2014, at 10:47 PM, Rainer Gerhards <[email protected]> >>>> >>> wrote: >>> >>>> >>>> On Fri, Jan 31, 2014 at 7:37 AM, Rainer Gerhards >>>> <[email protected]>wrote: >>>> >>>> On Fri, Jan 31, 2014 at 6:50 AM, Nathan Brown <[email protected]> >>>>>> >>>>> wrote: >>> >>>> >>>>>> Fully agree that some plugins will be brain dead simple and will only >>>>>> require an incredibly small feature set of what the plugin system may >>>>>> offer. But having multiple communications options, certainly ones that >>>>>> are entirely unstructured, will be difficult to maintain. Forward >>>>>> >>>>> I forgot to answer this: it's not difficult to maintain, as we simply >>>> use >>>> the template system that's already in place, and needs to be in place >>>> for >>>> all further versions in any case. So there is exactly 0% overhead for >>>> >>> this. >>> >>>> >>>> Rainer >>>> >>>> >>>> compatibility of a plugin that may start simple but eventually will >>>>> >>>>>> want to grow into something more complex is a good enough reason to >>>>>> have a structured plugin protocol. Let alone rsyslog's side that may >>>>>> increase the feature set that it can offer a plugin. JSON has proven >>>>>> it's a good candidate for this sort of thing, lightweight with decent >>>>>> native types. >>>>>> >>>>> IMO rsyslog is still about speed, even though we sacrifice it for the >>>>> >>>> sake >>> >>>> of making it easy to write plugins. I agree that JSON is the most >>>>> >>>> versatile >>> >>>> format and something to recommend for the general case. HOWEVER, there >>>>> >>>> is >>> >>>> quite a bit of overhead associated with crafting an all-JSON >>>>> >>>> representation >>> >>>> of a message object. >>>>> >>>>> So I think we should not FORCE plugins to use JSON format. There are >>>>> >>>> many >>> >>>> cases where this makes sense, but there are also many cases where simply >>>>> -and much faster!- formats make more sense. Given the simplicity of the >>>>> interface, I think anyone should understand what he is doing. Even more >>>>> >>>> so, >>> >>>> for real novices, getting the JSON parser to work might be a stumbling >>>>> stone, whereas pulling pre-formatted data from stdin makes them happy. >>>>> >>>>> This is why I think we should not enforce the use of JSON on the >>>>> >>>> interface. >>> >>>> >>>>> Rainer >>>>> >>>>> On Jan 30, 2014, at 9:11 PM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> The thing is that with the rsyslog templates, you can do a lot of >>>>>>> >>>>>> formatting and manipulation of the data before you send it to the >>>>>> >>>>> module. >>> >>>> so you should configure rsyslog to send your module whatever it can >>>>>> >>>>> handle >>> >>>> the best. >>>>>> >>>>>>> >>>>>>> If you have typical needs (take this entire line and send it there) >>>>>>> >>>>>> you >>> >>>> can get by with a very simple output module. If you have very complex >>>>>> needs, you may need to use JSON to structure things so that it's >>>>>> always >>>>>> clear what's what. >>>>>> >>>>>>> >>>>>>> the author of the output module needs to document what that module >>>>>>> >>>>>> requires as it's input. >>>>>> >>>>>>> >>>>>>> overall this looks pretty good. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> On Wed, 29 Jan 2014, Nathan Brown wrote: >>>>>>>> >>>>>>>> I would avoid having multiple communication options and use only >>>>>>>> line >>>>>>>> delimited JSON to talk between the plugin and rsyslog. >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jan 29, 2014 at 3:21 AM, Rainer Gerhards >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I've written a short presentation on writing rsyslog output >>>>>>>>> plugins. >>>>>>>>> >>>>>>>> It is >>>>>> >>>>>>> targeted towards people who would like to know at least a bit about >>>>>>>>> >>>>>>>> how >>>>>> >>>>>>> this works. I also plan to do an ultra-fasttrack document for those >>>>>>>>> >>>>>>>> that >>>>>> >>>>>>> just want to get it going somehow and don't about how it works. They >>>>>>>>> >>>>>>>> are >>>>>> >>>>>>> *not* the focus of the current presentation: >>>>>>>>> >>>>>>>>> http://www.slideshare.net/rainergerhards1/external-plugins >>>>>>>>> >>>>>>>>> I will include this information in the Fedora DevConf Talk next >>>>>>>>> >>>>>>>> week, >>> >>>> and >>>>>> >>>>>>> would really like your feedback. Where can we improve it, how to >>>>>>>>> >>>>>>>> make >>> >>>> even >>>>>> >>>>>>> more clear that it's simple. Note that I say "v8" only because it >>>>>>>>> >>>>>>>> already >>>>>> >>>>>>> covers a bit of what we will have in a month or so (the stdout >>>>>>>>> >>>>>>>> feedback). >>>>>> >>>>>>> When I give the presentation, I'll tell that omprog in older >>>>>>>>> >>>>>>>> versions >>> >>>> works >>>>>> >>>>>>> mostly the same. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Rainer >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> >>>>>>>> myriad >>>>>> >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> >>>>>>>> you >>> >>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> >>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>> POST >>>>>> >>>>> if >>> >>>> you DON'T LIKE THAT. >>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >>>>>> >>>>> if >>> >>>> you DON'T LIKE THAT. >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
