On Fri, Jan 24, 2014 at 9:51 AM, Radu Gheorghe <[email protected]>wrote:
> >> I don't know yet. Quite honestly, I don't try to design this fully > >> through. This time, I'd prefer to do some thing, see how they work out > and > >> what hurts, refactor and begin a new cycle. In this mode, I hope to have > >> something workable (even without real feedback, much like UDP) within a > >> day > >> or two. That would probably be enough to see the utility and if it is > >> being > >> used. A full design requires more time - time I don't have and I don't > >> know > >> if it would be well spent. You may consider this effort of a simple kind > >> of > >> "omprog" evolution but from the marketing PoV this seems not to be very > >> appealing... So let's do what everyone does and exaggregate the terms. > >> Still, I think it's technical far superior, and we could have a fairly > >> working SOLR script (with batches) very quickly... > >> > > > > how would you handle batches? > > > > This is all in the script, rsyslog only pushes messages to the script's > stdin and doesn't bother: > - one thread continuously reads from stdin and puts messages into a queue > - one or more threads write to Solr (or whatever). Logic would be something > like: > ######################### > output_array = [] > > try: > output_array += get_new_message_from_queue() > except QueueIsEmptyException: > if output_array.HasSomething: > PushArrayContentsToSolr() #batch is sent if the queue is empty > > if output_array.size() == max_batch_size: > PushArrayContentsToSolr() #batch is sent if the maximum batch size is > reached > ######################### > > Yes, it does reinvent the rsyslog batch logic (or some of it, at least, I'm > not super-intimate with how rsyslog does it). The point of providing such a > "skeleton" script is to spare people from re-doing this logic the third > time, fourth time and so on :) > > Basically, for each destination, one would have to modify the > PushArrayContentsToSolr() function from the snippet above. Which proves > that the name containing Solr is bad :D > > > This is the python plugin skeleton I've been working on: https://github.com/rsyslog/rsyslog/blob/master/plugins/external/skeletons/plugin.py I haven't used python for quite a while, but I think it now looks mostly OK (except probably for the app's global var handling). I think we could try push the solr handling into it and see if that qualifies? ;) It does not yet convey anything back to rsyslog, as omprog does not yet understand it in any case. I still think it's useful. Feedback appreciated. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
