Sent from phone, thus brief. Am 23.01.2014 19:09 schrieb "David Lang" <[email protected]>: > > On Thu, 23 Jan 2014, Rainer Gerhards wrote: > >> On Thu, Jan 23, 2014 at 6:46 PM, David Lang <[email protected]> wrote: >> >>> so what exactly is being proposed? >>> >>> It sounds as if we are talking about omprog, but that also captures stderr >>> of the program that's executed, and watches that stderr for specific >>> keywords. >>> >>> >> I think yes, that's bascially the idea. >> >> >>> what keywords are you talking about, and what actions will be taken? how >>> would these actions differ from the program just stalling reading of the >>> pipe or exiting with an error code? >>> >>> >> don't know yet - this needs to evolve/be specified. Currently working on a >> python test script and test integration. > > > Ok, probably a better question than qhat keywords, is what functionality you are looking for in this feedback. Radu, do you have thoughts?
I think ok vs. Error is needed in any case - and probably sufficient at first. > > >>> This is not going to be able to support batch mode, some number of logs >>> are going to be lost in any case because we will dequeue messages (consider >>> them delivered) when we put them in the pipe to the program, any feedback >>> we get from stderr can only affect logs that haven't been sent to the pipe >>> yet. >>> >> >> yeah, it's not totally reliable. It's the same scenario we have with omfwd >> in plain tcp syslog mode. Judging from that, it's still good for many >> applications. If that's a problem, we could even go down (via config >> option) to do a half-duplex mode, where only one message is posted at one >> time, and reply awaited. Obviously much slower, but there always is a price >> for reliability. > > > given this level of reliability, I'm not sure that it's really worth trying to get more feedback than we get with TCP (it died and we are restarting, or it blocked and we can't send to it) That is what I also think. > > >>> what other restrictions are there? >>> >> >> I don't know yet. Quite honestly, I don't try to design this fully >> through. This time, I'd prefer to do some thing, see how they work out and >> what hurts, refactor and begin a new cycle. In this mode, I hope to have >> something workable (even without real feedback, much like UDP) within a day >> or two. That would probably be enough to see the utility and if it is being >> used. A full design requires more time - time I don't have and I don't know >> if it would be well spent. You may consider this effort of a simple kind of >> "omprog" evolution but from the marketing PoV this seems not to be very >> appealing... So let's do what everyone does and exaggregate the terms. >> Still, I think it's technical far superior, and we could have a fairly >> working SOLR script (with batches) very quickly... > > > how would you handle batches? > Read stdin until a max nbr msgs is reached or no more data. Then process. Its not exactly the batch rsyslog sees, but you can submit multiple messages e.g. via http. I have a small python script that does this, but want to improve it a little bit. Rainer > David Lang > > >> Rainer >> >>> >>> David Lang >>> >>> >>> On Thu, 23 Jan 2014, Radu Gheorghe wrote: >>> >>> Or, maybe it's easy to add imthrift&omthrift? >>>> >>>> >>>> 2014/1/23 Radu Gheorghe <[email protected]> >>>> >>>> +1. Let's make what we already have (and maybe add some stuff with >>>>> >>>>> minimum >>>>> effort) easy to use. Let's see what the adoption is. Let's see people >>>>> complain about performance, if that's ever going to happen and then we >>>>> can >>>>> "do it right" according to demand. >>>>> >>>>> 2014/1/23 Boylan, James <[email protected]> >>>>> >>>>> I agree with Rainer. >>>>> >>>>>> >>>>>> While 'doing it right' from from the start and building from the ground >>>>>> up perspective is a fantastic goal, the reality of trying to fit that in >>>>>> with an entirely new direction is difficult at best. When you have a >>>>>> mounting list of items that need to be worked on already, sometimes >>>>>> taking >>>>>> that fast to implement first step to just show how much better something >>>>>> can be is critical to get stakeholders to back investing the kind of >>>>>> time >>>>>> required to make a major overhaul to something. >>>>>> >>>>>> -- James >>>>>> >>>>>> -----Original Message----- >>>>>> From: [email protected] [mailto: >>>>>> [email protected]] On Behalf Of Rainer Gerhards >>>>>> On Thu, Jan 23, 2014 at 9:09 AM, David Lang <[email protected]> wrote: >>>>>> >>>>>> On Thu, 23 Jan 2014, Rainer Gerhards wrote: >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 23, 2014 at 8:00 AM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> On Thu, 23 Jan 2014, Radu Gheorghe wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> 2014/1/22 Rainer Gerhards <[email protected]> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Wed, Jan 22, 2014 at 3:17 PM, Radu Gheorghe < >>>>>>>>>> [email protected] >>>>>>>>>> >>>>>>>>>> >>>>>>> I think it all depends on how much work we are going to have to do. >>>>>>> >>>>>>> why do we need to do anything? can't omprog with a JSON template >>>>>>> already do what's needed? >>>>>>> >>>>>>> >>>>>> IMHO quite some things, but it lacks some others (like a feedback >>>>>> capability, see below). My interest is to extend it in a way that mostly >>>>>> solves these issues. >>>>>> >>>>>> >>>>>> >>>>>>> as long as we stick to pipes, we aren't going to be able to get per >>>>>>> message success/failure messages (the buffering in the pipe will >>>>>>> prevent that from working well) >>>>>>> >>>>>>> >>>>>> That's right, but we get pretty close to "around when it begins to >>>>>> fail". >>>>>> Per message is hard in any case. Think e.g. TCP forwarding: we never >>>>>> know >>>>>> exactly when the connection broke, so this is also approximate. >>>>>> >>>>>> >>>>>> I guess I'm not fully understanding what's being proposed for the >>>>>>> >>>>>>> short term. >>>>>>> >>>>>>> >>>>>> Let me coin it into a non-technical statement: I try to do the minimal >>>>>> thing necessary to make it easier for people to connect to destinations >>>>>> for >>>>>> which rsyslog does not yet have any native plugins. Among the tech >>>>>> issues, >>>>>> this probably means make folks aware that they actually *can do that* >>>>>> (as >>>>>> you say, to a large extent even with what we have today, but that seems >>>>>> to >>>>>> be too well-hidden). Once this is done, I'd like to sit back, relax, >>>>>> gain >>>>>> some experience and then see how to proceed further. >>>>>> >>>>> _______________________________________________ >>> >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
