On Thu, 23 Jan 2014, Rainer Gerhards wrote:
On Thu, Jan 23, 2014 at 6:46 PM, David Lang <[email protected]> wrote:
so what exactly is being proposed?
It sounds as if we are talking about omprog, but that also captures stderr
of the program that's executed, and watches that stderr for specific
keywords.
I think yes, that's bascially the idea.
what keywords are you talking about, and what actions will be taken? how
would these actions differ from the program just stalling reading of the
pipe or exiting with an error code?
don't know yet - this needs to evolve/be specified. Currently working on a
python test script and test integration.
Ok, probably a better question than qhat keywords, is what functionality you are
looking for in this feedback. Radu, do you have thoughts?
This is not going to be able to support batch mode, some number of logs
are going to be lost in any case because we will dequeue messages (consider
them delivered) when we put them in the pipe to the program, any feedback
we get from stderr can only affect logs that haven't been sent to the pipe
yet.
yeah, it's not totally reliable. It's the same scenario we have with omfwd
in plain tcp syslog mode. Judging from that, it's still good for many
applications. If that's a problem, we could even go down (via config
option) to do a half-duplex mode, where only one message is posted at one
time, and reply awaited. Obviously much slower, but there always is a price
for reliability.
given this level of reliability, I'm not sure that it's really worth trying to
get more feedback than we get with TCP (it died and we are restarting, or it
blocked and we can't send to it)
what other restrictions are there?
I don't know yet. Quite honestly, I don't try to design this fully
through. This time, I'd prefer to do some thing, see how they work out and
what hurts, refactor and begin a new cycle. In this mode, I hope to have
something workable (even without real feedback, much like UDP) within a day
or two. That would probably be enough to see the utility and if it is being
used. A full design requires more time - time I don't have and I don't know
if it would be well spent. You may consider this effort of a simple kind of
"omprog" evolution but from the marketing PoV this seems not to be very
appealing... So let's do what everyone does and exaggregate the terms.
Still, I think it's technical far superior, and we could have a fairly
working SOLR script (with batches) very quickly...
how would you handle batches?
David Lang
Rainer
David Lang
On Thu, 23 Jan 2014, Radu Gheorghe wrote:
Or, maybe it's easy to add imthrift&omthrift?
2014/1/23 Radu Gheorghe <[email protected]>
+1. Let's make what we already have (and maybe add some stuff with
minimum
effort) easy to use. Let's see what the adoption is. Let's see people
complain about performance, if that's ever going to happen and then we
can
"do it right" according to demand.
2014/1/23 Boylan, James <[email protected]>
I agree with Rainer.
While 'doing it right' from from the start and building from the ground
up perspective is a fantastic goal, the reality of trying to fit that in
with an entirely new direction is difficult at best. When you have a
mounting list of items that need to be worked on already, sometimes
taking
that fast to implement first step to just show how much better something
can be is critical to get stakeholders to back investing the kind of
time
required to make a major overhaul to something.
-- James
-----Original Message-----
From: [email protected] [mailto:
[email protected]] On Behalf Of Rainer Gerhards
On Thu, Jan 23, 2014 at 9:09 AM, David Lang <[email protected]> wrote:
On Thu, 23 Jan 2014, Rainer Gerhards wrote:
On Thu, Jan 23, 2014 at 8:00 AM, David Lang <[email protected]> wrote:
On Thu, 23 Jan 2014, Radu Gheorghe wrote:
2014/1/22 Rainer Gerhards <[email protected]>
On Wed, Jan 22, 2014 at 3:17 PM, Radu Gheorghe <
[email protected]
I think it all depends on how much work we are going to have to do.
why do we need to do anything? can't omprog with a JSON template
already do what's needed?
IMHO quite some things, but it lacks some others (like a feedback
capability, see below). My interest is to extend it in a way that mostly
solves these issues.
as long as we stick to pipes, we aren't going to be able to get per
message success/failure messages (the buffering in the pipe will
prevent that from working well)
That's right, but we get pretty close to "around when it begins to
fail".
Per message is hard in any case. Think e.g. TCP forwarding: we never
know
exactly when the connection broke, so this is also approximate.
I guess I'm not fully understanding what's being proposed for the
short term.
Let me coin it into a non-technical statement: I try to do the minimal
thing necessary to make it easier for people to connect to destinations
for
which rsyslog does not yet have any native plugins. Among the tech
issues,
this probably means make folks aware that they actually *can do that*
(as
you say, to a large extent even with what we have today, but that seems
to
be too well-hidden). Once this is done, I'd like to sit back, relax,
gain
some experience and then see how to proceed further.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
