On Fri, 24 Jan 2014, Radu Gheorghe wrote:
David,
First of all, thanks a lot for your input in this matter. For me it's
eye-opening in many areas and it's very interesting for this subject in
general.
More inline.
2014/1/23 David Lang <[email protected]>
On Thu, 23 Jan 2014, Rainer Gerhards wrote:
On Thu, Jan 23, 2014 at 6:46 PM, David Lang <[email protected]> wrote:
so what exactly is being proposed?
It sounds as if we are talking about omprog, but that also captures
stderr
of the program that's executed, and watches that stderr for specific
keywords.
I think yes, that's bascially the idea.
what keywords are you talking about, and what actions will be taken? how
would these actions differ from the program just stalling reading of the
pipe or exiting with an error code?
don't know yet - this needs to evolve/be specified. Currently working
on a
python test script and test integration.
Ok, probably a better question than qhat keywords, is what functionality
you are looking for in this feedback. Radu, do you have thoughts?
With the way I've used omprog so far, I didn't ever try (or miss trying) to
communicate from the script back to rsyslog. The script's internal queue
would be tiny (say, 1000 messages) and if it gets full, bad luck, it just
stops.
If something catastrophic happens, throw an exception and rely on omprog to
restart it. Warnings would have been logged to its own logfile, and I guess
that's when it would be nice to have a channel back to rsyslog. But again,
I wouldn't stress too much on that. For example, a custom UDP port to
listen to, that binds to a ruleset that spits those logs to a file should
be enough. You wouldn't want to send the same logs to the same script,
because it would create an endless loop.
In short, what I needed was to discard malformed messages (which should be
a rare exception, indicating a bug somewhere - which is why it's nice to
log when that happens) and to stop and retry on temporary errors (like
network issues).
remember that it's also possible to create a /dev/log type socket which your
standard logging library can access without it feeding back to itself if you
don't want it to.
This is not going to be able to support batch mode, some number of logs
are going to be lost in any case because we will dequeue messages
(consider
them delivered) when we put them in the pipe to the program, any feedback
we get from stderr can only affect logs that haven't been sent to the
pipe
yet.
yeah, it's not totally reliable. It's the same scenario we have with omfwd
in plain tcp syslog mode. Judging from that, it's still good for many
applications. If that's a problem, we could even go down (via config
option) to do a half-duplex mode, where only one message is posted at one
time, and reply awaited. Obviously much slower, but there always is a
price
for reliability.
given this level of reliability, I'm not sure that it's really worth
trying to get more feedback than we get with TCP (it died and we are
restarting, or it blocked and we can't send to it)
+1
what other restrictions are there?
I don't know yet. Quite honestly, I don't try to design this fully
through. This time, I'd prefer to do some thing, see how they work out and
what hurts, refactor and begin a new cycle. In this mode, I hope to have
something workable (even without real feedback, much like UDP) within a
day
or two. That would probably be enough to see the utility and if it is
being
used. A full design requires more time - time I don't have and I don't
know
if it would be well spent. You may consider this effort of a simple kind
of
"omprog" evolution but from the marketing PoV this seems not to be very
appealing... So let's do what everyone does and exaggregate the terms.
Still, I think it's technical far superior, and we could have a fairly
working SOLR script (with batches) very quickly...
how would you handle batches?
This is all in the script, rsyslog only pushes messages to the script's
stdin and doesn't bother:
This is a good start for now, but I wouldn't spend too much work on it.
I'm thinking that it shouldn't be that hard for the rsyslog side to batch all
the messages into one structure, pass that structure to the other app, and then
take feedback as to the success or failure.
But I'm going to have to refresh myself on how rsyslog handles the batches
internally (and how this is changing with v8)
for now, as far as rsyslog is concerned, it isn't batched, it just writes a
series of logs out to the other app.
- one thread continuously reads from stdin and puts messages into a queue
- one or more threads write to Solr (or whatever). Logic would be something
A programmer had a problem. He thought “I know, I’ll solve it with threads!”.
has Now problems. two he
Also, creating an internal queue can lead to performance issues related to
locking of that queue. It was a substantial amount of (sponsored) work to change
rsyslog to do batching on it's queues, and it was also a huge speedup
internally.
remember that the pipe that the messages are being sent to you over is a fairly
substantial size, so you really can think about getting away with a much simpler
approach. remember that premature optimization is the root of all evil
tier 1 (simplest)
read messages until there are no more or I hit my limit
push those messages to my output
with no thread or async programming tricks involved.
Tier 2
the next more complicated version would push the messages to the output, and
then start working on reading the next batch of messages while it's waiting for
confirmation
Tier 3
This version would support multiple outbound sessions, they may be separate
connections or logical connections over one network connection, depending on
what you're connecting to.
tier 4
It's only if you need to overlap formatting of the new messages that have
arrived with reading those messages that you would need to resort to threads.
Before we get to something this complicated, we should do something that moves
away from JSON text as the protocol between rsyslog and the external app and
replace it with something faster.
David Lang
like:
#########################
output_array = []
try:
output_array += get_new_message_from_queue()
except QueueIsEmptyException:
if output_array.HasSomething:
PushArrayContentsToSolr() #batch is sent if the queue is empty
if output_array.size() == max_batch_size:
PushArrayContentsToSolr() #batch is sent if the maximum batch size is
reached
#########################
Yes, it does reinvent the rsyslog batch logic (or some of it, at least, I'm
not super-intimate with how rsyslog does it). The point of providing such a
"skeleton" script is to spare people from re-doing this logic the third
time, fourth time and so on :)
Basically, for each destination, one would have to modify the
PushArrayContentsToSolr() function from the snippet above. Which proves
that the name containing Solr is bad :D
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
