On 04/16/2015 04:48 AM, Etienne Champetier wrote:
Hi,
I'm using heka to read logs with logstreamer (nginx access log) and ship
them to another heka via TcpOutput.
I'm new to heka (and this ML) so here are my questions:
Is bufferedoutput (to disk) mandatory? (maybe i'm not understanding the
code correctly/missed a config)
TcpOutput always runs the messages through a disk buffer, yes. This might be
optional in the future, but it's true for now.
In my case it produces useless IO, ie read logs from disk, split them,
parse them, write them to disk, read them again and send them via tcp.
In our experience the performance impact from running the messages through a
disk buffer is small enough, and TCP connections are brittle enough, that it
wasn't worth introducing more complexity into the code to support both options.
We have plans to change the way buffering works entirely, however, so it's
optionally available to all plugins, this will change your options with regard
to the TcpOutput.
Is there plan for memory buffer output (buffered chan, ...).
There haven't been plans thus far, no. Buffered channels are always in play,
but there's no retry, messages that don't get through will be dropped. I'm
definitely open to considering in memory buffering options, but not until after
we get all of the details of the disk buffering situation worked out.
Output buffer shrink only every ticker_interval, which is 300sec, isn't
it too much by default? (on first run of heka with big input logs buffer
can grow a lot)
Whether or not this is a reasonable default depends on how much data is flowing
through, and how precious a commodity disk space is. I'm open to lowering it,
however, and explaining the implications a bit better in the docs.
queue_full_action = block doesn't recover (you have to restart heka) or
i'm not patient enough?
No, it's not you... I've done some testing and have been able to reproduce. I'm
currently working on a fix, more soon.
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
