(Sorry, I hadn't notice this issues when I browsed all them) Thank you, it's a great news about buffering!
Have you consider making heka pipeline synchronous or transactional? I mean something like that: - if output plugin cannot write the message pack (more than one) outside or do writing, it just blocks - all tied modules (encoders, filters, pipeline, decoders, inputs) also eventually blocks - while input plugin is blocked, it doesn't acknowledge input data (e.g. LogStreamer doesn't write position to journal or HttpListen doesn't response). This mode may prevent losing messages at all. (But it might decrease performance.) Now heka doesn't protected from crashes or something else. If OOM killer kills heka, I'll lose 3 messages at minimum (because of channels in the pipeline + input and output plugins). ---- On Wed, 15 Oct 2014 22:58:07 +0700 Rob Miller <[email protected]> wrote ---- Nimi is right that the TcpOutput actually does buffer messages to disk. Originally that functionality was built directly in to the TcpOutput, but we later abstracted it out so it could be used by other output plugins. What hasn't been mentioned so far is that we have plans to change the interaction btn encoders and outputs and, as part of that, we plan on making the output buffering automatically available as a configuration option for *every* output. There are already a couple of issues open to capture this: https://github.com/mozilla-services/heka/issues/930 and: https://github.com/mozilla-services/heka/issues/1103 which contains this relevant comment: https://github.com/mozilla-services/heka/issues/1103#issuecomment-58548339 This will be a fair amount of work. It is all intended to land before we release Heka 1.0, which is targeted for January 2015. -r On 10/15/2014 03:48 AM, Denis Shashkov wrote: > > Hello! > > AFAIK, now heka doesn't guarantee message delivering in case of output > plugin couldn't write message: > - output plugins didn't buffer or re-try write operations, > - there is no interior buffer between pipeline and output plugins (I > know about channels, but they have finite length and they located in > memory), > - you cannot write a buffering filter (because you cannot write all > messages back into pipeline). > > (Please, correct me if I wrong). > > I thought a lot about how not to lost messages if my storage (e.g. HTTP > server) is unavailable. I decided it will be great if some output plugin > will be special: if other outputs cannot write messages, this fallback > output will write instead of them. > > Can I do this without touching pipeline code? >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
