[
https://issues.apache.org/jira/browse/NIFI-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike de Rhino updated NIFI-856:
--------------------------------
Description:
It would be great if NIFI could support the [lumberjack
protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
so to enable the use of logstash forwarder as a source of data.
A lot of non Java shops tend to avoid installing Java at data producing nodes
and instead of Flume they end up using things like kafka, heka, fluentd or
logstash-forwarded as data shipping mechanisms.
Kafka is great but its architecture seem to be better focused on multi-DC
environments instead of multi-branch scenarios (imagine having to manager 80
Zookeeper quorum, one for each country where you operate?)
[Heka|https://github.com/mozilla-services/heka] is fine, it has decent
backpressure buffering but no concept of acknowledgement on the receiving side
of a TCP stream. If the other end of a TCP stream is capable of listening but
gets stuck with its messages it will keep spitting data through the pipe,
oblivious to the woes at the other end.
Logstash forwarded in the other hand, is a quite simple tool, with a reasonable
implementation of acknowledgments on the receiving side but... it depends on
Logstash(and logstash has its own issues).
It would be great if NIFI could serve as a middle man, receiving lumberjack
messages and offloading some of the hard work Logstash seems to struggle with
(e.g. using NIFI to save to HDFS while a downstream Logstash writes into ES).
was:
It would be great if NIFI could support the lumberjack protocol so to enable
the use of logstash forwarder as a source of data.
A lot of non Java shops tend to avoid installing Java at data producing nodes
and instead of Flume they end up using things like kafka, heka, fluentd or
logstash-forwarded as data shipping mechanisms.
Kafka is great but its architecture seem to be better focused on multi-DC
environments instead of multi-branch scenarios (imagine having to manager 80
Zookeeper quorum, one for each country where you operate?)
[Heka|https://github.com/mozilla-services/heka] is fine, it has decent
backpressure buffering but no concept of acknowledgement on the receiving side
of a TCP stream. If the other end of a TCP stream is capable of listening but
gets stuck with its messages it will keep spitting data through the pipe,
oblivious to the woes at the other end.
Logstash forwarded in the other hand, is a quite simple tool, with a reasonable
implementation of acknowledgments on the receiving side but... it depends on
Logstash(and logstash has its own issues).
It would be great if NIFI could serve as a middle man, receiving [lumberjack
messages|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
and offloading some of the hard work Logstash seems to struggle with (e.g.
using NIFI to save to HDFS while a downstream Logstash writes into ES).
Summary: Add Processor for Lumberjack protocol (was: Add suport to
Lumberjack protocol)
> Add Processor for Lumberjack protocol
> -------------------------------------
>
> Key: NIFI-856
> URL: https://issues.apache.org/jira/browse/NIFI-856
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Mike de Rhino
> Labels: features
>
> It would be great if NIFI could support the [lumberjack
> protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
> so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes
> and instead of Flume they end up using things like kafka, heka, fluentd or
> logstash-forwarded as data shipping mechanisms.
> Kafka is great but its architecture seem to be better focused on multi-DC
> environments instead of multi-branch scenarios (imagine having to manager 80
> Zookeeper quorum, one for each country where you operate?)
> [Heka|https://github.com/mozilla-services/heka] is fine, it has decent
> backpressure buffering but no concept of acknowledgement on the receiving
> side of a TCP stream. If the other end of a TCP stream is capable of
> listening but gets stuck with its messages it will keep spitting data through
> the pipe, oblivious to the woes at the other end.
> Logstash forwarded in the other hand, is a quite simple tool, with a
> reasonable implementation of acknowledgments on the receiving side but... it
> depends on Logstash(and logstash has its own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack
> messages and offloading some of the hard work Logstash seems to struggle with
> (e.g. using NIFI to save to HDFS while a downstream Logstash writes into ES).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
