Mike de Rhino created NIFI-856:
-----------------------------------
Summary: Add suport to Lumberjack protocol
Key: NIFI-856
URL: https://issues.apache.org/jira/browse/NIFI-856
Project: Apache NiFi
Issue Type: New Feature
Reporter: Mike de Rhino
It would be great if NIFI could support the lumberjack protocol so to enable
the use of logstash forwarder as a source of data.
A lot of non Java shops tend to avoid installing Java at data producing nodes
and instead of Flume they end up using things like kafka, heka, fluentd or
logstash-forwarded as data shipping mechanisms.
Kafka is great but its architecture seem to be better focused on multi-DC
environments instead of multi-branch scenarios (imagine having to manager 80
Zookeeper quorum, one for each country where you operate?)
[Heka|https://github.com/mozilla-services/heka] is fine, it has decent
backpressure buffering but no concept of acknowledgement on the receiving side
of a TCP stream. If the other end of a TCP stream is capable of listening but
gets stuck with its messages it will keep spitting data through the pipe,
oblivious to the woes at the other end.
Logstash forwarded in the other hand, is a quite simple tool, with a reasonable
implementation of acknowledgments on the receiving side but... it depends on
Logstash(and logstash has its own issues).
It would be great if NIFI could serve as a middle man, receiving [lumberjack
messages|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
and offloading some of the hard work Logstash seems to struggle with (e.g.
using NIFI to save to HDFS while a downstream Logstash writes into ES).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
