[
https://issues.apache.org/jira/browse/NIFI-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15101723#comment-15101723
]
Andre commented on NIFI-856:
----------------------------
Functional code has been uploaded to the URL above.
Runtime with default settings
{code}
2016/01/15 22:51:24.247008 Setting trusted CA from file: ../certs/selfsigned.crt
2016/01/15 22:51:24.247774 Connecting to [127.0.0.1]:12344 (development.shv01)
2016/01/15 22:51:24.389020 Connected to 127.0.0.1
2016/01/15 22:51:24.481649 Registrar: processing 1024 events
2016/01/15 22:51:26.046868 Registrar: processing 1024 events
2016/01/15 22:51:26.958385 Registrar: processing 1024 events
2016/01/15 22:51:27.829147 Registrar: processing 1024 events
2016/01/15 22:51:28.445449 Registrar: processing 1024 events
2016/01/15 22:51:29.272289 Registrar: processing 1024 events
2016/01/15 22:51:30.014493 Registrar: processing 1024 events
2016/01/15 22:51:30.499132 Registrar: processing 1024 events
2016/01/15 22:51:31.045010 Registrar: processing 1024 events
2016/01/15 22:51:31.737989 Registrar: processing 1024 events
2016/01/15 22:51:36.766126 Registrar: processing 260 events
{code}
Runtime with batches of 4096 messages per frame {{(-sv 4096)}}
{code}
2016/01/15 23:01:23.949294 Loading client ssl certificate:
../certs/selfsigned.crt and ../certs/server.key
2016/01/15 23:01:23.952709 Setting trusted CA from file: ../certs/selfsigned.crt
2016/01/15 23:01:23.953216 Connecting to [127.0.0.1]:12344 (development.shv01)
2016/01/15 23:01:24.045095 Connected to 127.0.0.1
2016/01/15 23:01:24.152665 Registrar: processing 4096 events
2016/01/15 23:01:25.835721 Registrar: processing 4096 events
2016/01/15 23:01:31.537973 Registrar: processing 2308 events
{code}
> Add Processor for Lumberjack protocol
> -------------------------------------
>
> Key: NIFI-856
> URL: https://issues.apache.org/jira/browse/NIFI-856
> Project: Apache NiFi
> Issue Type: New Feature
> Reporter: Mike de Rhino
> Labels: features
>
> It would be great if NIFI could support the [lumberjack
> protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md]
> so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes
> and instead of Flume they end up using things like kafka, heka, fluentd or
> logstash-forwarded as data shipping mechanisms.
> Kafka is great but its architecture seem to be better focused on multi-DC
> environments instead of multi-branch scenarios (imagine having to manager 80
> Zookeeper quorum, one for each country where you operate?)
> [Heka|https://github.com/mozilla-services/heka] is fine, it has decent
> backpressure buffering but no concept of acknowledgement on the receiving
> side of a TCP stream. If the other end of a TCP stream is capable of
> listening but gets stuck with its messages it will keep spitting data through
> the pipe, oblivious to the woes at the other end.
> Logstash forwarder in the other hand, is a quite simple tool, with a
> reasonable implementation of acknowledgments on the receiving side but... it
> depends on Logstash(and logstash has its own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack
> messages and offloading some of the hard work Logstash seems to struggle with
> (e.g. using NIFI to save to HDFS while a downstream Logstash writes into ES).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
