Andre, Very cool that you have made progress here. Being able to integrate with logstash will be very useful.
I think the refactoring I'm doing for the RELP stuff should help reduce the amount of code that had to be carried over from ListenSyslog. I'm happy to help you update your code once my changes are in. Sorry it hasn't gotten in sooner. -Bryan On Fri, Jan 15, 2016 at 8:39 AM, Andre <[email protected]> wrote: > Hey folks, > > I've managed to progress on ListenLumberjack. The code is a bit > 'spaghettic' at the moment, with some serious amount of logger.____ enabled > to allow some additional troubleshooting, but overall it "works". > > I am strongly considering refactor the code as whole once Bryan completes > the ListenRELP processor. > > Functional code (I guess? :D ) should be available in here: > > https://github.com/trixpan/nifi-lumberjack-bundle/ > > Known issues: > * If logstash-forwarder goes silent for too long the processor will raise a > Timeout. Couldn't find evidence of a keep alive within Lumberjack so I am > considering catching this error as debug. > * I suspect the code may have some memory leaks. > * Tests haven't been created yet. To be honest I never wrote unit tests in > my whole life so it will be another ride. :-) > > My results were the following: > > Single thread, 2 sec runs > 2016/01/15 23:52:27.589014 Registrar: processing 4000 events > 2016/01/15 23:52:29.169361 Registrar: processing 4000 events > 2016/01/15 23:52:30.552031 Registrar: processing 4000 events > 2016/01/15 23:52:32.998425 Registrar: processing 4000 events > 2016/01/15 23:52:35.411438 Registrar: processing 4000 events > 2016/01/15 23:52:37.062141 Registrar: processing 4000 events > 2016/01/15 23:52:39.468577 Registrar: processing 4000 events > 2016/01/15 23:52:40.940890 Registrar: processing 4000 events > 2016/01/15 23:52:43.480875 Registrar: processing 4000 events > 2016/01/15 23:52:45.026758 Registrar: processing 4000 events > > 4 threads, 2 sec runs > 2016/01/15 23:56:03.376303 Registrar: processing 4000 events > 2016/01/15 23:56:03.443074 Registrar: processing 4000 events > 2016/01/15 23:56:03.471795 Registrar: processing 4000 events > 2016/01/15 23:56:03.508283 Registrar: processing 4000 events > 2016/01/15 23:56:03.534002 Registrar: processing 4000 events > 2016/01/15 23:56:03.562387 Registrar: processing 4000 events > 2016/01/15 23:56:03.587744 Registrar: processing 4000 events > 2016/01/15 23:56:03.622716 Registrar: processing 4000 events > 2016/01/15 23:56:03.649074 Registrar: processing 4000 events > 2016/01/15 23:56:03.675780 Registrar: processing 4000 events > > Would anyone have a decent logstash testbed to put some extra pressure > against the processor? >
