Nick, I use it with:
filebeat: spool_size: 2048 output.logstash: compression_level: 2 bulk_max_size: 100 timeout: 30 On the NiFi side I have: Receive Buffer Size: 128KB Max Size of Message Queue: 10000 Max Size of Socket Buffer: 2MB Can you try and let me know? Perhaps something changed on the filebeat size that accentuated the issue. If that's the case I am happy to try to prioritise fixing it. Cheers On Fri, Aug 25, 2017 at 5:08 AM, Nick Carenza < [email protected]> wrote: > It seemed to hold up a little bit better but I ended up getting the same > errors. > > On Thu, Aug 24, 2017 at 11:35 AM, Nick Carenza < > [email protected]> wrote: > >> Okay, thank you Andre! >> >> - Nick >> >> On Wed, Aug 23, 2017 at 4:03 PM, Andre <[email protected]> wrote: >> >>> Nick, >>> >>> This is a bug (NIFI-3753). I am also able to confirm there is a >>> workaround: just change the compression settings to a lower compression >>> level (I believe anything between none and 3 should work) and/or reduce the >>> number of messages batched on a single beats payload. >>> >>> Cheers >>> >>> On 23 Aug 2017 09:11, "Nick Carenza" <[email protected]> >>> wrote: >>> >>>> I am trying to cut Logstash out of my stack because i have moved all of >>>> its functionality into Nifi but I haven't been able to get Filebeat to play >>>> nicely with the ListenBeats processor so I have to keep Logstash around >>>> just to forward messages to Nifi. >>>> >>>> I am using latest stable 5.5.2 but the ListenBeats processor throws >>>> these errors and gets yielded for 30s in an endless loop. >>>> >>>> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29] >>>>> o.a.nifi.processors.beats.ListenBeats >>>>> ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e] >>>>> Error reading Beats frames due to Error decoding Beats frame: Error >>>>> decompressing frame: Unexpected end of ZLIB input stream: >>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error >>>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB >>>>> input stream >>>>> >>>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error >>>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB >>>>> input stream >>>>> >>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process( >>>>> BeatsDecoder.java:123) >>>>> >>>>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH >>>>> andler.processBuffer(BeatsSocketChannelHandler.java:71) >>>>> >>>>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException: >>>>> Error decompressing frame: Unexpected end of ZLIB input stream >>>>> >>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP >>>>> AYLOAD(BeatsDecoder.java:292) >>>>> >>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process( >>>>> BeatsDecoder.java:103) >>>>> >>>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP >>>>> AYLOAD(BeatsDecoder.java:277) >>>>> >>>> while filebeat is logging >>>> >>>> >>>> >>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> sync.go:85: Failed to publish events caused by: read tcp >>>>> localhost:45416->remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> single.go:91: Error publishing events (retrying): read tcp >>>>> localhost:45416-> remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> metrics.go:39: Non-zero metrics in the last 30s: >>>>> libbeat.logstash.call_count.PublishEvents=1 >>>>> libbeat.logstash.publish.read_bytes=186 >>>>> libbeat.logstash.publish.read_errors=1 >>>>> libbeat.logstash.publish.write_bytes=31399 >>>>> libbeat.logstash.published_and_acked_events=98 >>>>> libbeat.logstash.published_but_not_acked_events=1950 >>>> >>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> sync.go:85: Failed to publish events caused by: read tcp >>>>> localhost:49790-> remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> single.go:91: Error publishing events (retrying): read tcp >>>>> localhost:49790-> remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> metrics.go:39: Non-zero metrics in the last 30s: >>>>> libbeat.logstash.call_count.PublishEvents=1 >>>>> libbeat.logstash.publish.read_bytes=138 >>>>> libbeat.logstash.publish.read_errors=1 >>>>> libbeat.logstash.publish.write_bytes=27456 >>>>> libbeat.logstash.published_and_acked_events=31 >>>>> libbeat.logstash.published_but_not_acked_events=1919 >>>> >>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> sync.go:85: Failed to publish events caused by: read tcp >>>>> localhost:54060-> remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> single.go:91: Error publishing events (retrying): read tcp >>>>> localhost:54060-> remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> metrics.go:39: Non-zero metrics in the last 30s: >>>>> libbeat.logstash.call_count.PublishEvents=1 >>>>> libbeat.logstash.publish.read_bytes=492 >>>>> libbeat.logstash.publish.read_errors=1 >>>>> libbeat.logstash.publish.write_bytes=63110 >>>>> libbeat.logstash.published_and_acked_events=23 >>>>> libbeat.logstash.published_but_not_acked_events=1896 >>>> >>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> sync.go:85: Failed to publish events caused by: read tcp >>>>> localhost:58359-> remotehost:5044: i/o timeout >>>> >>>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>>> single.go:91: Error publishing events (retrying): read tcp >>>>> localhost:58359-> remotehost:5044: i/o timeout >>>> >>>> >>>> has anyone seen something similar or have any idea what might be going >>>> on here? >>>> >>>> my processor params are: >>>> >>>> >>>> Port >>>> 5044 >>>> >>>> Receive Buffer Size >>>> 10 MB >>>> >>>> Max Size of Message Queue >>>> 10000 >>>> >>>> Max Size of Socket Buffer >>>> 8 MB >>>> >>>> Character Set >>>> UTF-8 >>>> >>>> Max Batch Size >>>> 1 >>>> >>>> Batching Message Delimiter >>>> \n >>>> >>>> Max Number of TCP Connections >>>> 100 >>>> >>>> SSL Context Service >>>> No value set >>>> >>>> Thanks, >>>> Nick >>>> >>> >> >
