Okay, thank you Andre! - Nick
On Wed, Aug 23, 2017 at 4:03 PM, Andre <[email protected]> wrote: > Nick, > > This is a bug (NIFI-3753). I am also able to confirm there is a > workaround: just change the compression settings to a lower compression > level (I believe anything between none and 3 should work) and/or reduce the > number of messages batched on a single beats payload. > > Cheers > > On 23 Aug 2017 09:11, "Nick Carenza" <[email protected]> > wrote: > >> I am trying to cut Logstash out of my stack because i have moved all of >> its functionality into Nifi but I haven't been able to get Filebeat to play >> nicely with the ListenBeats processor so I have to keep Logstash around >> just to forward messages to Nifi. >> >> I am using latest stable 5.5.2 but the ListenBeats processor throws these >> errors and gets yielded for 30s in an endless loop. >> >> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29] >>> o.a.nifi.processors.beats.ListenBeats >>> ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e] >>> Error reading Beats frames due to Error decoding Beats frame: Error >>> decompressing frame: Unexpected end of ZLIB input stream: >>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error >>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB >>> input stream >>> >>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error >>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB >>> input stream >>> >>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process( >>> BeatsDecoder.java:123) >>> >>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH >>> andler.processBuffer(BeatsSocketChannelHandler.java:71) >>> >>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException: >>> Error decompressing frame: Unexpected end of ZLIB input stream >>> >>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP >>> AYLOAD(BeatsDecoder.java:292) >>> >>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process( >>> BeatsDecoder.java:103) >>> >>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP >>> AYLOAD(BeatsDecoder.java:277) >>> >> while filebeat is logging >> >> >> >> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> sync.go:85: Failed to publish events caused by: read tcp >>> localhost:45416->remotehost:5044: i/o timeout >> >> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> single.go:91: Error publishing events (retrying): read tcp >>> localhost:45416-> remotehost:5044: i/o timeout >> >> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> metrics.go:39: Non-zero metrics in the last 30s: >>> libbeat.logstash.call_count.PublishEvents=1 >>> libbeat.logstash.publish.read_bytes=186 >>> libbeat.logstash.publish.read_errors=1 >>> libbeat.logstash.publish.write_bytes=31399 >>> libbeat.logstash.published_and_acked_events=98 >>> libbeat.logstash.published_but_not_acked_events=1950 >> >> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> sync.go:85: Failed to publish events caused by: read tcp >>> localhost:49790-> remotehost:5044: i/o timeout >> >> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> single.go:91: Error publishing events (retrying): read tcp >>> localhost:49790-> remotehost:5044: i/o timeout >> >> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> metrics.go:39: Non-zero metrics in the last 30s: >>> libbeat.logstash.call_count.PublishEvents=1 >>> libbeat.logstash.publish.read_bytes=138 >>> libbeat.logstash.publish.read_errors=1 >>> libbeat.logstash.publish.write_bytes=27456 >>> libbeat.logstash.published_and_acked_events=31 >>> libbeat.logstash.published_but_not_acked_events=1919 >> >> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> sync.go:85: Failed to publish events caused by: read tcp >>> localhost:54060-> remotehost:5044: i/o timeout >> >> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> single.go:91: Error publishing events (retrying): read tcp >>> localhost:54060-> remotehost:5044: i/o timeout >> >> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> metrics.go:39: Non-zero metrics in the last 30s: >>> libbeat.logstash.call_count.PublishEvents=1 >>> libbeat.logstash.publish.read_bytes=492 >>> libbeat.logstash.publish.read_errors=1 >>> libbeat.logstash.publish.write_bytes=63110 >>> libbeat.logstash.published_and_acked_events=23 >>> libbeat.logstash.published_but_not_acked_events=1896 >> >> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> sync.go:85: Failed to publish events caused by: read tcp >>> localhost:58359-> remotehost:5044: i/o timeout >> >> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: >>> single.go:91: Error publishing events (retrying): read tcp >>> localhost:58359-> remotehost:5044: i/o timeout >> >> >> has anyone seen something similar or have any idea what might be going >> on here? >> >> my processor params are: >> >> >> Port >> 5044 >> >> Receive Buffer Size >> 10 MB >> >> Max Size of Message Queue >> 10000 >> >> Max Size of Socket Buffer >> 8 MB >> >> Character Set >> UTF-8 >> >> Max Batch Size >> 1 >> >> Batching Message Delimiter >> \n >> >> Max Number of TCP Connections >> 100 >> >> SSL Context Service >> No value set >> >> Thanks, >> Nick >> >
