I am trying to cut Logstash out of my stack because i have moved all of its functionality into Nifi but I haven't been able to get Filebeat to play nicely with the ListenBeats processor so I have to keep Logstash around just to forward messages to Nifi.
I am using latest stable 5.5.2 but the ListenBeats processor throws these errors and gets yielded for 30s in an endless loop. 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29] > o.a.nifi.processors.beats.ListenBeats > ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e] Error reading Beats > frames due to Error decoding Beats frame: Error decompressing frame: > Unexpected end of ZLIB input stream: > org.apache.nifi.processors.beats.frame.BeatsFrameException: Error decoding > Beats frame: Error decompressing frame: Unexpected end of ZLIB input stream > > org.apache.nifi.processors.beats.frame.BeatsFrameException: Error decoding > Beats frame: Error decompressing frame: Unexpected end of ZLIB input stream > > at > org.apache.nifi.processors.beats.frame.BeatsDecoder.process(BeatsDecoder.java:123) > > at > org.apache.nifi.processors.beats.handler.BeatsSocketChannelHandler.processBuffer(BeatsSocketChannelHandler.java:71) > > Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException: > Error decompressing frame: Unexpected end of ZLIB input stream > > at > org.apache.nifi.processors.beats.frame.BeatsDecoder.processPAYLOAD(BeatsDecoder.java:292) > > at > org.apache.nifi.processors.beats.frame.BeatsDecoder.process(BeatsDecoder.java:103) > > at > org.apache.nifi.processors.beats.frame.BeatsDecoder.processPAYLOAD(BeatsDecoder.java:277) > while filebeat is logging Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: > sync.go:85: Failed to publish events caused by: read tcp > localhost:45416->remotehost:5044: i/o timeout Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: > single.go:91: Error publishing events (retrying): read tcp > localhost:45416-> remotehost:5044: i/o timeout Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]: > metrics.go:39: Non-zero metrics in the last 30s: > libbeat.logstash.call_count.PublishEvents=1 > libbeat.logstash.publish.read_bytes=186 > libbeat.logstash.publish.read_errors=1 > libbeat.logstash.publish.write_bytes=31399 > libbeat.logstash.published_and_acked_events=98 > libbeat.logstash.published_but_not_acked_events=1950 Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: > sync.go:85: Failed to publish events caused by: read tcp > localhost:49790-> remotehost:5044: i/o timeout Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: > single.go:91: Error publishing events (retrying): read tcp > localhost:49790-> remotehost:5044: i/o timeout Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]: > metrics.go:39: Non-zero metrics in the last 30s: > libbeat.logstash.call_count.PublishEvents=1 > libbeat.logstash.publish.read_bytes=138 > libbeat.logstash.publish.read_errors=1 > libbeat.logstash.publish.write_bytes=27456 > libbeat.logstash.published_and_acked_events=31 > libbeat.logstash.published_but_not_acked_events=1919 Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: > sync.go:85: Failed to publish events caused by: read tcp > localhost:54060-> remotehost:5044: i/o timeout Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: > single.go:91: Error publishing events (retrying): read tcp > localhost:54060-> remotehost:5044: i/o timeout Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]: > metrics.go:39: Non-zero metrics in the last 30s: > libbeat.logstash.call_count.PublishEvents=1 > libbeat.logstash.publish.read_bytes=492 > libbeat.logstash.publish.read_errors=1 > libbeat.logstash.publish.write_bytes=63110 > libbeat.logstash.published_and_acked_events=23 > libbeat.logstash.published_but_not_acked_events=1896 Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: > sync.go:85: Failed to publish events caused by: read tcp > localhost:58359-> remotehost:5044: i/o timeout Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: > single.go:91: Error publishing events (retrying): read tcp > localhost:58359-> remotehost:5044: i/o timeout has anyone seen something similar or have any idea what might be going on here? my processor params are: Port 5044 Receive Buffer Size 10 MB Max Size of Message Queue 10000 Max Size of Socket Buffer 8 MB Character Set UTF-8 Max Batch Size 1 Batching Message Delimiter \n Max Number of TCP Connections 100 SSL Context Service No value set Thanks, Nick
