I am having some major problems with our production Graylog server
processing incoming logs. We are seeing a large amount of errors like the
following. It is not clear to me what the source or the message that is
causing Graylog to choke.
2016-11-08 12:56:16,728 ERROR: org.graylog2.shared.buffers.processors.
DecodingProcessor - Unable to decode raw message RawMessage{id=8d8a9927-a5f5
-11e6-98e1- 0050568f1eb6, journalOffset=338236794, codec=gelf, payloadSize=8
, timestamp=2016-11-08T20:54:34.162Z} on input <56be878ce4b0e6e046749b5b>.
2016-11-08 12:56:16,728 ERROR: org.graylog2.shared.buffers.processors.
DecodingProcessor - Error processing message RawMessage{id=8d8a9927-a5f5-
11e6-98e1-0050 568f1eb6, journalOffset=338236794, codec=gelf, payloadSize=8,
timestamp=2016-11-08T20:54:34.162Z}
com.fasterxml.jackson.databind.JsonMappingException: No content to map due
to end-of-input
at [Source: ; line: 1, column: 0]
at com.fasterxml.jackson.databind.JsonMappingException.from(
JsonMappingException.java:255) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(
ObjectMapper.java:3851) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(
ObjectMapper.java:3792) ~[graylog.jar:?]
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper
.java:2332) ~[graylog.jar:?]
at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:120)
~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.
processMessage(DecodingProcessor.java:146) ~[graylog.jar:?]
at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(
DecodingProcessor.java:87) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.
onEvent(ProcessBufferProcessor.java:58) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.
onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [
graylog.jar:?]
at com.codahale.metrics.
InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory
.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Furthermore, periodically, Graylog server becomes unstable and repeatedly
spits out these errors. It seems to get stuck in this state for a bit then
recovers. However, during this time, it is not able to process logs.
2016-11-07 16:15:04,382 WARN : org.graylog2.periodical.NodePingThread - Did
not find meta info of this node. Re-registering.
I need a way to figure out what the sources and messages are the culprit of
these errors.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/294e30fe-4a22-44e3-b7b0-799692f0dcfa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
