Hi Jimmy,
- Which version of Graylog and Elasticsearch are you running?
- Which hardware are you using to run these?
- Are there other error or warning messages in the logs of your Graylog
and Elasticsearch nodes?
Cheers,
Jochen
On Tuesday, 8 November 2016 22:05:03 UTC+1, Jimmy Chen wrote:
>
> I am having some major problems with our production Graylog server
> processing incoming logs. We are seeing a large amount of errors like the
> following. It is not clear to me what the source or the message that is
> causing Graylog to choke.
>
> 2016-11-08 12:56:16,728 ERROR: org.graylog2.shared.buffers.processors.
> DecodingProcessor - Unable to decode raw message RawMessage{id=8d8a9927-
> a5f5-11e6-98e1- 0050568f1eb6, journalOffset=338236794, codec=gelf,
> payloadSize=8, timestamp=2016-11-08T20:54:34.162Z} on input <
> 56be878ce4b0e6e046749b5b>.
> 2016-11-08 12:56:16,728 ERROR: org.graylog2.shared.buffers.processors.
> DecodingProcessor - Error processing message RawMessage{id=8d8a9927-a5f5-
> 11e6-98e1-0050 568f1eb6, journalOffset=338236794, codec=gelf, payloadSize=
> 8, timestamp=2016-11-08T20:54:34.162Z}
> com.fasterxml.jackson.databind.JsonMappingException: No content to map
> due to end-of-input
> at [Source: ; line: 1, column: 0]
> at com.fasterxml.jackson.databind.JsonMappingException.from(
> JsonMappingException.java:255) ~[graylog.jar:?]
> at com.fasterxml.jackson.databind.ObjectMapper._initForReading(
> ObjectMapper.java:3851) ~[graylog.jar:?]
> at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(
> ObjectMapper.java:3792) ~[graylog.jar:?]
> at com.fasterxml.jackson.databind.ObjectMapper.readTree(
> ObjectMapper.java:2332) ~[graylog.jar:?]
> at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:120)
> ~[graylog.jar:?]
> at org.graylog2.shared.buffers.processors.DecodingProcessor.
> processMessage(DecodingProcessor.java:146) ~[graylog.jar:?]
> at org.graylog2.shared.buffers.processors.DecodingProcessor.
> onEvent(DecodingProcessor.java:87) [graylog.jar:?]
> at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.
> onEvent(ProcessBufferProcessor.java:58) [graylog.jar:?]
> at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.
> onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [
> graylog.jar:?]
> at com.codahale.metrics.
> InstrumentedThreadFactory$InstrumentedRunnable.run(
> InstrumentedThreadFactory.java:66) [graylog.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>
>
>
> Furthermore, periodically, Graylog server becomes unstable and repeatedly
> spits out these errors. It seems to get stuck in this state for a bit then
> recovers. However, during this time, it is not able to process logs.
>
> 2016-11-07 16:15:04,382 WARN : org.graylog2.periodical.NodePingThread -
> Did not find meta info of this node. Re-registering.
>
>
> I need a way to figure out what the sources and messages are the culprit
> of these errors.
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/db51c476-bc32-4803-a64c-63de2d5ecaa8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
