Graylog is on 2.1.2 which was just updated yesterday. ElasticSearch is
2.3.5.
The Graylog server is running as a VM with 8 cores, 4G of RAM, 1G assigned
to heap. We have the web front end running on another VM and connected as
Graylog slave.
The ES data nodes are all physical Dell R330 1Us with E3-1270v4 64G of RAM
and 1TB SSDs, 5 total.
I am not seeing any errors in the data nodes when this error is happening
on the Graylog server, and this happens constantly on it.
On Wednesday, November 9, 2016 at 1:16:28 AM UTC-8, Jochen Schalanda wrote:
>
> Hi Jimmy,
>
>
> - Which version of Graylog and Elasticsearch are you running?
> - Which hardware are you using to run these?
> - Are there other error or warning messages in the logs of your
> Graylog and Elasticsearch nodes?
>
>
> Cheers,
> Jochen
>
> On Tuesday, 8 November 2016 22:05:03 UTC+1, Jimmy Chen wrote:
>>
>> I am having some major problems with our production Graylog server
>> processing incoming logs. We are seeing a large amount of errors like the
>> following. It is not clear to me what the source or the message that is
>> causing Graylog to choke.
>>
>> 2016-11-08 12:56:16,728 ERROR: org.graylog2.shared.buffers.processors.
>> DecodingProcessor - Unable to decode raw message RawMessage{id=8d8a9927-
>> a5f5-11e6-98e1- 0050568f1eb6, journalOffset=338236794, codec=gelf,
>> payloadSize=8, timestamp=2016-11-08T20:54:34.162Z} on input <
>> 56be878ce4b0e6e046749b5b>.
>> 2016-11-08 12:56:16,728 ERROR: org.graylog2.shared.buffers.processors.
>> DecodingProcessor - Error processing message RawMessage{id=8d8a9927-a5f5-
>> 11e6-98e1-0050 568f1eb6, journalOffset=338236794, codec=gelf, payloadSize
>> =8, timestamp=2016-11-08T20:54:34.162Z}
>> com.fasterxml.jackson.databind.JsonMappingException: No content to map
>> due to end-of-input
>> at [Source: ; line: 1, column: 0]
>> at com.fasterxml.jackson.databind.JsonMappingException.from(
>> JsonMappingException.java:255) ~[graylog.jar:?]
>> at com.fasterxml.jackson.databind.ObjectMapper._initForReading(
>> ObjectMapper.java:3851) ~[graylog.jar:?]
>> at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(
>> ObjectMapper.java:3792) ~[graylog.jar:?]
>> at com.fasterxml.jackson.databind.ObjectMapper.readTree(
>> ObjectMapper.java:2332) ~[graylog.jar:?]
>> at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:120
>> ) ~[graylog.jar:?]
>> at org.graylog2.shared.buffers.processors.DecodingProcessor.
>> processMessage(DecodingProcessor.java:146) ~[graylog.jar:?]
>> at org.graylog2.shared.buffers.processors.DecodingProcessor.
>> onEvent(DecodingProcessor.java:87) [graylog.jar:?]
>> at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.
>> onEvent(ProcessBufferProcessor.java:58) [graylog.jar:?]
>> at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.
>> onEvent(ProcessBufferProcessor.java:35) [graylog.jar:?]
>> at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [
>> graylog.jar:?]
>> at com.codahale.metrics.
>> InstrumentedThreadFactory$InstrumentedRunnable.run(
>> InstrumentedThreadFactory.java:66) [graylog.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
>>
>>
>>
>> Furthermore, periodically, Graylog server becomes unstable and repeatedly
>> spits out these errors. It seems to get stuck in this state for a bit then
>> recovers. However, during this time, it is not able to process logs.
>>
>> 2016-11-07 16:15:04,382 WARN : org.graylog2.periodical.NodePingThread -
>> Did not find meta info of this node. Re-registering.
>>
>>
>> I need a way to figure out what the sources and messages are the culprit
>> of these errors.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/c520f001-bbe2-484a-95f9-1931ce490010%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
