Hi,
Thanks for reply.
I’m working with Алексей Лакустов.
We have reconfigured graylog with
processbuffer_processors = 10
outputbuffer_processors = 10
and set ES_HEAP_SIZE to 15GB.
Configuration of buffer you can see below:
sysctl -a | grep "udp\|net.core.rmem_max\|net.core.netdev_max_backlog"
net.core.rmem_max = 536870912
net.core.netdev_max_backlog = 30000
net.ipv4.udp_mem = 725361 967149 1450722
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
Graylog works fine after restarting of graylog service. But few minutes
late we see errors on Web interface:
- · “? Msg/s on 1 node”;
- · “The processing of stream *Deny* has taken too long for 3
times. To protect the stability of message processing, this stream has been
disabled. Please correct the stream rules and reenable the stream.” (Rule
of stream is “full_message must match regular expression .?Deny?.”);
- · A lot of “Indexer failures” with error
“SendRequestTransportException[[graylog2-server][inet[/10.6.134.99:9300]][bulk/shard]];
nested: OutOfMemoryError[GC overhead limit exceeded];”
Below you can see errors from graylog2-server.log
…
2014-08-11 11:39:54,984 WARN : org.graylog2.streams.StreamRouter -
Processing of stream <53e0933ce4b086e645c2c6f0> failed to return within
2000ms.
2014-08-11 11:39:54,984 WARN : org.graylog2.streams.StreamRouter -
Processing of stream <53e0933ce4b086e645c2c6f0> failed to return within
2000ms.
2014-08-11 11:42:54,650 INFO : org.elasticsearch.monitor.jvm -
[graylog2-server] [gc][old][567][9] duration [6s], collections [1]/[7.1s],
total [6s]/[25.7s], memory [5.6gb]->[4.6gb]/[6.5gb], all_pools {[young]
[689.3mb]->[1mb]/[838.5mb]}{[survivor] [378.9mb]->[0b]/[837.5mb]}{[old]
[4.5gb]->[4.6gb]/[4.9gb]}
2014-08-11 11:42:54,670 ERROR:
org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception
during jersey resource handling
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:118)
at
org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
at
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at
sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at
java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flush(BufferedWriter.java:254)
at
org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
at
org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:243)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:230)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:103)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:88)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1139)
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:574)
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381)
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
at
org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:306)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-08-11 11:42:54,675 ERROR:
org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception
during jersey resource handling
java.nio.channels.ClosedChannelException
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.discard(ChunkedWriteHandler.java:168)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:285)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:118)
at
org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
at
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at
sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at
java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flush(BufferedWriter.java:254)
at
org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
at
org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:243)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:230)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:103)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:88)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1139)
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:574)
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381)
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
at
org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:306)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-08-11 11:42:54,676 INFO :
org.graylog2.jersey.container.netty.NettyContainer - Not writing any
response, channel is already closed.
java.nio.channels.ClosedChannelException
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.discard(ChunkedWriteHandler.java:168)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:285)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:118)
at
org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
at
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at
sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at
java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flush(BufferedWriter.java:254)
at
org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
at
org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:243)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:230)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:103)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:88)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1139)
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:574)
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381)
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
at
org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:306)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-08-11 11:43:04,699 INFO : org.elasticsearch.monitor.jvm -
[graylog2-server] [gc][old][572][10] duration [5.5s], collections [1]/[6s],
total [5.5s]/[31.3s], memory [5.3gb]->[4.7gb]/[6.5gb], all_pools {[young]
[740.2mb]->[3.1mb]/[838.5mb]}{[survivor] [0b]->[0b]/[837.5mb]}{[old]
[4.6gb]->[4.7gb]/[4.9gb]}
2014-08-11 11:43:04,699 ERROR: org.graylog2.streams.StreamRouter -
Processing of stream <53e0933ce4b086e645c2c6f0> failed to return within
2000ms for more than 3 times. Disabling stream.
2014-08-11 11:43:04,708 ERROR:
org.graylog2.jersey.container.netty.NettyContainer - Uncaught exception
during jersey resource handling
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at
org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:146)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
at
org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:280)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:121)
at org.jboss.netty.channel.Channels.write(Channels.java:704)
at org.jboss.netty.channel.Channels.write(Channels.java:671)
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at
org.graylog2.jersey.container.netty.NettyContainer$NettyResponseWriter$1.write(NettyContainer.java:118)
at
org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
at
sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at
sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
at
sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at
java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flush(BufferedWriter.java:254)
at
org.glassfish.jersey.message.internal.ReaderWriter.writeToAsString(ReaderWriter.java:192)
at
org.glassfish.jersey.message.internal.AbstractMessageReaderWriterProvider.writeToAsString(AbstractMessageReaderWriterProvider.java:129)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:99)
at
org.glassfish.jersey.message.internal.StringMessageProvider.writeTo(StringMessageProvider.java:59)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:243)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:230)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:103)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:88)
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1139)
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:574)
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:381)
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:371)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:262)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at
org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:236)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1010)
at
org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:306)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
…
Also, few minutes late we can’t login to web interface. It say “Sorry,
those credentials are invalid.”.
Restart of service resolves all issues for short period of time.
Thanks for any suggestions.
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
