Hello,
I've been using graylog for a couple weeks now and started to notice some
unusual behavior today. I am currently running a single node setup.
The Issue:
Every once in awhile I start to notice that that graylog is dragging quite
a bit (the loading spinner is persisting much longer than usual) so I go
check the logs and find the following error message.
ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a
response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException:
java.io.IOException: Connection closed
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92)
~[graylog.jar:?]
at
org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162)
~[graylog.jar:?]
at
org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130)
~[graylog.jar:?]
at
org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711)
[graylog.jar:?]
at
org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
[graylog.jar:?]
at
org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
[graylog.jar:?]
at
org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329)
[graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
[graylog.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
[graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
[graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
[graylog.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
[graylog.jar:?]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
[graylog.jar:?]
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
[graylog.jar:?]
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
[graylog.jar:?]
at
org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384)
[graylog.jar:?]
at
org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224)
[graylog.jar:?]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
[graylog.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_91]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: java.io.IOException: Connection closed
at
org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:317)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:501)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:412)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:604)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.NIOConnection$5.run(NIOConnection.java:570)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:235)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:564)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:197)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344)
~[graylog.jar:?]
at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107)
~[graylog.jar:?]
at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
~[graylog.jar:?]
at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
~[graylog.jar:?]
at
org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
~[graylog.jar:?]
at
org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
~[graylog.jar:?]
at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
~[graylog.jar:?]
at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
~[graylog.jar:?]
... 1 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_91]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
~[?:1.8.0_91]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
~[?:1.8.0_91]
at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_91]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
~[?:1.8.0_91]
at
org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:187)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344)
~[graylog.jar:?]
at
org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107)
~[graylog.jar:?]
at
org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536)
~[graylog.jar:?]
at
org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
~[graylog.jar:?]
at
org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103)
~[graylog.jar:?]
at
org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348)
~[graylog.jar:?]
at
org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279)
~[graylog.jar:?]
at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591)
~[graylog.jar:?]
at
org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571)
~[graylog.jar:?]
... 1 more
At times I get a notification in the graylog web interface in conjunction
with this error even though my *is_master = true *is indeed set :
*There was no master Graylog server node detected in the
cluster. (triggered a few seconds ago)*
*Certain operations of Graylog server require the presence of a master
node, but no such master was started. Please ensure that one of your
Graylog server nodes contains the setting is_master = true in its
configuration and that it is running. Until this is resolved index cycling
will not be able to run, which means that the index retention mechanism is
also not running, leading to increased index sizes. Certain maintenance
functions as well as a variety of web interface pages (e.g. Dashboards) are
unavailable.*
After a couple minutes the notification just disappears and I continue to
use graylog as normal. My thinking is that there is a communication issue
somewhere along the tech stack. If anyone has any thoughts as to what might
be causing the issue or where else I could look for error logs to further
investigate please let me know. Thanks for your time.
Regards,
Ariel
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/9f5b54ed-facc-4661-8acd-9ac730302366%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
