After further investigation I think this was due to elasticsearch and graylog being overloaded. I have increased their heap sizes accordingly and will see how the system performs.
Ariel On Wednesday, July 6, 2016 at 12:21:11 PM UTC-5, Ariel Godinez wrote: > > Hello, > > I've been using graylog for a couple weeks now and started to notice some > unusual behavior today. I am currently running a single node setup. > > The Issue: > > Every once in awhile I start to notice that that graylog is dragging quite > a bit (the loading spinner is persisting much longer than usual) so I go > check the logs and find the following error message. > > ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a > response message entity to the container output stream. > org.glassfish.jersey.server.internal.process.MappableException: > java.io.IOException: Connection closed > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) > > ~[graylog.jar:?] > at > org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) > > ~[graylog.jar:?] > at > org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) > > ~[graylog.jar:?] > at > org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) > > [graylog.jar:?] > at > org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) > > [graylog.jar:?] > at > org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) > > [graylog.jar:?] > at > org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) > [graylog.jar:?] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) > [graylog.jar:?] > at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) > [graylog.jar:?] > at org.glassfish.jersey.internal.Errors.process(Errors.java:315) > [graylog.jar:?] > at org.glassfish.jersey.internal.Errors.process(Errors.java:297) > [graylog.jar:?] > at org.glassfish.jersey.internal.Errors.process(Errors.java:267) > [graylog.jar:?] > at > org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) > > [graylog.jar:?] > at > org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) > [graylog.jar:?] > at > org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) > > [graylog.jar:?] > at > org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) > > [graylog.jar:?] > at > org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) > [graylog.jar:?] > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > [graylog.jar:?] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > [?:1.8.0_91] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > [?:1.8.0_91] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > Caused by: java.io.IOException: Connection closed > at > org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:317) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:501) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:412) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:604) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.NIOConnection$5.run(NIOConnection.java:570) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:235) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:564) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:197) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) > ~[graylog.jar:?] > at > org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) > > ~[graylog.jar:?] > ... 1 more > Caused by: java.io.IOException: Broken pipe > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) > ~[?:1.8.0_91] > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) > ~[?:1.8.0_91] > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) > ~[?:1.8.0_91] > at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_91] > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) > ~[?:1.8.0_91] > at > org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:187) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) > ~[graylog.jar:?] > at > org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) > ~[graylog.jar:?] > at > org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) > > ~[graylog.jar:?] > at > org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) > > ~[graylog.jar:?] > ... 1 more > > At times I get a notification in the graylog web interface in conjunction > with this error even though my *is_master = true *is indeed set : > > *There was no master Graylog server node detected in the > cluster. (triggered a few seconds ago)* > *Certain operations of Graylog server require the presence of a master > node, but no such master was started. Please ensure that one of your > Graylog server nodes contains the setting is_master = true in its > configuration and that it is running. Until this is resolved index cycling > will not be able to run, which means that the index retention mechanism is > also not running, leading to increased index sizes. Certain maintenance > functions as well as a variety of web interface pages (e.g. Dashboards) are > unavailable.* > > After a couple minutes the notification just disappears and I continue to > use graylog as normal. My thinking is that there is a communication issue > somewhere along the tech stack. If anyone has any thoughts as to what might > be causing the issue or where else I could look for error logs to further > investigate please let me know. Thanks for your time. > > Regards, > Ariel > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d6ca264c-fce3-4ecd-893c-14da84c2f832%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
