Increasing the heap size on ES and Graylog respectively fixed the issue. On Friday, July 8, 2016 at 11:07:46 AM UTC-5, Ariel Godinez wrote: > > After further investigation I think this was due to elasticsearch and > graylog being overloaded. I have increased their heap sizes accordingly and > will see how the system performs. > > Ariel > > On Wednesday, July 6, 2016 at 12:21:11 PM UTC-5, Ariel Godinez wrote: >> >> Hello, >> >> I've been using graylog for a couple weeks now and started to notice some >> unusual behavior today. I am currently running a single node setup. >> >> The Issue: >> >> Every once in awhile I start to notice that that graylog is dragging >> quite a bit (the loading spinner is persisting much longer than usual) so I >> go check the logs and find the following error message. >> >> ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a >> response message entity to the container output stream. >> org.glassfish.jersey.server.internal.process.MappableException: >> java.io.IOException: Connection closed >> at >> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) >> >> ~[graylog.jar:?] >> at >> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) >> >> ~[graylog.jar:?] >> at >> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) >> >> ~[graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors.process(Errors.java:315) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors.process(Errors.java:297) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors.process(Errors.java:267) >> [graylog.jar:?] >> at >> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) >> >> [graylog.jar:?] >> at >> org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) >> [graylog.jar:?] >> at >> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) >> >> [graylog.jar:?] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> [?:1.8.0_91] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> [?:1.8.0_91] >> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] >> Caused by: java.io.IOException: Connection closed >> at >> org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:317) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:501) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:412) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:604) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.NIOConnection$5.run(NIOConnection.java:570) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:235) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:564) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:197) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) >> >> ~[graylog.jar:?] >> ... 1 more >> Caused by: java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> ~[?:1.8.0_91] >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >> ~[?:1.8.0_91] >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) >> ~[?:1.8.0_91] >> at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_91] >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) >> ~[?:1.8.0_91] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:187) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) >> >> ~[graylog.jar:?] >> ... 1 more >> >> At times I get a notification in the graylog web interface in conjunction >> with this error even though my *is_master = true *is indeed set : >> >> *There was no master Graylog server node detected in the >> cluster. (triggered a few seconds ago)* >> *Certain operations of Graylog server require the presence of a master >> node, but no such master was started. Please ensure that one of your >> Graylog server nodes contains the setting is_master = true in its >> configuration and that it is running. Until this is resolved index cycling >> will not be able to run, which means that the index retention mechanism is >> also not running, leading to increased index sizes. Certain maintenance >> functions as well as a variety of web interface pages (e.g. Dashboards) are >> unavailable.* >> >> After a couple minutes the notification just disappears and I continue to >> use graylog as normal. My thinking is that there is a communication issue >> somewhere along the tech stack. If anyone has any thoughts as to what might >> be causing the issue or where else I could look for error logs to further >> investigate please let me know. Thanks for your time. >> >> Regards, >> Ariel >> >>
-- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f27e8c99-78d1-411c-9d81-939b9b54b7fe%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
