Hi Yogesh, please post your complete Graylog configuration file and describe your setup in more detail.
Cheers, Jochen On Friday, 25 November 2016 02:34:53 UTC+1, Yogesh Rao wrote: > > Hi, > > We have just started using graylog2 (2.1.2) on centos 7. I keep getting > below exception after few hours > > 2016-11-24T17:25:13.737+05:30 WARN [ProxiedResource] Unable to call > http://10.101.160.95:9000/api/system on node > > java.net.SocketTimeoutException: timeout > > at okio.Okio$3.newTimeoutException(Okio.java:210) ~[graylog.jar:?] > > at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[graylog.jar:?] > > at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[graylog.jar:?] > > at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) > ~[graylog.jar:?] > > at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) > ~[graylog.jar:?] > > at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) > ~[graylog.jar:?] > > at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) > ~[graylog.jar:?] > > at > okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) > ~[graylog.jar:?] > > at > okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) > > ~[graylog.jar:?] > > at > okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) > > ~[graylog.jar:?] > > at > okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) > > ~[graylog.jar:?] > > at > org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) > > ~[graylog.jar:?] > > at > okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) > > ~[graylog.jar:?] > > at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) > ~[graylog.jar:?] > > at okhttp3.RealCall.execute(RealCall.java:60) ~[graylog.jar:?] > > at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?] > > at > org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) > > ~[graylog.jar:?] > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102] > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > [?:1.8.0_102] > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > [?:1.8.0_102] > > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] > > Caused by: java.net.SocketTimeoutException: Read timed out > > at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_102] > > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > ~[?:1.8.0_102] > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > ~[?:1.8.0_102] > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > ~[?:1.8.0_102] > > at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?] > > at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[graylog.jar:?] > > ... 29 more > > 2016-11-24T17:25:13.751+05:30 WARN [jvm] > [graylog-b8d88e71-dbbb-44ee-9951-2bcdb45b047a] [gc][old][19743][529] > duration [12.4s], collections [1]/[12.6s], total [12.4s]/[10.4m], memory > [3.7gb]->[3.7gb]/[3.8gb], all_pools {[young] > [1.6gb]->[1.6gb]/[1.6gb]}{[survivor] [188.7mb]->[191.2mb]/[204.7mb]}{[old] > [1.9gb]->[1.9gb]/[2gb]} > 2016-11-24T17:25:13.821+05:30 INFO [BeatsCodec] inside > decodeMessagesRawMessage{id=52182873-b20e-11e6-9b98-14feb5ea6076, > journalOffset=960249, codec=beats, payloadSize=4169, > timestamp=2016-11-24T06:22:05.798Z, remoteAddress=/10.101.160.6:49324} > > > Heap Size set for Graylog2 is 4G, I noticed the heap size jumps up from > 1.5G to 4G as soon as the exceptions are encountered, eventually leading to > slower GC's and finally ends up unresponsive. > > All the buffer processors are set to 1 > process_buffer and output buffer are set to 32768 > > Data pumped into graylog is around 4 GB > > Graylog is configured on a Core 2 Duo machine with 8GB Ram and is pumping > data into 3 elasticsearch (clustered) instances. > > > How can i troubleshoot this issue to figure out what the actual problem is > ? > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d23f0a4b-d7f0-4c7c-9699-4ebcc42c8e38%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
