Hi,
I am using graylog 1.3.3 with ES 1.7.5, from yesterday we are seeing the
process buffer filled up on the master node and the outgoing process is too
slow than normal, I have tried restarting GL and ES but did not fix the
issue, below are the log warn and errors we see that repeats continuously.
We have 4 graylog server and 7 elasticsearch nodes, Only the Master graylog
is processing slow and sometimes the 3rd node, rest of the nodes are
working fine.
Could you please help me on this, i have been breaking my head since
yesterday.
2016-07-08T01:53:21.355-06:00 WARN [GelfChunkAggregator] Error while
expiring GELF chunk entries
java.util.NoSuchElementException
at
java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
at
java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
at
org.graylog2.inputs.codecs.GelfChunkAggregator$ChunkEvictionTask.run(GelfChunkAggregator.java:288)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-07-08T05:37:47.803-06:00 ERROR [AnyExceptionClassMapper] Unhandled
exception in REST resource
org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.onFailure(TransportSearchQueryThenFetchAction.java:159)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:41)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException:
org.elasticsearch.search.aggregations.bucket.terms.LongTerms$Bucket cannot
be cast to
org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket
at
org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.compareTerm(StringTerms.java:85)
at
org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:87)
at
org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:83)
at
org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:284)
at
org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:270)
at
org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:37)
at
org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:26)
at
org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:225)
at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:133)
at
org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:149)
at
org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.reduce(InternalTerms.java:195)
at
org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
at
org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.reduce(InternalSingleBucketAggregation.java:79)
at
org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:407)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.doRun(TransportSearchQueryThenFetchAction.java:147)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
2016-07-08T05:43:30.757-06:00 WARN [DefaultExceptionMonitor] Unexpected
exception.
java.nio.channels.ClosedSelectorException
at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)
at
org.apache.mina.transport.socket.nio.NioProcessor.isSelectorEmpty(NioProcessor.java:107)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1139)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks, Hema.
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/fc0333b5-2c6b-4216-92cb-67f8fc57d296%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
