Hi Jan,
Upgrading to 2.x version will take at-least 6-7 months for us to
migrate. About the heap, it is at 70% and no issues with it. No logs are
showing up based on the slow rate, apart from what i have posted. There was
an error on indice which mentioned about not able to calculate the range, i
did the "Recalculate index cycles" which fixed that, but the slow
processing has accumulated about 100,000,000 messages in the journal on the
Master Node.
Still finding what is happening.. :-(
Thanks Hema.
On Friday, July 8, 2016 at 5:40:47 PM UTC+5:30, Hema Kumar wrote:
>
> Hi,
> I am using graylog 1.3.3 with ES 1.7.5, from yesterday we are seeing
> the process buffer filled up on the master node and the outgoing process is
> too slow than normal, I have tried restarting GL and ES but did not fix the
> issue, below are the log warn and errors we see that repeats continuously.
>
> We have 4 graylog server and 7 elasticsearch nodes, Only the Master
> graylog is processing slow and sometimes the 3rd node, rest of the nodes
> are working fine.
>
> Could you please help me on this, i have been breaking my head since
> yesterday.
>
>
> 2016-07-08T01:53:21.355-06:00 WARN [GelfChunkAggregator] Error while
> expiring GELF chunk entries
> java.util.NoSuchElementException
> at
> java.util.concurrent.ConcurrentSkipListMap.firstKey(ConcurrentSkipListMap.java:2036)
> at
> java.util.concurrent.ConcurrentSkipListSet.first(ConcurrentSkipListSet.java:396)
> at
> org.graylog2.inputs.codecs.GelfChunkAggregator$ChunkEvictionTask.run(GelfChunkAggregator.java:288)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-07-08T05:37:47.803-06:00 ERROR [AnyExceptionClassMapper] Unhandled
> exception in REST resource
> org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
> execute phase [fetch], [reduce]
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.onFailure(TransportSearchQueryThenFetchAction.java:159)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:41)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassCastException:
> org.elasticsearch.search.aggregations.bucket.terms.LongTerms$Bucket cannot
> be cast to
> org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket
> at
> org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.compareTerm(StringTerms.java:85)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:87)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$4.compare(InternalOrder.java:83)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:284)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalOrder$CompoundOrder$CompoundOrderComparator.compare(InternalOrder.java:270)
> at
> org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:37)
> at
> org.elasticsearch.search.aggregations.bucket.terms.support.BucketPriorityQueue.lessThan(BucketPriorityQueue.java:26)
> at
> org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:225)
> at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:133)
> at
> org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:149)
> at
> org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.reduce(InternalTerms.java:195)
> at
> org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
> at
> org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.reduce(InternalSingleBucketAggregation.java:79)
> at
> org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:140)
> at
> org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:407)
> at
> org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.doRun(TransportSearchQueryThenFetchAction.java:147)
> at
> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
>
> 2016-07-08T05:43:30.757-06:00 WARN [DefaultExceptionMonitor] Unexpected
> exception.
> java.nio.channels.ClosedSelectorException
> at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)
> at
> org.apache.mina.transport.socket.nio.NioProcessor.isSelectorEmpty(NioProcessor.java:107)
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1139)
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
> Thanks, Hema.
>
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/4f40a9a3-8d16-43e7-80a2-06f854cf707c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
