Other than the jmap -heap I didn't manage to look more specifically into it:
https://gist.github.com/theflow/b983d512ea344545f7f6#file-jmap The same process runs fine on much smaller machines in our staging environment, without the live traffic, of course. Anything particular I should run that would give more insights? Cheers, Florian On Tuesday, June 3, 2014 12:21:32 PM UTC+2, Mark Walkom wrote: > > How does your heap look during all this? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [email protected] <javascript:> > web: www.campaignmonitor.com > > > On 3 June 2014 20:14, Florian Munz <[email protected] <javascript:>> wrote: > >> Hello, >> >> we recently moved our ES cluster from dedicated hardware to AWS >> instances, they have less memory available, but use SSDs for the ES data >> directory. We kept JVM (1.7.0_17) and ES (0.90.9) version exactly the same. >> On the new hardware, after running a full re-index (creating a new index, >> pointing an alias to the new and one alias to the old index, sending >> realtime updates to both aliases and running a script to fill up the new >> index) our cluster gets stuck. >> >> 10 minutes after the re-index finishes and we move both aliases to the >> new index, ES stops answering any search or index queries, no errors in the >> logs apart from it not answering queries anymore: >> >> org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: >> rejected execution (queue capacity 1000) on >> org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4@172018e5 >> >> CPU load is low, it doesn't look like it's doing anything expensive. A >> request to hot_threads times out. I've put the output from jstack and jmap >> here: >> >> https://gist.github.com/theflow/b983d512ea344545f7f6 >> >> We tried upgrading to 0.90.13, since the changelog mentioned a problem >> with infinite loops, but same behavior. We're planning to upgrade to a more >> recent version of ES soon, but it'll take a bit to fully test that. >> >> >> Any ideas what could be causing this? >> >> >> thanks, >> Florian >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/7a347529-df1a-4a21-9ac1-d3af882a035a%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/7a347529-df1a-4a21-9ac1-d3af882a035a%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04b3d0a2-a47e-47c6-8411-eb619c3c54bc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
