Thanks for your response. GC was my first thought too. I have looked through the logs and ran the app through a profiler, I am not seeing any spike in GC activity or any other background thread when performance degrades. Also, the fact that the slowdown occurs exactly every minute at the same second would point me towards a more deliberate timeout or heartbeat.
I am running these tests in a controlled performance environment with constant light to moderate load. There is no change in the behaviour when under very light load. I have turned on slow logging for queries/fetches but am not seeing any slow queries corresponding with the problem. The only time I see a slow query is post-cold start of the search node, so it is at least working. On Wednesday, April 15, 2015 at 1:00:00 AM UTC-4, Mark Walkom wrote: > > Have you checked the logs for GC events or similar? What about the web > logs for events coming in? > > On 15 April 2015 at 09:03, Daryl Robbins <[email protected] > <javascript:>> wrote: > >> I am seeing a consistent bottleneck in requests (taking about 2+ seconds) >> at the same second every minute across all four of my client nodes who are >> connecting using the transport client from Java. These nodes are completely >> independent aside from their reliance on the ElasticSearch cluster and >> consequently they all happen to pause at the exact same second every >> minute. The exact second when this happens varies over time, but the four >> nodes always pause at the same time. >> >> I have 4 web nodes that connect to my ES cluster via transport. They >> connect to a load balancer fronting our 3 dedicated master nodes. The >> cluster contains 2 or more data nodes dependent on the configuration. >> Regardless of the number, I am seeing the same symptoms. >> >> Any hints on how to proceed to troubleshoot this issue on the >> ElasticSearch side would be greatly appreciated. Thanks very much! >> >> >> <https://lh3.googleusercontent.com/-GKiOcsPXBjI/VS2ak04mzBI/AAAAAAAAAhQ/aLDlD82AddY/s1600/Screenshot%2B2015-04-14%2B18.53.24.png> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/af209904-9113-43d0-8cbc-0c85afe52611%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/af209904-9113-43d0-8cbc-0c85afe52611%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c78aa9dd-4a77-467c-ba88-41534db0f1ad%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
