Also here is what we are seeing in our logs. If it's an issue I'm more then
happy to raise a ticket in github:
>From the master elasticsearch log:
[2014-04-16 18:57:17,298][WARN ][*transport* ] [Byrrah] Received response
for a request that has timed out, sent [31116ms] ago, timed out [1115ms]ago,
action
[discovery/zen/fd/ping], node [[Aleta Ogord][EvVSzG4lQoq9A4gpCaVPsw]
[ip-xxxxx.compute.internal][inet[ip-xxxxxx.compute.internal/xxxxx:9300]]
[master=false]], id [38161759]
[2014-04-17 00:00:53,471][WARN ][*transport* ] [Byrrah] Received response
for a request that has timed out, sent [32850ms] ago, timed out [2850ms]ago,
action
[discovery/zen/fd/ping], node [[Aleta Ogord][EvVSzG4lQoq9A4gpCaVPsw]
[ip-xxxxxx.compute.internal][inet[ip-xxxxx.compute.internal/xxxxxx:9300]]
[master=false]], id [38401500]
The above was repeated consistently until this:
[2014-04-17 07:07:21,179][INFO ][*cluster.service* ] [Byrrah] removed
{[Aleta
Ogord][EvVSzG4lQoq9A4gpCaVPsw][ip-xxxxxx.compute.internal][inet[ip-xxxxxx.compute.internal/xxxxxx:9300]]{master=false},},
reason: zen-disco-node_failed([Aleta Ogord][EvVSzG4lQoq9A4gpCaVPsw]
[ip-10-0-105-89.eu-west-1.compute.internal][inet
[ip-xxxxxx.compute.internal/xxxxxx:9300]][master=false]), reason failed to
ping, tried [3] times, each with maximum [30s] timeout
On the data node we saw this logs:
[monitor.jvm ] [Aleta Ogord] [gc][old][31491][1023] duration
[20s], collections [1]/[20.3s], total [20s]/[4.6h], memory
[4.8gb]->[4.8gb]/[4.9gb], all_pools {[young]
[25.4mb]->[14.6mb]/[133.1mb]}{[survivor] [0b]->[0b]/[16.6mb]}{[old]
[4.8gb]->[4.8gb]/[4.8gb]}
Until finally we got this:
[2014-04-17 08:42:53,630][WARN ][indices.ttl ] [Aleta Ogord] failed to
execute ttl purge
java.lang.OutOfMemoryError: Java heap space
[2014-04-17 08:52:42,761][WARN ][index.engine.internal ] [Aleta Ogord]
[pyxis-aggregator-2014-04-17][1] failed engine
java.lang.OutOfMemoryError: Java heap space
Throughout this entire period the cluster was in green state but was not
usable.
Thanks
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b4739458-0a85-405e-ae92-9543e55cfddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
