The issue I am seeing seems similar to what was reported
at
https://groups.google.com/forum/#!searchin/elasticsearch/sporadic$20/elasticsearch/jUsoUV3_mbo/nM1OtJ9tmW0J
I enabled more logging on the transport layer and I see the following
exceptions when the disconnect/reconnect happens.
[2015-01-08 17:32:18,216][TRACE][transport.netty ] [es1] close
connection exception caught on transport layer [[id: 0xc4e4b9a1,
/10.152.16.37:59038 => /10.109.172.201:9300]], disconnecting from relevant
node
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-01-08 17:32:18,217][DEBUG][transport.netty ] [es1]
disconnecting from
[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]],
channel closed event
[2015-01-08 17:32:18,217][TRACE][transport.netty ] [es1]
disconnected from
[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]],
channel closed event
.......
[2015-01-08 17:32:23,400][DEBUG][transport.netty ] [es1] connected
to node
[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]]
[2015-01-08 17:32:24,546][INFO ][cluster.service ] [es1] removed
{[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]],},
reason:
zen-disco-node_failed([es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]),
reason transport disconnected
[2015-01-08 17:32:26,979][DEBUG][transport.netty ] [es1]
disconnecting from
[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]]
due to explicit disconnect call
[2015-01-08 17:32:26,980][TRACE][transport.netty ] [es1]
disconnected from
[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]]
due to explicit disconnect call
[2015-01-08 17:32:27,075][TRACE][transport.netty ] [es1] channel
closed: [id: 0xdbb6e4b0, /10.109.172.201:45127 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0x211fc69e, /10.109.172.201:45128 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0x8bba710f, /10.109.172.201:45130 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0x79f540c2, /10.109.172.201:45129 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0xc65857d4, /10.109.172.201:45133 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0xbca56b4c, /10.109.172.201:45131 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0xa3b3f93f, /10.109.172.201:45132 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0x1400e583, /10.109.172.201:45134 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0x72d6fccf, /10.109.172.201:45138 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0xfb686698, /10.109.172.201:45126 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0xb1633576, /10.109.172.201:45136 => /10.152.16.37:9300]
[2015-01-08 17:32:27,076][TRACE][transport.netty ] [es1] channel
closed: [id: 0x782b9080, /10.109.172.201:45137 => /10.152.16.37:9300]
[2015-01-08 17:32:27,077][TRACE][transport.netty ] [es1] channel
closed: [id: 0x7ded7f98, /10.109.172.201:45135 => /10.152.16.37:9300]
[2015-01-08 17:32:27,301][TRACE][transport.netty ] [es1] channel
opened: [id: 0x9662aeb5, /10.109.172.201:45156 => /10.152.16.37:9300]
[2015-01-08 17:32:27,301][TRACE][transport.netty ] [es1] channel
opened: [id: 0x62567a47, /10.109.172.201:45155 => /10.152.16.37:9300]
[2015-01-08 17:32:27,301][TRACE][transport.netty ] [es1] channel
opened: [id: 0x09ad440e, /10.109.172.201:45157 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0x30d6d25d, /10.109.172.201:45158 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0x421bb20b, /10.109.172.201:45159 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0xd83d71b5, /10.109.172.201:45160 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0xde607f1d, /10.109.172.201:45162 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0x161c9ae0, /10.109.172.201:45163 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0x960d6ce3, /10.109.172.201:45161 => /10.152.16.37:9300]
[2015-01-08 17:32:27,302][TRACE][transport.netty ] [es1] channel
opened: [id: 0x8277c29e, /10.109.172.201:45165 => /10.152.16.37:9300]
[2015-01-08 17:32:27,303][TRACE][transport.netty ] [es1] channel
opened: [id: 0xe046cbc3, /10.109.172.201:45164 => /10.152.16.37:9300]
[2015-01-08 17:32:27,303][TRACE][transport.netty ] [es1] channel
opened: [id: 0x22233acf, /10.109.172.201:45166 => /10.152.16.37:9300]
[2015-01-08 17:32:27,303][TRACE][transport.netty ] [es1] channel
opened: [id: 0x5880bff7, /10.109.172.201:45167 => /10.152.16.37:9300]
[2015-01-08 17:32:42,422][TRACE][transport.netty ] [es1] channel
opened: [id: 0xe02aca38, /10.109.172.201:45168 => /10.152.16.37:9300]
[2015-01-08 17:32:57,535][TRACE][transport.netty ] [es1] channel
opened: [id: 0x489f219c, /10.109.172.201:45169 => /10.152.16.37:9300]
[2015-01-08 17:32:57,536][TRACE][transport.netty ] [es1] channel
closed: [id: 0xe02aca38, /10.109.172.201:45168 => /10.152.16.37:9300]
[2015-01-08 17:32:57,537][TRACE][transport.netty ] [es1] channel
closed: [id: 0x489f219c, /10.109.172.201:45169 => /10.152.16.37:9300]
\[2015-01-08 17:32:57,540][DEBUG][transport.netty ] [es1]
connected to node
[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]]
[2015-01-08 17:32:57,541][INFO ][cluster.service ] [es1] added
{[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]],},
reason: zen-disco-receive(join from
node[[es2][jAhWlTiKTASdHDQaZGVncw][ip-10-109-172-201][inet[/10.109.172.201:9300]]])
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ca1af751-4210-42c7-8c45-e9e2df3a1c15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
