I see the same behavior on 0.90.7. Any hints from ES members on this? Thanks.
On Friday, August 16, 2013 2:27:57 AM UTC-4, Robert Deaton wrote: > > Hi all, > > I booted up a brand new cluster, all running the same version of > elasticsearch, with no cruft around from any previous clusters or anything. > I'm seeing a lot of the following in my logs, shortly before nodes start > disconnecting. > > [root@qtau ~] # java -version > java version "1.7.0_25" > Java(TM) SE Runtime Environment (build 1.7.0_25-b15) > Java HotSpot(TM) Server VM (build 23.25-b01, mixed mode) > [root@qtau ~] # /opt/elasticsearch/bin/elasticsearch -v > Version: 0.90.3, Build: 5c38d60/2013-08-06T13:18:31Z, JVM: 1.7.0_25 > [root@qtau ~] # uname -a > SunOS qtau.quizlet.com 5.11 joyent_20130226T234312Z i86pc i386 i86pc > Solaris > > > > [2013-08-16 06:20:09,017][WARN ][transport.netty ] [qtau] Message > not fully read (response) for [87] handler > org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$4@1b9865f9, > > error [false], resetting > [2013-08-16 06:20:09,017][DEBUG][action.admin.cluster.node.info] [qtau] > failed to execute on node [Ho3YGCj8RXqVypaVyw693w] > org.elasticsearch.transport.RemoteTransportException: Failed to > deserialize response of type > [org.elasticsearch.action.admin.cluster.node.info.NodeInfo] > Caused by: org.elasticsearch.transport.TransportSerializationException: > Failed to deserialize response of type > [org.elasticsearch.action.admin.cluster.node.info.NodeInfo] > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:147) > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:124) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Expected handle header, got [105] > at > org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65) > at > org.elasticsearch.common.io.stream.StreamInput.readOptionalString(StreamInput.java:220) > at > org.elasticsearch.threadpool.ThreadPool$Info.readFrom(ThreadPool.java:737) > at > org.elasticsearch.threadpool.ThreadPoolInfo.readFrom(ThreadPoolInfo.java:65) > at > org.elasticsearch.threadpool.ThreadPoolInfo.readThreadPoolInfo(ThreadPoolInfo.java:55) > at > org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readFrom(NodeInfo.java:220) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:145) > ... 23 more > [2013-08-16 06:20:09,014][DEBUG][action.admin.cluster.node.info] [qtau] > failed to execute on node [GNcFyx5oQAS9gBPLXviJqw] > org.elasticsearch.transport.RemoteTransportException: Failed to > deserialize response of type > [org.elasticsearch.action.admin.cluster.node.info.NodeInfo] > Caused by: org.elasticsearch.transport.TransportSerializationException: > Failed to deserialize response of type > [org.elasticsearch.action.admin.cluster.node.info.NodeInfo] > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:147) > at > org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:124) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > at > org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) > at > org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) > at > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90) > at > org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) > at > org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) > at > org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: Expected handle header, got [104] > at > org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:65) > at > org.elasticsearch.common.io.stream.StreamInput.readOptionalString(StreamInput.java:220) > at > org.elasticsearch.threadpool.ThreadPool$Info.readFrom(ThreadPool.java:737) > at > org.elasticsearch.threadpool.ThreadPoolInfo.readFrom(ThreadPoolInfo.java:65) > at > org.elasticsearch.threadpool.ThreadPoolInfo.readThreadPoolInfo(ThreadPoolInfo.java:55) > at > org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readFrom(NodeInfo.java:220) > at > org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:145) > > Thanks, > -- > Robert Deaton > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db9ebb2e-60bc-43e2-83b7-fabd7b3ff213%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
