Using elasticsearch embedded in tomcat 7 where we have custom classloader that shares spring application beans with many webapps. The API's to these implementations are in shared but the implementations are in a separate classloader "ComponentLoader". Our search implementation is loaded from ComponentLoader where elasticsearch has been promoted as our default search engine.
Everything works fine on a single node but when in a cluster seeing the following ClassNotFoundException issue when shards are trying to update on other nodes. Caused by: java.lang.ClassNotFoundException: org.elasticsearch.transport.RemoteTransportException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547) at org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93) at org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601) I thought this might have been related to https://github.com/elasticsearch/elasticsearch/issues/4634 but that doesn't seem to be the case. My next idea is to look at why org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass is invoking WebappClassLoader instead of our custom ComponentLoader and coerce it to use ours. But I wanted to get some opinions on this strategy as I am new to elasticsearch ;) my thanks, Earle Full stack trace: 2014-08-05 01:25:26,685 WARN elasticsearch[app02][generic][T#1] org.elasticsearch.indices.cluster - [app02] [sakai_index][0] failed to start shard org.elasticsearch.indices.recovery.RecoveryFailedException: [sakai_index][0]: Recovery failed from [app01][0YtJIFeHSuehUfjjfMgv6A][ip-10-93-162-196][inet[/10.93.162.196:9300]]{local=false} into [app02][yshRulD0QjaNyu40Z6EGWQ][ip-10-7-174-145][inet[/10.7.174.145:9300]]{local=false} at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:307) at org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65) at org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.InvalidClassException: failed to read class descriptor at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1603) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:167) ... 23 more Caused by: java.lang.ClassNotFoundException: org.elasticsearch.transport.RemoteTransportException at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1702) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1547) at org.elasticsearch.common.io.ThrowableObjectInputStream.loadClass(ThrowableObjectInputStream.java:93) at org.elasticsearch.common.io.ThrowableObjectInputStream.readClassDescriptor(ThrowableObjectInputStream.java:67) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601) ... 28 more -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17542a40-7b6c-4c31-9e3e-1fdd2fe7f90b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
