Hi Sebastian,
I've continued to struggle with this on several levels (both local and
on Hadoop), and in the end tried to change the way the classloader works
(changed from parent-first to child-first) and this fixed the issues for
me.
I've created an issue[1] + patch for it. Can you let me know what you
think of it?
Best regards,
Jurian
[1] https://issues.apache.org/jira/browse/NUTCH-2378
On 24-04-17 17:37, Sebastian Nagel wrote:
Hi Jurjan,
it's a known problem that plugin dependencies and main dependencies may cause
conflicts,
e.g, https://issues.apache.org/jira/browse/NUTCH-2316.
Every plugin is run in it's own class loader inheriting from the main class
loader
but that's only prevents from conflicts between plugins.
It's hard to say how and whether this can be fixed easily.
I would first try adjust or exclude netty from the main ivy/ivy.xmx,
or also just remove it from runtime/local/lib/
Best,
Sebastian
On 04/24/2017 01:25 PM, Jurian Broertjes wrote:
Hi list,
I'm trying to bump the ES lib version of the indexer-elastic plugin from 2.x to
5.x (in this case
5.3), but am running into runtime dependency issues. What is the correct way of
handling these kinds
of errors? I'm running Nutch 1.13
Exception:
java.lang.Exception: java.lang.NoSuchMethodError:
io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio/netty/buffer/CompositeByteBuf;
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.lang.NoSuchMethodError:
io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio/netty/buffer/CompositeByteBuf;
at
org.elasticsearch.transport.netty4.Netty4Utils.toByteBuf(Netty4Utils.java:78)
at
org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:422)
at
org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:93)
at
org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1058)
at
org.elasticsearch.transport.TcpTransport.sendRequestToChannel(TcpTransport.java:1040)
at
org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1555)
at
org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:502)
at
org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:460)
at
org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:314)
at
org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:408)
at
org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:354)
at
org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:195)
at
org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:322)
at
org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.makeClient(ElasticIndexWriter.java:170)
at
org.apache.nutch.indexwriter.elastic.ElasticIndexWriter.open(ElasticIndexWriter.java:120)
at org.apache.nutch.indexer.IndexWriters.open(IndexWriters.java:77)
at
org.apache.nutch.indexer.IndexerOutputFormat.getRecordWriter(IndexerOutputFormat.java:39)
at
org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:484)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:414)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Relevant parts:
ivy.xml:
<dependencies>
<dependency org="org.elasticsearch" name="elasticsearch" rev="5.3.0"
conf="*->default"/>
<dependency org="org.elasticsearch.client" name="transport" rev="5.3.0"/>
<dependency org="org.apache.logging.log4j" name="log4j-api" rev="2.7" />
<dependency org="org.apache.logging.log4j" name="log4j-core" rev="2.7" />
<dependency org="io.netty" name="netty-all" rev="4.1.7.Final"/>
</dependencies>
plugin.xml:
<runtime>
<library name="indexer-elastic.jar">
<export name="*" />
</library>
<library name="rest-5.3.0.jar"/>
<library name="commons-codec-1.10.jar"/>
<library name="commons-logging-1.1.3.jar"/>
<library name="httpasyncclient-4.1.2.jar"/>
<library name="httpclient-4.5.2.jar"/>
<library name="httpcore-4.4.5.jar"/>
<library name="httpcore-nio-4.4.5.jar"/>
<library name="elasticsearch-5.3.0.jar"/>
<library name="compiler-0.9.3.jar"/>
<library name="HdrHistogram-2.1.6.jar"/>
<library name="hppc-0.7.1.jar"/>
<library name="jackson-core-2.8.6.jar"/>
<library name="jackson-dataformat-cbor-2.8.6.jar"/>
<library name="jackson-dataformat-smile-2.8.6.jar"/>
<library name="jackson-dataformat-yaml-2.8.6.jar"/>
<library name="jna-4.2.2.jar"/>
<library name="joda-time-2.9.5.jar"/>
<library name="jopt-simple-5.0.2.jar"/>
<library name="log4j-api-2.7.jar"/>
<library name="log4j-core-2.7.jar"/>
<library name="lucene-analyzers-common-6.4.1.jar"/>
<library name="lucene-backward-codecs-6.4.1.jar"/>
<library name="lucene-core-6.4.1.jar"/>
<library name="lucene-grouping-6.4.1.jar"/>
<library name="lucene-highlighter-6.4.1.jar"/>
<library name="lucene-join-6.4.1.jar"/>
<library name="lucene-memory-6.4.1.jar"/>
<library name="lucene-misc-6.4.1.jar"/>
<library name="lucene-queries-6.4.1.jar"/>
<library name="lucene-queryparser-6.4.1.jar"/>
<library name="lucene-sandbox-6.4.1.jar"/>
<library name="lucene-spatial3d-6.4.1.jar"/>
<library name="lucene-spatial-6.4.1.jar"/>
<library name="lucene-spatial-extras-6.4.1.jar"/>
<library name="lucene-suggest-6.4.1.jar"/>
<library name="securesm-1.1.jar"/>
<library name="snakeyaml-1.15.jar"/>
<library name="t-digest-3.0.jar"/>
<library name="percolator-client-5.3.0.jar"/>
<library name="reindex-client-5.3.0.jar"/>
<library name="lang-mustache-client-5.3.0.jar"/>
<library name="transport-5.3.0.jar"/>
<library name="transport-netty3-client-5.3.0.jar"/>
<library name="transport-netty4-client-5.3.0.jar"/>
<library name="netty-all-4.1.7.Final.jar"/>
<library name="netty-3.10.6.Final.jar"/>
<library name="netty-buffer-4.1.7.Final.jar"/>
<library name="netty-codec-4.1.7.Final.jar"/>
<library name="netty-codec-http-4.1.7.Final.jar"/>
<library name="netty-common-4.1.7.Final.jar"/>
<library name="netty-handler-4.1.7.Final.jar"/>
<library name="netty-resolver-4.1.7.Final.jar"/>
<library name="netty-transport-4.1.7.Final.jar"/>
</runtime>
I think the problem might be caused by a conflicting version of netty jars.
HDFS seems to pull
4.0.23 in, but ES needs 4.1.7. What can be done to alleviate this?
If more info or src is needed, please let me know.
Best regards,
Jurian
