sebastian-nagel commented on a change in pull request #515:
URL: https://github.com/apache/nutch/pull/515#discussion_r415537853
##########
File path: src/plugin/indexer-solr/ivy.xml
##########
@@ -36,9 +36,9 @@
</publications>
<dependencies>
- <dependency org="org.apache.solr" name="solr-solrj" rev="7.3.1"/>
- <dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.6"
conf="*->default"/>
- <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.3"
conf="*->default"/>
+ <dependency org="org.apache.solr" name="solr-solrj" rev="8.5.1"/>
+ <dependency org="org.apache.httpcomponents" name="httpcore" rev="4.4.12"
conf="*->default"/>
+ <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.10"
conf="*->default"/>
Review comment:
Should be httpmime 4.5.10 (httpcore 4.4.12), otherwise you'll see:
```
$> bin/nutch indexchecker
-Dplugin.includes='protocol-okhttp|parse-tika|index-(basic|more|anchor)|indexer-solr'
\
-doIndex https://localhost/
...
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/http/entity/mime/content/ContentBody
at
org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:952)
at
org.apache.nutch.indexwriter.solr.SolrUtils.getHttpSolrClient(SolrUtils.java:59)
at
org.apache.nutch.indexwriter.solr.SolrIndexWriter.open(SolrIndexWriter.java:107)
at org.apache.nutch.indexer.IndexWriters.open(IndexWriters.java:216)
at
org.apache.nutch.indexer.IndexingFiltersChecker.process(IndexingFiltersChecker.java:296)
at
org.apache.nutch.util.AbstractChecker.processSingle(AbstractChecker.java:85)
at
org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:132)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:305)
Caused by: java.lang.ClassNotFoundException:
org.apache.http.entity.mime.content.ContentBody
```
##########
File path: src/plugin/indexer-solr/plugin.xml
##########
@@ -22,15 +22,36 @@
<library name="indexer-solr.jar">
<export name="*" />
</library>
- <library name="commons-io-2.5.jar"/>
- <library name="httpclient-4.5.3.jar"/>
+ <library name="commons-io-2.6.jar"/>
+ <library name="netty-buffer-4.1.29.Final.jar"/>
+ <library name="netty-codec-4.1.29.Final.jar"/>
+ <library name="netty-common-4.1.29.Final.jar"/>
+ <library name="netty-handler-4.1.29.Final.jar"/>
+ <library name="netty-resolver-4.1.29.Final.jar"/>
+ <library name="netty-transport-4.1.29.Final.jar"/>
+ <library name="netty-transport-native-epoll-4.1.29.Final.jar"/>
+ <library name="netty-transport-native-unix-common-4.1.29.Final.jar"/>
+ <library name="commons-math3-3.6.1.jar"/>
+ <library name="httpclient-4.5.10.jar"/>
<library name="httpcore-4.4.6.jar"/>
<library name="httpmime-4.5.3.jar"/>
- <library name="noggit-0.8.jar"/>
- <library name="solr-solrj-7.3.1.jar"/>
+ <library name="zookeeper-3.5.5.jar"/>
+ <library name="zookeeper-jute-3.5.5.jar"/>
<library name="stax2-api-3.1.4.jar"/>
<library name="woodstox-core-asl-4.4.1.jar"/>
- <library name="zookeeper-3.4.11.jar"/>
+ <library name="jetty-alpn-client-9.4.24.v20191120.jar"/>
+ <library name="jetty-alpn-java-client-9.4.24.v20191120.jar"/>
+ <library name="jetty-client-9.4.24.v20191120.jar"/>
+ <library name="jetty-http-9.4.24.v20191120.jar"/>
+ <library name="jetty-io-9.4.24.v20191120.jar"/>
+ <library name="jetty-util-9.4.24.v20191120.jar"/>
+ <library name="http2-client-9.4.24.v20191120.jar"/>
+ <library name="http2-common-9.4.24.v20191120.jar"/>
+ <library name="http2-hpack-9.4.24.v20191120.jar"/>
+ <library name="http2-http-client-transport-9.4.24.v20191120.jar"/>
+ <library name="jcl-over-slf4j-1.7.24.jar"/>
+ <library name="slf4j-api-1.7.24.jar"/>
Review comment:
```
$> bin/nutch indexchecker
-Dplugin.includes='protocol-okhttp|parse-tika|index-(basic|more|anchor)|indexer-solr'
\
-doIndex https://localhost/
...
Exception in thread "main" java.lang.LinkageError: loader constraint
violation: when resolving method
"org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()Lorg/slf4j/ILoggerFactory;"
the class loader (instance of org/apache/nutch/plugin/PluginClassLoader) of
the current class, org/slf4j/LoggerFactory, and the class loader (instance of
sun/misc/Launcher$AppClassLoader) for the method's defining class,
org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type
org/slf4j/ILoggerFactory used in the signature
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:418)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
at
org.apache.nutch.indexwriter.solr.SolrIndexWriter.<clinit>(SolrIndexWriter.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.nutch.plugin.Extension.getExtensionInstance(Extension.java:164)
at org.apache.nutch.indexer.IndexWriters.<init>(IndexWriters.java:97)
at
org.apache.nutch.indexer.IndexWriters.lambda$get$0(IndexWriters.java:60)
at java.util.Map.computeIfAbsent(Map.java:957)
at org.apache.nutch.indexer.IndexWriters.get(IndexWriters.java:60)
at
org.apache.nutch.indexer.IndexingFiltersChecker.process(IndexingFiltersChecker.java:295)
at
org.apache.nutch.util.AbstractChecker.processSingle(AbstractChecker.java:85)
at
org.apache.nutch.indexer.IndexingFiltersChecker.run(IndexingFiltersChecker.java:132)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.nutch.indexer.IndexingFiltersChecker.main(IndexingFiltersChecker.java:305)
```
Nutch already provides slf4j-api (version 1.7.26) which causes this error.
When class implementations are shared between plugins and the Nutch core
classes (obvious for logging classes because all write into the same log file)
it's mandatory to add an exclusion.
I've successfully tried using the following exclusions:
```xml
<dependency org="org.apache.solr" name="solr-solrj" rev="8.5.1">
<!-- exclusions of dependencies provided in Nutch core (ivy/ivy.xml)
-->
<exclude org="org.apache.commons" name="commons-codec" />
<exclude org="org.apache.commons" name="commons-logging" />
<exclude org="org.slf4j" name="slf4j-api" />
</dependency>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
