[
https://issues.apache.org/jira/browse/NUTCH-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17930697#comment-17930697
]
ASF GitHub Bot commented on NUTCH-3108:
---------------------------------------
tballison commented on PR #849:
URL: https://github.com/apache/nutch/pull/849#issuecomment-2685115436
Thank you @sebastian-nagel for beating me to it. Y, we had to shade
commons-io because hadoop was using an old version, and Tika and POI were using
some of the newer API calls.
I just released this shim for 2.9.3 and 3.1.0. Maybe give those a try?
I did notice this:
https://github.com/tballison/hadoop-safe-tika/blob/main/tika-parsers-standard-package-shaded/pom.xml#L67
Which may be causing the problems. If we have to downgrade slf4j to match
hadoop, we can do that...or maybe we shade logging too?
I'm happy to make those updates and release a 2.9.3.1 and/or 3.1.0.1. :D
Let me know what makes sense.
> Fix SLF4J Class Loader Conflict in language-identifier
> ------------------------------------------------------
>
> Key: NUTCH-3108
> URL: https://issues.apache.org/jira/browse/NUTCH-3108
> Project: Nutch
> Issue Type: Bug
> Components: plugin
> Affects Versions: 1.20
> Reporter: Maciej Puzianowski
> Priority: Major
>
> When running Apache Nutch 1.20 on a distributed Hadoop cluster with the
> language-identifier plugin enabled, a class loader conflict occurs during the
> parse process. This results in the following error:
> {code:java}
> 2025-02-24 08:58:59,152 INFO mapreduce.Job: Task Id :
> attempt_1740061418437_0135_m_000001_0, Status : FAILED
> Error: loader constraint violation: when resolving method
> 'org.slf4j.ILoggerFactory
> org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()' the class loader
> org.apache.nutch.plugin.PluginClassLoader @4c5228e7 of the current class,
> org/slf4j/LoggerFactory, and the class loader 'app' for the method's defining
> class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for
> the type org/slf4j/ILoggerFactory used in the signature
> (org.slf4j.LoggerFactory is in unnamed module of loader
> org.apache.nutch.plugin.PluginClassLoader @4c5228e7, parent loader 'app';
> org.slf4j.impl.StaticLoggerBinder is in unnamed module of loader 'app')
> {code}
> I have managed to resolve this issue by modifying following files:
> ivy.xml ->
> {code:xml}
> <dependency org="org.apache.tika" name="tika-langdetect-optimaize"
> rev="2.9.0" conf="*->default">
> <!-- exclusions of dependencies provided in Nutch core (ivy/ivy.xml) -->
> <exclude org="org.apache.tika" name="tika-core" />
> <exclude org="com.google.guava" name="guava" />
> <exclude org="org.slf4j" name="slf4j-api" />
> <!-- exclusions of dependencies provided in Nutch core (ivy/ivy.xml) -->
> </dependency>
> {code}
> and plugin.xml ->
> {code:xml}
> <library name="annotations-12.0.jar"/>
> <library name="checker-qual-3.33.0.jar"/>
> <library name="error_prone_annotations-2.18.0.jar"/>
> <library name="failureaccess-1.0.1.jar"/>
> <library name="j2objc-annotations-2.8.jar"/>
> <library name="jsonic-1.2.11.jar"/>
> <library name="jsr305-3.0.2.jar"/>
> <library name="language-detector-0.6.jar"/>
> <library
> name="listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar"/>
> <library name="tika-langdetect-optimaize-2.9.0.jar"/>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
