[ https://issues.apache.org/jira/browse/HADOOP-16765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Íñigo Goiri resolved HADOOP-16765. ---------------------------------- Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix curator dependencies for gradle projects using hadoop-minicluster > --------------------------------------------------------------------- > > Key: HADOOP-16765 > URL: https://issues.apache.org/jira/browse/HADOOP-16765 > Project: Hadoop Common > Issue Type: Bug > Reporter: Mate Szalay-Beko > Assignee: Mate Szalay-Beko > Priority: Major > Fix For: 3.3.0 > > > *The Problem:* > The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava > dependency issue in the `hadoop-minicluster` module. > {code:java} > java.lang.NoSuchMethodError: > com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V > at > org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167) > at > org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156) > at > org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709) > at > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911) > at > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:518) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477) > at > org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216) > {code} > The issue in that change is that even though Guava was excluded from the > `curator-client` module, just below that the `curator-framework` module is > defined and doesn't exclude Gauva: > [https://github.com/apache/hadoop/blob/fccccc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414] > This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined > by Hadoop: > {noformat} > +--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT > | +--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT > | | +--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT > | | +--- com.google.guava:guava:11.0.2 -> 27.0.1-jre > {noformat} > {noformat} > +--- org.apache.curator:curator-framework:4.2.0 > | \--- org.apache.curator:curator-client:4.2.0 > | +--- org.apache.zookeeper:zookeeper:3.5.4-beta -> > 3.5.5.7.1.0.0-SNAPSHOT (*) > | +--- com.google.guava:guava:27.0.1-jre (*) > | \--- org.slf4j:slf4j-api:1.7.25{noformat} > > *The root cause:* > I was able to reproduce this issue with some dummy projects, see > [https://github.com/symat/transitive-dependency-test] > It seems that gradle behaves in this case differently than maven. If someone > is using maven, then he will not see this problem, as the exclude rules > defined for the {{curator-client}} will be enforced even if the > {{curator-client}} comes transitively through the {{curator-framework}}. > While using the hadoop-minicluster in a gradle project will lead to this > problem (unless extra excludes / dependencies gets defined in the gradle > project). > *The proposed solution* is to add the exclude rules for all Curator > dependencies, preventing other gradle projects using Hadoop from breaking > because of the Curator upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org