[jira] [Created] (HADOOP-16765) Fix curator dependencies for gradle projects using hadoop-minicluster
Mate Szalay-Beko created HADOOP-16765:
-
Summary: Fix curator dependencies for gradle projects using
hadoop-minicluster
Key: HADOOP-16765
URL: https://issues.apache.org/jira/browse/HADOOP-16765
Project: Hadoop Common
Issue Type: Bug
Components: common
Reporter: Mate Szalay-Beko
*The Problem:*
The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava
dependency issue in the `hadoop-minicluster` module.
{code:java}
java.lang.NoSuchMethodError:
com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
at
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
at
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
at
org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:518)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477)
at
org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216)
{code}
The issue in that change is that even though Guava was excluded from the
`curator-client` module, just below that the `curator-framework` module is
defined and doesn't exclude Gauva:
[https://github.com/apache/hadoop/blob/fc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414]
This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined by
Hadoop:
{noformat}
+--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT
|+--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT
||+--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT
||+--- com.google.guava:guava:11.0.2 -> 27.0.1-jre
{noformat}
{noformat}
+--- org.apache.curator:curator-framework:4.2.0
|\--- org.apache.curator:curator-client:4.2.0
| +--- org.apache.zookeeper:zookeeper:3.5.4-beta ->
3.5.5.7.1.0.0-SNAPSHOT (*)
| +--- com.google.guava:guava:27.0.1-jre (*)
| \--- org.slf4j:slf4j-api:1.7.25{noformat}
*The root cause:*
I was able to reproduce this issue with some dummy projects, see
[https://github.com/symat/transitive-dependency-test]
It seems that gradle behaves in this case differently than maven. If someone is
using maven, then he will not see this problem, as the exclude rules defined
for the {{curator-client}} will be enforced even if the {{curator-client}}
comes transitively through the {{curator-framework}}. While using the
hadoop-minicluster in a gradle project will lead to this problem (unless extra
excludes / dependencies gets defined in the gradle project).
*The proposed solution* is to add the exclude rules for all Curator
dependencies, preventing other gradle projects using Hadoop from breaking
because of the Curator upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
Mate Szalay-Beko created HADOOP-16579: - Summary: Upgrade to Apache Curator 4.2.0 in Hadoop Key: HADOOP-16579 URL: https://issues.apache.org/jira/browse/HADOOP-16579 Project: Hadoop Common Issue Type: Improvement Reporter: Mate Szalay-Beko Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
