[jira] [Commented] (HADOOP-17612) Bump default Zookeeper version to 3.7.0
[ https://issues.apache.org/jira/browse/HADOOP-17612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311740#comment-17311740 ] Mate Szalay-Beko commented on HADOOP-17612: --- Thanks [~weichiu] for the ping and also thanks [~vjasani] for the PR! I'm not an active member in the Hadoop community, but I can provide some insight from ZooKeeper... 3.7.0 was just released very recently. I think I would recommend the latest 3.5 or 3.6 version instead, as those can be considered more stable. I know about many large companies and open-source projects and distributions using 3.5 now (migrated from 3.4 in the last 1-2 years), and also I know some that are using 3.6 in production. But I maybe wouldn't go with 3.7.0 as a requirement just yet... as 3.5 and 3.6 are still supported by the ZooKeeper community and seem to me more safe versions for Hadoop. Unless you want to use a specific feature in 3.7 that you miss from earlier ZooKeepers. cc. [~eolivelli] and [~andor] who are active PMC members in ZooKeeper, they might have more input about versions / end-of-life decisions. > Bump default Zookeeper version to 3.7.0 > --- > > Key: HADOOP-17612 > URL: https://issues.apache.org/jira/browse/HADOOP-17612 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > We can bump Zookeeper version to 3.7.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16918) Dependency update for Hadoop 2.10
[ https://issues.apache.org/jira/browse/HADOOP-16918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193492#comment-17193492 ] Mate Szalay-Beko commented on HADOOP-16918: --- Hello guys! I'm missing the context here a bit... still FYI: * ZooKeeper 3.4 is EOL now, I think no more new security updates / CVE fixes will be provided for the 3.4 line. The last 3.4 ZooKeeper is the 3.4.14 version, released on April, 2019. * The 3.5 / 3.6 ZooKeeper versions are still active, we provide relatively frequent releases with bugfixes and security fixes. > Dependency update for Hadoop 2.10 > - > > Key: HADOOP-16918 > URL: https://issues.apache.org/jira/browse/HADOOP-16918 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Priority: Major > Labels: release-blocker > Attachments: dependency-check-report.html, > dependency-check-report.html > > > A number of dependencies can be updated. > nimbus-jose-jwt > jetty > netty > zookeeper > hbase-common > jackson-databind > and many more. They should be updated in the 2.10.1 release. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16765) Fix curator dependencies for gradle projects using hadoop-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-16765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mate Szalay-Beko updated HADOOP-16765:
--
Component/s: (was: common)
> Fix curator dependencies for gradle projects using hadoop-minicluster
> -
>
> Key: HADOOP-16765
> URL: https://issues.apache.org/jira/browse/HADOOP-16765
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Mate Szalay-Beko
>Priority: Major
>
> *The Problem:*
> The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava
> dependency issue in the `hadoop-minicluster` module.
> {code:java}
> java.lang.NoSuchMethodError:
> com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
> at
> org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
> at
> org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:518)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477)
> at
> org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216)
> {code}
> The issue in that change is that even though Guava was excluded from the
> `curator-client` module, just below that the `curator-framework` module is
> defined and doesn't exclude Gauva:
> [https://github.com/apache/hadoop/blob/fc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414]
> This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined
> by Hadoop:
> {noformat}
> +--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT
> |+--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT
> ||+--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT
> ||+--- com.google.guava:guava:11.0.2 -> 27.0.1-jre
> {noformat}
> {noformat}
> +--- org.apache.curator:curator-framework:4.2.0
> |\--- org.apache.curator:curator-client:4.2.0
> | +--- org.apache.zookeeper:zookeeper:3.5.4-beta ->
> 3.5.5.7.1.0.0-SNAPSHOT (*)
> | +--- com.google.guava:guava:27.0.1-jre (*)
> | \--- org.slf4j:slf4j-api:1.7.25{noformat}
>
> *The root cause:*
> I was able to reproduce this issue with some dummy projects, see
> [https://github.com/symat/transitive-dependency-test]
> It seems that gradle behaves in this case differently than maven. If someone
> is using maven, then he will not see this problem, as the exclude rules
> defined for the {{curator-client}} will be enforced even if the
> {{curator-client}} comes transitively through the {{curator-framework}}.
> While using the hadoop-minicluster in a gradle project will lead to this
> problem (unless extra excludes / dependencies gets defined in the gradle
> project).
> *The proposed solution* is to add the exclude rules for all Curator
> dependencies, preventing other gradle projects using Hadoop from breaking
> because of the Curator upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16765) Fix curator dependencies for gradle projects using hadoop-minicluster
Mate Szalay-Beko created HADOOP-16765:
-
Summary: Fix curator dependencies for gradle projects using
hadoop-minicluster
Key: HADOOP-16765
URL: https://issues.apache.org/jira/browse/HADOOP-16765
Project: Hadoop Common
Issue Type: Bug
Components: common
Reporter: Mate Szalay-Beko
*The Problem:*
The Kudu unit tests that use the `MiniDFSCluster` are broken due to a guava
dependency issue in the `hadoop-minicluster` module.
{code:java}
java.lang.NoSuchMethodError:
com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
at
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
at
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
at
org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:166)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2794)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2709)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1669)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:911)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:518)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:477)
at
org.apache.kudu.backup.HDFSTestKuduBackupLister.setUp(TestKuduBackupLister.scala:216)
{code}
The issue in that change is that even though Guava was excluded from the
`curator-client` module, just below that the `curator-framework` module is
defined and doesn't exclude Gauva:
[https://github.com/apache/hadoop/blob/fc97034b29243a0509633849de55aa734859/hadoop-project/pom.xml#L1391-L1414]
This causes Guava 27.0.1-jre to be pulled in instead of Guava 11.0.2 defined by
Hadoop:
{noformat}
+--- org.apache.hadoop:hadoop-minicluster:3.1.1.7.1.0.0-SNAPSHOT
|+--- org.apache.hadoop:hadoop-common:3.1.1.7.1.0.0-SNAPSHOT
||+--- org.apache.hadoop:hadoop-annotations:3.1.1.7.1.0.0-SNAPSHOT
||+--- com.google.guava:guava:11.0.2 -> 27.0.1-jre
{noformat}
{noformat}
+--- org.apache.curator:curator-framework:4.2.0
|\--- org.apache.curator:curator-client:4.2.0
| +--- org.apache.zookeeper:zookeeper:3.5.4-beta ->
3.5.5.7.1.0.0-SNAPSHOT (*)
| +--- com.google.guava:guava:27.0.1-jre (*)
| \--- org.slf4j:slf4j-api:1.7.25{noformat}
*The root cause:*
I was able to reproduce this issue with some dummy projects, see
[https://github.com/symat/transitive-dependency-test]
It seems that gradle behaves in this case differently than maven. If someone is
using maven, then he will not see this problem, as the exclude rules defined
for the {{curator-client}} will be enforced even if the {{curator-client}}
comes transitively through the {{curator-framework}}. While using the
hadoop-minicluster in a gradle project will lead to this problem (unless extra
excludes / dependencies gets defined in the gradle project).
*The proposed solution* is to add the exclude rules for all Curator
dependencies, preventing other gradle projects using Hadoop from breaking
because of the Curator upgrade.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated deeply these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but [the latest PR|https://github.com/apache/hadoop/pull/1656] seems to be stable. ZooKeeper 3.5.6 just got released during our work. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the [Apache ZooKeeper ftp site|https://www-eu.apache.org/dist/zookeeper/]). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the [release notes|https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md]. Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released during our work. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the [apache zookeeper ftp site|[https://www-eu.apache.org/dist/zookeeper/]]). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release notes: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released during our work. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the [apache zookeeper ftp site|[https://www-eu.apache.org/dist/zookeeper/]]). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release notes: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the apache zookeeper ftp site). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release note: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the apache zookeeper ftp site). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release note: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Summary: Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop (was: Upgrade to Apache Curator 4.2.0 in Hadoop) > Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952702#comment-16952702 ] Mate Szalay-Beko commented on HADOOP-16579: --- I fixed all the failing unit tests (the ways of starting a test ZooKeeperServer changed a bit, so I had to update these parts). Let's see what Yetus thinks :) > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951708#comment-16951708 ] Mate Szalay-Beko commented on HADOOP-16579: --- I fixed some checkstyle errors. We also run into some deprecated function calls in the curator API, I hope I was able to find / fix all of these as well. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951029#comment-16951029
]
Mate Szalay-Beko edited comment on HADOOP-16579 at 10/14/19 2:22 PM:
-
I was able to find the root cause of the {{TestZKFailoverController}} failures,
I created a new PR ([PR-1656|https://github.com/apache/hadoop/pull/1656]).
Let's see if any other tests will fail.
was (Author: symat):
I was able to find the root cause of the `TestZKFailoverController` failures, I
created a new PR. Let's see if any other tests will fail.
> Upgrade to Apache Curator 4.2.0 in Hadoop
> -
>
> Key: HADOOP-16579
> URL: https://issues.apache.org/jira/browse/HADOOP-16579
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Mate Szalay-Beko
>Assignee: Norbert Kalmár
>Priority: Major
>
> Currently in Hadoop we are using [ZooKeeper version
> 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90].
> ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains
> many new features (including SSL related improvements which can be very
> important for production use; see [the release
> notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]).
> Apache Curator is a high level ZooKeeper client library, that makes it easier
> to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator
> 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91]
> and [in Ozone we use Curator
> 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146].
> Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator
> 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately,
> the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and
> 3.5.x. (see [the relevant Curator
> page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects
> have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other
> components are doing it right now (e.g. Hive).
> *The aims of this task are* to:
> - change Curator version in Hadoop to the latest stable 4.x version
> (currently 4.2.0)
> - also make sure we don't have multiple ZooKeeper versions in the classpath
> to avoid runtime problems (it is
> [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the
> ZooKeeper which come with Curator, so that there will be only a single
> ZooKeeper version used runtime in Hadoop)
> In this ticket we still don't want to change the default ZooKeeper version in
> Hadoop, we only want to make it possible for the community to be able to
> build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the
> ZooKeeper communication with SSL, what is only supported in the new ZooKeeper
> version). Upgrading to Curator 4.x should keep Hadoop to be compatible with
> both ZooKeeper 3.4 and 3.5.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951029#comment-16951029 ] Mate Szalay-Beko commented on HADOOP-16579: --- I was able to find the root cause of the `TestZKFailoverController` failures, I created a new PR. Let's see if any other tests will fail. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934207#comment-16934207 ] Mate Szalay-Beko commented on HADOOP-16579: --- There are some... I compared the dependencies which changed between 4.2.0 ([see here|https://mvnrepository.com/artifact/org.apache.curator/apache-curator/4.2.0]) and 2.12.0 ([see here|https://mvnrepository.com/artifact/org.apache.curator/apache-curator/2.12.0]): ||Lib||Curator 2.12.0||Curator 4.2.0|| |com.facebook.swift|0.12.0|0.23.1| |com.fasterxml.jackson|n/a|2.9.8| |com.google.guava|16.0.1|27.0.1-jre| |com.sun.jersey|1.18.1|1.19.4| |io.dropwizard (configuration and logging)|0.7.0|1.3.7| |org.apache.commons|2.2|2.2| |org.jboss.resteasy|2.3.0.GA|2.3.5.Final| |org.slf4j|1.7.6|1.7.25| We actually built Hadoop already with the new Curator and new ZooKeeper (I think we even deployed and executed some smoke tests) without touching the rest of the dependencies. But I think it would be important to review these changes above in the table and decide for each Curator dependency if we want to include it as it is or if we want to exclude and use Hadoop's version. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. > Upgrade to Apache Curator 4.2.0 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version >
[jira] [Created] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 in Hadoop
Mate Szalay-Beko created HADOOP-16579: - Summary: Upgrade to Apache Curator 4.2.0 in Hadoop Key: HADOOP-16579 URL: https://issues.apache.org/jira/browse/HADOOP-16579 Project: Hadoop Common Issue Type: Improvement Reporter: Mate Szalay-Beko Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
