[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HADOOP-16579: - Fix Version/s: (was: 3.3.6) > Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > Fix For: 3.3.0 > > > *Update:* the original idea was to only update Curator but keep the old > ZooKeeper version in Hadoop. However, we encountered some run-time > backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper > 3.5.5. We haven't really investigated deeply these issues, but upgraded to > ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the > unit tests (and also had to change some deprecated Curator API calls), but > [the latest PR|https://github.com/apache/hadoop/pull/1656] seems to be stable. > ZooKeeper 3.5.6 just got released during our work. (I think the official > announcement will get out maybe tomorrow, but it is already available in > maven central or on the [Apache ZooKeeper ftp > site|https://www-eu.apache.org/dist/zookeeper/]). It is considered to be a > stable version, contains some minor fixes and improvements, plus some CVE > fixes. See the [release > notes|https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md]. > > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated deeply these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but [the latest PR|https://github.com/apache/hadoop/pull/1656] seems to be stable. ZooKeeper 3.5.6 just got released during our work. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the [Apache ZooKeeper ftp site|https://www-eu.apache.org/dist/zookeeper/]). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the [release notes|https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md]. Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released during our work. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the [apache zookeeper ftp site|[https://www-eu.apache.org/dist/zookeeper/]]). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release notes: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released during our work. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the [apache zookeeper ftp site|[https://www-eu.apache.org/dist/zookeeper/]]). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release notes: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the apache zookeeper ftp site). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release note: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Description: *Update:* the original idea was to only update Curator but keep the old ZooKeeper version in Hadoop. However, we encountered some run-time backward-incompatibility during unit tests with Curator 4.2.0 and ZooKeeper 3.5.5. We haven't really investigated these issues, but upgraded to ZooKeeper 3.5.5 (and later to 3.5.6). We had to do some minor fixes in the unit tests (and also had to change some deprecated Curator API calls), but the latest PR seems to be stable. ZooKeeper 3.5.6 just got released. (I think the official announcement will get out maybe tomorrow, but it is already available in maven central or on the apache zookeeper ftp site). It is considered to be a stable version, contains some minor fixes and improvements, plus some CVE fixes. See the release note: [https://github.com/apache/zookeeper/blob/branch-3.5.6/zookeeper-docs/src/main/resources/markdown/releasenotes.md] Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single ZooKeeper version used runtime in Hadoop) In this ticket we still don't want to change the default ZooKeeper version in Hadoop, we only want to make it possible for the community to be able to build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the ZooKeeper communication with SSL, what is only supported in the new ZooKeeper version). Upgrading to Curator 4.x should keep Hadoop to be compatible with both ZooKeeper 3.4 and 3.5. was: Currently in Hadoop we are using [ZooKeeper version 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains many new features (including SSL related improvements which can be very important for production use; see [the release notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). Apache Curator is a high level ZooKeeper client library, that makes it easier to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] and [in Ozone we use Curator 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and 3.5.x. (see [the relevant Curator page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other components are doing it right now (e.g. Hive). *The aims of this task are* to: - change Curator version in Hadoop to the latest stable 4.x version (currently 4.2.0) - also make sure we don't have multiple ZooKeeper versions in the classpath to avoid runtime problems (it is [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the ZooKeeper which come with Curator, so that there will be only a single
[jira] [Updated] (HADOOP-16579) Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-16579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mate Szalay-Beko updated HADOOP-16579: -- Summary: Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop (was: Upgrade to Apache Curator 4.2.0 in Hadoop) > Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.6 in Hadoop > - > > Key: HADOOP-16579 > URL: https://issues.apache.org/jira/browse/HADOOP-16579 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mate Szalay-Beko >Assignee: Norbert Kalmár >Priority: Major > > Currently in Hadoop we are using [ZooKeeper version > 3.4.13|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L90]. > ZooKeeper 3.5.5 is the latest stable Apache ZooKeeper release. It contains > many new features (including SSL related improvements which can be very > important for production use; see [the release > notes|https://zookeeper.apache.org/doc/r3.5.5/releasenotes.html]). > Apache Curator is a high level ZooKeeper client library, that makes it easier > to use the low level ZooKeeper API. Currently [in Hadoop we are using Curator > 2.13.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/hadoop-project/pom.xml#L91] > and [in Ozone we use Curator > 2.12.0|https://github.com/apache/hadoop/blob/7f9073132dcc9db157a6792635d2ed099f2ef0d2/pom.ozone.xml#L146]. > Curator 2.x is supporting only the ZooKeeper 3.4.x releases, while Curator > 3.x is compatible only with the new ZooKeeper 3.5.x releases. Fortunately, > the latest Curator 4.x versions are compatible with both ZooKeeper 3.4.x and > 3.5.x. (see [the relevant Curator > page|https://curator.apache.org/zk-compatibility.html]). Many Apache projects > have already migrated to Curator 4 (like HBase, Phoenix, Druid, etc.), other > components are doing it right now (e.g. Hive). > *The aims of this task are* to: > - change Curator version in Hadoop to the latest stable 4.x version > (currently 4.2.0) > - also make sure we don't have multiple ZooKeeper versions in the classpath > to avoid runtime problems (it is > [recommended|https://curator.apache.org/zk-compatibility.html] to exclude the > ZooKeeper which come with Curator, so that there will be only a single > ZooKeeper version used runtime in Hadoop) > In this ticket we still don't want to change the default ZooKeeper version in > Hadoop, we only want to make it possible for the community to be able to > build / use Hadoop with the new ZooKeeper (e.g. if they need to secure the > ZooKeeper communication with SSL, what is only supported in the new ZooKeeper > version). Upgrading to Curator 4.x should keep Hadoop to be compatible with > both ZooKeeper 3.4 and 3.5. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
