This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/helix.git
The following commit(s) were added to refs/heads/master by this push:
new 695b1fd3b Upgrade ZooKeeper from 3.6.3 to 3.9.3 (#3093)
695b1fd3b is described below
commit 695b1fd3bfc8bb1b91e37de2827738dcd56260b9
Author: Junkai Xue <[email protected]>
AuthorDate: Fri Nov 6 01:25:27 2026 -0800
Upgrade ZooKeeper from 3.6.3 to 3.9.3 (#3093)
ZOOKEEPER-4712 (Follower/Observer shutdown does not stop the sync processor,
which can leave the data tree inconsistent) affects every release Helix
could
pick from the 3.6, 3.7 and 3.8 lines; the fix only shipped in 3.9.3 and
3.10.0. 3.9.3 is therefore the smallest bump that contains it. The 3.6 line
has also been end of life since December 2022.
No Helix API changes. Every ZooKeeper type that Helix exposes (Stat, Op,
OpResult, ACL, CreateMode, Watcher, KeeperException, ZooKeeper) is unchanged
between the two versions, and all modules compile without source changes,
including the embedded ZkServer and the tools that read transaction logs and
snapshots. Paginated getChildren is still not part of the ZooKeeper client,
so
ZkClient keeps using the regular API. The OSGi Import-Package range for
org.apache.zookeeper in zookeeper-api and helix-rest moves to [3.9,3.10).
Dependency changes: 3.9.3 drops the log4j 1 binding (logback is an optional
dependency and is not pulled in), brings netty 4.1.113 and commons-io 2.17
(Helix already manages both), and newly adds
netty-tcnative-boringssl-static,
a multi-platform native OpenSSL binding only used for TLS between client and
server. Helix does not configure ZooKeeper TLS, so that artifact is excluded
to keep the dependency footprint the same as before. The checked-in ivy
files
and the meta-client KeeperException code mapping are updated for 3.9.3; the
two codes added since 3.6 (QUOTAEXCEEDED, THROTTLEDOP) are API errors and
map to DB_USER_ERROR, which is what the default branch already produced.
Behavior change verified by tests: ZooKeeper 3.9.0 fixed the server so that
a
one time watch on a path no longer replaces a persistent watch on the same
path (ZOOKEEPER-4466) and persistent recursive watches no longer receive
NodeChildrenChanged (ZOOKEEPER-4475). TestZooKeeperConnection asserted the
old behavior (persistent watchers silently missing events after a one time
watch was added) and now asserts that no event is missed. ZkClient itself is
unaffected: in persist mode it registers child subscriptions with
AddWatchMode.PERSISTENT, refuses one time watches, and never relied on
NodeChildrenChanged for recursive persistent listeners.
ZkTestHelper.getZkWatch (helix-core and zookeeper-api test helpers) read the
client's watch registry through the ZooKeeper#watchManager field, which 3.7
moved into ClientCnxn behind the package-private getWatchManager(); the
helper now calls that accessor.
Tests run locally against 3.9.3: the whole zookeeper-api module, and in
helix-core the ZK client, session, watcher, reconnect, flapping, connection
loss, data accessor, cache, bucket accessor, size limit, admin and ZkCopy
test classes.
---
helix-core/helix-core-2.0.2-SNAPSHOT.ivy | 2 +-
.../test/java/org/apache/helix/ZkTestHelper.java | 17 ++++++------
helix-rest/pom.xml | 2 +-
.../metaclient/impl/zk/util/ZkMetaClientUtil.java | 6 ++++-
zookeeper-api/pom.xml | 10 ++++++--
.../zookeeper/impl/TestZooKeeperConnection.java | 30 ++++++++++++----------
.../apache/helix/zookeeper/impl/ZkTestHelper.java | 17 ++++++------
zookeeper-api/zookeeper-api-2.0.2-SNAPSHOT.ivy | 2 +-
8 files changed, 49 insertions(+), 37 deletions(-)
diff --git a/helix-core/helix-core-2.0.2-SNAPSHOT.ivy
b/helix-core/helix-core-2.0.2-SNAPSHOT.ivy
index fff0a3652..0d1e248a3 100755
--- a/helix-core/helix-core-2.0.2-SNAPSHOT.ivy
+++ b/helix-core/helix-core-2.0.2-SNAPSHOT.ivy
@@ -52,7 +52,7 @@ under the License.
<dependency org="org.apache.logging.log4j" name="log4j-slf4j-impl"
rev="2.17.1" force="true"
conf="compile->compile(*),master(*);runtime->runtime(*)">
<artifact name="log4j-slf4j-impl" ext="jar"/>
</dependency>
- <dependency org="org.apache.zookeeper" name="zookeeper" rev="3.6.3"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
+ <dependency org="org.apache.zookeeper" name="zookeeper" rev="3.9.3"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
<dependency org="com.fasterxml.jackson.core"
name="jackson-databind" rev="2.12.6.1"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
<dependency org="com.fasterxml.jackson.core" name="jackson-core"
rev="2.12.6"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
<dependency org="commons-io" name="commons-io" rev="2.11.0"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
diff --git a/helix-core/src/test/java/org/apache/helix/ZkTestHelper.java
b/helix-core/src/test/java/org/apache/helix/ZkTestHelper.java
index 74158bc45..3632ebfa1 100644
--- a/helix-core/src/test/java/org/apache/helix/ZkTestHelper.java
+++ b/helix-core/src/test/java/org/apache/helix/ZkTestHelper.java
@@ -399,24 +399,23 @@ public class ZkTestHelper {
ZkConnection connection = ((ZkConnection) zkClient.getConnection());
ZooKeeper zk = connection.getZookeeper();
- java.lang.reflect.Field field = getField(zk.getClass(), "watchManager");
- field.setAccessible(true);
- Object watchManager = field.get(zk);
+ // Since ZooKeeper 3.7 the client watch manager lives in ClientCnxn and is
reached through the
+ // package-private ZooKeeper#getWatchManager() instead of a field.
+ java.lang.reflect.Method getWatchManager =
ZooKeeper.class.getDeclaredMethod("getWatchManager");
+ getWatchManager.setAccessible(true);
+ Object watchManager = getWatchManager.invoke(zk);
java.lang.reflect.Field field2 = getField(watchManager.getClass(),
"dataWatches");
field2.setAccessible(true);
- HashMap<String, Set<Watcher>> dataWatches =
- (HashMap<String, Set<Watcher>>) field2.get(watchManager);
+ Map<String, Set<Watcher>> dataWatches = (Map<String, Set<Watcher>>)
field2.get(watchManager);
field2 = getField(watchManager.getClass(), "existWatches");
field2.setAccessible(true);
- HashMap<String, Set<Watcher>> existWatches =
- (HashMap<String, Set<Watcher>>) field2.get(watchManager);
+ Map<String, Set<Watcher>> existWatches = (Map<String, Set<Watcher>>)
field2.get(watchManager);
field2 = getField(watchManager.getClass(), "childWatches");
field2.setAccessible(true);
- HashMap<String, Set<Watcher>> childWatches =
- (HashMap<String, Set<Watcher>>) field2.get(watchManager);
+ Map<String, Set<Watcher>> childWatches = (Map<String, Set<Watcher>>)
field2.get(watchManager);
lists.put("dataWatches", new ArrayList<>(dataWatches.keySet()));
lists.put("existWatches", new ArrayList<>(existWatches.keySet()));
diff --git a/helix-rest/pom.xml b/helix-rest/pom.xml
index ab50e4524..c10724aef 100644
--- a/helix-rest/pom.xml
+++ b/helix-rest/pom.xml
@@ -51,7 +51,7 @@
org.slf4j*;version="[1.7,2)",
org.apache.logging.log4j*;version="[2.17,3)",
org.apache.logging.slf4j*;version="[2.17,3)",
- org.apache.zookeeper*;version="[3.6,3.7)",
+ org.apache.zookeeper*;version="[3.9,3.10)",
org.apache.commons.io*;version="[1.4,2)",
*
</osgi.import>
diff --git
a/meta-client/src/main/java/org/apache/helix/metaclient/impl/zk/util/ZkMetaClientUtil.java
b/meta-client/src/main/java/org/apache/helix/metaclient/impl/zk/util/ZkMetaClientUtil.java
index aee0c698a..f1693dbba 100644
---
a/meta-client/src/main/java/org/apache/helix/metaclient/impl/zk/util/ZkMetaClientUtil.java
+++
b/meta-client/src/main/java/org/apache/helix/metaclient/impl/zk/util/ZkMetaClientUtil.java
@@ -277,7 +277,7 @@ public class ZkMetaClientUtil {
/**
* This function translate and group Zk exception code to metaclient code.
- * It currently includes all ZK code on 3.6.3.
+ * It currently includes all ZK code on 3.9.3.
*/
public static MetaClientException.ReturnCode
translateZooKeeperCodeToMetaClientCode(
KeeperException.Code zkCode) {
@@ -341,6 +341,10 @@ public class ZkMetaClientUtil {
case NOTREADONLY:
case EPHEMERALONLOCALSESSION:
case RECONFIGDISABLED:
+ // QUOTAEXCEEDED (the znode quota is exhausted) and THROTTLEDOP (the
server is overloaded and did
+ // not execute the operation at all) were added after 3.6; both are API
errors in ZK.
+ case QUOTAEXCEEDED:
+ case THROTTLEDOP:
return MetaClientException.ReturnCode.DB_USER_ERROR;
/*
diff --git a/zookeeper-api/pom.xml b/zookeeper-api/pom.xml
index c5e24bb39..b58f48ae4 100644
--- a/zookeeper-api/pom.xml
+++ b/zookeeper-api/pom.xml
@@ -45,7 +45,7 @@
<properties>
<osgi.import>
org.slf4j*;version="[1.7,2)",
- org.apache.zookeeper*;version="[3.6,3.7)",
+ org.apache.zookeeper*;version="[3.9,3.10)",
*
</osgi.import>
<osgi.export>org.apache.helix.zookeeper*;version="${project.version};-noimport:=true</osgi.export>
@@ -64,7 +64,7 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
- <version>3.6.3</version>
+ <version>3.9.3</version>
<exclusions>
<exclusion>
<groupId>junit</groupId>
@@ -78,6 +78,12 @@
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
+ <!-- Native OpenSSL binding for all platforms, only used for
client/server TLS which Helix
+ does not configure. -->
+ <exclusion>
+ <groupId>io.netty</groupId>
+ <artifactId>netty-tcnative-boringssl-static</artifactId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
diff --git
a/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/TestZooKeeperConnection.java
b/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/TestZooKeeperConnection.java
index 0ea8524db..0454167e5 100644
---
a/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/TestZooKeeperConnection.java
+++
b/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/TestZooKeeperConnection.java
@@ -27,6 +27,9 @@ public class TestZooKeeperConnection extends ZkTestBase {
1. Register a persist watcher on a path and create 100 children Znode, edit
the ZNode for 100 times.
Expecting 200 events.
2. register a one time listener on the path. Make the same change and count
the total number of event.
+ Since ZooKeeper 3.9.0 (ZOOKEEPER-4466) a one time watch on the same path no
longer replaces the
+ persistent watch on the server, so the persistent watcher keeps receiving
every event. Before that
+ fix it silently missed events, which is why ZkClient refuses one time
watches in persist mode.
*/
@Test
@@ -46,18 +49,18 @@ public class TestZooKeeperConnection extends ZkTestBase {
Assert.assertTrue(countDownLatch.await(50000, TimeUnit.MILLISECONDS));
// register a one time listener on the path. Count the total number of
event.
- // ZK will over write the persist watcher and only trigger event once for
child and data change.
+ // The one time watch coexists with the persist watcher (ZOOKEEPER-4466),
so the persist
+ // watcher still gets all 200 data change and 200 child change events.
_zk.readData(path, null, true);
_zk.getChildren(path, true);
for (int i=0; i<200; ++i) {
_zk.writeData(path, ("datat"+i).getBytes(), -1);
_zk.create(path+"/c2_" +i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
}
- System.out.println("testPersistWatcher: rafter register one time listener,
original listener received event count: " + get_count[0]);
- // total number of event is 400. We will miss event now
- Assert.assertTrue(TestHelper.verify(() -> {
- return (get_count[0].get() >= 202 & get_count[0].get() < 400);
- }, TestHelper.WAIT_DURATION));
+ System.out.println("testPersistWatcher: after register one time listener,
original listener received event count: " + get_count[0]);
+ // total number of events is 200 + 400 = 600 and none of them is missed
+ Assert.assertTrue(TestHelper.verify(() -> get_count[0].get() == 600,
TestHelper.WAIT_DURATION),
+ "Persist watcher received " + get_count[0].get() + " events, expected
600");
zkClient.close();
}
@@ -70,8 +73,9 @@ public class TestZooKeeperConnection extends ZkTestBase {
IZkConnection _zk = zkClient.getConnection();
String path="/testRecursivePersistWatcher";
_zk.create(path, null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
- // register a persist listener on a path, change the ZNode 100 times,
create 100 child ZNode,
- // and expecting 200 events
+ // register a recursive persist listener on a path, change the ZNode 100
times, create 100 child
+ // ZNode and 100 grandchild ZNode, and expecting 300 events. A recursive
persist watcher only gets
+ // NodeDataChanged/NodeCreated/NodeDeleted, no NodeChildrenChanged
(ZOOKEEPER-4475).
_zk.addWatch(path, watcher1, AddWatchMode.PERSISTENT_RECURSIVE);
for (int i=0; i<count; ++i) {
_zk.writeData(path, "datat".getBytes(), -1);
@@ -81,7 +85,8 @@ public class TestZooKeeperConnection extends ZkTestBase {
Assert.assertTrue(countDownLatch2.await(50000, TimeUnit.MILLISECONDS));
// register a one time listener on the path. Count the total number of
event.
- // ZK will over write the persist watcher and only trigger event once for
child and data change.
+ // The one time watch coexists with the persist watcher (ZOOKEEPER-4466),
so the persist
+ // watcher still gets all 200 data change and 200 child creation events.
_zk.readData(path, null, true);
_zk.getChildren(path, true);
for (int i=0; i<200; ++i) {
@@ -89,10 +94,9 @@ public class TestZooKeeperConnection extends ZkTestBase {
_zk.create(path+"/c2_" +i, null, ZooDefs.Ids.OPEN_ACL_UNSAFE,
CreateMode.PERSISTENT);
}
System.out.println("testRecursivePersistWatcherWithOneTimeWatcher: after
register one time listener, original listener received event count: " +
get_count[0]);
- // total number of event is 500. We will miss event now
- Assert.assertTrue(TestHelper.verify(() -> {
- return (get_count[0].get() >= 302 && get_count[0].get() < 500);
- }, TestHelper.WAIT_DURATION));
+ // total number of events is 300 + 400 = 700 and none of them is missed
+ Assert.assertTrue(TestHelper.verify(() -> get_count[0].get() == 700,
TestHelper.WAIT_DURATION),
+ "Recursive persist watcher received " + get_count[0].get() + " events,
expected 700");
zkClient.close();
}
diff --git
a/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/ZkTestHelper.java
b/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/ZkTestHelper.java
index 32e2b8acc..5624c9eca 100644
---
a/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/ZkTestHelper.java
+++
b/zookeeper-api/src/test/java/org/apache/helix/zookeeper/impl/ZkTestHelper.java
@@ -360,24 +360,23 @@ public class ZkTestHelper {
ZkConnection connection = ((ZkConnection) zkClient.getConnection());
ZooKeeper zk = connection.getZookeeper();
- java.lang.reflect.Field field = getField(zk.getClass(), "watchManager");
- field.setAccessible(true);
- Object watchManager = field.get(zk);
+ // Since ZooKeeper 3.7 the client watch manager lives in ClientCnxn and is
reached through the
+ // package-private ZooKeeper#getWatchManager() instead of a field.
+ java.lang.reflect.Method getWatchManager =
ZooKeeper.class.getDeclaredMethod("getWatchManager");
+ getWatchManager.setAccessible(true);
+ Object watchManager = getWatchManager.invoke(zk);
java.lang.reflect.Field field2 = getField(watchManager.getClass(),
"dataWatches");
field2.setAccessible(true);
- HashMap<String, Set<Watcher>> dataWatches =
- (HashMap<String, Set<Watcher>>) field2.get(watchManager);
+ Map<String, Set<Watcher>> dataWatches = (Map<String, Set<Watcher>>)
field2.get(watchManager);
field2 = getField(watchManager.getClass(), "existWatches");
field2.setAccessible(true);
- HashMap<String, Set<Watcher>> existWatches =
- (HashMap<String, Set<Watcher>>) field2.get(watchManager);
+ Map<String, Set<Watcher>> existWatches = (Map<String, Set<Watcher>>)
field2.get(watchManager);
field2 = getField(watchManager.getClass(), "childWatches");
field2.setAccessible(true);
- HashMap<String, Set<Watcher>> childWatches =
- (HashMap<String, Set<Watcher>>) field2.get(watchManager);
+ Map<String, Set<Watcher>> childWatches = (Map<String, Set<Watcher>>)
field2.get(watchManager);
lists.put("dataWatches", new ArrayList<>(dataWatches.keySet()));
lists.put("existWatches", new ArrayList<>(existWatches.keySet()));
diff --git a/zookeeper-api/zookeeper-api-2.0.2-SNAPSHOT.ivy
b/zookeeper-api/zookeeper-api-2.0.2-SNAPSHOT.ivy
index 1b581384f..cecff16bc 100644
--- a/zookeeper-api/zookeeper-api-2.0.2-SNAPSHOT.ivy
+++ b/zookeeper-api/zookeeper-api-2.0.2-SNAPSHOT.ivy
@@ -43,7 +43,7 @@ under the License.
<dependency org="org.apache.logging.log4j" name="log4j-slf4j-impl"
rev="2.17.1" force="true"
conf="compile->compile(*),master(*);runtime->runtime(*)">
<artifact name="log4j-slf4j-impl" ext="jar"/>
</dependency>
- <dependency org="org.apache.zookeeper" name="zookeeper" rev="3.6.3"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
+ <dependency org="org.apache.zookeeper" name="zookeeper" rev="3.9.3"
conf="compile->compile(default);runtime->runtime(default);default->default"/>
<dependency org="com.fasterxml.jackson.core"
name="jackson-databind" rev="2.12.6.1" force="true"
conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="com.fasterxml.jackson.core"
name="jackson-core" rev="2.12.6" force="true"
conf="compile->compile(*),master(*);runtime->runtime(*)"/>
<dependency org="commons-cli" name="commons-cli" rev="1.2"
force="true" conf="compile->compile(*),master(*);runtime->runtime(*)"/>
