JackieTien97 commented on code in PR #18921:
URL: https://github.com/apache/iotdb/pull/18921#discussion_r4321474810
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/NodesSupplier.java:
##########
@@ -212,45 +357,77 @@ private boolean createConnection(TEndPoint endPoint) {
enableRPCCompression,
zoneId,
version);
- return true;
+ if (closed) {
+ destroyClient(current);
+ return null;
+ }
+ return current;
} catch (Exception e) {
- LOGGER.warn(SessionMessages.FAILED_TO_CREATE_CONNECTION, endPoint);
- destroyCurrentClient();
- return false;
+ if (!closed) {
+ LOGGER.warn(SessionMessages.FAILED_TO_CREATE_CONNECTION, endPoint);
+ }
+ destroyClient(current);
+ return null;
}
}
- private synchronized void destroyCurrentClient() {
- if (client != null) {
- client.close();
- client = null;
+ private void destroyClient(ThriftConnection current) {
+ synchronized (clientLock) {
+ if (client == current) {
+ client = null;
+ }
}
+ current.close();
}
@Override
public void close() {
- closed = true;
- destroyCurrentClient();
+ ThriftConnection current;
+ boolean inUse;
+ synchronized (clientLock) {
+ closed = true;
+ current = client != null ? client : activeClient;
+ inUse = current != null && current == activeClient;
+ client = null;
+ }
+ if (current != null) {
+ if (inUse) {
+ current.abort();
+ } else {
+ current.close();
+ }
+ }
}
@Override
public Optional<TEndPoint> getQueryEndPoint() {
- if (availableNodes == null || availableNodes.isEmpty()) {
+ List<TEndPoint> snapshot = availableNodes;
+ if (snapshot.isEmpty()) {
return Optional.empty();
} else {
- return Optional.of(policy.chooseOne(get()));
+ return Optional.of(policy.chooseOne(snapshot));
}
}
- private boolean updateDataNodeList() {
+ private void updateDataNodeList(ThriftConnection current) throws
IoTDBConnectionException {
try (SessionDataSet sessionDataSet =
- client.executeQueryStatement(SHOW_AVAILABLE_URLS_COMMAND,
TIMEOUT_IN_MS, FETCH_SIZE)) {
+ current.executeQueryStatement(SHOW_AVAILABLE_URLS_COMMAND,
TIMEOUT_IN_MS, FETCH_SIZE)) {
updateAvailableNodes(sessionDataSet);
- } catch (Exception e1) {
- LOGGER.warn(SessionMessages.FAILED_TO_FETCH_DATA_NODE_LIST,
client.endPoint);
- return false;
+ } catch (IoTDBConnectionException e) {
+ if (!closed) {
+ LOGGER.warn(SessionMessages.FAILED_TO_FETCH_DATA_NODE_LIST,
current.endPoint);
+ }
+ if (!(e.getCause() instanceof TApplicationException)) {
+ throw e;
+ }
+ // A Thrift application error also means the peer responded; it is not a
transport failure.
+ } catch (Exception e) {
+ // A query error (for example no ConfigNode leader) is not evidence of a
dead endpoint.
+ // Keep this connection and snapshot, and let the next scheduled refresh
retry the query.
Review Comment:
**A node that keeps failing the discovery query now pins discovery.** Both
branches here keep the connection and end the round. The next scheduled refresh
starts from the same `client`, so discovery never moves away from this node
while it stays reachable.
A typical case is a DataNode that is cut off from the ConfigNode while
clients can still reach it. The ConfigNode marks it `Unknown`, which is exactly
the node the new server-side filter is meant to drop.
A/B with the same harness:
- Discovery first connects to A while A is healthy; snapshot `[B, C, A]`.
- Then only A's query keeps failing, while B and C answer `[B, C]`.
- Three refreshes follow.
| | snapshot after 3 refreshes | queries A / B / C |
|---|---|---|
| base `bacbd5a6dc` | `[B, C]` | 1 / 2 / 0 |
| `c600593` | `[B, C]` | 1 / 3 / 0 |
| `c240197` | `[B, C, A]` (frozen) | 3 / 0 / 0 |
While discovery is pinned, the snapshot is frozen: A stays in the query
rotation, and restarts of other nodes are not picked up.
`testQueryStatusFailureDoesNotRotateConnections` and
`testQueryApplicationFailureDoesNotRotateConnections` assert this behaviour.
Ending the round on a query error is right; it removed the sweep. The
problem is keeping the connection for the next round. Suggestion: after a query
error, close the connection and start the next round from a different
candidate, for example by putting the failed endpoint last. A cluster-wide
failure then costs at most one new connection and one query per refresh, which
is still bounded.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/NodesSupplier.java:
##########
@@ -164,43 +230,122 @@ private NodesSupplier(
this.version = version;
}
- // the method will only be called while reconnecting, so it's ok to copy set
each time
- // and the List needn't be thread-safe, because it will only be used in one
thread.
+ // Readers retain an immutable snapshot while discovery publishes
replacements.
@Override
public List<TEndPoint> get() {
return availableNodes;
}
@Override
- public void run() {
- if (closed) {
- if (client != null) {
- destroyCurrentClient();
+ public List<TEndPoint> getSeedNodes() {
+ return seedNodes;
+ }
+
+ @Override
+ public void requestRefresh() {
+ if (!closed && refreshRequested.compareAndSet(false, true)) {
+ try {
+ executorService.execute(
+ () -> {
+ try {
+ run();
+ } finally {
+ refreshRequested.set(false);
+ }
+ });
+ } catch (RejectedExecutionException e) {
+ // The session or pool may have shut down its executor concurrently.
+ refreshRequested.set(false);
}
+ }
+ }
+
+ @Override
+ public void run() {
+ if (closed || !refreshLock.tryLock()) {
Review Comment:
**`run()` is now best-effort.** With `tryLock()`, a direct call returns
immediately, without refreshing, while another refresh holds the lock.
Scheduled and requested refreshes share the session/pool executor, so they are
already serialized and unaffected. Callers outside that executor are affected.
`SessionSeedNodeFallbackIT` calls `sessionSupplier.run()` /
`poolSupplier.run()` directly and asserts right afterwards (L100-101 and
L108-109). With `synchronized` in `c600593`, such a call waited for the
in-flight refresh and then refreshed again. Now, if a scheduled refresh happens
to be running, it can return before the snapshot changes. The unit tests
already poll (`awaitSnapshot`); polling in the IT as well would keep it
deterministic.
##########
RELEASE_NOTES.md:
##########
@@ -19,6 +19,32 @@
-->
+# Unreleased
Review Comment:
nit: on master, `RELEASE_NOTES.md` only changes in the release-info PRs
(#18164, #17468, #17275, ...), so an `# Unreleased` section added by a feature
PR may be moved or rewritten by the release manager. The upgrade-order note
(clients before servers) might fit better in the PR description or the user
docs, where it gets picked up at release time.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/Session.java:
##########
@@ -599,11 +606,9 @@ private static List<String> shuffleNodeUrls(List<String>
endPoints) {
}
private List<TEndPoint> getNodeUrls() {
- if (defaultEndPoint != null) {
- return Collections.singletonList(defaultEndPoint);
- } else {
- return SessionUtils.parseSeedNodeUrls(nodeUrls);
- }
+ return defaultEndPoint == null
+ ? SessionUtils.parseSeedNodeUrls(initialNodeUrls)
+ : Collections.singletonList(defaultEndPoint.deepCopy());
Review Comment:
nit, pre-existing: with `enableAutoFetch(false)`, `open()` builds `new
DummyNodesSupplier(getNodeUrls())`, and `DummyNodesSupplier` has no seeds.
After a failover and `close()`/`open()`, this method returns only
`[defaultEndPoint]`, so the static list shrinks to a single endpoint.
Checked with `nodeUrls = [A, B]`: after a failover to B and a reopen, the
static list is `[B]`. When B then stops, `reconnect()` fails even though A is
back. Base behaves the same way. `c600593` happened to keep the full list,
because `getNodeUrls()` always returned `initialNodeUrls` there.
Passing `[defaultEndPoint] + initialNodeUrls` (deduplicated) to
`DummyNodesSupplier` would keep failover working after a reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]