JackieTien97 commented on code in PR #18921:
URL: https://github.com/apache/iotdb/pull/18921#discussion_r4321154646
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/NodesSupplier.java:
##########
@@ -266,9 +311,7 @@ private void updateAvailableNodes(SessionDataSet
sessionDataSet) throws Exceptio
}
}
}
- // replace the older ones.
- if (!res.isEmpty()) {
- availableNodes = res;
- }
+ // An empty successful result is authoritative too; seeds remain available
for recovery.
+ availableNodes = Collections.unmodifiableList(res);
Review Comment:
**An empty result used to mean "no information"; now it means "no nodes".**
In practice an empty filtered result appears while the responding node itself
is alive:
- With `dn_rpc_address=0.0.0.0`, the wildcard filter above drops every row.
The official `docker/src/main/Dockerfile-1c1d` sets exactly this.
- Right after a ConfigNode leader change, node statistics default to
`Unknown` (`NodeStatistics.generateDefaultNodeStatistics()`). Until the first
heartbeats arrive, the server filters out every node.
Measured with all rows `0.0.0.0`: base keeps the seeds, so
`getQueryEndPoint()` keeps round-robining. Head publishes `[]`, and
`getQueryEndPoint()` returns empty.
Consequences:
- Queries are no longer spread across nodes.
- Some RPCs go through `callWithRetryAndReconnect`: createTimeseries,
templates, `setTimeZone`, raw/last/aggregation queries and others. For these,
the `!availableNodes.get().contains(this.endPoint)` check
(`SessionConnection.java` L1034) is always true, so every retryable status
forces a reconnect plus a refresh request.
- The empty snapshot stays until the next periodic refresh, up to 60 s.
Suggestion: if every row was a wildcard, treat the result as no information
and keep the previous snapshot. If the server genuinely returned no rows, keep
the previous snapshot, or at least publish the responder (`client.endPoint`).
`testEmptyDiscoveryFallsBackWithoutLosingSeeds` and
`testQuerySelectionDuringEmptyDiscoveryRefresh` would need to follow.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/NodesSupplier.java:
##########
@@ -172,24 +186,54 @@ public List<TEndPoint> get() {
}
@Override
- public void run() {
+ public List<TEndPoint> getSeedNodes() {
+ return seedNodes;
+ }
+
+ @Override
+ public void requestRefresh() {
+ if (!closed && refreshRequested.compareAndSet(false, true)) {
+ try {
+ executorService.execute(
+ () -> {
+ try {
+ run();
+ } finally {
+ refreshRequested.set(false);
+ }
+ });
+ } catch (RejectedExecutionException e) {
+ // The session or pool may have shut down its executor concurrently.
+ refreshRequested.set(false);
+ }
+ }
+ }
+
+ @Override
+ public synchronized void run() {
if (closed) {
if (client != null) {
destroyCurrentClient();
}
return;
}
- if (client == null) {
- for (TEndPoint endPoint : availableNodes) {
- if (createConnection(endPoint)) {
- break;
- }
+ Set<TEndPoint> attempted = new HashSet<>();
+ if (client != null) {
+ attempted.add(client.endPoint);
+ if (updateDataNodeList()) {
+ return;
}
+ destroyCurrentClient();
}
- // make sure the following code block can run thread-safely with close()
method.
- synchronized (this) {
- if (client != null && !updateDataNodeList()) {
+ // A stale discovery snapshot can omit the only node that has finished a
rolling restart.
+ Set<TEndPoint> candidates = new LinkedHashSet<>(availableNodes);
+ candidates.addAll(seedNodes);
+ for (TEndPoint endPoint : candidates) {
+ if (attempted.add(endPoint)) {
+ if (createConnection(endPoint) && updateDataNodeList()) {
Review Comment:
**A query failure is treated like an unreachable node.**
`createConnection(endPoint) && updateDataNodeList()` moves on to the next
candidate in both cases: when the node can't be reached, and when it was
reached but `SHOW AVAILABLE URLS` failed.
So a cluster-wide query failure opens a new session on every discovered node
and every seed. One example is a cluster with no ConfigNode leader, where
`showDataNodes` fails on every DataNode. This happens on every refresh of every
`Session`, while the monitor from the L213 thread is held.
Measured for one refresh with 3 nodes whose query fails after 1 s:
- base: 1 failed query, 0 new sessions, ~1.0 s
- head: 3 failed queries + 2 new sessions, ~3.0 s
Suggestion: move to the next candidate only when the *connection* fails. If
a reachable node fails the query, end this round and keep the current snapshot.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/SessionConnection.java:
##########
@@ -1143,18 +1145,18 @@ private boolean reconnect() {
for (int i = 1; i <= SessionConfig.RETRY_NUM; i++) {
if (transport != null) {
transport.close();
- endPointList = availableNodes.get();
- int currHostIndex = random.nextInt(endPointList.size());
- int tryHostNum = 0;
- for (int j = currHostIndex; j < endPointList.size(); j++) {
- if (tryHostNum == endPointList.size()) {
- break;
- }
- this.endPoint = endPointList.get(j);
- if (j == endPointList.size() - 1) {
- j = -1;
- }
- tryHostNum++;
+ List<TEndPoint> discoveredNodes = availableNodes.get();
+ endPointList = new ArrayList<>(discoveredNodes);
+ if (!endPointList.isEmpty()) {
+ Collections.rotate(endPointList,
-random.nextInt(endPointList.size()));
+ }
+ if (availableNodes instanceof INodeSupplier) {
+ endPointList.addAll(((INodeSupplier) availableNodes).getSeedNodes());
+ }
+ // Try every discovered endpoint before falling back to seeds, once
per endpoint per round.
+ endPointList = new ArrayList<>(new LinkedHashSet<>(endPointList));
Review Comment:
**Deduplication is by literal `TEndPoint`.** If `nodeUrls` use host names
and `dn_rpc_address` uses IPs (or the reverse), every node appears twice: here,
and in the candidates of `NodesSupplier.run()`.
Measured with seeds `localhost:p1/p2` and discovery returning
`127.0.0.1:p1/p2`, all nodes down: one `reconnect()` makes 12 attempts on head
versus 6 on base. Each attempt can be a full connect timeout.
There are two more side effects:
- Connecting through the host-name form of an already discovered node also
calls `requestRefresh()`.
- After that, `this.endPoint` is never in the discovered list, so
`callWithRetryAndReconnect` reconnects on every retryable status.
Suggestion: document that seeds should use the same address form as
`dn_rpc_address`, or deduplicate on resolved addresses.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/SessionConnection.java:
##########
@@ -1143,18 +1145,18 @@ private boolean reconnect() {
for (int i = 1; i <= SessionConfig.RETRY_NUM; i++) {
if (transport != null) {
transport.close();
- endPointList = availableNodes.get();
- int currHostIndex = random.nextInt(endPointList.size());
- int tryHostNum = 0;
- for (int j = currHostIndex; j < endPointList.size(); j++) {
- if (tryHostNum == endPointList.size()) {
- break;
- }
- this.endPoint = endPointList.get(j);
- if (j == endPointList.size() - 1) {
- j = -1;
- }
- tryHostNum++;
+ List<TEndPoint> discoveredNodes = availableNodes.get();
+ endPointList = new ArrayList<>(discoveredNodes);
+ if (!endPointList.isEmpty()) {
+ Collections.rotate(endPointList,
-random.nextInt(endPointList.size()));
+ }
+ if (availableNodes instanceof INodeSupplier) {
+ endPointList.addAll(((INodeSupplier) availableNodes).getSeedNodes());
Review Comment:
nit: seeds are appended in a fixed order, and for `SessionPool` the shared
supplier's seeds are never shuffled. When all discovered endpoints fail, every
pooled session falls back to the same first live seed, and nothing rebalances
them afterwards. Rotating the seed part randomly in each round, as already done
for the discovered part, would spread that load.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/config/metadata/ShowAvailableUrlsTask.java:
##########
@@ -58,7 +58,9 @@ public static void buildTsBlock(
if (showDataNodesResp.getDataNodesInfoList() != null) {
for (TDataNodeInfo dataNodeInfo :
showDataNodesResp.getDataNodesInfoList()) {
String status = dataNodeInfo.getStatus();
- if (RegionStatus.Removing.getStatus().equals(status)) {
+ if (NodeStatus.Removing.getStatus().equals(status)
+ || NodeStatus.Unknown.getStatus().equals(status)
+ || NodeStatus.Stopped.getStatus().equals(status)) {
Review Comment:
**A node in graceful shutdown is still advertised.**
`DataNodeShutdownHook.run()` does three things in this order:
1. It stops `ExternalRPCService` (L117).
2. It sets `ReadOnly` with reason `NodeStatus.STOPPING` (L120-122).
3. It reports `Stopped` to the ConfigNode leader at the very end, after
`DataNode.stop()` (L203).
Between steps 2 and 3 the node runs the WAL flush, `syncCloseAllProcessor()`
and Ratis snapshots, and waits for pipes. Heartbeats keep reporting
`ReadOnly`/`Stopping`, so this filter still returns the node although its
client port is already closed. `SessionSeedNodeFallbackIT` waits for `Stopped`
before calling `run()`, so it does not cover this window.
`TDataNodeInfo` already carries `statusReason`;
`NodeManager.getRegisteredDataNodeInfoList()` fills it from the heartbeat
sample. This filter could also skip `ReadOnly` when
`NodeStatus.STOPPING.equals(dataNodeInfo.getStatusReason())`. An older
ConfigNode that doesn't send the reason keeps today's behaviour. A
`ShowAvailableUrlsTaskTest` case for `ReadOnly` + `Stopping` would cover it.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -564,7 +565,7 @@ public SessionPool(AbstractSessionPoolBuilder builder) {
if (builder.nodeUrls.isEmpty()) {
throw new IllegalArgumentException(SessionMessages.NODE_URLS_EMPTY);
}
- this.nodeUrls = builder.nodeUrls;
+ this.nodeUrls = Collections.unmodifiableList(new
ArrayList<>(builder.nodeUrls));
Review Comment:
nit: only the builder constructor copies `nodeUrls`. The `List<String>`
constructors (L506) still keep the caller's list, and later
`constructNewSession()` calls read it. Copying there too would match
"independent copies" in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]