loserwang1024 opened a new issue, #3389: URL: https://github.com/apache/fluss/issues/3389
### Search before asking - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar. ### Fluss version 0.9.0 (latest release) ### Please describe the bug ð During a rolling upgrade of Fluss servers, the Admin client suffers from two problems: ### Stale metadata never refreshed â permanent failure example: Admin#listPartitionInfosãgetLatestLakeSnapshotãgetLatestKvSnapshotsãgetTableInfo `readOnlyGateway` is backed by `metadataUpdater::getRandomTabletServer` as its node supplier. When all tablet server IPs change (e.g., pods restarting with new IPs in Kubernetes), the cached Cluster still holds the old IPs. The RPC fails with a network error, but nothing triggers updateMetadata() â the CompletableFuture simply completes exceptionally and the stale Cluster remains unchanged. Subsequent calls keep resolving the same stale nodes, making this a permanent failure that no amount of caller-side retries can fix. ### Stale leader routing â transient failure Example: Admin#listOffsets ```java // FlussAdmin.java:529 metadataUpdater.updateTableOrPartitionMetadata(physicalTablePath.getTablePath(), null); // ... then prepareListOffsetsRequests() calls leaderFor() and sends to that leader ``` Each call starts by refreshing metadata, so it picks up the latest leader assignment. However, if a leader-follower switch occurs in the small window between the metadata refresh and the actual RPC send, the request is routed to the old leader (now a follower). **This is a transient failure â a caller-side retry will trigger a fresh metadata update at the beginning of the next call, resolve the new leader, and succeed.** ### Solution ⺠Therefore, we plan to address this in two steps: 1. Urgently fix Problem 1 â refresh metadata upon network errors on the readOnlyGateway path, so the client can recover from stale server addresses. 2. Problem 2 is less urgent â we can address it later by introducing a more robust metadata mechanism at the framework level. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
