Jackie-Jiang opened a new pull request, #20106: URL: https://github.com/apache/pinot/pull/20106
A server that already initialized a table can receive another partition during rebalance and start replaying before its older segments finish loading. The existing partial-upsert/dedup readiness check latches for the table manager's lifetime, so it never checks the newly assigned history. Loading the immediate predecessor also does not prove that every older segment is loaded. For partial upsert, replaying `B += 1` before the base `B = 1` arrives can produce `B = 1` instead of `B = 2`. Recheck required history before each partial-upsert or dedup consumer starts: - Wait for assigned ONLINE segments from the same partition with lower sequence numbers; conservatively include non-LLC segments that may contain base rows. - Exclude the consumer itself and its successors to avoid waiting on their completion. Re-read the assignment while waiting so dropped or reassigned prerequisites do not block forever. - Preserve parallel consumption during segment building by recognizing each local predecessor that explicitly closed consumption and released its permit. Merely finding a local segment is insufficient because immutable segments can be published before their upsert metadata finishes loading. - Keep the server readiness gate and retry ZooKeeper read failures through the existing five-second polling loop. Full-upsert and non-upsert tables retain their existing behavior. Follow-up to #20097; related to #20095. The controller's once-per-partition force commit behavior remains unchanged. This change runs on the receiving server and requires no configuration or protocol changes. ## Validation - The missing-base regression fails at the readiness assertion with the original table-wide latch, exposed through a test-only adapter. With the fix, it waits for the delayed base and verifies `B = 2` using real immutable/mutable segments and partial-upsert metadata. - 81 distinct focused cases passed across the recorded runs: `TableStateUtilsTest`, `RealtimeTableReadinessTest`, `RealtimeSegmentDataManagerTest`, `ConsumerCoordinatorTest`, and `RealtimeTableDataManagerTest`. Coverage includes dedup, full-upsert bypass, pauseless release, assignment changes, missing states, and ZooKeeper recovery. - JDK 25 compilation with deprecation/warning flags: no warnings on added lines. Spotless, checkstyle, license formatting, and license checks passed for both affected modules. Tests control Helix state publication synchronously; a full-cluster rebalance run was not performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
