suvodeep-pyne opened a new pull request, #19738: URL: https://github.com/apache/pinot/pull/19738
## Summary Every IdealState update attempt clones the IdealState it just read before applying the updaters (`IdealStateGroupCommit` and `IdealStateSingleCommit`, through `HelixHelper.cloneIdealState`). The clone was a `ZNRecordSerializer` round trip: it pretty-prints the whole ZNRecord as JSON, gzips it when compression is enabled (always the case above 1000 segments), then decompresses and parses it back. This PR replaces the round trip with a direct deep copy of the record. ## Motivation On large tables the clone is about half of the CPU time of an update attempt, and a longer attempt is more likely to lose to a concurrent writer with `Version changed while updating ideal state`. On a pauseless REALTIME table with ~550K segments (IdealState ~5 MB compressed), update attempts took 6–11 s while other controllers were deleting segments, and group-commit batches lost all 10 attempts, failing every segment commit in the batch. Measured on a synthetic IdealState with 547K segments × 3 replicas (4.6 MB compressed), JDK 25, median of 7 runs: | Step of one update attempt | Time | |---|---| | Read from ZK (gunzip + parse) | 1.1 s | | Clone, serialization round trip (before) | 2.7 s | | Clone, deep copy (after) | 0.07 s | | Write (serialize + gzip) | 1.2 s | ## Changes - `HelixHelper.cloneIdealState` copies the id, simple fields, map fields, list fields and raw payload directly, producing the same container types and iteration order as the round trip: TreeMap outer maps from the `IdealState` constructor, `LinkedHashMap` inner maps in source order, `ArrayList` lists. Like the round trip, it does not copy the ZK stat fields (e.g. version); the update path uses the version of the record it read. - The serializer's write policies (list field bound, size limit) are no longer applied to the copy. They still apply when the IdealState is written, and no caller relied on them at clone time. - `IdealStateSingleCommit.cloneIdealState` delegates to `HelixHelper.cloneIdealState`. ## Testing - `HelixHelperTest#testCloneIdealState`: the copy equals the round-trip clone, including iteration order of outer and inner maps, raw payload and version. It also checks that changing the copy's simple fields, map fields and list fields leaves the original unchanged. - `IdealStateGroupCommitTest`, `PinotHelixResourceManagerStatelessTest` and the controller `HelixHelperTest` pass. These update IdealStates on a real ZK through both commit paths. Related to #<PR1> and #<PR2>, which make segment commits and their repair tolerate failed IdealState updates on large tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
