Vamsi-klu opened a new pull request, #19084: URL: https://github.com/apache/pinot/pull/19084
## Why Upsert compaction must apply a server `validDocIds` bitmap only to a matching on-disk segment body. CRC mismatches across ZK / deepstore / replica servers caused tasks to **skip work while looking healthy**, or to fail opaquely on reload races. Ops could not tell “benign skip” from “hard failure,” and space reclaim stalled. Much of the original “WARN then COMPLETED” path has hardened over time; this PR finishes **observability**, **data-CRC alignment**, and **targeted retry** without redesigning segment commit. ## Impact - **Clear skip vs fail signals** (meters + progress messages) for ZK CRC change, empty valid docs, deepstore mismatch, server bitmap mismatch. - **Fewer false failures** when segment CRC differs but **data CRC** matches (`crcMatches`). - **Better success rate** on transient server reload races via bounded bitmap-fetch retry. - **Safer ops**: does not expand `ignoreCrcMismatch` defaults; still refuses to compact mismatched bytes+bitmap pairs. ## How - Align deepstore Check B with `MinionTaskUtils.crcMatches` (segment CRC or data CRC). - Add minion meters for CRC skip/fail/empty-doc paths; surface skip reasons on progress observer. - Bounded retry on Check C server bitmap CRC mismatch before failing the task. - Unit coverage for executor convert/CRC branches (previously nearly untested). ## Test plan - [x] `UpsertCompactionTaskExecutorTest` — CRC match/mismatch, ignore flag, empty bitmap, null bitmap fail. - [x] `MinionTaskUtilsTest` / `BaseSingleSegmentConversionExecutorTest` extensions for skip metering paths. - [ ] `./mvnw -pl pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks -am -Dtest=UpsertCompactionTaskExecutorTest,MinionTaskUtilsTest,BaseSingleSegmentConversionExecutorTest -Dsurefire.failIfNoSpecifiedTests=false test` ## Related fixes: #13491 ## Reviewers Suggested: tibrewalpratik17 (issue author) --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Grok Build (xAI) Generated-by: Grok Build (xAI) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
