xiangfu0 opened a new pull request, #19118: URL: https://github.com/apache/pinot/pull/19118
## Summary Preserve stable record ordering for OFFLINE FULL upsert tables that do not configure a comparison column or time column. - Add the reserved internal `$upsertComparisonTime` column to the effective upsert schema. - Synthesize it from the authoritative segment push, creation, or index creation time for legacy segments. - Prefer a physically materialized value when a rewritten segment contains the internal column. - Preserve the physical internal column when it is absent from the user-facing table schema. - Use the same comparison-time reader for segment add, replacement/revert, TTL, and deleted-key cleanup paths. ## Root cause OFFLINE FULL upsert currently treats the segment push or creation time as a constant comparison value when no comparison or time column is configured. A Minion rewrite can rebuild unchanged records into a replacement segment with a newer creation or push time. After replacement or reload, those unchanged records are therefore treated as newly updated and can incorrectly replace records that arrived after the original source segment. The ordering timestamp only existed in mutable segment metadata, so it was not stable across physical segment rewrites. ## How to reproduce 1. Create an OFFLINE FULL upsert table without a comparison column or time column. 2. Upload a source segment containing a key at ordering time `T_A`. 3. Upload another segment containing a newer value for the same key at `T_C`. 4. Rebuild the original source through a Minion task and replace it with an output whose metadata time is `T_B`, where `T_A < T_C < T_B`. 5. Reload or replace the rebuilt segment. Before this change, the rebuilt row is compared using `T_B` and can incorrectly replace the value from `T_C`. With this change, a rewrite can physically preserve `T_A`; legacy segments continue to derive their value from segment metadata. ## Compatibility - Configured comparison columns and configured time columns are unchanged. - Empty and absent `comparisonColumns` both follow Pinot's existing normalization to no configured comparison column. - The user-facing table schema is not mutated. - Existing segments require no migration because the virtual provider supplies the legacy value. - Deploy this reader/loader compatibility before enabling the follow-up Minion writer that materializes the reserved column. ## Validation - Focused `pinot-segment-local` reactor: 51 tests passed. - `spotless:apply` for `pinot-spi,pinot-segment-local`. - `checkstyle:check` for `pinot-spi,pinot-segment-local`. - `license:format` and `license:check` for `pinot-spi,pinot-segment-local`. - `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
