krishan1390 opened a new pull request, #18850: URL: https://github.com/apache/pinot/pull/18850
## Summary `MinionTaskUtils.getValidDocIdsType` previously defaulted to `SNAPSHOT_WITH_DELETE` for delete-enabled upsert tables and silently overrode any user-specified `SNAPSHOT`/`IN_MEMORY` to `SNAPSHOT_WITH_DELETE`. This change makes the default `SNAPSHOT` for all upsert tables and honors the user-specified value as-is. ## Why `SNAPSHOT_WITH_DELETE` (queryableDocIds) excludes delete-marker (tombstone) records immediately, so upsert compaction physically drops tombstones regardless of `deletedKeysTTL`. `SNAPSHOT` (validDocIds) retains a tombstone until it expires via `deletedKeysTTL`. Dropping a tombstone before it expires loses the record of the deletion: when upsert metadata is rebuilt from segments (server restart, new replica, rebalance), the deleted primary key is no longer known, and an out-of-order or replayed record older than the delete is accepted as live data — resurrecting the invalidated record. Deferring tombstone removal to `deletedKeysTTL` prevents this. ## Behavior - Default `validDocIdsType` is now `SNAPSHOT` for all upsert tables (delete-enabled or not). - A user-specified `SNAPSHOT` / `IN_MEMORY` / `SNAPSHOT_WITH_DELETE` / `IN_MEMORY_WITH_DELETE` is honored as configured (no silent override). - Preconditions are unchanged: `SNAPSHOT` / `SNAPSHOT_WITH_DELETE` require upsert snapshots enabled; the `*_WITH_DELETE` types require a `deleteRecordColumn`. ## Tests Updated `MinionTaskUtilsTest`: the default-with-delete case now asserts `SNAPSHOT`, and the former override test now asserts that user-specified values are honored (renamed `testGetValidDocIdsTypeNotOverriddenForDeleteTable`). The validation-error tests for the retained preconditions are unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
