Keston375 opened a new issue, #14885: URL: https://github.com/apache/pinot/issues/14885
# Title Bugfix: Construct `queryableDocIds` When Adding Segments Out of TTL in Upsert Tables ## Repository Version release-1.0.0 ## Key Words / Search Terms upsert, TTL, `queryableDocIds`, segment addition, deletion, `upsertConfig`, `deleteRecordColumn` ## Description In Apache Pinot, upsert tables support both soft-deletes via a designated boolean `deleteRecordColumn` and a time-to-live (TTL) mechanism to manage the lifecycle of primary keys. Previously, when adding segments that fall outside the defined TTL, the system did not construct the `queryableDocIds`, leading to potential inconsistencies in query results. This bugfix addresses the issue by ensuring that `queryableDocIds` are properly constructed when segments beyond the TTL are added, thereby maintaining accurate and consistent query responses. ## Expected Behavior - When segments that exceed the defined TTL are added to an upsert table, the system should construct the `queryableDocIds` to reflect the current state of data accurately. - Queries executed against the upsert table should return results consistent with the applied deletions and TTL configurations. ## Current Behavior - Segments added beyond the TTL do not trigger the construction of `queryableDocIds`, resulting in potential discrepancies in query outcomes. - Soft-deleted records may still appear in query results due to the absence of updated `queryableDocIds`. ## Use Cases / Motivating Examples (for Feature Requests) - Ensuring that upsert tables with both deletion and TTL configurations maintain data integrity and provide accurate query results. - Preventing scenarios where outdated or soft-deleted records are inadvertently included in query responses due to missing `queryableDocIds`. - Enhancing the reliability of upsert tables in handling complex data lifecycle management involving both deletions and TTL policies. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
