tibrewalpratik17 commented on issue #11736: URL: https://github.com/apache/pinot/issues/11736#issuecomment-1747480797
> Enabling tTLs for soft deletion keys is similar to regular keys, it will need to have consistency enforcement as metadataTTL does What does consistency mean here? Consistency of HashMap during server restarts? The goal is to remove keys marked for deletion after a TTL window + mark the validDocIds as invalid. Once we mark the validDocID as invalid, i think during server restart the HashMap won't have the primary-key as well. Even if it does, the key will be removed in the next cycle. > do you think we can enable the deleted key hard deletion during segment compaction minion job? Segment compaction minion job doesn't remove primary-keys from the HashMap. It picks one segment at a time, looks at the validDocIDs and only persists those records back into the segment. Doing this via segment compaction, means using `queyrableDocIDs` to compact segments and also how will we still update the ConcurrentHashMap via minion nodes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
