yupeng9 commented on PR #10572: URL: https://github.com/apache/pinot/pull/10572#issuecomment-1504340869
> Currently the deduplication is handled using the same way as upsert as a short term solution (not production ready). We have done a lot of bugfixes to the upsert implementation, but not actively maintain the dedup implementation. > > My suggestion would be to redesign dedup from scratch since it is not the same as upsert (no need to maintain valid docs, no need to track segment etc.), and TTL (dedup window) should be a must have for dedup. If we have proper TTL, the key size should be much smaller. > > After that if we still need disk based KV store we can introduce that as a plug-in. We don't want to introduce RocksDB dependency in default distribution I think the redesign of dedup makes sense but it'll be a much larger and involved effort, and unlikely for @raghavgautam to take. I think it makes sense to start a separate effort of dedup V2, but in the meantime it's fine for the community to add features and make improvements over V1, assuming it'll take some time for V2 to be ready. For depedency on KV store, I left a similar comment that we'd better move this to a plugin for simpler dependencies of the core. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
