raghavgautam opened a new issue, #10571: URL: https://github.com/apache/pinot/issues/10571
This problem was reported by @ybrankov Currently Pinot stream ingestion with exact deduplication requires a lot of memory because it uses [ConcurrentHashMap](https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/dedup/PartitionDedupMetadataManager.java#L45). - Let use say we want to ingest 1K/node and want to keep data for 11 days, we want to be able to deduplication about 1K/sec * 86400 sec/day * 11 days ~ 1B rows. - Assuming the primary key is a UUID string, it takes about 36 byes. - So, this will require 36 GB just for keys. There is also java overhead and memory needed for storing values. All this adds up to become too expensive and does not work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
