raghavgautam opened a new pull request, #10572: URL: https://github.com/apache/pinot/pull/10572
This patch addresses memory problem as discussed in https://github.com/apache/pinot/issues/10571 It uses rocksdb as an on disk key-value store instead of ConcurrentHashMap to reduce memory requirement The memory requirement is number_of_keys * 10 bits + 64MB for each column family. So, for 1B keys it will use <1.5GB RAM. Source: - https://www.bookstack.cn/read/rocksdb-en/5dd063f6b6ed224d.md - https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB#indexes-and-filter-blocks For testing, I was able to ingest at 8-10K/sec, with 265 million records already loaded on my M1 max with 10 cores and 64 GB RAM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
