TheR1sing3un commented on issue #11715: URL: https://github.com/apache/hudi/issues/11715#issuecomment-2264428887
> @TheR1sing3un hi, this is a great capability, but before starting, it is necessary to confirm in which scenarios it can bring benefits. Could you provide a few scenarios? > Assume the following scenario - some high-frequency keys are updated frequently - the first appearance of these keys in log files came relatively late - MaxMemoryForCompaction relatively small When a Compaction/Log-Compaction happens and these keys were not initially loaded into the memory map, the subsequent frequent 'get' in compaction logic results in a large number of random reads on the disk (both `bitcask` and `rocksdb` are log-structure-append-only-file data structure, high performance for reading but low performance for writing), which will cause great performance degradation. If the map in the memory has a certain elimination and update mechanism, such as lru or lfu, the memory hit ratio will be improved in some scenarios for better compaction performance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
