empiredan opened a new pull request #646: URL: https://github.com/apache/incubator-pegasus/pull/646
### What problem does this PR solve? <!--add issue link with summary if exists--> While multiple services are deployed on a node, memory usage of each service has to be restricted. As for pegasus, memory is consumed by both rDSN and rocksdb. We first restrict rocksdb's memory since it's easier to implement. ### What is changed and how it works? This PR involves some rocksdb's options, to manage memtable, block cache, index and filter blocks into block cache size, by which we can take precise control over the memory used by rocksdb. If `rocksdb_enable_write_buffer_manager = true`, memtable will be managed into block cache size, and `rocksdb_block_cache_capacity` will be considered as the total memory assigned to rocksdb, `rocksdb_total_size_across_write_buffer` is the maximum memory assigned to memtables. If `rocksdb_cache_index_and_filter_blocks = true`, index and filter blocks will be stored in block cache, and meanwhile you'd better use partitioned filters; If `rocksdb_cache_index_and_filter_blocks = false`, you have to use `rocksdb_max_open_files` to take control. As for details of both write buffer manager and index and filter blocks, see the docs as below: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Unit test - Integration test - Manual test (add detailed scripts or steps below) - No code Code changes - Has exported function/method change - Has exported variable/fields change - Has interface methods change - Has persistent data change Side effects - Possible performance regression - Increased code complexity - Breaking backward compatibility Related changes - Need to cherry-pick to the release branch - Need to update the documentation - Need to be included in the release note ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
