rmahindra123 opened a new pull request #3117:
URL: https://github.com/apache/hudi/pull/3117
## What is the purpose of the pull request
This pull request adds a new alternative based on RockDb for the Disk Based
Map that is used within the ExternalSpillableMap. Our benchmark results shows
that RockDb may improve performance significantly when the data set is large
while available memory may be scarce. RockDb supports compression, efficient
memory usage and native library, that may be more efficient in certain
situations. By default, disk based map will be used, and a config change will
be required to enable rocksDb.
In this PR, the rocksDB support is only enabled for HoodieMergeHandle, and a
subsequent PR will extend it to all consumers of ExternalSpillableMap (tracked
here HUDI-2044)
## Brief change log
- Adds a new alternative based on RockDb for the Disk Based Map that is
used within the ExternalSpillableMap.
- The support is currently added only for HoodieMergeHandle
## Verify this pull request
This change added tests and can be verified as follows:
Added the unit test in TestSpillableRocksDBBasedMap
Updated the test for TestExternalSpillableMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
