HuangZhenQiu commented on PR #19765: URL: https://github.com/apache/hudi/pull/19765#issuecomment-5480463920
It is a good question. As we described in the RFC, we probably need to decouple the dynamic partitioned record index load and final lookup into two operators. When a record of old partition comes, it will trigger the partition record index load, these record index should be keyed by and shuttle to all of other task managers. In this way, the RocksDB in each of task managers only needs to store the state it required, and one partition need to load once. The RocksDBPartitionedIndexBackend will be a configurable backend owned by BucketAssignFunction and the new function PreLookup. The RocksDB Dao will be initialized in singleton and shared by both BucketAssignFunction and PreLookup. How do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
