GitHub user vinothchandar added a comment to the discussion: RocksDB as The Replica of MDT/RLI
Starting a thread here in the discussion around issues flagged in RFC-107 (https://github.com/apache/hudi/pull/19046) For background, we landed the first version of RFC-107 that adds a rocksDB backed partitioned index backend. The idea is simple, we cache fully all "hot" partitions affected by a write such that the rocksDB state scales `O(num_storage_partitions_actively_written_to)` . RocksDB is the source of truth i.e assume if there is a location entry in local rocksDB state, it is correct. **Few scenarios complicate this:** 1. Concurrent async clustering, that can change the file group location of a record. 2. Concurrent writer (OCC), which can write records into same file group as a writer with Rocksdb record index backend. But, this does not change the location i.e key to file group mapping. 3. Concurrent writer (NBCC) which can write records into same file group as writer with Rocksdb record index backend. Even this should not change the record location. **Questions that arise:** 1. From https://github.com/apache/hudi/pull/19046#issuecomment-5351517103 , `when the exception throws because of the confclit resolution, the commit procedure aborted and trigger the job to restart, but the job by default restore from the last successful checkpoint ckp_n which is actually invalid to Hudi.` => What does this mean for scenarios 2,3? Flink won't replay uncommitted input streams, right since checkpoint is successful. Hudi cannot commit either because writes are conflicting? 2. For scenario 1, we need some safe, atomic recovery protocol? What is that? we cannot disallow clustering altogether. 3. We can throw an exception for OCC, with rocksDB RecordIndexBackend is enabled? i.e I am saying we only support NBCC (this does not need the OCC like error-retry loop and is way more streaming friendly). (note this is 180* from my original thinking that NBCC is the harder of the two. I realize now that the record location does not change.). Poke holes at this? GitHub link: https://github.com/apache/hudi/discussions/18296#discussioncomment-18240183 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
