GitHub user cshuo added a comment to the discussion: RocksDB as The Replica of MDT/RLI
> What does this mean for scenarios 2,3? Flink won't replay uncommitted input > streams, right since checkpoint is successful. Hudi cannot commit either > because writes are conflicting? Yes, that's a common issue for flink writer, regardless of the index type. > (note this is 180* from my original thinking that NBCC is the harder of the > two. I realize now that the record location does not change.). Poke holes at > this? “Record location does not change” only covers existing keys. For a new key, multiple writers may all miss it in their local RLI replicas and independently assign it to different file groups. NBCC cannot reconcile these writes because they do not conflict on the same file group, potentially creating duplicates. This is why NBCC currently requires the simple bucket index: its deterministic key -> bucket -> file group mapping makes every writer choose the same location. Supporting Flink RLI with NBCC therefore requires a logically shared, atomic allocator for new keys—`record key -> file group`, so all writers agree on the first assignment. GitHub link: https://github.com/apache/hudi/discussions/18296#discussioncomment-18243039 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
