codope commented on issue #848: URL: https://github.com/apache/hudi-rs/issues/848#issuecomment-6531047845
Thanks @yihua, +1 on making the C ABI the one native boundary. ### Who uses `hudi-cpp` today As far as I know, only the in-repo demo. It isn't published, and nothing depends on cxx-specific behavior, so deprecating it in 0.7.0 costs little. I'd keep a thin header-only C++ wrapper over the C ABI rather than drop C++ entirely: engines end up writing RAII handles and exception translation anyway, and one canonical header is better than each engine writing its own. The C++ engine that would have mattered most has gone another way for now. The native Hudi MOR effort for Presto C++ (facebookincubator/velox#18530, prestodb/rfcs#66) proposes a pure C++ connector in Velox. RFC-0066 considered hudi-rs via C FFI and set it aside for three reasons: 1. a Rust toolchain in the Velox/Prestissimo build 2. buffers allocated outside Velox's `MemoryPool`, invisible to its arbitrator 3. a Tokio runtime inside a synchronous reader path The RFC keeps its connector layer independent of the reader, so hudi-rs could be swapped in later. I think those three points are a good bar for `hudi-ffi`. cc: @jja725 ### What a Velox-class engine needs In engines like Prestissimo and Spark on Gluten+Velox, a JVM coordinator plans and native workers read splits. The per-row read path wants to stay inside the engine. Its own Parquet reader pushes filters into decoding, loads columns lazily, keeps dictionary encoding and uses the engine's cache. And Arrow batches don't import into Velox for free, since strings, timestamps and nested types are laid out differently. So the most useful C ABI pieces for these engines are: 1. **Plan-only (your point 5).** I'd rank it higher. It lets the engine read base-file-only slices natively, which is most of the bytes for most tables. 2. **Log-decode-only.** Return a slice's log records, with positions and delete markers when the blocks carry them, and do no merge, so the engine can apply them over its own base-file read. 3. **Host-supplied limits at open.** A memory budget, the runtime's thread count, and bounded batch sizes. Together with plan-only, that addresses objections (2) and (3). Two smaller points: - #767 found that a per-read time window was silently ignored, because it's applied from the reader's configs. It would help if the C ABI said, per option, whether it applies at open, at plan or per read. - RFC-0066 plans golden fixtures from Hudi Java as its correctness oracle, and hudi-rs already checks its fixtures against Spark gold snapshots. Sharing one fixture set across hudi-rs and Velox would keep the readers from drifting. Also, I think #830 (byte-range split reads) should target the C ABI rather than the cxx bridge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
