XiaoHongbo-Hope commented on PR #455: URL: https://github.com/apache/paimon-rust/pull/455#issuecomment-4886288975
> Thanks @XiaoHongbo-Hope! A question on necessity first. > > The direction of the read effort (#413) is to let pypaimon run its DataFrame read on the Rust core — initially as a **basic, opt-in path behind a config flag**, not a wholesale replacement, so it can mature alongside the pure-Python path. In that model Rust both plans _and_ reads: PR3 already exposes `new_read().read(splits)` returning Arrow from the Rust `TableRead`, and splits stay opaque on the Python side — a transport token, nothing more — so pypaimon never needs to look inside one. > > `to_dict()` exposes the full internal split contents, which is only needed if something **rebuilds the split and reads the files outside Rust** — i.e. reading in Python, which is the opposite of the direction #413 is moving in. > > So: **what's the use case for exposing split internals?** If it's for a Rust-plans / Python-reads path, I think we should align on that direction first — otherwise it risks pulling us away from the opt-in Rust read path we're building toward. Thanks, Junrui. Got your concern, will discuss with Jingsong and then back to you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
