dbtsai commented on PR #108: URL: https://github.com/apache/spark-connect-rust/pull/108#issuecomment-6483825292
Thanks @viirya, implemented your suggestion in d3eabc88eacb73a10b3df6a1f6eee9587da7c9d0. The handoff now uses `Box<dyn RecordBatchReader + Send>::into_pyarrow()` followed by `read_all()`, directly in `pyspark-rs`. I removed the custom callbacks, separate converter crate/workflow, and generic ordering/callback tests. The schema consistency check, NUL name/timezone IPC fallback, and `schema.to_pyarrow(py)?` preflight remain. PyO3 stays at 0.28.3. The updated wheel passed 294 tests on each of Python 3.12/PyArrow 26 and Python 3.9/PyArrow 10, including buffer-sharing/lifetime checks. All 360 samples in the fresh C Data / previous C Stream / upstream-reader comparison passed. Performance was broadly comparable, rather than an established general speedup; cold-wide medians were essentially unchanged. One memory caveat remains: a 128 MiB workload sample reached 285.7 MiB RSS with the upstream reader versus a 239.6 MiB maximum for the previous stream in the same round, despite close medians. That sample passed validation and is retained; the cause is unresolved. These are shared-desktop, canned-localhost receive/conversion measurements, not Spark execution. I've updated the description with the results and limitations. Keeping this draft while the RSS tail, full official parity, and minimum-Rust validation remain open. This is substantially simpler to maintain—thanks for pointing out the upstream reader path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
