dbtsai commented on PR #108: URL: https://github.com/apache/spark-connect-rust/pull/108#issuecomment-6502074780
Thanks @viirya and @tdcmeehan. Addressed the regression-test and documentation items in e60aa53: - Added a normal-handoff guard that makes `pyarrow.ipc.open_file` fail. It passes on this revision and fails against master `d884e34`, so returning to the old IPC handoff is detected. The NUL tests separately assert that IPC is used exactly once. - Added list-of-struct and map-value-struct NUL-name cases, plus nullability mismatch cases with and without NUL names. - Missing `protoc` now fails when `CI` is set and skips locally; two tests cover those outcomes. - Clarified that the NUL panic occurs inside the C Stream `get_schema` callback and aborts Python, rather than producing a Python exception. - Removed the redundant buffer-address assertions; retained the value checks after dropping the table/session and forcing GC. - Documented the tested combinations in `docs/architecture.md`: PyArrow 10.0.0/Python 3.9 and PyArrow 26.0.0/Python 3.12. CI currently tests the latest PyArrow, not a pinned minimum. I also ran the identical 28-call RSS probe on rebuilt master and this revision in fresh processes, with the same 128 MiB input, environment and four-call warmup. Post-GC RSS on master rose **1129.23 → 3976.06 MiB**; C Stream rose **418.81 → 569.47 MiB**. All calls passed validation. Growth is not unique to this PR, but the combined mock-server/client process still prevents attributing it to the client, server, or allocator. I am not claiming flat RSS or leak freedom. The methodology and reproducer are committed in [dev/design/arrow-handoff-rss.md](https://github.com/dbtsai/spark-connect-rust/blob/e60aa53/dev/design/arrow-handoff-rss.md). Validation: **299 passed, 1 skipped on each of PyArrow 10 and 26**, including 26 handoff/fixture cases; **392 Rust unit tests passed**; all integrations compiled with and without `wasm-udf`; **3 native live-server smoke tests passed**; official collection parity **15 passed, 1 skipped**. Known baseline Clippy failures and the existing `SparkSession.Hook` API gap remain. **Still open:** wiring `test_arrow_e2e.py` into CI. Per the author's request to leave `.github/workflows` unchanged, this follow-up documents those tests as manual-only rather than claiming that the official parity gate covers native `toArrow()`. The live-server tests were rerun locally; this does not close the CI-wiring request. The PyArrow-version documentation follows the alternative tdcmeehan suggested to a pinned CI job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
