dbtsai opened a new pull request, #107: URL: https://github.com/apache/spark-connect-rust/pull/107
### What changes were proposed in this pull request? Upgrade PyO3 from 0.28.3 to 0.30.1 in a standalone dependency change: - Update the workspace dependency and five PyO3 lockfile entries, including the required Syn 3 dependency. - Declare `rust-version = "1.85"` for `pyspark-rs`, matching PyO3 0.30.1's published requirement. The workspace default remains 1.82. - Keep `extension-module` and `abi3-py39`, and remove an outdated version-specific comment. There are no Arrow handoff, scalar conversion, or Python wrapper implementation changes. This is separate from #103, which remains unchanged for now. ### Why are the changes needed? Track the PyO3 upgrade under [SPARK-60596](https://issues.apache.org/jira/browse/SPARK-60596), independently of the Arrow optimization in #103, so it can be reviewed and reverted separately. ### Does this PR introduce _any_ user-facing change? The Python extension's declared minimum Rust version becomes 1.85. Its Python 3.9+ stable-ABI target is unchanged. No Python API changes are intended. **Performance concern: the isolated benchmark reproduces a `collect()` slowdown with this upgrade. This draft is not ready to merge on performance grounds.** The four selected collection workloads were about 15-31% slower in the initial run and 19-30% slower in a separate slot-swapped confirmation. The exact mechanism has not been diagnosed. The two `toArrow()` workloads did not show a clear timing difference in this sample. ### How was this patch tested? Built both unmodified upstream `fbd6af5763bae5cd572ba4438026e675165ba70a` and the four-file upgrade offline as optimized release wheels (`opt-level=3`, `abi3-py39`) on macOS arm64, using Rust 1.99.1 and maturin 1.15.0. - `cargo test -p apache-spark-connect --release --lib`: **388 passed**. - `cargo test -p apache-spark-connect --release --no-run`: all integration tests compiled. - `cargo test -p apache-spark-connect --release --features wasm-udf --no-run`: all integration tests compiled with WASM enabled. - `cargo fmt --all --check`: passed. - Existing `python/tests/test_dropin_offline.py` against the built wheel: **273 passed** on Python 3.9 / PyArrow 10, and **273 passed** on Python 3.12 / PyArrow 26. - `scripts/run_official_tests.py --spark <Spark-4.2.0-tree> --remote sc://localhost:15084 --only pyspark.sql.tests.connect.test_parity_collection`: **15 passed, 1 skipped**, using the candidate extension and a local Spark Connect server. This exercises official PySpark through the Rust transport, not the entire native drop-in API. - Public API-name comparison across 12 classes: unchanged from the baseline. The existing `SparkSession.Hook` gap remains; `Column.to_plan` is intentionally absent. This is not a claim of complete behavioral/signature parity. - `cargo clippy --workspace --all-targets`: not green. The same 150 error diagnostics, including their source locations, reproduce on unmodified upstream (workspace plus a separate `pyspark-rs` check to finish collecting baseline errors). No new tests were added: this change updates the binding dependency without changing application logic, and uses the existing unit, integration, offline Python, and official parity suites. #### Isolated benchmark Used [dbtsai/pyspark-connect-benchmark](https://github.com/dbtsai/pyspark-connect-benchmark/tree/1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4), with the same upstream source before and after except for this PR. Six fresh-process samples per client/case, matching Python 3.12.14, PyArrow 26.0.0, pandas 2.3.3, NumPy 2.5.4 and psutil 7.2.2. These public client actions receive canned localhost gRPC responses; they do not benchmark Spark execution. Initial run (median time and sampled client-process peak RSS): | Workload | Time before / after | Change | Peak RSS before / after | |---|---:|---:|---:| | `core-return-long-w4-b1000-collect` | 12.18 / 15.89 ms | +30.5% | 137.12 / 139.41 MiB | | `core-return-string-w4-b1000-collect` | 22.50 / 26.40 ms | +17.3% | 144.67 / 146.52 MiB | | `core-return-string-w4-b10000-collect` | 21.10 / 25.23 ms | +19.6% | 146.58 / 147.80 MiB | | `allocation-first-100000x8-collect` | 23.36 / 26.90 ms | +15.2% | 157.56 / 159.23 MiB | | `core-return-long-w50-b10000-toArrow` | 29.49 / 29.39 ms | -0.4% | 268.93 / 270.27 MiB | | `return-buffer-134217728-toArrow` | 59.27 / 62.06 ms | +4.7% | 628.98 / 629.78 MiB | The four collection cases were rerun with baseline/candidate harness slots swapped: slowdowns were respectively **30.2%, 19.1%, 20.6%, and 20.0%**. Before/after timing ranges did not overlap for those cases in either run. All **180 samples across the three clients passed** validation; Rust request counters matched and source/wheel identities remained unchanged. The runs are not pooled. RSS is sampled resident memory, not allocation counts, and may miss short spikes. This selected benchmark does not establish performance on every workload or platform. #### Draft checklist - [ ] Investigate the reproduced collection slowdown before merging. - [ ] Run the full official parity gate; only the collection suite was run locally. - [ ] Validate the declared Rust 1.85 minimum; local builds used 1.99.1. - [ ] Review CI results and the reproduced upstream Clippy/API gaps. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (application version unavailable). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
