[
https://issues.apache.org/jira/browse/SPARK-60596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DB Tsai updated SPARK-60596:
----------------------------
Description:
Upgrade the Spark Connect Rust Python extension from PyO3 0.28.3 to 0.30.1 in a
standalone dependency change.
Scope:
* Bump the workspace PyO3 dependency and the five PyO3 lockfile entries, adding
the required Syn 3 dependency.
* Declare Rust 1.85 for pyspark-rs to match PyO3 0.30.1's minimum compiler
requirement. Leave the workspace default unchanged.
* Preserve extension-module and abi3-py39, and remove an outdated
version-specific comment.
This separates the dependency upgrade from SPARK-60586 / spark-connect-rust PR
#103. It does not change the Arrow handoff, scalar conversion, or Python
wrapper implementation.
Validation so far: optimized baseline and upgraded wheels built offline; 388
core unit tests passed; 273 existing offline Python tests passed on each of
Python 3.9 with PyArrow 10 and Python 3.12 with PyArrow 26; integration tests
compiled with and without wasm-udf; official collection parity passed (15
passed, 1 skipped); formatting passed. Public API names are unchanged. Full
official parity and a Rust 1.85 build have not been verified locally. Workspace
Clippy errors also reproduce on the unchanged upstream baseline.
Performance concern to resolve before merge: an isolated six-sample comparison
using dbtsai/pyspark-connect-benchmark at
1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4 found four collect() cases 15-30%
slower after the upgrade. A separate six-sample confirmation with benchmark
slots swapped reproduced 19-30% slowdowns. The before/after sample ranges did
not overlap in either run. Two toArrow() cases did not show a clear timing
difference. Peak RSS increased by about 1-2 MiB. Both wheels used the same
fbd6af5763bae5cd572ba4438026e675165ba70a source plus only this upgrade,
optimized release builds, Python 3.12.14 and matching dependencies. All 180
samples passed validation; source/wheel identities and Rust RPC counters
matched. This measures client operations against canned localhost gRPC
responses, not real Spark execution. The exact cause of the slowdown remains
undiagnosed.
was:
Upgrade the Spark Connect Rust Python extension from PyO3 0.28.3 to 0.30.1 in a
standalone dependency change.
Scope:
* Bump the workspace PyO3 dependency and the five PyO3 lockfile entries, adding
the required Syn 3 dependency.
* Declare Rust 1.85 for pyspark-rs to match PyO3 0.30.1's minimum compiler
requirement. Leave the workspace default unchanged.
* Preserve extension-module and abi3-py39, and remove an outdated
version-specific comment.
This separates the dependency upgrade from SPARK-60586 / spark-connect-rust PR
#103. It does not change the Arrow handoff, scalar conversion, or Python
wrapper implementation.
Validation so far: optimized baseline and upgraded wheels built offline; 388
core unit tests passed; 273 existing offline Python tests passed on each of
Python 3.9 with PyArrow 10 and Python 3.12 with PyArrow 26; integration tests
compiled with and without wasm-udf; official collection parity passed (15
passed, 1 skipped); formatting passed. Public API names are unchanged. Full
official parity and a Rust 1.85 build have not been verified locally. Workspace
Clippy errors also reproduce on the unchanged upstream baseline.
> Upgrade PyO3 to 0.30.1 in the Spark Connect Rust client
> -------------------------------------------------------
>
> Key: SPARK-60596
> URL: https://issues.apache.org/jira/browse/SPARK-60596
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> Upgrade the Spark Connect Rust Python extension from PyO3 0.28.3 to 0.30.1 in
> a standalone dependency change.
> Scope:
> * Bump the workspace PyO3 dependency and the five PyO3 lockfile entries,
> adding the required Syn 3 dependency.
> * Declare Rust 1.85 for pyspark-rs to match PyO3 0.30.1's minimum compiler
> requirement. Leave the workspace default unchanged.
> * Preserve extension-module and abi3-py39, and remove an outdated
> version-specific comment.
> This separates the dependency upgrade from SPARK-60586 / spark-connect-rust
> PR #103. It does not change the Arrow handoff, scalar conversion, or Python
> wrapper implementation.
> Validation so far: optimized baseline and upgraded wheels built offline; 388
> core unit tests passed; 273 existing offline Python tests passed on each of
> Python 3.9 with PyArrow 10 and Python 3.12 with PyArrow 26; integration tests
> compiled with and without wasm-udf; official collection parity passed (15
> passed, 1 skipped); formatting passed. Public API names are unchanged. Full
> official parity and a Rust 1.85 build have not been verified locally.
> Workspace Clippy errors also reproduce on the unchanged upstream baseline.
> Performance concern to resolve before merge: an isolated six-sample
> comparison using dbtsai/pyspark-connect-benchmark at
> 1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4 found four collect() cases 15-30%
> slower after the upgrade. A separate six-sample confirmation with benchmark
> slots swapped reproduced 19-30% slowdowns. The before/after sample ranges did
> not overlap in either run. Two toArrow() cases did not show a clear timing
> difference. Peak RSS increased by about 1-2 MiB. Both wheels used the same
> fbd6af5763bae5cd572ba4438026e675165ba70a source plus only this upgrade,
> optimized release builds, Python 3.12.14 and matching dependencies. All 180
> samples passed validation; source/wheel identities and Rust RPC counters
> matched. This measures client operations against canned localhost gRPC
> responses, not real Spark execution. The exact cause of the slowdown remains
> undiagnosed.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
