[
https://issues.apache.org/jira/browse/SPARK-59167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59167:
-----------------------------------
Labels: pull-request-available (was: )
> Add an end-to-end Spark Connect client benchmark harness (Rust drop-in vs
> reference PySpark)
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-59167
> URL: https://issues.apache.org/jira/browse/SPARK-59167
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Add a reproducible end-to-end benchmark harness that compares the Rust-native
> Spark Connect client (pyspark-client-rust) against the reference Python Spark
> Connect client (pyspark-client). Both clients drive the same Spark Connect
> server, so the only variable is the client implementation (plan building,
> gRPC transport, Arrow IPC decoding, and Python result construction).
> scripts/benchmark_suite.py measures, across multiple dimensions:
> - transport payload-size sweep (64 KB - 16 MB)
> - row-count sweep
> - concurrent client-process counts
> - UDF complexity classes (identity, arithmetic, strings, expensive pure
> Python, NumPy, pandas_udf)
> - cold-start latency
> reporting rows/sec, GB/sec, CPU/GB, p50/p99/p999 latency, client CPU (Rust vs
> Python), a local server-JVM CPU proxy, and peak RSS. Results are written as
> JSON.
> scripts/bench_report.py renders a Markdown comparison table from two result
> JSON files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
