zhengruifeng commented on PR #57781: URL: https://github.com/apache/spark/pull/57781#issuecomment-5190148250
Benchmark results Environment: conda spark-dev-313, Python 3.13.12, PyArrow 24.0.0, ASV with 3 repeats. End-to-end cases use local[4], 4 input rows, and 4 partitions so broadcast work dominates. End-to-end broadcast of a native PyArrow table | Path | Broadcast rows | Pickle | Arrow | | --- | ---: | ---: | ---: | | RDD | 100k | 41.7 +/- 2 ms | 40.5 +/- 2 ms | | RDD | 1M | 121 +/- 2 ms | 117 +/- 2 ms | | Arrow UDF | 100k | 69.3 +/- 4 ms | 78.2 +/- 4 ms | | Arrow UDF | 1M | 150 +/- 8 ms | 142 +/- 5 ms | The larger payload is about 44 MB. Arrow is approximately 3-5% faster there, although the difference is modest. Isolated serde for a native PyArrow table, 1M rows | Operation | Pickle | Arrow | | --- | ---: | ---: | | Driver serialization | 26.2 +/- 2 ms | 23.2 +/- 0.6 ms | | Worker-path deserialization | 3.85 +/- 0.08 ms | 23.2 +/- 1 ms | | Serialized size | 44 MB | 44 MB | Native PyArrow values already have efficient pickle support, so this is not the main target case. Isolated serde for the custom protocol example backed by list[float] | Values | Operation | Pickle | Arrow | | ---: | --- | ---: | ---: | | 100k | Driver serialization | 1.44 +/- 0.04 ms | 1.65 +/- 0.1 ms | | 100k | Worker-path deserialization | 2.21 +/- 0.09 ms | 0.136 +/- 0.04 ms | | 100k | Serialized size | 900 KB | 800 KB | | 1M | Driver serialization | 14.1 +/- 0.2 ms | 13.8 +/- 0.3 ms | | 1M | Worker-path deserialization | 46.4 +/- 10 ms | 0.762 +/- 0.07 ms | | 1M | Serialized size | 9.00 MB | 8.00 MB | For 1M floats, combined serde falls from about 60.5 ms to 14.6 ms, approximately 4.2x faster. Retained worker payload memory falls from 32,289,080 bytes for the Python list and float objects to 8,000,000 bytes for the Arrow float64 buffer, approximately 75% less. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
