Yicong-Huang commented on PR #57911: URL: https://github.com/apache/spark/pull/57911#issuecomment-5289730760
> What python version you tested this on? Also do you have results for larger benchmarks? Like non micro ones. Either E2E or even the full worker path. I'm curious about how this really matters in real workload. I tested with Python 3.12 and PyArrow 25.0.1. I also added the following end-to-end benchmark results to the PR description: > | workload | before | after | speedup | |---|---|---|---| | `array<string>`, 500K rows x 20 short elems (10M elems) | 1763 ms | 1323 ms | **-25%** | | `array<binary>`, 500K rows x 20 short elems (10M elems) | 2066 ms | 1361 ms | **-34%** | | `array<array<string>>`, 200K rows x 4 x 4 | 1312 ms | 1133 ms | -14% | | `array<array<binary>>`, 200K rows x 4 x 4 | 1220 ms | 1014 ms | -17% | | `array<array<string>>`, 100% `int` (coercion, worst) | 1184 ms | 1154 ms | -2.5% | | `array<array<binary>>`, 100% `bytearray` (coercion, worst) | 1394 ms | 1451 ms | +4% | These results are expected because per-element string/binary conversion is a hot path for nested data types containing strings or binary values. Even in the worst-case misuse scenario, `array<array<binary>> with 100% bytearray values`, where every one is a misused type, the regression was only 4%. I think these end-to-end results show that this is not merely a microbenchmark improvement; it can significantly improve performance in the Python output path for real workloads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
