Yicong-Huang opened a new pull request, #57911: URL: https://github.com/apache/spark/pull/57911
### What changes were proposed in this pull request? `LocalDataToArrowConversion` converts local Python rows to Arrow element by element. For `string` and `binary` leaf types the per-element converter is effectively a no-op when the element is already the target Python type, yet it still pays a Python function call (and, for binary, a copy) per element: - `convert_string` returns `str` inputs unchanged, since `str(s) is s`. - `convert_binary` calls `bytes(value)`, which copies even though `bytes` is immutable. This PR adds a type-check fast path: - `convert_string`: `type(value) is str` returns the value directly, skipping the `None`/`bool` checks. - `convert_binary`: `type(value) is bytes` returns the value directly, skipping the copy. `bytearray` still falls through and is copied into an immutable `bytes`. - `convert_array`: for `string`/`binary` element types, the fast path is inlined into the list comprehension so already-target-typed elements skip the per-element converter call entirely. Non-target elements (e.g. a `bool` coerced to string) still fall back to the reused `element_conv`. This is orthogonal to SPARK-52796, which optimized the outer `convert()` assembly rather than the element-level conversion. ### Why are the changes needed? Converting local Python data to Arrow is the hot path of Spark Connect `createDataFrame` and Arrow-optimized Python UDF / DataSource output. `string` is the most common non-trivial leaf type, and the per-element converter dominates the cost for `array<string>` and similar nested columns. A new ASV benchmark (`python/benchmarks/bench_local_data_to_arrow.py`) on `LocalDataToArrowConversion.convert` (1M rows), before vs after: | leaf | schema | before | after | speedup | |---|---|---|---|---| | string | scalar | 188+-2ms | 165+-3ms | 1.14x | | string | array | 627+-3ms | 475+-4ms | 1.32x | | string | map | 987+-3ms | 872+-7ms | 1.13x | | string | struct | 716+-3ms | 633+-7ms | 1.13x | | binary | scalar | 238+-2ms | 158+-2ms | 1.51x | | binary | array | 805+-2ms | 453+-2ms | 1.78x | | binary | map | 1.16+-0s | 880+-8ms | 1.32x | | binary | struct | 839+-3ms | 624+-3ms | 1.34x | ### Does this PR introduce _any_ user-facing change? No. The conversion results are unchanged; this is a performance-only change. Non-target-typed elements (bool/int coerced to string, `bytearray`, `None`) still go through the existing converter, preserving the current behavior. ### How was this patch tested? - Added `test_string_binary_identity_fast_path` in `python/pyspark/sql/tests/test_conversion.py`, covering the identity fast path for `str`/`bytes` and the fallback for non-target elements (bool/int coerced to string, `bytearray` materialized as immutable `bytes`) across scalar, array, map-value and struct leaves. - Existing `test_conversion.py` suite passes. - Added `bench_local_data_to_arrow.py` ASV benchmark. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
