[
https://issues.apache.org/jira/browse/SPARK-58050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58050:
-----------------------------------
Labels: pull-request-available (was: )
> Fuse per-row converters into bulk Arrow-to-rows conversion and bulk-assemble
> Arrow Python UDF results
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-58050
> URL: https://issues.apache.org/jira/browse/SPARK-58050
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> Follow-up of SPARK-58019/SPARK-58023/SPARK-58024. Even with bulk
> {{_to_pylist}}, the Arrow Python UDF worker still runs per-row Python
> converters on both sides: on input, {{_create_converter}} wraps every map row
> into a dict and every struct row into a Row one value at a time; on output,
> {{LocalDataToArrowConversion}} converters copy every returned list, convert
> every dict to an entry list and every Row to a dict before {{pa.array}}.
> Worker profiling shows these per-row converters dominate the remaining gap vs
> pickled UDFs on nested types (pickle currently beats arrow by 1.4-3.1x on
> array/map/struct).
> This change:
> * adds {{ArrowTableToRowsConversion._to_rows_column}}, fusing the per-row
> input converter into the bulk conversion (map rows become dicts and struct
> rows become Rows built from flattened child columns; child converters are
> applied per flattened column),
> * bulk-assembles UDF results when element/field/value converters are
> identity: arrays pass the returned lists to {{pa.array}} directly, map
> results pass dicts directly (PyArrow accepts dicts for map types), struct
> results are transposed and assembled via {{StructArray.from_arrays}};
> anything else falls back to the existing per-row path.
> Microbenchmark (400k rows): input map->dict 3.0x, input struct->Row 1.8x,
> output array 7.5x, output map 4.2x, output struct 4.0x — outputs identical.
> End-to-end UDF benchmark (6.4M rows): array_string 2.39s -> 1.77s,
> map<string,int> 3.65s -> 2.40s, struct 6.03s -> 4.87s on top of
> SPARK-58019/58023/58024; arrow-vs-pickle outputs verified identical with
> injected nulls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
