L. C. Hsieh created SPARK-58024:
-----------------------------------
Summary: Convert Arrow struct and map columns to Python rows in
bulk
Key: SPARK-58024
URL: https://issues.apache.org/jira/browse/SPARK-58024
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Follow-up of SPARK-58019. Extend {{ArrowTableToRowsConversion._to_pylist}} with
bulk paths for struct and map columns:
* Struct: convert each child field in bulk (recursively reusing the list/leaf
fast paths), zip field values into one dict per row, masked by the validity
bitmap. Duplicate field names fall back to {{to_pylist}} so they raise the same
{{ValueError}} as {{StructScalar.as_py}}.
* Map: same offsets layout as lists; flattened keys/items children are
converted in bulk and each row becomes a list of (key, value) tuples, matching
{{MapScalar.as_py}} exactly.
ASV microbenchmark (1M rows, 10% nulls): {{struct<int64,string>}} 914ms ->
175ms (5.2x); {{map<string,int64>}} 2.07s -> 303ms (6.8x).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
