Yicong Huang created SPARK-55169:
------------------------------------
Summary: Refactor ArrowStreamArrowUDTFSerializer to use
ArrowBatchTransformer.flatten_struct
Key: SPARK-55169
URL: https://issues.apache.org/jira/browse/SPARK-55169
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.2.0
Reporter: Yicong Huang
{{ArrowStreamArrowUDTFSerializer.load_stream}} has inline flatten logic that
duplicates {{ArrowBatchTransformer.flatten_struct}}:
{code:python}
# Current code (line 203-208)
struct = batch.column(i)
flattened_batch = pa.RecordBatch.from_arrays(
struct.flatten(), schema=pa.schema(struct.type)
)
{code}
This should reuse {{ArrowBatchTransformer.flatten_struct}} to eliminate
duplicated transformation logic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
