Yicong Huang created SPARK-55170:
------------------------------------
Summary: Extract grouped stream reading pattern from serializers
Key: SPARK-55170
URL: https://issues.apache.org/jira/browse/SPARK-55170
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.2.0
Reporter: Yicong Huang
Multiple serializers share the same pattern for reading grouped data from
stream:
{code:python}
# Pattern found in: GroupArrowUDFSerializer, ArrowStreamAggArrowUDFSerializer,
# ArrowStreamAggPandasUDFSerializer, CogroupArrowUDFSerializer,
CogroupPandasUDFSerializer, etc.
dataframes_in_group = None
while dataframes_in_group is None or dataframes_in_group > 0:
dataframes_in_group = read_int(stream)
if dataframes_in_group == EXPECTED_COUNT:
# process group
yield result
elif dataframes_in_group != 0:
raise PySparkValueError(
errorClass="INVALID_NUMBER_OF_DATAFRAMES_IN_GROUP",
messageParameters={"dataframes_in_group": str(dataframes_in_group)},
)
{code}
This could be extracted into a reusable component (e.g.,
{{GroupedStreamReader}}) to reduce code duplication across serializers.
This is part of Phase 2: Reduce serializer complexity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
