d80tb7 commented on a change in pull request #24981: [WIP][SPARK-27463][PYTHON]
Support Dataframe Cogroup via Pandas UDFs- Arrow Stream Impl
URL: https://github.com/apache/spark/pull/24981#discussion_r302736415
##########
File path: python/pyspark/serializers.py
##########
@@ -401,6 +427,22 @@ def __repr__(self):
return "ArrowStreamPandasUDFSerializer"
+class InterleavedArrowStreamPandasSerializer(ArrowStreamPandasUDFSerializer):
+
+ def __init__(self, timezone, safecheck, assign_cols_by_name):
+ super(InterleavedArrowStreamPandasSerializer, self).__init__(timezone,
safecheck, assign_cols_by_name)
+
+ def load_stream(self, stream):
+ """
+ Deserialize ArrowRecordBatches to an Arrow table and return as a list
of pandas.Series.
+ """
+ reader = InterleavedArrowReader(stream)
Review comment:
I'm not sure how this would work as the ArrowStreamSerializer wouldn't know
how we signal the number of dataframes nor that 0 dataframes means end of
stream?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
