Yicong-Huang commented on code in PR #54172:
URL: https://github.com/apache/spark/pull/54172#discussion_r2800743011
##########
python/pyspark/sql/pandas/serializers.py:
##########
@@ -129,33 +151,25 @@ def dump_stream(self, iterator, stream):
writer.close()
def load_stream(self, stream):
- import pyarrow as pa
+ """Load batches: plain stream if num_dfs=0, grouped otherwise."""
+ if self._num_dfs == 0:
+ import pyarrow as pa
- reader = pa.ipc.open_stream(stream)
- for batch in reader:
- yield batch
+ reader = pa.ipc.open_stream(stream)
+ for batch in reader:
+ yield batch
+ elif self._num_dfs == 2:
Review Comment:
make sense. changed to be explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
