zhengruifeng commented on code in PR #53952:
URL: https://github.com/apache/spark/pull/53952#discussion_r2730330031
##########
python/pyspark/sql/conversion.py:
##########
@@ -63,17 +63,18 @@ class ArrowBatchTransformer:
"""
@staticmethod
- def flatten_struct(batch: "pa.RecordBatch") -> "pa.RecordBatch":
+ def flatten_struct(batch: "pa.RecordBatch", column_index: int = 0) ->
"pa.RecordBatch":
"""
- Flatten a single struct column into a RecordBatch.
+ Flatten a struct column at given index into a RecordBatch.
Used by:
- ArrowStreamUDFSerializer.load_stream
- GroupArrowUDFSerializer.load_stream
+ - ArrowStreamArrowUDTFSerializer.load_stream
"""
import pyarrow as pa
- struct = batch.column(0)
+ struct = batch.column(column_index)
return pa.RecordBatch.from_arrays(struct.flatten(),
schema=pa.schema(struct.type))
Review Comment:
Aren't code still duplicated with the new
`ArrowBatchTransformer.flatten_struct`?
I think it is only worthwhile to add complex logic in such helper class, and
I am assuming we will have a group of complicated functions in the new
transformer classes.
I don't think it necessary to just replace `pa.RecordBatch.from_arrays` with
`ArrowBatchTransformer.flatten_struct`, and it is possible to refactor the code
to be more clearer without such replacement.
For simple ones, I don't think it improve the readability because I have to
go to another place to check how the function is defined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
