zhengruifeng commented on code in PR #52391:
URL: https://github.com/apache/spark/pull/52391#discussion_r2378126932
##########
python/pyspark/sql/pandas/serializers.py:
##########
@@ -1630,8 +1633,23 @@ def row_stream():
yield (batch_key, row)
for batch_key, group_rows in groupby(row_stream(), key=lambda x:
x[0]):
- df = pd.DataFrame([row for _, row in group_rows])
- yield (batch_key, df)
+ rows = []
Review Comment:
> Did we do this in other UDF?
I don't think so
> Is it beneficial or probably over-thinking?
I remember @HyukjinKwon discussed it before, it should be beneficial if the
size is properly estimated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
