grundprinzip commented on code in PR #152:
URL: https://github.com/apache/spark-connect-go/pull/152#discussion_r2203503791
##########
spark/sql/dataframe.go:
##########
@@ -213,6 +213,7 @@ type DataFrame interface {
Take(ctx context.Context, limit int32) ([]types.Row, error)
// ToArrow returns the Arrow representation of the DataFrame.
ToArrow(ctx context.Context) (*arrow.Table, error)
+ ToLocalIterator(ctx context.Context) (types.RowIterator, error)
Review Comment:
The second question I had was about using Golangs native iterator types to
support `iter.Seq` or `iter.Seq2` essentially.
One good improvement to the current code (but can be done in a follow-up)
would be to replace the `All()` implementation using this iterator based one so
that we avoid buffering the whole table before returning the iterator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
