caldempsey opened a new pull request, #151:
URL: https://github.com/apache/spark-connect-go/pull/151
### What changes were proposed in this pull request?
* **Incremental result streaming for the Spark Connect Go client**
* Introduces a new `types.RowIterator` interface plus a concrete
`arrowRowIterator` that streams rows out of an Arrow table one-by-one.
* Adds `ReadArrowTableToIterator(arrow.Table)` in `types/arrow.go`, the
lazy counterpart to `ReadArrowTableToRows`.
* Extends the `DataFrame` API with **`ToLocalRowIterator(ctx)`** (renamed
from the initially-suggested `ToLocalIterator` to make it crystal-clear that
*rows* are iterated from the position of selection in your IDE or favorite text
editor, I still think the naming convention doesn't make clear this is
incrementally, as they arrive though).
* **Tests**
* New unit-tests in `types/arrow_test.go` cover:
* partial consumption (fetch a single row, then the remainder), and
* equivalence between the lazy iterator and eager conversion.
There are **no modifications to existing public methods or proto contracts**
– only additive, backwards-compatible extensions. I would suggest we merge this
behavior with the current Collect function though.
### Why are the changes needed?
The Go client could previously retrieve results only through `Collect()`,
which loads every row into memory before returning. This is impractical for
large OLAP workloads that routinely return GBs of data and prevents lightweight
services (API gateways, lambda-style functions, etc.) from using Spark Connect
efficiently.
Scala/Java clients solve this with `toLocalIterator()`. This PR brings the
same capability to Go:
* **Constant-memory iteration** – rows are deserialised batch-by-batch and
released immediately after consumption.
* **Back-pressure friendly** – external consumers can pull at their own pace
(e.g. stream over gRPC/HTTP).
* **API parity** with the other Spark Connect language bindings.
* **Clearer naming** – the new method is called `ToLocalRowIterator` so
users instantly understand they are iterating **row by row**.
### Does this PR introduce *any* user-facing change?
Yes, a **new opt-in API**:
```go
it, err := df.ToLocalRowIterator(ctx)
for {
row, err := it.Next()
if err == io.EOF { break }
// process row ...
}
```
Again.. Existing code that uses `Collect()` or `ToArrow()` continues to work
unchanged.
### How was this patch tested?
* **Unit tests** in `types/arrow_test.go` verify:
* Streaming access returns rows in the correct order and signals `io.EOF`
properly.
* The set of rows produced by the iterator is exactly equal to the rows
produced by the existing eager converter.
* The full Go test suite (`go test ./...`) passes locally on Linux with Go
1.21.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
