This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 3d0693d4 [SPARK-61160][CONNECT][PYTHON] Close the local iterator of
toLocalIterator as a generator, so contextlib.closing and close() end the
execution in the Python client as PySpark does
3d0693d4 is described below
commit 3d0693d423c122d8b03485dd52c12d13efd6e003
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:26:04 2026 +0900
[SPARK-61160][CONNECT][PYTHON] Close the local iterator of toLocalIterator
as a generator, so contextlib.closing and close() end the execution in the
Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61160](https://issues.apache.org/jira/browse/SPARK-61160): Close the
local iterator of `toLocalIterator` as a generator, so `contextlib.closing` and
`close()` end the execution in the Python client as PySpark does.
`DataFrame.toLocalIterator()` returns a `LocalRowIterator` that implements
only `__iter__` and `__next__`. PySpark returns a generator. The change adds
`close()` to `LocalRowIterator`:
- It takes the underlying row stream, so the server execution is released,
as dropping the iterator already did. A query that has not started sends
nothing.
- It marks the iterator finished, so `next()` then raises `StopIteration`,
as a closed generator does.
- Closing it again does nothing.
`send()` and `throw()` are not added. PySpark's generator does not
implement `send()` with a value after its first row either (the delegated list
iterator raises `AttributeError`), so they are left out.
Official PySpark 4.3.0 gives `None` for `close()` and `StopIteration` for
the `next()` after it, the same as this change.
### Why are the changes needed?
```python
from contextlib import closing
with closing(spark.range(1000, numPartitions=10).toLocalIterator()) as rows:
next(rows)
# AttributeError: 'LocalRowIterator' object has no attribute 'close'
```
### Does this PR introduce _any_ user-facing change?
Yes. `close()` exists on the iterator of `toLocalIterator`, so
`contextlib.closing` and `close()` work and end the iteration early, as in
PySpark.
### How was this patch tested?
- Offline `test_local_iterator_close_ends_it_before_it_runs_as_pyspark`:
closing an iterator before its first row, twice and with `contextlib.closing`,
ends it, and no query is sent.
- Live `test_local_iterator_closes_as_a_generator_as_pyspark`:
`contextlib.closing` after one row ends the iteration, `next()` after the close
gives nothing, and a second close does nothing. The same results with official
PySpark 4.3.0.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and
the live tests of toLocalIterator, collect and toPandas (16 passed) against a
local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in
the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #348 from HyukjinKwon/SPARK-61160-local-iterator-close.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 8 ++++++++
python/tests/test_dropin_e2e.py | 12 ++++++++++++
python/tests/test_dropin_offline.py | 17 +++++++++++++++++
3 files changed, 37 insertions(+)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 756b2f97..88141df3 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -4181,6 +4181,14 @@ impl PyLocalRowIterator {
slf
}
+ /// Close the iterator, as a generator does: the rows that are left are
not fetched, so the
+ /// execution of the query is released, and the iteration ends. Closing it
again does nothing.
+ fn close(&mut self, py: Python<'_>) {
+ let iterator = self.iterator.take();
+ self.finished = true;
+ py.detach(move || drop(iterator));
+ }
+
/// Fetch the next row from the iterator.
///
/// Releases the GIL during the blocking fetch to allow other threads to
run.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 7d637d39..2de54daa 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2478,6 +2478,18 @@ def
test_create_dataframe_of_a_numpy_array_as_pyspark(session):
session.createDataFrame(np.array([[1, 2]]), ["x"])
assert e.value.getCondition() == "AXIS_LENGTH_MISMATCH"
+def test_local_iterator_closes_as_a_generator_as_pyspark(session):
+ # As PySpark's generator: contextlib.closing ends the iteration early, so
the rows that are
+ # left are not fetched; the iterator then ends, and closing it again does
nothing.
+ import contextlib
+
+ df = session.range(1000, numPartitions=10)
+ with contextlib.closing(df.toLocalIterator()) as rows:
+ assert next(rows).id == 0
+ assert list(rows) == []
+ rows.close()
+
+
def test_create_dataframe_takes_the_ddl_of_spark_as_pyspark(session):
# As PySpark Connect's, which parses a DDL schema on the server: nested
structs, arrays and
# maps of them, backtick-quoted names, and NOT NULL and COMMENT clauses.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 9a099a57..31174616 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9911,6 +9911,23 @@ def
test_nearest_by_join_checks_its_arguments_as_pyspark():
a.nearestByJoin(b, ranking, 1, "approx", "distance", "inner")
+def test_local_iterator_close_ends_it_before_it_runs_as_pyspark(mock_connect):
+ # As PySpark's generator: a toLocalIterator closed before its first row
ends, and it
+ # never sends the query.
+ import contextlib
+
+ spark, service = mock_connect
+ df = spark.range(3)
+ rows = df.toLocalIterator()
+ rows.close()
+ rows.close()
+ assert list(rows) == []
+ with contextlib.closing(df.toLocalIterator()) as rows:
+ pass
+ assert list(rows) == []
+ assert service.execute_requests == []
+
+
def test_dataframes_of_other_sessions_raise_session_not_same_as_pyspark():
# As PySpark's `_check_same_session`, of the DataFrames that the methods
take, before any
# request: SESSION_NOT_SAME for a DataFrame of another session, and for
what is not one.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]