This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 3d0693d4 [SPARK-61160][CONNECT][PYTHON] Close the local iterator of 
toLocalIterator as a generator, so contextlib.closing and close() end the 
execution in the Python client as PySpark does
3d0693d4 is described below

commit 3d0693d423c122d8b03485dd52c12d13efd6e003
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 13:26:04 2026 +0900

    [SPARK-61160][CONNECT][PYTHON] Close the local iterator of toLocalIterator 
as a generator, so contextlib.closing and close() end the execution in the 
Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61160](https://issues.apache.org/jira/browse/SPARK-61160): Close the 
local iterator of `toLocalIterator` as a generator, so `contextlib.closing` and 
`close()` end the execution in the Python client as PySpark does.
    
    `DataFrame.toLocalIterator()` returns a `LocalRowIterator` that implements 
only `__iter__` and `__next__`. PySpark returns a generator. The change adds 
`close()` to `LocalRowIterator`:
    
    - It takes the underlying row stream, so the server execution is released, 
as dropping the iterator already did. A query that has not started sends 
nothing.
    - It marks the iterator finished, so `next()` then raises `StopIteration`, 
as a closed generator does.
    - Closing it again does nothing.
    
    `send()` and `throw()` are not added. PySpark's generator does not 
implement `send()` with a value after its first row either (the delegated list 
iterator raises `AttributeError`), so they are left out.
    
    Official PySpark 4.3.0 gives `None` for `close()` and `StopIteration` for 
the `next()` after it, the same as this change.
    
    ### Why are the changes needed?
    
    ```python
    from contextlib import closing
    with closing(spark.range(1000, numPartitions=10).toLocalIterator()) as rows:
        next(rows)
    # AttributeError: 'LocalRowIterator' object has no attribute 'close'
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `close()` exists on the iterator of `toLocalIterator`, so 
`contextlib.closing` and `close()` work and end the iteration early, as in 
PySpark.
    
    ### How was this patch tested?
    
    - Offline `test_local_iterator_close_ends_it_before_it_runs_as_pyspark`: 
closing an iterator before its first row, twice and with `contextlib.closing`, 
ends it, and no query is sent.
    - Live `test_local_iterator_closes_as_a_generator_as_pyspark`: 
`contextlib.closing` after one row ends the iteration, `next()` after the close 
gives nothing, and a second close does nothing. The same results with official 
PySpark 4.3.0.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2202 passed, 8 skipped), and 
the live tests of toLocalIterator, collect and toPandas (16 passed) against a 
local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics in 
the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #348 from HyukjinKwon/SPARK-61160-local-iterator-close.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs  |  8 ++++++++
 python/tests/test_dropin_e2e.py     | 12 ++++++++++++
 python/tests/test_dropin_offline.py | 17 +++++++++++++++++
 3 files changed, 37 insertions(+)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 756b2f97..88141df3 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -4181,6 +4181,14 @@ impl PyLocalRowIterator {
         slf
     }
 
+    /// Close the iterator, as a generator does: the rows that are left are 
not fetched, so the
+    /// execution of the query is released, and the iteration ends. Closing it 
again does nothing.
+    fn close(&mut self, py: Python<'_>) {
+        let iterator = self.iterator.take();
+        self.finished = true;
+        py.detach(move || drop(iterator));
+    }
+
     /// Fetch the next row from the iterator.
     ///
     /// Releases the GIL during the blocking fetch to allow other threads to 
run.
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 7d637d39..2de54daa 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2478,6 +2478,18 @@ def 
test_create_dataframe_of_a_numpy_array_as_pyspark(session):
         session.createDataFrame(np.array([[1, 2]]), ["x"])
     assert e.value.getCondition() == "AXIS_LENGTH_MISMATCH"
 
+def test_local_iterator_closes_as_a_generator_as_pyspark(session):
+    # As PySpark's generator: contextlib.closing ends the iteration early, so 
the rows that are
+    # left are not fetched; the iterator then ends, and closing it again does 
nothing.
+    import contextlib
+
+    df = session.range(1000, numPartitions=10)
+    with contextlib.closing(df.toLocalIterator()) as rows:
+        assert next(rows).id == 0
+    assert list(rows) == []
+    rows.close()
+
+
 def test_create_dataframe_takes_the_ddl_of_spark_as_pyspark(session):
     # As PySpark Connect's, which parses a DDL schema on the server: nested 
structs, arrays and
     # maps of them, backtick-quoted names, and NOT NULL and COMMENT clauses.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 9a099a57..31174616 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9911,6 +9911,23 @@ def 
test_nearest_by_join_checks_its_arguments_as_pyspark():
         a.nearestByJoin(b, ranking, 1, "approx", "distance", "inner")
 
 
+def test_local_iterator_close_ends_it_before_it_runs_as_pyspark(mock_connect):
+    # As PySpark's generator: a toLocalIterator closed before its first row 
ends, and it
+    # never sends the query.
+    import contextlib
+
+    spark, service = mock_connect
+    df = spark.range(3)
+    rows = df.toLocalIterator()
+    rows.close()
+    rows.close()
+    assert list(rows) == []
+    with contextlib.closing(df.toLocalIterator()) as rows:
+        pass
+    assert list(rows) == []
+    assert service.execute_requests == []
+
+
 def test_dataframes_of_other_sessions_raise_session_not_same_as_pyspark():
     # As PySpark's `_check_same_session`, of the DataFrames that the methods 
take, before any
     # request: SESSION_NOT_SAME for a DataFrame of another session, and for 
what is not one.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to