This is an automated email from the ASF dual-hosted git repository.

gaogaotiantian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new a82c0f97d4d6 [SPARK-60157][PYTHON][TEST] Test ListState existence and 
clear behavior
a82c0f97d4d6 is described below

commit a82c0f97d4d672c14e8973ce1add696c16822355
Author: Billy Shen <[email protected]>
AuthorDate: Mon Nov 2 16:07:21 2026 -0800

    [SPARK-60157][PYTHON][TEST] Test ListState existence and clear behavior
    
    ### What changes were proposed in this pull request?
    
    This PR adds dedicated tests for `ListState.exists()` and `clear()` in both
    Pandas and Row processors.
    
    ### Why are the changes needed?
    
    The tests verify that:
    - exists() returns false before writing and true after writing.
    - clear() makes exists() return false and leaves no values.
    - The state can be populated and cleared again in the next batch.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    Executed `python/run-tests-with-coverage` in local
    
    File: `pyspark/sql/streaming/list_state_client.py`
    |        | Stmts | Miss | Branch | BrPart | Cover |
    |--------|:-----:|------|--------|--------|-------|
    | Before |  114  | 30   | 24     | 6      | 70%   |
    |  After |  114  | 6    | 24     | 6      | 91%   |
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: gpt-6-astra
    Using AI tool for context gathering. Code changes/verifications/PR are 
driven by human.
    
    Closes #59368 from 
billygoatish/SPARK-60157-test-list-state-existence-and-clear.
    
    Authored-by: Billy Shen <[email protected]>
    Signed-off-by: Tian Gao <[email protected]>
---
 .../pandas/helper/helper_pandas_transform_with_state.py      | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git 
a/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py 
b/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
index aa8918ba7fa5..f8962f1d4df3 100644
--- 
a/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
+++ 
b/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
@@ -897,6 +897,12 @@ class PandasListStateProcessor(StatefulProcessor):
     def handleInputRows(self, key, rows, timerValues) -> 
Iterator[pd.DataFrame]:
         import datetime
 
+        self.list_state1.put([(1,)])
+        assert self.list_state1.exists()
+        self.list_state1.clear()
+        assert not self.list_state1.exists()
+        assert list(self.list_state1.get()) == []
+
         count = 0
         time_list = []
         for pdf in rows:
@@ -963,6 +969,12 @@ class RowListStateProcessor(StatefulProcessor):
     def handleInputRows(self, key, rows, timerValues) -> Iterator[Row]:
         import datetime
 
+        self.list_state1.put([(1,)])
+        assert self.list_state1.exists()
+        self.list_state1.clear()
+        assert not self.list_state1.exists()
+        assert list(self.list_state1.get()) == []
+
         count = 0
         time_list = []
         for row in rows:


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to