This is an automated email from the ASF dual-hosted git repository.
gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6ca2d9a72436 [SPARK-60157][PYTHON][TEST] Test ListState existence and
clear behavior
6ca2d9a72436 is described below
commit 6ca2d9a724364f7fcf49c1a4f4641f05a056b55c
Author: Billy Shen <[email protected]>
AuthorDate: Mon Nov 2 16:07:21 2026 -0800
[SPARK-60157][PYTHON][TEST] Test ListState existence and clear behavior
### What changes were proposed in this pull request?
This PR adds dedicated tests for `ListState.exists()` and `clear()` in both
Pandas and Row processors.
### Why are the changes needed?
The tests verify that:
- exists() returns false before writing and true after writing.
- clear() makes exists() return false and leaves no values.
- The state can be populated and cleared again in the next batch.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Executed `python/run-tests-with-coverage` in local
File: `pyspark/sql/streaming/list_state_client.py`
| | Stmts | Miss | Branch | BrPart | Cover |
|--------|:-----:|------|--------|--------|-------|
| Before | 114 | 30 | 24 | 6 | 70% |
| After | 114 | 6 | 24 | 6 | 91% |
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: gpt-6-astra
Using AI tool for context gathering. Code changes/verifications/PR are
driven by human.
Closes #59368 from
billygoatish/SPARK-60157-test-list-state-existence-and-clear.
Authored-by: Billy Shen <[email protected]>
Signed-off-by: Tian Gao <[email protected]>
(cherry picked from commit a82c0f97d4d672c14e8973ce1add696c16822355)
Signed-off-by: Tian Gao <[email protected]>
---
.../pandas/helper/helper_pandas_transform_with_state.py | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git
a/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
b/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
index aa8918ba7fa5..f8962f1d4df3 100644
---
a/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
+++
b/python/pyspark/sql/tests/pandas/helper/helper_pandas_transform_with_state.py
@@ -897,6 +897,12 @@ class PandasListStateProcessor(StatefulProcessor):
def handleInputRows(self, key, rows, timerValues) ->
Iterator[pd.DataFrame]:
import datetime
+ self.list_state1.put([(1,)])
+ assert self.list_state1.exists()
+ self.list_state1.clear()
+ assert not self.list_state1.exists()
+ assert list(self.list_state1.get()) == []
+
count = 0
time_list = []
for pdf in rows:
@@ -963,6 +969,12 @@ class RowListStateProcessor(StatefulProcessor):
def handleInputRows(self, key, rows, timerValues) -> Iterator[Row]:
import datetime
+ self.list_state1.put([(1,)])
+ assert self.list_state1.exists()
+ self.list_state1.clear()
+ assert not self.list_state1.exists()
+ assert list(self.list_state1.get()) == []
+
count = 0
time_list = []
for row in rows:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]