This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7cc72bd37804 [SPARK-51684][PYTHON][SS] Fix test failure in
test_pandas_transform_with_state
7cc72bd37804 is described below
commit 7cc72bd37804b12af58dfa389490a4da2d1a65ab
Author: bogao007 <[email protected]>
AuthorDate: Wed Apr 2 08:45:38 2025 +0800
[SPARK-51684][PYTHON][SS] Fix test failure in
test_pandas_transform_with_state
### What changes were proposed in this pull request?
Fixed test failure in test_pandas_transform_with_state mentioned in
https://github.com/apache/spark/pull/50349#issuecomment-2769393040
### Why are the changes needed?
To unblock Spark 4.0.0 release
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Test only change.
### Was this patch authored or co-authored using generative AI tooling?
Closes #50482 from bogao007/test-fix-list-state.
Authored-by: bogao007 <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
.../pyspark/sql/tests/pandas/test_pandas_transform_with_state.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
b/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
index bd55e20cf3cd..55948891089e 100644
--- a/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
+++ b/python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py
@@ -1046,10 +1046,10 @@ class TransformWithStateInPandasTestsMixin:
metadata_df.select("operatorProperties").collect()[0][0]
)
state_var_list = operator_properties_json_obj["stateVariables"]
- assert len(state_var_list) == 3
+ assert len(state_var_list) == 4
for state_var in state_var_list:
- if state_var["stateName"] in ["listState1", "listState2"]:
- state_var["stateVariableType"] == "ListState"
+ if state_var["stateName"] in ["listState1", "listState2",
"listStateTimestamp"]:
+ assert state_var["stateVariableType"] == "ListState"
else:
assert state_var["stateName"] ==
"$procTimers_keyToTimestamp"
assert state_var["stateVariableType"] == "TimerState"
@@ -1099,7 +1099,7 @@ class TransformWithStateInPandasTestsMixin:
self._test_transform_with_state_in_pandas_basic(
ListStateProcessor(),
check_results,
- True,
+ False,
"processingTime",
checkpoint_path=checkpoint_path,
initial_state=None,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
