This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 517c5e932381 [SPARK-60609][SS][PYTHON][FOLLOWUP] Fix simple stream
reader test on branch-4.1
517c5e932381 is described below
commit 517c5e93238165265f02dc074505b5167835595c
Author: Jungtaek Lim <[email protected]>
AuthorDate: Fri Oct 30 11:38:37 2026 +0900
[SPARK-60609][SS][PYTHON][FOLLOWUP] Fix simple stream reader test on
branch-4.1
### What changes were proposed in this pull request?
Use `latestOffset()` in the list-reader test on branch-4.1.
### Why are the changes needed?
The #59855 backport copied calls to a newer API, including undefined
`ReadAllAvailable`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Five Python-only datasource tests passed. Full JVM suite not run.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: OpenAI Codex 0.161.0.
Closes #59948 from
HeartSaVioR/SPARK-60609-fix-branch-4.1-stream-reader-test.
Authored-by: Jungtaek Lim <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
python/pyspark/sql/tests/test_python_streaming_datasource.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/sql/tests/test_python_streaming_datasource.py
b/python/pyspark/sql/tests/test_python_streaming_datasource.py
index 9298f95245c8..80a4aeb22630 100644
--- a/python/pyspark/sql/tests/test_python_streaming_datasource.py
+++ b/python/pyspark/sql/tests/test_python_streaming_datasource.py
@@ -333,20 +333,20 @@ class BasePythonStreamingDataSourceTestsMixin:
# Non-empty list with an advanced end offset is cached and readable.
wrapper = _SimpleStreamReaderWrapper(ListReader([(0,), (1,)],
advance=True))
- end = wrapper.latestOffset(start, ReadAllAvailable())
+ end = wrapper.latestOffset()
self.assertEqual(end, {"offset": 2})
self.assertEqual(list(wrapper.getCache(start, end)), [(0,), (1,)])
# Empty list with end == start is allowed and not cached.
wrapper = _SimpleStreamReaderWrapper(ListReader([], advance=False))
- end = wrapper.latestOffset(start, ReadAllAvailable())
+ end = wrapper.latestOffset()
self.assertEqual(end, start)
self.assertEqual(len(wrapper.cache), 0)
# Non-empty list with end == start raises the offset validation error.
wrapper = _SimpleStreamReaderWrapper(ListReader([(0,)], advance=False))
with self.assertRaises(PySparkException) as cm:
- wrapper.latestOffset(start, ReadAllAvailable())
+ wrapper.latestOffset()
self.assertEqual(
cm.exception.getCondition(),
"SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]