vinodkc opened a new pull request, #59855: URL: https://github.com/apache/spark/pull/59855
### What changes were proposed in this pull request? In `_SimpleStreamReaderWrapper.add_result_to_cache()`, check whether the batch returned by `SimpleDataSourceStreamReader.read()` is empty with `next(iter(it))` instead of `next(it)`, so that the offset validation added by SPARK-55416 also works when `read()` returns a list (or any other iterable) rather than an iterator. ### Why are the changes needed? SPARK-55416 (#54237, released in 4.2.0) added a check in `add_result_to_cache()` that calls `next(it)` when `read()` returns `end == start`. `next()` only accepts iterators, so a reader that returns a list now fails with: ``` TypeError: 'list' object is not an iterator ``` Before 4.2.0 a list return worked in every case: the wrapper only cached the result and later consumed it via `copy.copy()` + `itertools.chain`, both of which accept any iterable. So this is a regression for existing readers, in particular: - `read()` returning `([], start)` to signal "no new data" -- explicitly allowed by SPARK-55416 for an empty iterator -- now fails the streaming query with a `TypeError`. - A buggy reader returning a non-empty list with `end == start` gets a `TypeError` instead of the intended `SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE` error. The `end != start` path is unaffected and the cached object is kept as returned, matching the pre-4.2.0 behavior. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a regression in 4.2.0. A `SimpleDataSourceStreamReader.read()` that returns a list with `end == start` no longer fails with `TypeError`: an empty list is accepted as "no new data", and a non-empty list raises `SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE`, the same as for an iterator. ### How was this patch tested? Added `test_simple_stream_reader_returns_list` in `test_python_streaming_datasource.py`, covering a non-empty list with an advanced end offset (cached and readable), an empty list with `end == start` (allowed, not cached), and a non-empty list with `end == start` (raises `SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE`). Without the fix, the last two cases fail with `TypeError: 'list' object is not an iterator`. ### Was this patch authored or co-authored using generative AI tooling? Yes, Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
