AlejandroMorgante opened a new pull request, #60193:
URL: https://github.com/apache/spark/pull/60193
### What changes were proposed in this pull request?
`VALUE_NOT_POSITIVE` errors raised by
`GroupState.setTimeoutDuration`/`setTimeoutTimestamp`
and by
`StreamingQuery`/`StreamingQueryManager.awaitTermination`/`awaitAnyTermination`
(classic and Spark Connect) set the `arg_value` message parameter to
`type(value).__name__` instead of the value itself, so the error always read
`"...got 'int'."` regardless of what was actually passed (e.g. `-5`). This
PR passes the
actual value (`str(value)`) instead, matching every other
`VALUE_NOT_POSITIVE` call site in
the codebase (e.g. `DataFrame.repartition`).
Fixed call sites:
- `python/pyspark/sql/streaming/state.py`: `GroupState.setTimeoutDuration`,
`GroupState.setTimeoutTimestamp`
- `python/pyspark/sql/streaming/query.py`: `StreamingQuery.awaitTermination`,
`StreamingQueryManager.awaitAnyTermination`
- `python/pyspark/sql/connect/streaming/query.py`: same two methods, Spark
Connect client
### Why are the changes needed?
The error message is supposed to tell the user what value they passed so
they can see what
went wrong; reporting `'int'` for every input (whether `-5`, `0`, or any
other int) gives no
diagnostic information. This was introduced by a one-line oversight in
SPARK-48934, which
renamed the message parameter key from `arg_type` to `arg_value` but left
the value
expression as `type(x).__name__` (correct for the old `arg_type` key, wrong
for the new
`arg_value` key), and the same `type(x).__name__` pattern was then copied
into the
`StreamingQuery`/`StreamingQueryManager` timeout validation.
### Does this PR introduce _any_ user-facing change?
Yes, an error message correction. Before (e.g.
`state.setTimeoutDuration(-5)`):
```
[VALUE_NOT_POSITIVE] Value for `durationMs` must be positive, got 'int'.
```
After:
```
[VALUE_NOT_POSITIVE] Value for `durationMs` must be positive, got '-5'.
```
No other behavior change: same error condition, same trigger, only the
reported value is
corrected.
### How was this patch tested?
Added `pyspark.sql.tests.streaming.test_group_state.GroupStateTests` (2
tests) and
`pyspark.sql.tests.streaming.test_streaming_query_validation
.StreamingQueryTimeoutValidationTests` (4 tests, covering classic and
Connect), which
construct each class directly with `-5` and assert the corrected `arg_value`
via
`checkError`. None of these require a `SparkSession`, since every fixed call
site raises
before touching its JVM/Connect handle. Verified locally with `python -m
unittest` and
`ruff check`; the full `python/run-tests` / CI suite will run on this PR.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Code (Claude Sonnet 5)
---
This contribution is my original work and I license the work to the project
under the
project's open source license.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
