Hyukjin Kwon created SPARK-61984:
------------------------------------
Summary: Compute the timeout of awaitTermination and
awaitAnyTermination as PySpark's int(timeout * 1000), so that a NaN and an int
beyond a float raise its errors
Key: SPARK-61984
URL: https://issues.apache.org/jira/browse/SPARK-61984
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark's StreamingQuery.awaitTermination and
StreamingQueryManager.awaitAnyTermination
(connect/streaming/query.py:79-89 and :214-224) refuse a timeout only when it
is not an int or a
float, or when `timeout <= 0`, with VALUE_NOT_POSITIVE. Then they set
`int(timeout * 1000)` in the
int64 field `timeout_ms` of the command. The drop-in's binding
(positive_timeout,
crates/pyspark-rs/src/streaming.rs) took the timeout as a float and refused it
unless
`seconds > 0.0`. So it differs in three cases:
- A NaN is not `<= 0`, so PySpark raises the ValueError of `int`. The drop-in
raised
VALUE_NOT_POSITIVE.
- An int beyond a float, such as 10**400: PySpark raises protobuf's ValueError
of the field,
with the digits of the int. The drop-in raised the OverflowError of the float
conversion.
- An int beyond 2**53 milliseconds: PySpark sends it exactly. The drop-in sent
it through a
float, which rounded it.
The Rust API's own await_termination and await_any_termination raise PySpark's
errors since
SPARK-61961. The binding's check came first.
Repro (a running streaming query `q`)
q.awaitTermination(float("nan"))
spark.streams.awaitAnyTermination(float("nan"))
q.awaitTermination(10**400)
q.awaitTermination(2**53 + 1) # the timeout_ms of the request
Actual (drop-in):
- `PySparkValueError: [VALUE_NOT_POSITIVE] Value for `timeout` must be
positive, got 'float'.`
for both NaN calls;
- `OverflowError: int too large to convert to float`;
- timeout_ms 9007199254740992000.
Expected (pyspark-client 4.3.0):
- `ValueError: cannot convert float NaN to integer` for both NaN calls;
- `ValueError: Value out of range: 1000...0` (the 404 digits of 10**403);
- timeout_ms 9007199254740993000.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]