Hyukjin Kwon created SPARK-61984:
------------------------------------

             Summary: Compute the timeout of awaitTermination and 
awaitAnyTermination as PySpark's int(timeout * 1000), so that a NaN and an int 
beyond a float raise its errors
                 Key: SPARK-61984
                 URL: https://issues.apache.org/jira/browse/SPARK-61984
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


Problem
PySpark's StreamingQuery.awaitTermination and 
StreamingQueryManager.awaitAnyTermination
(connect/streaming/query.py:79-89 and :214-224) refuse a timeout only when it 
is not an int or a
float, or when `timeout <= 0`, with VALUE_NOT_POSITIVE. Then they set 
`int(timeout * 1000)` in the
int64 field `timeout_ms` of the command. The drop-in's binding 
(positive_timeout,
crates/pyspark-rs/src/streaming.rs) took the timeout as a float and refused it 
unless
`seconds > 0.0`. So it differs in three cases:
- A NaN is not `<= 0`, so PySpark raises the ValueError of `int`. The drop-in 
raised
  VALUE_NOT_POSITIVE.
- An int beyond a float, such as 10**400: PySpark raises protobuf's ValueError 
of the field,
  with the digits of the int. The drop-in raised the OverflowError of the float 
conversion.
- An int beyond 2**53 milliseconds: PySpark sends it exactly. The drop-in sent 
it through a
  float, which rounded it.

The Rust API's own await_termination and await_any_termination raise PySpark's 
errors since
SPARK-61961. The binding's check came first.

Repro (a running streaming query `q`)
    q.awaitTermination(float("nan"))
    spark.streams.awaitAnyTermination(float("nan"))
    q.awaitTermination(10**400)
    q.awaitTermination(2**53 + 1)   # the timeout_ms of the request
Actual (drop-in):
- `PySparkValueError: [VALUE_NOT_POSITIVE] Value for `timeout` must be 
positive, got 'float'.`
  for both NaN calls;
- `OverflowError: int too large to convert to float`;
- timeout_ms 9007199254740992000.
Expected (pyspark-client 4.3.0):
- `ValueError: cannot convert float NaN to integer` for both NaN calls;
- `ValueError: Value out of range: 1000...0` (the 404 digits of 10**403);
- timeout_ms 9007199254740993000.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to