[ 
https://issues.apache.org/jira/browse/SPARK-61984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61984:
------------------------------------

    Assignee: Hyukjin Kwon

> Compute the timeout of awaitTermination and awaitAnyTermination as PySpark's 
> int(timeout * 1000), so that a NaN and an int beyond a float raise its errors
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61984
>                 URL: https://issues.apache.org/jira/browse/SPARK-61984
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> Problem
> PySpark's StreamingQuery.awaitTermination and 
> StreamingQueryManager.awaitAnyTermination
> (connect/streaming/query.py:79-89 and :214-224) refuse a timeout only when it 
> is not an int or a
> float, or when `timeout <= 0`, with VALUE_NOT_POSITIVE. Then they set 
> `int(timeout * 1000)` in the
> int64 field `timeout_ms` of the command. The drop-in's binding 
> (positive_timeout,
> crates/pyspark-rs/src/streaming.rs) took the timeout as a float and refused 
> it unless
> `seconds > 0.0`. So it differs in three cases:
> - A NaN is not `<= 0`, so PySpark raises the ValueError of `int`. The drop-in 
> raised
>   VALUE_NOT_POSITIVE.
> - An int beyond a float, such as 10**400: PySpark raises protobuf's 
> ValueError of the field,
>   with the digits of the int. The drop-in raised the OverflowError of the 
> float conversion.
> - An int beyond 2**53 milliseconds: PySpark sends it exactly. The drop-in 
> sent it through a
>   float, which rounded it.
> The Rust API's own await_termination and await_any_termination raise 
> PySpark's errors since
> SPARK-61961. The binding's check came first.
> Repro (a running streaming query `q`)
>     q.awaitTermination(float("nan"))
>     spark.streams.awaitAnyTermination(float("nan"))
>     q.awaitTermination(10**400)
>     q.awaitTermination(2**53 + 1)   # the timeout_ms of the request
> Actual (drop-in):
> - `PySparkValueError: [VALUE_NOT_POSITIVE] Value for `timeout` must be 
> positive, got 'float'.`
>   for both NaN calls;
> - `OverflowError: int too large to convert to float`;
> - timeout_ms 9007199254740992000.
> Expected (pyspark-client 4.3.0):
> - `ValueError: cannot convert float NaN to integer` for both NaN calls;
> - `ValueError: Value out of range: 1000...0` (the 404 digits of 10**403);
> - timeout_ms 9007199254740993000.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to