[ 
https://issues.apache.org/jira/browse/SPARK-61443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61443.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 511
[https://github.com/apache/spark-connect-rust/pull/511]

> Fix the races of the progress handlers and streaming listeners, and the 
> swallowed errors of the error conversion, of the Python client of Spark 
> Connect Rust
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61443
>                 URL: https://issues.apache.org/jira/browse/SPARK-61443
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> The PyO3 extension of the Python client of spark-connect-rust 
> (crates/pyspark-rs) declares gil_used = false, so two threads can be in the 
> same object at once, and several of its paths also run Python code (__eq__, 
> __del__) between two steps that were meant to be one.
> * Progress handlers: the number of handlers that the client keeps beside the 
> list is stored from a length read before other changes of the list. A handler 
> registered while another change is in flight (another thread, or the __del__ 
> of a handler that clearProgressHandlers drops) can be left uncounted, and is 
> then never called.
> * toLocalIterator: a second thread that calls next or close while a thread 
> waits for a row gets "RuntimeError: Already borrowed". PySpark's generator 
> raises "ValueError: generator already executing".
> * Streaming listeners: addListener stores the ids of its registrations in a 
> _rust_listener_ids attribute of the user's listener. That changes a user 
> object, fails for an object that cannot take the attribute (__slots__), and 
> loses registrations when several threads add and remove the same listener. 
> PySpark's bus keeps its listeners in its own list and removes the first that 
> is equal to the one given.
> * Error conversion: a failure to make the exception of the connect classes, 
> or of the class of the kind of the error, is dropped silently, as is a 
> failure to set the __context__ of the done error of a progress handler or the 
> _operation_id of an exception.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to