L. C. Hsieh created SPARK-61443:
-----------------------------------
Summary: Fix the races of the progress handlers and streaming
listeners, and the swallowed errors of the error conversion, of the Python
client of Spark Connect Rust
Key: SPARK-61443
URL: https://issues.apache.org/jira/browse/SPARK-61443
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
The PyO3 extension of the Python client of spark-connect-rust
(crates/pyspark-rs) declares gil_used = false, so two threads can be in the
same object at once, and several of its paths also run Python code (__eq__,
__del__) between two steps that were meant to be one.
* Progress handlers: the number of handlers that the client keeps beside the
list is stored from a length read before other changes of the list. A handler
registered while another change is in flight (another thread, or the __del__ of
a handler that clearProgressHandlers drops) can be left uncounted, and is then
never called.
* toLocalIterator: a second thread that calls next or close while a thread
waits for a row gets "RuntimeError: Already borrowed". PySpark's generator
raises "ValueError: generator already executing".
* Streaming listeners: addListener stores the ids of its registrations in a
_rust_listener_ids attribute of the user's listener. That changes a user
object, fails for an object that cannot take the attribute (__slots__), and
loses registrations when several threads add and remove the same listener.
PySpark's bus keeps its listeners in its own list and removes the first that is
equal to the one given.
* Error conversion: a failure to make the exception of the connect classes, or
of the class of the kind of the error, is dropped silently, as is a failure to
set the __context__ of the done error of a progress handler or the
_operation_id of an exception.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]