Hyukjin Kwon created SPARK-61145:
------------------------------------
Summary: Calls from several Python threads on one DataFrameWriter,
DataFrameWriterV2 or DataStreamWriter fail with "Already borrowed"
Key: SPARK-61145
URL: https://issues.apache.org/jira/browse/SPARK-61145
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The writer methods of the Python client that release the GIL while the request
runs (save, parquet, csv, saveAsTable, create, append, start, toTable, ...)
took an exclusive Rust borrow of the writer object for the whole request. A
second thread that called a method of the same writer, a write or a builder
such as format or mode, got RuntimeError: Already borrowed. PySpark's writers
are plain Python objects, so these calls run and each write goes through. The
writer state is now behind a lock that is held only to copy or replace it, so
that a request runs on its own copy.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
