Hyukjin Kwon created SPARK-61928:
------------------------------------
Summary: Do not read the state of the readers and writers that
Python's garbage collector traverses while they are created, which crashed the
interpreter
Key: SPARK-61928
URL: https://issues.apache.org/jira/browse/SPARK-61928
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
On CPython 3.11, Python's garbage collector can traverse an object of the
drop-in's extension while the object is being created. PyO3 makes an object of
a #[pyclass(dict)] class, or of a Python subclass of it, with object.__new__.
That call tracks the object that it allocates, and then allocates the object's
__dict__. When the free list of dicts is empty, that allocation can start a
collection, before PyO3 writes the object's value. The collector then calls the
class's __traverse__ on zeroed memory.
The __traverse__ of four classes read their state there:
- DataFrameWriter and DataFrameWriterV2 read a null Arc (Arc::strong_count of
the kept arguments);
- DataFrameReader and DataStreamReader read their zeroed Option of the unsent
format as Some of a null Py, and so gave the collector NULL.
So creating one of them, for example df.write, spark.read, df.writeTo("t"),
spark.readStream or a copy of one, crashed the interpreter with SIGSEGV when a
collection started at that point. It is rare in normal use, as it needs a
collection that starts at that allocation, but a script whose threshold or
allocation pattern makes one start there crashes every time. For example,
gc.set_threshold(1) with the free list of dicts used up crashes `df.write` on
master. pyspark-client 4.3.0 has no such crash, as its classes are Python
classes.
PyColumn and PyDataFrame already guarded their __traverse__ with a flag that
zeroed memory leaves false. The writers and readers should check such a flag
too, before they read anything else of the object.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]