[
https://issues.apache.org/jira/browse/SPARK-61928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61928:
------------------------------------
Assignee: Hyukjin Kwon
> Do not read the state of the readers and writers that Python's garbage
> collector traverses while they are created, which crashed the interpreter
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61928
> URL: https://issues.apache.org/jira/browse/SPARK-61928
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> On CPython 3.11, Python's garbage collector can traverse an object of the
> drop-in's extension while the object is being created. PyO3 makes an object
> of a #[pyclass(dict)] class, or of a Python subclass of it, with
> object.__new__. That call tracks the object that it allocates, and then
> allocates the object's __dict__. When the free list of dicts is empty, that
> allocation can start a collection, before PyO3 writes the object's value. The
> collector then calls the class's __traverse__ on zeroed memory.
> The __traverse__ of four classes read their state there:
> - DataFrameWriter and DataFrameWriterV2 read a null Arc (Arc::strong_count of
> the kept arguments);
> - DataFrameReader and DataStreamReader read their zeroed Option of the unsent
> format as Some of a null Py, and so gave the collector NULL.
> So creating one of them, for example df.write, spark.read, df.writeTo("t"),
> spark.readStream or a copy of one, crashed the interpreter with SIGSEGV when
> a collection started at that point. It is rare in normal use, as it needs a
> collection that starts at that allocation, but a script whose threshold or
> allocation pattern makes one start there crashes every time. For example,
> gc.set_threshold(1) with the free list of dicts used up crashes `df.write` on
> master. pyspark-client 4.3.0 has no such crash, as its classes are Python
> classes.
> PyColumn and PyDataFrame already guarded their __traverse__ with a flag that
> zeroed memory leaves false. The writers and readers should check such a flag
> too, before they read anything else of the object.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]