Hyukjin Kwon created SPARK-61928:
------------------------------------

             Summary: Do not read the state of the readers and writers that 
Python's garbage collector traverses while they are created, which crashed the 
interpreter
                 Key: SPARK-61928
                 URL: https://issues.apache.org/jira/browse/SPARK-61928
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


On CPython 3.11, Python's garbage collector can traverse an object of the 
drop-in's extension while the object is being created. PyO3 makes an object of 
a #[pyclass(dict)] class, or of a Python subclass of it, with object.__new__. 
That call tracks the object that it allocates, and then allocates the object's 
__dict__. When the free list of dicts is empty, that allocation can start a 
collection, before PyO3 writes the object's value. The collector then calls the 
class's __traverse__ on zeroed memory.

The __traverse__ of four classes read their state there:
- DataFrameWriter and DataFrameWriterV2 read a null Arc (Arc::strong_count of 
the kept arguments);
- DataFrameReader and DataStreamReader read their zeroed Option of the unsent 
format as Some of a null Py, and so gave the collector NULL.

So creating one of them, for example df.write, spark.read, df.writeTo("t"), 
spark.readStream or a copy of one, crashed the interpreter with SIGSEGV when a 
collection started at that point. It is rare in normal use, as it needs a 
collection that starts at that allocation, but a script whose threshold or 
allocation pattern makes one start there crashes every time. For example, 
gc.set_threshold(1) with the free list of dicts used up crashes `df.write` on 
master. pyspark-client 4.3.0 has no such crash, as its classes are Python 
classes.

PyColumn and PyDataFrame already guarded their __traverse__ with a flag that 
zeroed memory leaves false. The writers and readers should check such a flag 
too, before they read anything else of the object.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to