[ 
https://issues.apache.org/jira/browse/SPARK-62063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62063.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 876
[https://github.com/apache/spark-connect-rust/pull/876]

> Report the option keys and values that are objects, and the arguments that 
> the writers keep, to Python's garbage collector in the readers, writers and 
> session builder, also of their shallow copies
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62063
>                 URL: https://issues.apache.org/jira/browse/SPARK-62063
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark's readers, writers and Connect session builder are plain Python 
> objects. Their option
> dicts keep a key that is not a str as it is (DataFrameReader.option, 
> DataStreamReader.option,
> DataFrameWriter.option, DataFrameWriterV2.option and tableProperty, 
> Builder.config;
> python/pyspark/sql/connect/readwriter.py, streaming/readwriter.py, 
> session.py), and so do a table
> property's value. Their writers keep a format, a save mode, partitioning and 
> clustering columns
> and a provider that are not strs as they are until the write. Python's 
> garbage collector sees all
> of these through the objects' __dict__, also when a shallow copy (copy.copy) 
> shares the dict.
> The drop-in keeps them in native state that the shallow copies share 
> (crate::options::
> UnencodableOptions, the writers' UnsentArgs and kept columns, the builder's 
> BuilderOptions), which
> the collector did not see:
> - a cycle through an option key, such as a key that keeps its reader, was 
> never collected;
> - the V1 and V2 writers reported a few kept arguments only when no copy 
> shared them, so a cycle
>   through a writer and a copy of it was never collected either.
> Repro (master 80046f2f, against a Spark 4.3.0 server; these holders are the 
> same at 4332de49)
>   import copy, gc, weakref
>   class Key:
>       def __hash__(self): return 1
>       def __eq__(self, other): return self is other
>   def collected(make):
>       key = Key(); key.held = make(key); freed = weakref.ref(key); del key; 
> gc.collect()
>       return freed() is None
>   collected(lambda k: spark.read.option(k, "v"))
>   collected(lambda k: spark.readStream.option(k, "v"))
>   collected(lambda k: df.write.option(k, "v"))
>   collected(lambda k: df.writeTo("t").tableProperty("p", k))
>   from pyspark.sql.connect.session import SparkSession as ConnectSession
>   collected(lambda k: ConnectSession.builder.config(k, "v"))
>   collected(lambda k: (lambda w: (w, copy.copy(w)))(df.write.format(k)))
>   collected(lambda k: (lambda w: (w, copy.copy(w)))(df.writeTo("t").using(k)))
> Actual (drop-in)
>   False for each.
> Expected (pyspark-client 4.3.0)
>   True for each. df.writeStream.option(key, "v") raises the TypeError of 
> protobuf's map in both
>   clients, and bucketBy and sortBy refuse such a column in both.
> Fix
> - A native object holds each state that the copies share 
> (crate::options::SharedObjects,
>   PySharedObjects). Each copy holds the object and reports it in its 
> __traverse__, and the object
>   reports the Python objects of the state, each once, and clears them in its 
> __clear__.
> - So each object is freed with the last copy that holds the state.
> - The V1 and V2 writers report all of their kept arguments through it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to