Hyukjin Kwon created SPARK-61237:
------------------------------------
Summary: Release the relation of a checkpointed DataFrame on the
server when its last DataFrame is dropped, as PySpark does
Key: SPARK-61237
URL: https://issues.apache.org/jira/browse/SPARK-61237
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
checkpoint() and localCheckpoint() in the Rust client (and the Python drop-in,
which calls the same core code) return a DataFrame over a CachedRemoteRelation,
but nothing ever sent RemoveCachedRemoteRelationCommand, so every checkpoint
stayed on the server, with its data, until the session ended.
Repro: a DataFrame from localCheckpoint() is dropped, then its relation is
still readable on the server (count succeeds), while official PySpark 4.3.0
gives CONNECT_INVALID_PLAN.DATAFRAME_NOT_FOUND once the checkpoint and its
derived DataFrames are dropped and garbage-collected.
Fix: the checkpoint plan owns a reference that removes the relation when the
last plan of it is dropped, as CachedRemoteRelation.__del__ does in PySpark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
