[ https://issues.apache.org/jira/browse/SPARK-47190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821286#comment-17821286 ]
Nicholas Chammas commented on SPARK-47190: ------------------------------------------ [~gurwls223] - Is there some design reason we do _not_ want to support checkpointing in Spark Connect? Or is it just a matter of someone taking the time to implement support? If the latter, do we do so via a new method directly on {{SparkSession}}, or shall we somehow expose a limited version of {{spark.sparkContext}} so users can call the existing {{setCheckpointDir()}} method? > Add support for checkpointing to Spark Connect > ---------------------------------------------- > > Key: SPARK-47190 > URL: https://issues.apache.org/jira/browse/SPARK-47190 > Project: Spark > Issue Type: Improvement > Components: Connect > Affects Versions: 4.0.0 > Reporter: Nicholas Chammas > Priority: Minor > > The {{sparkContext}} that underlies a given {{SparkSession}} is not > accessible over Spark Connect. This means you cannot call > {{spark.sparkContext.setCheckpointDir(...)}}, which in turn means you cannot > checkpoint a DataFrame. > We should add support for this somehow to Spark Connect. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org