[
https://issues.apache.org/jira/browse/SPARK-45249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kondziolka9ld updated SPARK-45249:
----------------------------------
Description:
Hi!
I consider why `isCheckpointed` method return `false` - could someone explain
it? I would expect `true`.
{code:java}
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.0
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.13)
Type in expressions to have them evaluated.
Type :help for more information.
scala> sc.setCheckpointDir("file:///tmp/")
scala> val df1 = Seq(1,2,3,4).toDF
df1: org.apache.spark.sql.DataFrame = [value: int]
scala> val df2 = df1.checkpoint()
df2: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
scala> df2.rdd.isCheckpointed
res2: Boolean = false // why false?{code}
was:
Hi!
I consider why `isCheckpointed` method return `false` - could someone explain
it? I would expect `true`.
{code:java}
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.3.0
/_/
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.13)
Type in expressions to have them evaluated.
Type :help for more information.scala>
sc.setCheckpointDir("file:///tmp/")scala> val df1 = Seq(1,2,3,4).toDF
df1: org.apache.spark.sql.DataFrame = [value: int]scala> val df2 =
df1.checkpoint()
df2: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
scala> df2.rdd.isCheckpointed
res2: Boolean = false // why false?{code}
> Why isCheckpointed returns false while as checkpointing is eager by default?
> ----------------------------------------------------------------------------
>
> Key: SPARK-45249
> URL: https://issues.apache.org/jira/browse/SPARK-45249
> Project: Spark
> Issue Type: Question
> Components: Spark Core
> Affects Versions: 3.3.0
> Reporter: kondziolka9ld
> Priority: Major
>
> Hi!
> I consider why `isCheckpointed` method return `false` - could someone explain
> it? I would expect `true`.
> {code:java}
> Welcome to
> ____ __
> / __/__ ___ _____/ /__
> _\ \/ _ \/ _ `/ __/ '_/
> /___/ .__/\_,_/_/ /_/\_\ version 3.3.0
> /_/
>
> Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.13)
> Type in expressions to have them evaluated.
> Type :help for more information.
> scala> sc.setCheckpointDir("file:///tmp/")
> scala> val df1 = Seq(1,2,3,4).toDF
> df1: org.apache.spark.sql.DataFrame = [value: int]
> scala> val df2 = df1.checkpoint()
> df2: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [value: int]
> scala> df2.rdd.isCheckpointed
> res2: Boolean = false // why false?{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
