[ https://issues.apache.org/jira/browse/SPARK-21484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16844487#comment-16844487 ]
Liang-Chi Hsieh commented on SPARK-21484: ----------------------------------------- I think this issue isn't going to be fixed, at least, in foreseen future. I will close it now. > Wrong query plans of Dataset after persist/unpersist > ---------------------------------------------------- > > Key: SPARK-21484 > URL: https://issues.apache.org/jira/browse/SPARK-21484 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Reporter: Liang-Chi Hsieh > Priority: Major > Labels: bulk-closed > > After the cal of persist/unpersis, the query plans of a Dataset should be > changed accordingly. But currently the query plans are the same. So you will > see the inconsistent query plans like: > {code} > scala> val x1 = Seq(1).toDF() > x1: org.apache.spark.sql.DataFrame = [value: int] > scala> println(x1.queryExecution.executedPlan) // query plans are > materialized before persist() > LocalTableScan [value#1] > scala> x1.persist() > scala> x1.count() > scala> println(x1.queryExecution.executedPlan) > LocalTableScan [value#1] > {code} > {code} > scala> val x1 = Seq(1).toDF() > x1: org.apache.spark.sql.DataFrame = [value: int] > scala> x1.persist() > scala> x1.count() > scala> println(x1.queryExecution.executedPlan) // query plans are > materialized after persist() > InMemoryTableScan [value#24] > +- InMemoryRelation [value#24], true, 10000, StorageLevel(disk, memory, > deserialized, 1 replicas) > +- LocalTableScan [value#1] > scala> x1.unpersist() > scala> println(x1.queryExecution.executedPlan) > InMemoryTableScan [value#24] > +- InMemoryRelation [value#24], true, 10000, StorageLevel(disk, memory, > deserialized, 1 replicas) > +- LocalTableScan [value#1] > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org