[
https://issues.apache.org/jira/browse/SPARK-21579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-21579.
-------------------------------
Resolution: Not A Problem
> dropTempView has a critical BUG
> -------------------------------
>
> Key: SPARK-21579
> URL: https://issues.apache.org/jira/browse/SPARK-21579
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.1, 2.2.0
> Reporter: ant_nebula
> Priority: Critical
> Attachments: screenshot-1.png
>
>
> when I dropTempView dwd_table1 only, sub table dwd_table2 also disappear from
> http://127.0.0.1:4040/storage/.
> It affect version 2.1.1 and 2.2.0, 2.1.0 is ok for this problem.
> {code:java}
> val spark =
> SparkSession.builder.master("local").appName("sparkTest").getOrCreate()
> val rows = Seq(Row("p1", 30), Row("p2", 20), Row("p3", 25), Row("p4", 10),
> Row("p5", 40), Row("p6", 15))
> val schema = new StructType().add(StructField("name",
> StringType)).add(StructField("age", IntegerType))
> val rowRDD = spark.sparkContext.parallelize(rows, 3)
> val df = spark.createDataFrame(rowRDD, schema)
> df.createOrReplaceTempView("ods_table")
> spark.sql("cache table ods_table")
> spark.sql("cache table dwd_table1 as select * from ods_table where age>=25")
> spark.sql("cache table dwd_table2 as select * from dwd_table1 where
> name='p1'")
> spark.catalog.dropTempView("dwd_table1")
> //spark.catalog.dropTempView("ods_table")
> spark.sql("select * from dwd_table2").show()
> {code}
> It will keep ods_table1 in memory, although it will not been used anymore. It
> waste memory, especially when my service diagram much more complex
> !screenshot-1.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
