ant_nebula created SPARK-21579:
----------------------------------
Summary: dropTempView has a critical BUG
Key: SPARK-21579
URL: https://issues.apache.org/jira/browse/SPARK-21579
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.0, 2.1.1
Reporter: ant_nebula
Priority: Critical
when I dropTempView dwd_table1 only, sub table dwd_table2 also disappear from
http://127.0.0.1:4040/storage/.
It affect version 2.1.1 and 2.2.0, 2.1.0 is ok for this problem.
{code:java}
val spark =
SparkSession.builder.master("local").appName("sparkTest").getOrCreate()
val rows = Seq(Row("p1", 30), Row("p2", 20), Row("p3", 25), Row("p4", 10),
Row("p5", 40), Row("p6", 15))
val schema = new StructType().add(StructField("name",
StringType)).add(StructField("age", IntegerType))
val rowRDD = spark.sparkContext.parallelize(rows, 3)
val df = spark.createDataFrame(rowRDD, schema)
df.createOrReplaceTempView("ods_table")
spark.sql("cache table ods_table")
spark.sql("cache table dwd_table1 as select * from ods_table where age>=25")
spark.sql("cache table dwd_table2 as select * from dwd_table1 where name='p1'")
spark.catalog.dropTempView("dwd_table1")
//spark.catalog.dropTempView("ods_table")
spark.sql("select * from dwd_table2").show()
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
