[ https://issues.apache.org/jira/browse/SPARK-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin resolved SPARK-9689. -------------------------------- Resolution: Fixed Assignee: (was: Cheng Hao) Fix Version/s: 2.0.0 I think this one has been fixed in 2.0 already. > Cache doesn't refresh for HadoopFsRelation based table > ------------------------------------------------------ > > Key: SPARK-9689 > URL: https://issues.apache.org/jira/browse/SPARK-9689 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.4.1, 1.5.0 > Reporter: Cheng Hao > Fix For: 2.0.0 > > > {code:title=example|borderStyle=solid} > // create a HadoopFsRelation based table > sql(s""" > |CREATE TEMPORARY TABLE jsonTable (a int, b string) > |USING org.apache.spark.sql.json.DefaultSource > |OPTIONS ( > | path '${path.toString}' > |)""".stripMargin) > > // give the value from table jt > sql( > s""" > |INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt > """.stripMargin) > // cache the HadoopFsRelation Table > sqlContext.cacheTable("jsonTable") > > // update the HadoopFsRelation Table > sql( > s""" > |INSERT OVERWRITE TABLE jsonTable SELECT a * 2, b FROM jt > """.stripMargin) > // Even this will fail > sql("SELECT a, b FROM jsonTable").collect() > // This will fail, as the cache doesn't refresh > checkAnswer( > sql("SELECT a, b FROM jsonTable"), > sql("SELECT a * 2, b FROM jt").collect()) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org