[ https://issues.apache.org/jira/browse/SPARK-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Hao updated SPARK-9689: ----------------------------- Description: {code:title=example|borderStyle=solid} // create a HadoopFsRelation based table sql(s""" |CREATE TEMPORARY TABLE jsonTable (a int, b string) |USING org.apache.spark.sql.json.DefaultSource |OPTIONS ( | path '${path.toString}' |)""".stripMargin) // give the value from table jt sql( s""" |INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt """.stripMargin) // cache the HadoopFsRelation Table sqlContext.cacheTable("jsonTable") // update the HadoopFsRelation Table sql( s""" |INSERT OVERWRITE TABLE jsonTable SELECT a * 2, b FROM jt """.stripMargin) // Even this will fail sql("SELECT a, b FROM jsonTable").collect() // This will fail, as the cache doesn't refresh checkAnswer( sql("SELECT a, b FROM jsonTable"), sql("SELECT a * 2, b FROM jt").collect()) {code} was: {code:title=example|borderStyle=solid} // create a HadoopFsRelation based table sql(s""" |CREATE TEMPORARY TABLE jsonTable (a int, b string) |USING org.apache.spark.sql.json.DefaultSource |OPTIONS ( | path '${path.toString}' |)""".stripMargin) // give the value from table jt sql( s""" |INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt """.stripMargin) // cache the HadoopFsRelation Table sqlContext.cacheTable("jsonTable") // update the HadoopFsRelation Table sql( s""" |INSERT OVERWRITE TABLE jsonTable SELECT a * 2, b FROM jt """.stripMargin) // This will fail, as the cache doesn't refresh checkAnswer( sql("SELECT a, b FROM jsonTable"), sql("SELECT a * 2, b FROM jt").collect()) {code} > Cache doesn't refresh for HadoopFsRelation based table > ------------------------------------------------------ > > Key: SPARK-9689 > URL: https://issues.apache.org/jira/browse/SPARK-9689 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: Cheng Hao > > {code:title=example|borderStyle=solid} > // create a HadoopFsRelation based table > sql(s""" > |CREATE TEMPORARY TABLE jsonTable (a int, b string) > |USING org.apache.spark.sql.json.DefaultSource > |OPTIONS ( > | path '${path.toString}' > |)""".stripMargin) > > // give the value from table jt > sql( > s""" > |INSERT OVERWRITE TABLE jsonTable SELECT a, b FROM jt > """.stripMargin) > // cache the HadoopFsRelation Table > sqlContext.cacheTable("jsonTable") > > // update the HadoopFsRelation Table > sql( > s""" > |INSERT OVERWRITE TABLE jsonTable SELECT a * 2, b FROM jt > """.stripMargin) > // Even this will fail > sql("SELECT a, b FROM jsonTable").collect() > // This will fail, as the cache doesn't refresh > checkAnswer( > sql("SELECT a, b FROM jsonTable"), > sql("SELECT a * 2, b FROM jt").collect()) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org