[
https://issues.apache.org/jira/browse/SPARK-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010919#comment-15010919
]
Sean Owen commented on SPARK-11777:
-----------------------------------
If there is no update to the metastore, what update can a REFRESH read? I
understand you're saying that files are added. Maybe your description doesn't
mean what I'm reading, since you say that REFRESH causes Impala to see the
metadata update (not file data) you expect it to see.
What do you propose as a resolution?
> HiveContext.saveAsTable does not update the metastore on overwrite
> ------------------------------------------------------------------
>
> Key: SPARK-11777
> URL: https://issues.apache.org/jira/browse/SPARK-11777
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.1
> Reporter: Stanislav Hadjiiski
>
> Consider the following code:
> {quote}
> case class Bean(cdata: String)
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sparkContext)
> val df = hiveContext.createDataFrame(Bean("test10") :: Bean("test20") :: Nil)
> df.write.mode(SaveMode.Overwrite).saveAsTable("db_name.table")
> {quote}
> This works as expected - if the table does not exist it is created, otherwise
> it's content is replaced. However, only in the first case the data is
> accessible through impala (i.e. outside of spark environment). To get it
> working after overwriting a
> {quote}
> REFRESH db_name.table
> {quote}
> should be issued in impala-shell. Neither
> {quote}
> hiveContext.refreshTable("db_name.table")
> {quote}
> nor
> {quote}
> hiveContext.sql("REFRESH TABLE db_name.table")
> {quote}
> fixes the issue. The same applies if the {{default}} database is used (and
> {{db_name.}} is omiited everywhere)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
