Stanislav Hadjiiski created SPARK-11777:
-------------------------------------------
Summary: HiveContext.saveAsTable does not update the metastore on
overwrite
Key: SPARK-11777
URL: https://issues.apache.org/jira/browse/SPARK-11777
Project: Spark
Issue Type: Bug
Affects Versions: 1.5.1
Reporter: Stanislav Hadjiiski
Consider the following code:
{quote}
case class Bean(cdata: String)
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sparkContext)
val df = hiveContext.createDataFrame(Bean("test10") :: Bean("test20") :: Nil)
df.write.mode(SaveMode.Overwrite).saveAsTable("db_name.table")
{quote}
This works as expected - if the table does not exist it is created, otherwise
it's content is replaced. However, only in the first case the data is
accessible through impala (i.e. outside of spark environment). To get it
working after overwriting a
{quote}
REFRESH db_name.table
{quote}
should be issued in impala-shell. Neither
{quote}
hiveContext.refreshTable("db_name.table")
{quote}
nor
{quote}
hiveContext.sql("REFRESH TABLE db_name.table")
{quote}
fixes the issue. The same applies if the {{default}} database is used (and
{{db_name.}} is omiited everywhere)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
