Joachim Hereth created SPARK-20808:

             Summary: External Table unnecessarily not create in 
Hive-compatible way
                 Key: SPARK-20808
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.1.1, 2.1.0
            Reporter: Joachim Hereth
            Priority: Minor

In Spark 2.1.0 and 2.1.1 {{spark.catalog.createExternalTable}} creates tables 
unnecessarily in a hive-incompatible way.

For instance executing in a spark shell

val database = "default"
val table = "table_name"
val path = "/user/daki/"  + database + "/" + table

var data = Array(("Alice", 23), ("Laura", 33), ("Peter", 54))
val df = sc.parallelize(data).toDF("name","age") 


spark.sql("DROP TABLE IF EXISTS " + database + "." + table)

spark.catalog.createExternalTable(database + "."+ table, path)

issues the warning

Search Subject for Kerberos V5 INIT cred (<<DEF>>,
17/05/19 11:01:17 WARN hive.HiveExternalCatalog: Could not persist 
`default`.`table_name` in a Hive compatible way. Persisting it into Hive 
metastore in Spark SQL specific format.
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:User 
daki does not have privileges for CREATETABLE)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(

The Exception (user does not have privileges for CREATETABLE) is misleading (I 
do have the CREATE TABLE privilege).

Querying the table with Hive does not return any result. With Spark one can 
access the data.

The following code creates the table correctly (workaround):
def sqlStatement(df : org.apache.spark.sql.DataFrame, database : String, table: 
String, path: String) : String = {
  val rows = (for(col <- df.schema) 
                    yield "`" + + "` " + 
  val sqlStmnt = ("CREATE EXTERNAL TABLE `%s`.`%s` (%s) " +
    "Location 'hdfs://nameservice1%s'").format(database, table, rows, path)
  return sqlStmnt

spark.sql("DROP TABLE IF EXISTS " + database + "." + table)
spark.sql(sqlStatement(df, database, table, path))

The code is executed via YARN against a Cloudera CDH 5.7.5 cluster with Sentry 
enabled (in case this matters regarding the privilege warning). Spark was built 
against the CDH libraries.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to