[
https://issues.apache.org/jira/browse/SPARK-30436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved SPARK-30436.
----------------------------------
Resolution: Duplicate
> CREATE EXTERNAL TABLE doesn't work without STORED AS
> ----------------------------------------------------
>
> Key: SPARK-30436
> URL: https://issues.apache.org/jira/browse/SPARK-30436
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> CREATE EXTERNAL TABLE doesn't work as before since Spark 3.0.0 preview 2.
> SPARK-30098 describes the goal as `spark.sql.source.default` as default
> provider if there's no explicit "USING provider", but parser rule seems to be
> modified more than the goal which breaks CREATE EXTERNAL TABLE.
> {noformat}
> scala> sql("CREATE EXTERNAL TABLE aaa (key INT, value STRING) USING HIVE
> LOCATION '/tmp/aaa'")
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: CREATE EXTERNAL TABLE ...(line 1, pos 0)
> == SQL ==
> CREATE EXTERNAL TABLE aaa (key INT, value STRING) USING HIVE LOCATION
> '/tmp/aaa'
> ^^^ at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:41)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.$anonfun$visitCreateTable$1(AstBuilder.scala:2669)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitCreateTable(AstBuilder.scala:2666)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.super$visitCreateTable(SparkSqlParser.scala:188)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.$anonfun$visitCreateTable$1(SparkSqlParser.scala:188)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateTable(SparkSqlParser.scala:184)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateTable(SparkSqlParser.scala:55)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$CreateTableContext.accept(SqlBaseParser.java:1537)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.$anonfun$visitSingleStatement$1(AstBuilder.scala:77)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:77)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.$anonfun$parsePlan$1(ParseDriver.scala:77)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:113)
> at
> org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:76)
> at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:605)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:605)
> ... 47 elided{noformat}
>
> {noformat}
> scala> sql("CREATE EXTERNAL TABLE aaa (key INT, value STRING) LOCATION
> '/tmp/aaa'")
> org.apache.spark.sql.catalyst.parser.ParseException:
> Operation not allowed: CREATE EXTERNAL TABLE ...(line 1, pos 0)== SQL ==
> CREATE EXTERNAL TABLE aaa (key INT, value STRING) LOCATION '/tmp/aaa'
> ^^^ at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.operationNotAllowed(ParserUtils.scala:41)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.$anonfun$visitCreateTable$1(AstBuilder.scala:2669)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitCreateTable(AstBuilder.scala:2666)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.super$visitCreateTable(SparkSqlParser.scala:188)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.$anonfun$visitCreateTable$1(SparkSqlParser.scala:188)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateTable(SparkSqlParser.scala:184)
> at
> org.apache.spark.sql.execution.SparkSqlAstBuilder.visitCreateTable(SparkSqlParser.scala:55)
> at
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$CreateTableContext.accept(SqlBaseParser.java:1537)
> at
> org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.$anonfun$visitSingleStatement$1(AstBuilder.scala:77)
> at
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
> at
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:77)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.$anonfun$parsePlan$1(ParseDriver.scala:77)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:113)
> at
> org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:48)
> at
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:76)
> at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:605)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:605)
> ... 47 elided
> {noformat}
>
> It requires unique option for Hive table to let parser use the rule for Hive
> table. For example, STORED AS would work.
> {noformat}
> scala> sql("CREATE EXTERNAL TABLE aaa (key INT, value STRING) STORED AS
> parquet LOCATION '/tmp/aaa'")
> 20/01/06 21:08:36 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout
> does not exist
> 20/01/06 21:08:36 WARN HiveConf: HiveConf of name hive.stats.retries.wait
> does not exist
> 20/01/06 21:08:40 WARN ObjectStore: Version information not found in
> metastore. hive.metastore.schema.verification is not enabled so recording the
> schema version 2.3.0
> 20/01/06 21:08:40 WARN ObjectStore: setMetaStoreSchemaVersion called but
> recording version is disabled: version = 2.3.0, comment = Set by MetaStore
> [email protected]
> 20/01/06 21:08:41 WARN SessionState: METASTORE_FILTER_HOOK will be ignored,
> since hive.security.authorization.manager is set to instance of
> HiveAuthorizerFactory.
> 20/01/06 21:08:41 WARN HiveConf: HiveConf of name
> hive.internal.ss.authz.settings.applied.marker does not exist
> 20/01/06 21:08:41 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout
> does not exist
> 20/01/06 21:08:41 WARN HiveConf: HiveConf of name hive.stats.retries.wait
> does not exist
> res3: org.apache.spark.sql.DataFrame = []
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
