[
https://issues.apache.org/jira/browse/SPARK-27020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783098#comment-16783098
]
Hyukjin Kwon commented on SPARK-27020:
--------------------------------------
Yes, please make it self-contained reproducer if possible so that it can check
if Apache Spark has the same issue or not. Otherwise, no one can verify this.
FWIW, Apache Spark 2.3.2 does not support Hive 3.
Current Spark master support Hive 3.1 metastore as of SPARK-24360
> Unable to insert data with partial dynamic partition with Spark & Hive 3
> ------------------------------------------------------------------------
>
> Key: SPARK-27020
> URL: https://issues.apache.org/jira/browse/SPARK-27020
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.2
> Environment: Hortonwork HDP 3.1.0
> Spark 2.3.2
> Hive 3
> Reporter: Truong Duc Kien
> Priority: Major
>
> When performing inserting data with dynamic partition, the operation fails if
> all partitions are not dynamic. For example:
> The query
> {code:sql}
> insert overwrite table t1 (part_a='a', part_b) select * from t2
> {code}
> will fails with errors
> {code:xml}
> Cannot create partition spec from hdfs://xxxx/ ; missing keys [part_a]
> Ignoring invalid DP directory <path to staging directory>
> {code}
> On the other hand, if I remove the static value of part_a to make the insert
> fully dynamic, the following query will success.
> {code:sql}
> insert overwrite table t1 (part_a, part_b) select * from t2
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
