maomaodev commented on code in PR #7622:
URL: https://github.com/apache/kyuubi/pull/7622#discussion_r3756836578
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/write/HiveWrite.scala:
##########
@@ -66,8 +67,14 @@ case class HiveWrite(
private val tableLocation = hiveTable.getDataLocation
private val allColumns = info.schema().toAttributes
- private val dataColumns = allColumns.take(allColumns.length -
hiveTable.getPartCols.size())
- private val partColumns = allColumns.takeRight(hiveTable.getPartCols.size())
+ // Split data and partition columns by name rather than by position. Users
can create a
+ // partitioned table whose partition columns are not the trailing columns of
the schema
+ // (e.g. df.write.partitionBy("mid_col")), so a positional take/takeRight
would mis-classify
+ // columns and lead the metastore to reject the written partition spec. See
KYUUBI #6403.
+ private val dataColumns: Seq[AttributeReference] =
+ allColumns.filterNot(col => table.partitionColumnNames.contains(col.name))
+ private val partColumns: Seq[AttributeReference] =
+ allColumns.filter(col => table.partitionColumnNames.contains(col.name))
Review Comment:
Thanks for the fix, does this handle the case where the partition column
name contains upper-case letters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
