nsivabalan commented on code in PR #6163:
URL: https://github.com/apache/hudi/pull/6163#discussion_r926699569
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -96,10 +97,24 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
val partitionFields = partitionColumns.get().map(column =>
StructField(column, StringType))
StructType(partitionFields)
} else {
- val partitionFields = partitionColumns.get().map(column =>
- nameFieldMap.getOrElse(column, throw new
IllegalArgumentException(s"Cannot find column: '" +
- s"$column' in the schema[${schema.fields.mkString(",")}]")))
- StructType(partitionFields)
+ val partitionFields = partitionColumns.get().filter(column =>
nameFieldMap.contains(column))
+ .map(column => nameFieldMap.apply(column))
+
+ if (partitionFields.size != partitionColumns.get().size) {
+ val isBootstrapTable =
BootstrapIndex.getBootstrapIndex(metaClient).useIndex()
+ if (isBootstrapTable) {
+ // For bootstrapped tables its possible the schema does not
contain partition field when source table
Review Comment:
if yes, I agree. but if its feasible to generate a partitioned hudi table,
we can't proceed w/ this fix right. can you help me understand please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
