nsivabalan commented on code in PR #6163:
URL: https://github.com/apache/hudi/pull/6163#discussion_r926698953
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkHoodieTableFileIndex.scala:
##########
@@ -96,10 +97,24 @@ class SparkHoodieTableFileIndex(spark: SparkSession,
val partitionFields = partitionColumns.get().map(column =>
StructField(column, StringType))
StructType(partitionFields)
} else {
- val partitionFields = partitionColumns.get().map(column =>
- nameFieldMap.getOrElse(column, throw new
IllegalArgumentException(s"Cannot find column: '" +
- s"$column' in the schema[${schema.fields.mkString(",")}]")))
- StructType(partitionFields)
+ val partitionFields = partitionColumns.get().filter(column =>
nameFieldMap.contains(column))
+ .map(column => nameFieldMap.apply(column))
+
+ if (partitionFields.size != partitionColumns.get().size) {
+ val isBootstrapTable =
BootstrapIndex.getBootstrapIndex(metaClient).useIndex()
+ if (isBootstrapTable) {
+ // For bootstrapped tables its possible the schema does not
contain partition field when source table
Review Comment:
I haven't played much w/ bootstrapped table. help me clarify something. in
this case, hudi table is actually non-partitioned is it? i.e. when source table
has hive style partitioned, but does not contain the actual partition field in
the dataframe ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
