umehrot2 commented on a change in pull request #1702:
URL: https://github.com/apache/hudi/pull/1702#discussion_r447560016



##########
File path: hudi-spark/src/main/scala/org/apache/hudi/IncrementalRelation.scala
##########
@@ -71,13 +78,16 @@ class IncrementalRelation(val sqlContext: SQLContext,
     optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME_OPT_KEY, 
lastInstant.getTimestamp))
     .getInstants.iterator().toList
 
-  // use schema from latest metadata, if not present, read schema from the 
data file
-  private val latestSchema = {
-    val schemaUtil = new TableSchemaResolver(metaClient)
-    val tableSchema = 
HoodieAvroUtils.createHoodieWriteSchema(schemaUtil.getTableAvroSchemaWithoutMetadataFields);
-    AvroConversionUtils.convertAvroSchemaToStructType(tableSchema)
+  // use schema from a file produced in the latest instant
+  val latestSchema: StructType = {
+    log.info("Inferring schema..")
+    val schemaResolver = new TableSchemaResolver(metaClient)
+    val tableSchema = schemaResolver.getTableAvroSchemaWithoutMetadataFields
+    val dataSchema = 
AvroConversionUtils.convertAvroSchemaToStructType(tableSchema)
+    StructType(skeletonSchema.fields ++ dataSchema.fields)

Review comment:
       Not really. The reason for doing it this way is to intentionally only 
read the **user data schema** and then append the **metadata/skeleton schema** 
to it. This avoids us to have unnecessary checks here, because if we read the 
whole schema then there will be differences. For regular hudi files, the schema 
would have both **skeleton + user data schema** whereas for **bootstrapped 
files** the schema would only have **user data schema** read from the source 
file. So to keep things simple I did it this way.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Reply via email to