[
https://issues.apache.org/jira/browse/HUDI-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo closed HUDI-7073.
---------------------------
Resolution: Fixed
> Fix schema projection in file group reader-based parquet file format
> --------------------------------------------------------------------
>
> Key: HUDI-7073
> URL: https://issues.apache.org/jira/browse/HUDI-7073
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.0.0
>
>
> When turning on the file group reader-based parquet file format for MOR
> snapshot queries, the following exception is thrown in a few tests although
> ts exists in the schema
> {code:java}
> 2023-11-09T21:08:42.7345689Z - Test Call show_logfile_metadata Procedure ***
> FAILED ***
> 2023-11-09T21:08:42.7347213Z java.lang.IllegalArgumentException: Field ts
> does not exist in the data schema
> 2023-11-09T21:08:42.7349324Z at
> org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.$anonfun$generateRequiredSchemaWithMandatory$4(HoodieFileGroupReaderBasedParquetFileFormat.scala:277)
> 2023-11-09T21:08:42.7407685Z at scala.Option.getOrElse(Option.scala:189)
> 2023-11-09T21:08:42.7409315Z at
> org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.$anonfun$generateRequiredSchemaWithMandatory$3(HoodieFileGroupReaderBasedParquetFileFormat.scala:277)
> 2023-11-09T21:08:42.7412345Z at
> org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.$anonfun$generateRequiredSchemaWithMandatory$3$adapted(HoodieFileGroupReaderBasedParquetFileFormat.scala:272)
> 2023-11-09T21:08:42.7414126Z at
> scala.collection.immutable.List.foreach(List.scala:392)
> 2023-11-09T21:08:42.7424879Z at
> org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.generateRequiredSchemaWithMandatory(HoodieFileGroupReaderBasedParquetFileFormat.scala:272)
> 2023-11-09T21:08:42.7428280Z at
> org.apache.spark.sql.execution.datasources.parquet.HoodieFileGroupReaderBasedParquetFileFormat.buildReaderWithPartitionValues(HoodieFileGroupReaderBasedParquetFileFormat.scala:92)
> 2023-11-09T21:08:42.7430151Z at
> org.apache.spark.sql.execution.FileSourceScanExec.inputRDD$lzycompute(DataSourceScanExec.scala:426)
> 2023-11-09T21:08:42.7431345Z at
> org.apache.spark.sql.execution.FileSourceScanExec.inputRDD(DataSourceScanExec.scala:417)
> 2023-11-09T21:08:42.7432454Z at
> org.apache.spark.sql.execution.FileSourceScanExec.doExecute(DataSourceScanExec.scala:495)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
