[
https://issues.apache.org/jira/browse/HUDI-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo closed HUDI-3398.
---------------------------
Resolution: Fixed
> Schema validation fails for metadata table base file
> ----------------------------------------------------
>
> Key: HUDI-3398
> URL: https://issues.apache.org/jira/browse/HUDI-3398
> Project: Apache Hudi
> Issue Type: Bug
> Components: writer-core
> Reporter: Ethan Guo
> Assignee: Yue Zhang
> Priority: Blocker
> Labels: HUDI-bug, pull-request-available
> Fix For: 0.11.0
>
>
> Stacktrace:
> {code:java}
> java.lang.IllegalArgumentException: Unknown file format
> :file:/Users/ethan/Work/data/hudi/metadata_test_ds_mor_continuous_4/.hoodie/metadata/files/files-0000_0-93-815_20220208164926830001.hfile
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableParquetSchemaFromDataFile(TableSchemaResolver.java:103)
> at
> org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:119)
> at
> org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:480)
> at
> org.apache.hudi.common.table.TableSchemaResolver.<init>(TableSchemaResolver.java:65)
> at org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:682)
> at
> org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:698)
> at
> org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:173)
> at
> org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:154)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:663)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:675)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.lambda$writeTableMetadata$0(BaseHoodieWriteClient.java:273)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:96) at
> org.apache.hudi.client.BaseHoodieWriteClient.writeTableMetadata(BaseHoodieWriteClient.java:273)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:229)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:199)
> at
> org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:127)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:609)
> at
> org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:329)
> at
> org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:652)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
> full logs: https://gist.github.com/yihua/e00a1caddacbdc570b5b757049750f39
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
