umehrot2 commented on issue #1764: URL: https://github.com/apache/hudi/issues/1764#issuecomment-649928569
Wow, I just started running into this same issue as I was running performance tests for RFC-12 bootstrap implementation. cc @bvaradar ``` org.apache.hudi.exception.HoodieCommitException: Failed to complete commit 00000000000001 due to finalize errors. at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.finalizeWrite(BaseCommitActionExecutor.java:255) at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commit(BaseCommitActionExecutor.java:216) at org.apache.hudi.table.action.bootstrap.BootstrapCommitActionExecutor.commit(BootstrapCommitActionExecutor.java:205) at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:178) at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.updateIndexAndCommitIfNeeded(BaseCommitActionExecutor.java:172) at org.apache.hudi.table.action.bootstrap.BootstrapCommitActionExecutor.metadataBootstrap(BootstrapCommitActionExecutor.java:178) at org.apache.hudi.table.action.bootstrap.BootstrapCommitActionExecutor.execute(BootstrapCommitActionExecutor.java:136) at org.apache.hudi.table.HoodieCopyOnWriteTable.bootstrap(HoodieCopyOnWriteTable.java:143) at org.apache.hudi.client.HoodieWriteClient.bootstrap(HoodieWriteClient.java:158) at org.apache.hudi.HoodieSparkSqlWriter$.bootstrap(HoodieSparkSqlWriter.scala:203) at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:139) at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:173) at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:169) at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:197) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:194) at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:169) at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:114) at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:112) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676) at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676) at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$executeQuery$1(SQLExecution.scala:83) at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1$$anonfun$apply$1.apply(SQLExecution.scala:94) at org.apache.spark.sql.execution.QueryExecutionMetrics$.withMetrics(QueryExecutionMetrics.scala:141) at org.apache.spark.sql.execution.SQLExecution$.org$apache$spark$sql$execution$SQLExecution$$withMetrics(SQLExecution.scala:178) at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:93) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:200) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:92) at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229) ... 49 elided Caused by: org.apache.hudi.exception.HoodieIOException: Consistency check failed to ensure all files APPEAR at org.apache.hudi.table.HoodieTable.waitForAllFiles(HoodieTable.java:553) at org.apache.hudi.table.HoodieTable.cleanFailedWrites(HoodieTable.java:495) at org.apache.hudi.table.HoodieTable.finalizeWrite(HoodieTable.java:398) at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.finalizeWrite(BaseCommitActionExecutor.java:252) ... 84 more ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org