rubenssoto opened a new issue #2944:
URL: https://github.com/apache/hudi/issues/2944
Hello guys,
I have a lot of Hudi jobs, one job failed yesterday and after some hours
started to show this error:
Apache Hudi 0.8
EMR 6.2
Apache Spark 3.0
Hudi Options:
Map(hoodie.datasource.hive_sync.database -> raw_freshchat,
hoodie.parquet.small.file.limit -> 402653184,
hoodie.copyonwrite.record.size.estimate -> 1024,
hoodie.datasource.hive_sync.support_timestamp -> true,
hoodie.datasource.write.precombine.field -> linecreatedtimestamp,
hoodie.datasource.hive_sync.partition_extractor_class ->
org.apache.hudi.hive.NonPartitionedExtractor, hoodie.parquet.max.file.size ->
419430400,
hoodie.parquet.block.size -> 402653184,
hoodie.datasource.hive_sync.table -> conversation_created,
hoodie.datasource.write.operation -> upsert,
hoodie.datasource.hive_sync.enable -> false,
hoodie.datasource.write.recordkey.field -> conversation_id,created_at,
hoodie.table.name -> conversation_created,
hoodie.datasource.hive_sync.jdbcurl ->
jdbc:hive2://ip-10-0-92-222.us-west-2.compute.internal:10000,
hoodie.datasource.write.hive_style_partitioning -> false,
hoodie.datasource.write.table.name -> conversation_created,
hoodie.datasource.write.keygenerator.class ->
org.apache.hudi.keygen.NonpartitionedKeyGenerator, hoodie.keep.max.commits ->
38,
hoodie.upsert.shuffle.parallelism -> 100,
hoodie.cleaner.commits.retained -> 36,
hoodie.keep.min.commits -> 37, hoodie.clean.async -> true)
`21/05/13 13:21:00 ERROR HoodieTimelineArchiveLog: Failed to archive
commits, .commit file: 20210504033941.rollback.inflight
java.io.IOException: Not an Avro data file
at
org.apache.avro.file.DataFileReader.openReader(DataFileReader.java:50)
at
org.apache.hudi.common.table.timeline.TimelineMetadataUtils.deserializeAvroMetadata(TimelineMetadataUtils.java:175)
at
org.apache.hudi.client.utils.MetadataConversionUtils.createMetaWrapper(MetadataConversionUtils.java:84)
at
org.apache.hudi.table.HoodieTimelineArchiveLog.convertToAvroRecord(HoodieTimelineArchiveLog.java:370)
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archive(HoodieTimelineArchiveLog.java:311)
at
org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:128)
at
org.apache.hudi.client.AbstractHoodieWriteClient.postCommit(AbstractHoodieWriteClient.java:430)
at
org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:186)
at
org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:121)
at
org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:479)
at
org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:223)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
at
org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:46)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at
org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:90)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$execute$1(SparkPlan.scala:180)
at
org.apache.spark.sql.execution.SparkPlan.$anonfun$executeQuery$1(SparkPlan.scala:218)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:215)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:176)
at
org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:124)
at
org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:123)
at
org.apache.spark.sql.DataFrameWriter.$anonfun$runCommand$1(DataFrameWriter.scala:963)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:104)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:227)
at
org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:107)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:132)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:104)
at
org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:227)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:132)
at
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:248)
at
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:131)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
at
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at
org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:963)
at
org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:415)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:399)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:288)
at hudiwriter.HudiWriter.merge(HudiWriter.scala:98)
at hudiwriter.HudiContext.writeToHudi(HudiContext.scala:39)
at jobs.TableProcessor.start(TableProcessor.scala:88)
at
TableProcessorWrapper$.$anonfun$main$2(TableProcessorWrapper.scala:23)
at
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at
java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
21/05/13 13:21:00 INFO S3NativeFileSystem: Opening
's3://bucket/freshchat/response_time/.hoodie/20210512023309.clean.inflight' for
reading
21/05/13 13:21:00 INFO S3NativeFileSystem: Opening
's3://bucket/freshchat/response_time/.hoodie/20210512023309.clean' for reading`
Coudl you help me?
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
