[
https://issues.apache.org/jira/browse/HUDI-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kwang updated HUDI-6659:
------------------------
Description:
Spark offline compaction rollbacks failed inflight instant throw this error
info when mdt enabled.
{code:java}
org.apache.hudi.exception.HoodieMetadataException: Commit being rolled back
20230802174236306 is earlier than the latest compaction 20230803201423881001.
There are 3 deltacommits after this compaction:
[[20230803201545303__deltacommit__COMPLETED__20230803201721395],
[20230803201729007__deltacommit__COMPLETED__20230803201848687],
[20230803201852499__deltacommit__COMPLETED__20230803202010862]]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.validateRollback(HoodieBackedTableMetadataWriter.java:1034)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1002)
at
org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:256)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:118)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:139)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:218)
at
org.apache.hudi.table.HoodieTable.rollbackInflightInstant(HoodieTable.java:650)
at
org.apache.hudi.table.HoodieTable.rollbackInflightCompaction(HoodieTable.java:623)
at
org.apache.hudi.client.SparkRDDTableServiceClient.compact(SparkRDDTableServiceClient.java:80)
at
org.apache.hudi.client.SparkRDDWriteClient.compact(SparkRDDWriteClient.java:307)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1034)
at
org.apache.hudi.utilities.HoodieCompactor.doCompact(HoodieCompactor.java:306)
{code}
was:
Spark offline compaction rollback failed inflight instant throw next error info
when mdt enabled.
{code:java}
org.apache.hudi.exception.HoodieMetadataException: Commit being rolled back
20230802174236306 is earlier than the latest compaction 20230803201423881001.
There are 3 deltacommits after this compaction:
[[20230803201545303__deltacommit__COMPLETED__20230803201721395],
[20230803201729007__deltacommit__COMPLETED__20230803201848687],
[20230803201852499__deltacommit__COMPLETED__20230803202010862]]
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.validateRollback(HoodieBackedTableMetadataWriter.java:1034)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1002)
at
org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:256)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:118)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:139)
at
org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:218)
at
org.apache.hudi.table.HoodieTable.rollbackInflightInstant(HoodieTable.java:650)
at
org.apache.hudi.table.HoodieTable.rollbackInflightCompaction(HoodieTable.java:623)
at
org.apache.hudi.client.SparkRDDTableServiceClient.compact(SparkRDDTableServiceClient.java:80)
at
org.apache.hudi.client.SparkRDDWriteClient.compact(SparkRDDWriteClient.java:307)
at
org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1034)
at
org.apache.hudi.utilities.HoodieCompactor.doCompact(HoodieCompactor.java:306)
{code}
> Remove the validateRollback restriction for Spark/Flink MDT rollback
> --------------------------------------------------------------------
>
> Key: HUDI-6659
> URL: https://issues.apache.org/jira/browse/HUDI-6659
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: kwang
> Priority: Major
> Fix For: 0.14.0
>
>
> Spark offline compaction rollbacks failed inflight instant throw this error
> info when mdt enabled.
> {code:java}
> org.apache.hudi.exception.HoodieMetadataException: Commit being rolled back
> 20230802174236306 is earlier than the latest compaction 20230803201423881001.
> There are 3 deltacommits after this compaction:
> [[20230803201545303__deltacommit__COMPLETED__20230803201721395],
> [20230803201729007__deltacommit__COMPLETED__20230803201848687],
> [20230803201852499__deltacommit__COMPLETED__20230803202010862]]
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.validateRollback(HoodieBackedTableMetadataWriter.java:1034)
> at
> org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1002)
> at
> org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
> at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
> at
> org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
> at
> org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:256)
> at
> org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:118)
> at
> org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:139)
> at
> org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:218)
> at
> org.apache.hudi.table.HoodieTable.rollbackInflightInstant(HoodieTable.java:650)
> at
> org.apache.hudi.table.HoodieTable.rollbackInflightCompaction(HoodieTable.java:623)
> at
> org.apache.hudi.client.SparkRDDTableServiceClient.compact(SparkRDDTableServiceClient.java:80)
> at
> org.apache.hudi.client.SparkRDDWriteClient.compact(SparkRDDWriteClient.java:307)
> at
> org.apache.hudi.client.BaseHoodieWriteClient.compact(BaseHoodieWriteClient.java:1034)
> at
> org.apache.hudi.utilities.HoodieCompactor.doCompact(HoodieCompactor.java:306)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
