ravjot28 opened a new issue, #20584:
URL: https://github.com/apache/hudi/issues/20584
### Bug Description
**What happened:**
`restoreToSavepoint` (and `CALL rollback_to_savepoint`) fails after the data
table rollbacks, in its metadata step, and
leaves a pending restore on the data table, in three cases:
- A. The restore deletes and rebuilds the metadata table (savepoint at or
before its oldest compaction), and the rebuilt
table compacts right after its bootstrap delta commits.
- B. The metadata table was initialized after the savepoint and holds a
later data table delta commit.
- C. Table version 6, whenever the restore deletes and rebuilds the metadata
table.
Cases A and B (table versions 8 and later):
```
org.apache.hudi.exception.HoodieRestoreException: Failed to restore to
<savepoint>
at
org.apache.hudi.client.BaseHoodieWriteClient.restoreToSavepoint(BaseHoodieWriteClient.java)
...
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java)
at
org.apache.hudi.table.action.restore.BaseRestoreActionExecutor.finishRestore(BaseRestoreActionExecutor.java)
Caused by: org.apache.hudi.exception.HoodieException: Unable to restore to
instant: <savepoint>
at
org.apache.hudi.table.action.rollback.RestorePlanActionExecutor.execute(RestorePlanActionExecutor.java)
Caused by: org.apache.hudi.exception.HoodieException: Unable to find
completion time for instant: <savepoint>
```
Case C:
```
org.apache.hudi.exception.HoodieMetadataException: Can't restore to
<savepoint> since there is no base file in MDT lesser than the commit to
restore to. Please delete metadata table and retry
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java)
```
In case C the log first shows `Deleting MDT before restore to <savepoint>`,
so the metadata table the message asks to
delete is the one the same restore has just rebuilt.
**What you expected:**
The restore completes. A metadata table rebuilt for the restore, or
initialized after the savepoint, has nothing to
roll back.
**Steps to reproduce:**
Case A:
1. Copy-on-write table with the metadata table enabled and
`hoodie.metadata.compact.max.delta.commits=1` (any value at
or below the number of metadata partitions initialized at bootstrap;
files, column stats and partition stats give 3).
2. Insert commit C1, savepoint C1, insert commits C2 and C3.
3. `client.restoreToSavepoint(C1)` or `CALL rollback_to_savepoint(table =>
't', instant_time => 'C1')`.
Case B:
1. Copy-on-write table with `hoodie.metadata.enable=false`. Insert commit
C1, savepoint C1, insert commit C2.
2. Set `hoodie.metadata.enable=true` and insert commit C3.
3. `client.restoreToSavepoint(C1)`.
Case C: case B on a table at table version 6
(`hoodie.write.table.version=6`).
**Root cause (master d5230b69e22):**
-
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:875-880`
deletes the
metadata table and asks for no re-initialization, but
`initTable(operationType, instantTime, initMetadataTable)`
ignores the flag (`BaseHoodieWriteClient.java:1500-1502`), so the metadata
table is rebuilt, and its table services
run
(`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:398-399`),
before
the data table rollbacks.
-
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RestorePlanActionExecutor.java:113-116`:
`isMetadataTableRecreatedDuringRestore` decides from the last instant
only. In case A it is the compaction after the
bootstrap commits, so the table counts as not recreated and
`RestorePlanActionExecutor.java:88-89` looks up the
completion time of the data savepoint, which the rebuilt metadata timeline
never held.
- `BaseHoodieWriteClient.java:903` (`shouldDeleteMdtBeforeRestore`) deletes
the metadata table only when the target is
at or before its oldest compaction or before its timeline start. A
metadata table initialized after the savepoint
starts at `00000000000000000`, so it is kept although its bootstrap state
includes C2 (case B); the plan then fails
the same lookup.
- Table version 6 initializes a metadata table at the latest data table
instant plus a suffix (`<instant>010`), later
than the savepoint, so the base file check in
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:1206-1213`
throws for the rebuilt table (case C).
The recreated-table rule came with #13691.
Related, found while fixing the cases above: a metadata table rebuilt for a
restore is built before the data table
rollbacks, from files that include the rolled back commits, and the sync
after the restore covers the files and column
stats partitions only (`HoodieBackedTableMetadataWriter.java:1239-1251`;
`MetadataPartitionType.java:425-430` keeps the
record index on restore). With the record index enabled it keeps entries for
keys that only the rolled back commits
wrote. Once the cases above complete, more restores take that path, so the
fix drops the record index partition of
such a table during the restore, with the other index drops and before the
metadata restore step.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1
**Query engine:** Spark 3.5 (write client and Spark SQL procedures); the
planning code is engine independent
**Relevant configs:** `hoodie.metadata.enable=true`; case A needs
`hoodie.metadata.compact.max.delta.commits` at or
below
the number of metadata partitions initialized at bootstrap (with the default
of 10, ten or more partitions). Cases A and
B reproduced on table versions 9 and 10, case C on table version 6.
### Logs and Stack Trace
See "What happened" above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]