terrytlu opened a new pull request, #4293:
URL: https://github.com/apache/amoro/pull/4293
Fixes #4292
## What changes were proposed in this pull request?
IcebergTableMaintainer.cleanOrphanFiles() lists and deletes files under a
table's location. When two Iceberg tables share the same location (e.g. one
table is created from another's metadata directory), the cleanup of table A
can delete files that actually
belong to table B, silently corrupting B.
This PR adds a pre-check, `IcebergTableUtil.hasOtherTableInLocation(Table)`:
- lists the table's metadata directory via its own (storage-agnostic)
`FileIO`
(requires `SupportsPrefixOperations`; otherwise a `ValidationException` is
thrown so the caller can decide);
- compares the `table-uuid` of every other `metadata.json` with the current
table's uuid;
- an unreadable / legacy / compressed metadata file (uuid cannot be
determined)
is treated as a conflict (fail-safe).
If another table is detected in the same location, `cleanOrphanFiles()` now
**skips cleanup by default (fail-closed)** instead of risking another table's
files. A new table property `clean-orphan-file.ignore-location-conflict`
(default `false`) lets users opt out of the check when they are certain the
location is not shared.
## Why are the changes needed?
Without the guard, a single misconfigured / duplicated table location leads
to
silent cross-table data loss during scheduled orphan-file cleanup.
## How was this patch tested?
- Added `TestIcebergOrphanFileLocationConflict` (5 cases × 2 Iceberg
parameterizations = 10 tests): no conflict, conflict via corrupt metadata,
`ValidationException` when FileIO lacks prefix support, default
skip-on-conflict,
and ignore-conflict-when-property-enabled proceeds with cleanup.
- `mvn test -pl amoro-ams -Dtest=TestIcebergOrphanFileLocationConflict`
→ Tests run: 10, Failures: 0, Errors: 0.
- `spotless:check` passes.
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
