terrytlu opened a new issue, #4292:
URL: https://github.com/apache/amoro/issues/4292
### What happened?
## Bug description
Amoro's `IcebergTableMaintainer.cleanOrphanFiles()` can delete files
belonging to
*other* Iceberg tables when two tables share the same `LOCATION`.
It computes reachable files from only the target table's metadata, then
recursively
lists the whole location and deletes everything else — including a second
table's
data, manifest, and metadata files under the same directory.
This happens easily in practice: a table is created by copying `SHOW CREATE
TABLE`
of another table but forgetting to change `LOCATION`. We hit this in
production
(irreversible data loss) and found 100+ shared-location pairs in our
environment.
## Proposed fix
Add a default-on guard in `IcebergTableMaintainer` before orphan cleanup:
1. List `metadata/*.metadata.json` (+ `.metadata.json.gz`) under
`table.location()/metadata`.
2. For each file not in the table's own version set, read its `table-uuid`:
- Different uuid → another table shares the location → **skip cleanup**
(default).
- Unreadable (legacy/compressed/corrupt) → treat as suspicious → skip
(fail-safe).
- Same uuid → older version of this table → ignore.
3. Only proceed when no foreign uuid is found.
Add table property `clean-orphan-file.ignore-location-conflict` (default
`false`)
to opt out when a shared location is intentional.
### Affects Versions
all
### What table formats are you seeing the problem on?
Iceberg
### What engines are you seeing the problem on?
AMS
### How to reproduce
1. Create Iceberg table A at `hdfs://ns/tmp/shared`.
2. Create Iceberg table B at the **same** location `hdfs://ns/tmp/shared`.
3. Configure orphan file cleanup and trigger it on A:
- Set catalog-level table property `clean-orphan-file.enabled=true` to
enable periodic cleanup.
- Set `clean-orphan-file.min-existing-time-minutes=2` so orphan files are
eligible for
deletion within minutes (instead of the default 2 days), for quick
observation.
4. Restart AMS so the configuration takes effect, and let the scheduled
orphan file
cleanup task run
5. B's files under the shared directory are deleted → B becomes
corrupted/unloadable.
### Relevant log output
```shell
```
### Anything else
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
