aakashofficial-k01 opened a new pull request, #4374:
URL: https://github.com/apache/amoro/pull/4374
## Why are the changes needed?
Close #4373.
In `IcebergTableMaintainer.expireSnapshots`, table snapshots are planned and
expired concurrently using `IcebergThreadPools.getMaintenanceExecutor()`. In
`RollingFileCleaner`, `doCleanFiles()` was unsynchronized and directly operated
on `collectedFiles`. When worker threads concurrently called `addFile()` while
remote bulk deletion was in flight, newly submitted files were added to
`collectedFiles` and subsequently wiped by `finally { collectedFiles.clear();
}` before being deleted, causing silent orphan storage leaks.
## Brief change log
- In `RollingFileCleaner`, replace `collectedFiles` with
`BlockingQueue<String>`.
- In `doCleanFiles()`, synchronize execution and atomically drain pending
files into a local batch via `collectedFiles.drainTo(toClean)` before deletion.
- Synchronize `clear()`.
- Add `testConcurrentCleanFiles` in `TestRollingFileCleaner` to verify
concurrent additions during bulk deletion.
## How was this patch tested?
- [x] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
