zhuxiangyi opened a new pull request, #968:
URL: https://github.com/apache/paimon-rust/pull/968
### Purpose
Linked issue: #964 (part 4 of 4)
> Stacked on #965, which provides the shared helpers (`try_get_snapshot`,
`parse_older_than`,
> reassign-plan detection). Only the last commit, "feat(table): remove
orphan files and add
> sys.remove_orphan_files", is new here.
Failed or interrupted writes leave files that no snapshot references, and
nothing in paimon-rust
removes them. This PR ports Java's `LocalOrphanFilesClean` and adds `CALL
sys.remove_orphan_files`.
### Brief change log
- `Table::new_remove_orphan_files()` / `RemoveOrphanFiles`
(`table/orphan_files_clean.rs`):
- Candidates: files last modified before `older_than` in these places:
- the `manifest`, `index` and `statistics` directories;
- every bucket directory, walking `partition_keys` levels of `key=value`
directories, as
Java's `listFileDirs` does;
- bucket directories under `data-file.external-paths`.
- Non-snapshot files in each branch's `snapshot` and `changelog`
directories (temporary files of
interrupted commits) are removed too, as in Java's
`cleanBranchSnapshotDir`.
- Used files: everything referenced by the snapshots, tags, and long-lived
changelogs
(`changelog/changelog-*`, the snapshot JSON format) of every branch:
- manifest lists, manifests and their extra files;
- index manifests and index files;
- statistics files and reassign plans;
- data files and their extra files from all manifest entries.
Files are matched by name, as in Java.
- Safety:
- `older_than` defaults to one day ago and must be in the past;
- a store that reports no modification time never exposes a file;
- a branch without a schema aborts the run (Java);
- managed BLOB packs (`*.managed.blob`) are skipped (Java);
- directories are never deleted.
- Unlike Java, which reads a missing manifest as empty, a missing metadata
file of a snapshot or
changelog that still exists aborts the run. Otherwise that snapshot's
data files would look
unreferenced and be deleted. If the owning snapshot vanished at the same
time (concurrent
expiration), it is simply skipped.
- `dry_run` reports without deleting; `parallelism` bounds concurrent
reads and deletions.
- DataFusion `CALL sys.remove_orphan_files(table, older_than, dry_run,
parallelism, mode)` returns
`deletedFileCount` and `deletedFileTotalLenInBytes`, like Java. It
supports `mode => 'local'` and
one table per call; `db.*` returns a clear "not supported yet" error.
- Not included: removing empty directories. It is left for later, because on
object stores
"check empty, then delete" races with writers.
### Tests
- `table::orphan_files_clean::tests` (7 tests) run on a real local file
system, because they need
modification times:
- planted orphans (a data file, a manifest, an index file, a statistics
file, and a temporary
snapshot file) are found by a dry run, then removed. Afterwards the
directory holds exactly
the referenced files, the table reads correctly, and a second run finds
nothing;
- recent files are kept, and a future `older_than` is rejected;
- after a snapshot is removed without cleanup, only its own manifest lists
become orphans. Its
data file is still named by the next delta's DELETE entry, as in Java;
- a tag, a branch, and a long-lived changelog each protect their files;
once the changelog is
removed, its files become orphans;
- partitioned bucket directories are scanned, and managed BLOB packs are
skipped;
- a missing manifest of a live snapshot aborts the run with nothing
deleted;
- a branch without a schema aborts the run.
- Making the cleanup ignore branches makes the branch/tag/changelog test
fail.
- DataFusion `tests/procedures.rs`:
- an end-to-end test: the default cut-off keeps a fresh orphan; a dry run
reports it; a real run
with `parallelism` and `mode => 'local'` removes it and the table stays
readable;
- argument validation: `db.*`, `mode => 'distributed'`, a future
`older_than`, a bad `dry_run`,
and `parallelism => 0`.
- `cargo test -p paimon --all-targets --features fulltext,vortex` passes
(3784 tests), and so does
clippy with `-D warnings`.
### API and Format
New public API: `Table::new_remove_orphan_files()`, `RemoveOrphanFiles`,
`OrphanFilesCleanResult`;
new `sys.remove_orphan_files` procedure. No format change.
### Documentation
`docs/src/sql.md` documents `remove_orphan_files`: its arguments, what
counts as an orphan, and
the safety rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
