SteNicholas opened a new pull request, #1312:
URL: https://github.com/apache/paimon-rust/pull/1312
### Purpose
Linked issue: close #1309
DataFusion SQL users can't trigger table compaction yet, even though Paimon
core already exposes it through `Table::new_compact_builder()`. This PR adds
`CALL sys.compact`, a thin SQL adapter over that core API that matches Java
Paimon's Spark/Flink procedure.
```sql
CALL sys.compact(table => 'db.tbl');
CALL sys.compact(table => 'db.tbl', partitions =>
'dt=2026-10-10,hr=1;dt=2026-10-11,hr=2');
```
### Brief change log
- **Procedure** (`procedures.rs`): adds `proc_compact`.
- It plans, runs every task, then commits all results in a single COMPACT
snapshot. If the plan has no tasks, it doesn't commit.
- It returns one row: `tasks`, `files_removed`, `files_added`,
`rows_written`.
- The optional `partitions` argument uses the same syntax and parser as
`drop_global_index`. The shared parser helpers lose their `global_index` prefix.
- **Memory**: compaction now goes through the session memory pool. The new
`compact_resources` is passed via `CompactBuilder::with_resources`, and the
reader/writer/compact resource setup in `memory.rs` now shares one helper.
- **Failure cleanup**:
- New `TableCommit::abort_compact` in core. It deletes the data, changelog
and newly added index files of compaction results that were never submitted.
- It tries every file and returns an error naming any file it couldn't
delete. `abort`'s existing best-effort semantics are unchanged.
- If a later task fails, the procedure cleans up the outputs of the tasks
that already finished, and logs a warning if that cleanup fails.
- Once the commit starts, files are kept, because an uncertain commit may
still reference them.
- **Logging**: on completion, the procedure logs the table, task count,
duration and metrics.
### Tests
- `crates/integrations/datafusion/tests/procedures.rs`:
- Append table, including that a second call with nothing to compact
doesn't commit.
- Primary-key table, checking that updates are merged.
- Append and PK tables with `partitions`: other partitions are untouched,
multiple tasks go into a single commit, and PK compaction is incremental.
- Data-evolution table with deletion vectors: deletions are kept, and
`rows_written` counts physical rows.
- Task failure: outputs of earlier tasks are deleted and no snapshot is
created.
- The session memory limit is enforced.
- Invalid arguments: `partitions` on an unpartitioned table, an unknown
argument, a missing `table`.
- `crates/paimon/tests/append_compact_test.rs`: `abort_compact` tries every
output and reports any file it couldn't delete. The deletion failure is
injected through a custom `FileIOProvider`.
Note: the `procedures` integration tests need `RUST_MIN_STACK=16777216` in
local debug builds, matching CI.
### API and Format
- New public API: `TableCommit::abort_compact(&[CompactResult])`.
- New SQL procedure: `sys.compact`.
- No storage format changes.
### Documentation
- `docs/src/sql.md` gets a `compact` section covering usage, partition
filters, result columns, cleanup semantics, PK incremental compaction and the
local-cache requirement.
- The procedure list in the `procedures.rs` module docs is updated.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
