SteNicholas opened a new issue, #1309: URL: https://github.com/apache/paimon-rust/issues/1309
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation Paimon core already exposes table compaction through `Table::new_compact_builder()` (`CompactPlanner` / `CompactExecutor` / `TableCommit`). It covers append, primary-key and data-evolution tables (#1247, #1249). However, DataFusion SQL users cannot trigger it. `procedures.rs` supports tag, branch, rollback, consumer, global-index and permission procedures, but it does not support `compact`. Users therefore have to write Rust code to compact a table. This matters for aggregation and partial-update tables, which need compaction to merge results. It also matters for tables that accumulate many small files from frequent writes. Java Paimon exposes the same operation as `CALL sys.compact(...)` in Spark and Flink. ### Solution Add a `compact` procedure to the DataFusion integration as a thin SQL adapter over the existing core API: ```sql CALL sys.compact(table => 'db.tbl'); CALL sys.compact(table => 'db.tbl', partitions => 'dt=2026-10-10,hr=1;dt=2026-10-11,hr=2'); ``` - `table` (required): the table identifier, resolved in the same way as the other procedures. - `partitions` (optional): partition alternatives. This uses the same syntax and parser as `drop_global_index` (`parse_global_index_partitions`) and is passed to `CompactBuilder::with_partition_filter`. Omit it to compact all partitions. - Execution: plan → execute tasks → commit through `CompactBuilder`. If the plan contains no tasks, no commit is made. - Result: return a single row with the outcome, such as the new snapshot id (NULL when nothing was compacted) and the number of compaction tasks or rewritten files. - Errors from core (unsupported table types, branch references, lookup/DV compaction without `local-cache.enabled`, missing write authorization) are returned as DataFusion errors without changes. - Update the procedure list in the module docs and add tests in `crates/integrations/datafusion/tests/procedures.rs` for append and primary-key tables, with and without `partitions`, plus an unpartitioned table with `partitions`. Not in scope (possible follow-ups): Java's `order_strategy` / `order_by` (sort compaction), `where`, and `options` arguments. ### Anything else? Python (#286), C and Go bindings could later expose the same core API in a similar way. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
