sundapeng opened a new pull request, #863:
URL: https://github.com/apache/paimon-rust/pull/863
### Purpose
Linked issue: close #862
`TRUNCATE TABLE` on a Format Table ran the Paimon snapshot overwrite commit.
A Format Table has no snapshot, so the commit found nothing to delete and
returned `Ok`. Users got success and kept every row. This implements TRUNCATE
for Format Tables, following Java `FormatTableCommit#truncateTable` and
`#truncatePartitions` from apache/paimon#9330.
### Brief change log
- `TRUNCATE TABLE t [PARTITION (...)]` on a Format Table goes to
`format_table_truncate.rs`. It deletes data files only. Partition directories
and catalog registrations stay.
- With catalog-managed partitions, the registered partitions are emptied
and reported through `Catalog::create_partitions_with_statistics` with zero
statistics and `replaceStatistics=true`. The last file creation time is the
truncation time, as Java. A directory nobody registered is not part of the
table and is not touched.
- Without catalog-managed partitions, the partition directories found
under the table are emptied. An unpartitioned table loses the files in its
table directory.
- `PARTITION (...)` takes a leading run of partition values, the same rule
as ANALYZE. No matching partition is an error. A column named without a value,
like `PARTITION (dt)`, is rejected, because reading it as every value would
empty the table.
- A selected partition at a custom location fails the statement before
anything is deleted. Java resets such a partition to its default directory,
which needs partition options in `CreatePartitionsRequest`; the Rust request
does not carry them yet.
- If deleting fails in the middle, the partitions already emptied are
still reported, so the catalog does not describe files that are gone. The
deletion error is returned.
- `FormatTableTruncator`, public in `paimon::table`, does the deleting. It
removes every non-hidden file below a partition directory whatever its
extension, because a Java reader reads those files too. Staging entries such as
`_temporary` stay, with the same hidden-path rule as the scan listing (#813).
- The scan listing is split so the truncator can list without the extension
filter (`list_format_table_files`). A scan reads the same files as before.
- The leading prefix parsing of ANALYZE moves to
`format_partition_ddl::leading_partition_prefix`, shared by both statements.
ANALYZE behaves the same, error messages included.
- `TableCommit::truncate_table` and `truncate_partitions` refuse a Format
Table instead of reporting success. This covers the C and Go bindings, which
call them directly.
- `docs/src/sql.md` gets a TRUNCATE TABLE section under Format Table
Partitions.
### Tests
`crates/integrations/datafusion/tests/rest_format_partition_sql.rs`:
- whole-table TRUNCATE with catalog-managed partitions: rows gone, a file
without extension deleted, a `_temporary` file and an unregistered directory
kept, registrations kept, statistics replaced by zero after an earlier ANALYZE;
- `PARTITION` with a complete spec and with a prefix, only those partitions
emptied and reported, and the errors (non-leading key, non-partition column, no
match by prefix and by complete spec, column without value);
- a partition at a custom location refused before any file is deleted or any
statistics is reported, and a `PARTITION` clause that does not select it still
works;
- a table that discovers partitions from directories: partition and
whole-table TRUNCATE, a non-partition directory kept, nothing sent to the
catalog;
- an unpartitioned table, and `PARTITION` on it rejected;
- `TableCommit::truncate_table` and `truncate_partitions` refusing a Format
Table.
With the SQL dispatch and the `TableCommit` guard disabled, all six new
tests fail.
Commands run on the top commit:
```
cargo fmt --all -- --check
cargo clippy --locked -p paimon -p paimon-datafusion --all-targets
--features paimon/fulltext,paimon/vortex -- -D warnings
cargo test -p paimon-datafusion --test rest_format_partition_sql # 22
passed
cargo test -p paimon-datafusion --test table_type_routing # 38
passed
cargo test -p paimon-datafusion --lib truncate # 6 passed
cargo test -p paimon --lib format_ # 47
passed
cargo test -p paimon --lib table_commit # 72
passed
```
I also ran the statements against a real REST catalog (Aliyun DLF) with the
table on OSS: data written as `dt=a/part-0.parquet`, `dt=a/part-00001` (no
extension), `dt=a/_temporary/0/part-9.parquet` and `dt=b/part-0.parquet`. After
`ANALYZE`, `t$partitions` showed `2/1` and `1/1`. `TRUNCATE TABLE t PARTITION
(dt = 'a')` deleted both data files of `dt=a`, kept the staging file, and
`t$partitions` showed `0/0` and `1/1` with `COUNT(*)` 1. `TRUNCATE TABLE t`
left only the staging file, `COUNT(*)` 0, both partitions still in `SHOW
PARTITIONS` with `0/0`.
### API and Format
New public `paimon::table::FormatTableTruncator`.
`TableCommit::truncate_table` and `truncate_partitions` now return an error for
a Format Table, where they used to return `Ok` without effect. No storage
format change.
### Documentation
`docs/src/sql.md`, new "TRUNCATE TABLE" section under "Format Table
Partitions".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
