sundapeng opened a new pull request, #9330:
URL: https://github.com/apache/paimon/pull/9330
### Purpose
`TRUNCATE TABLE` did not work on a Format Table, and the two statements
failed differently.
`TRUNCATE TABLE t` was rejected in the planner: `PaimonFormatTable` does not
implement
`TruncatableTable`, so `DataSourceV2Strategy` failed evaluating
`asTruncatable` with
`Table does not support truncates: <table>.`
`TRUNCATE TABLE t PARTITION (...)` got further, into
`PaimonPartitionManagement`, which serves only
`FileStoreTable` and failed there with `Only FileStoreTable supports
partitions.` — a message about
an internal type, on a table that has partitions and lists them under `SHOW
PARTITIONS`.
Underneath both, `FormatTableCommit.truncateTable()` and
`truncatePartitions()` threw an empty
`UnsupportedOperationException`, and Flink's `FlinkFormatTableSink` did not
implement
`SupportsTruncate` either.
The workaround was `INSERT OVERWRITE ... PARTITION (...) SELECT ... WHERE
false`, which does not
extend to the whole table: without a static partition, an overwrite deletes
only the directories it
actually wrote a file into, so an empty result clears nothing.
### Approach
`FormatTableCommit` deletes the data files, reusing
`deletePreviousDataFile`, the primitive a static
`INSERT OVERWRITE` already clears partition directories with. Only data
files go: the partition
directories stay, and so do their catalog registrations, so `SHOW
PARTITIONS` returns what it
returned before (SPARK-34418). Staging trees of concurrent writers are left
alone, on the same
judgement `FormatTableScan` reads with.
Which partitions the table has is answered by whatever the table reads its
partitions from. For a
catalog-managed table that is the catalog, so truncating the whole table
empties the registered
partitions and leaves a directory still awaiting `MSCK REPAIR TABLE` alone —
matching what
`TruncateTableCommand` does for a v1 partitioned table. Under filesystem
partition discovery the
directory is the answer, and truncating clears the partition levels below
the table root.
On the Spark side `PaimonFormatTable` implements `TruncatableTable` and
overrides
`SupportsAtomicPartitionManagement.truncatePartition(s)` rather than taking
the statement over in
`PaimonStrategy`: `TruncatePartitionExec` already expands a partial spec and
refreshes the cache,
and managed tables reach truncation through the same interface. Truncating a
named partition the
table does not have is an error, reported per entry point as Spark's
`TruncateTableSuiteBase`
expects.
Flink has no `TRUNCATE TABLE ... PARTITION`, so `FlinkFormatTableSink`
implements only whole-table
truncation.
### Tests
- `FormatTableCommitTest`: whole-table truncate under filesystem discovery
and on a catalog-managed
table (registered partitions emptied, an unregistered directory
untouched), an unpartitioned
table, named partitions, a leading-prefix spec, and the value-only default
partition; staging
trees survive throughout.
- `FormatTablePartitionManagementTest`, `CatalogManagedPartitionTest`:
registrations survive,
unregistered partitions are refused, a `..` value is rejected before
anything is deleted, and an
end-to-end truncate through the REST catalog.
- `FormatTableTestBase`: `TRUNCATE TABLE`, `TRUNCATE TABLE ... PARTITION`
with a full and a partial
spec, and a partition the table does not have.
- `FormatTableITCase`: `TRUNCATE TABLE` on a partitioned and an
unpartitioned Format Table in Flink.
### API and Format
No format change. `PaimonFormatTable` gains `TruncatableTable` and
`FlinkFormatTableSink` gains
`SupportsTruncate` — both engine-side interfaces already implemented for
managed tables.
`format-table.implementation = engine` is unchanged and still does not
support truncation.
### Documentation
`docs/docs/spark/sql-write.md` and `docs/docs/flink/sql-write.mdx`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
