sundapeng opened a new pull request, #8751: URL: https://github.com/apache/paimon/pull/8751
## What Gives a Format Table whose partitions come from the catalog (#8750) the partition DDL that implies: ```sql ALTER TABLE t ADD PARTITION (dt='2026-07-21'); ALTER TABLE t DROP PARTITION (dt='2026-07-21'); MSCK REPAIR TABLE t; SHOW PARTITIONS t; ``` `SHOW PARTITIONS` needs no special support — it already asks the scan, which now asks the catalog. ## Semantics worth reviewing **ADD** forwards the batch and its `IF NOT EXISTS` flag to the catalog without looking anything up first, so rejecting a duplicate stays the catalog's decision and the batch stays atomic. It then creates the partition directory, so a freshly added partition reads as empty rather than as a missing path — Hive's `ADD PARTITION` semantics. **DROP** unregisters before deleting. A deletion that fails therefore leaves the partition unregistered rather than half-deleted and still readable. Only partitions the catalog actually knows are touched, so data merely waiting to be registered is never removed. Partial specs expand to the registered leaves they cover. **MSCK REPAIR TABLE** diffs the directories against the registration: `ADD` registers what exists on disk but not in the catalog, `DROP` unregisters what the catalog has but the filesystem does not. It never touches data files, and it refuses to act on a listing it could not complete — a transient LIST failure must not look like "these partitions are gone". Partitions are discovered from the raw directory names rather than through a scan, because a scan casts values to their column type and back (`month=01` becomes `1`) and would then diff against something that never round-trips to the real directory. **Failures keep the reason.** Refreshing Spark's cached plans after partition DDL used to sit in a plain `finally`, where a refresh that threw replaced the failure that actually mattered. The operation's failure now propagates with the refresh's attached to it. A Format Table whose partitions come from the filesystem keeps its previous behaviour and says so when asked for this DDL. ## Tests paimon-spark-common: 33, covering the repair engine's diff, ordering, batching and fail-closed behaviour, and the dynamic-option rules. paimon-spark-ut: 47, covering planning and rewrite routing, ADD/DROP through the partition manager including path-escape rejection and ambiguous catalog responses, MSCK end to end against a REST catalog with fault injection, and `ALTER TABLE ... ADD PARTITION` succeeding through SQL — registered, directory created, listed by `SHOW PARTITIONS`, read as empty, and rejected by the catalog on a duplicate. ## Relationship Stacked on #8750 — its commit is included here and should be reviewed there. Splits #8713. Supersedes #8729 and #8730, whose review comments are addressed across this and #8750. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
