sundapeng opened a new pull request, #8713:
URL: https://github.com/apache/paimon/pull/8713
### Purpose
Follow-up of #8707. That PR added REST-catalog partition metadata APIs
(create/drop) for Format Tables; this PR makes the catalog the partition source
of truth end to end and integrates the partition lifecycle with Spark SQL.
- Setting `metastore.partitioned-table=true` on a Format Table in a REST
catalog marks its partitions as catalog-managed: scans list partitions from the
catalog instead of filesystem discovery, partition predicates are pushed down
as a prefix pattern, and a missing directory for a registered partition fails
with a partition-drift error instead of silently returning partial data.
- Writes register the produced partitions in the catalog on commit. If
registration fails after data files were committed, the files are preserved and
the error points to MSCK repair; abort() no longer removes files once
registration has started.
- Spark SQL: `ALTER TABLE ... ADD/DROP PARTITION` (DROP honors `IF EXISTS`,
resolves partial specs, unregisters metadata and then deletes partition
directories), `SHOW PARTITIONS` (bounded by
`spark.paimon.format-table.show-partitions.max-results`), `MSCK REPAIR TABLE
... [ADD|DROP|SYNC] PARTITIONS`, and procedures
`sys.list_format_table_partitions` (paged listing with partition predicate) and
`sys.sync_format_table_metadata` (reconciliation with `dry_run` preview).
- Managed Format Tables require a REST catalog. Create/alter/replace
validate the effective options including catalog `table-default` options;
non-REST catalogs must remove the option before upgrading, and `ALTER TABLE ...
RESET ('metastore.partitioned-table')` keeps working on such tables as the
migration path.
### Tests
- Core: managed scan (paging, pattern pushdown, drift detection),
commit-time registration and failure preservation, catalog validation, API
compatibility tests.
- Spark: partition DDL planning and management suites, SHOW PARTITIONS,
metadata procedures, and MSCK repair end-to-end against the in-repo REST
catalog server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
