sundapeng opened a new pull request, #8730:
URL: https://github.com/apache/paimon/pull/8730
## Relationship
Third part of splitting #8713; stacked on the DDL part #8729 (the diff shown
here includes the
two previous commits until they merge — please review only the `[spark] Add
managed
Format Table partition administration` commit). Marked draft until the DDL
part lands.
## Purpose
Administration and repair on top of the DDL part: two procedures, `MSCK
REPAIR TABLE`
and `SHOW PARTITIONS`.
## Main changes
- `sys.list_format_table_partitions(table, where?, limit?, page_token?)`:
paged listing.
Filtered pagination carries a fingerprint token (predicate + page content
hashes) so a
continuation token cannot be silently replayed against a different `where`
or a
shifted page — the failure mode is an explicit error, never a wrong page.
- `sys.sync_format_table_metadata(table, target, mode=ADD|DROP|SYNC, scope?,
dry_run)`:
filesystem-to-catalog reconciliation. `dry_run` defaults to **true**;
discovery uses
raw directory values (round-trips with writer-registered values) and
**fails on a
mid-scan listing error** instead of producing a truncated diff (a DROP
diff from a
truncated listing would unregister live partitions). Applies diffs in
bounded batches;
both directions converge on rerun. Spark cache is refreshed after any
attempted
mutation, including ambiguous failures.
- `MSCK REPAIR TABLE [ADD|DROP|SYNC] PARTITIONS` reuses the sync engine
(bare MSCK =
ADD, Hive semantics; DROP is metadata-only). Unmanaged Format Tables and
native
Paimon tables keep Spark's rejection.
- `SHOW PARTITIONS` backed by the catalog, capped with a pointer to the
paged procedure.
## Behavior notes
The fingerprint token machinery in the list procedure is deliberately
strict; we
preferred an explicit error over any chance of silently misaligned pages
when a `where`
filter is combined with pagination. Feedback on simplifying this contract is
welcome.
## Tests
Procedure suite (36), MSCK end-to-end against a mock REST server (10),
planning suite
extension (30), SHOW partitions exec tests (8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
