sundapeng opened a new pull request, #8728:
URL: https://github.com/apache/paimon/pull/8728
## Relationship
First part of splitting #8713 into independently reviewable pieces (this PR
is the
engine-independent foundation; the Spark DDL and administration parts follow
in separate
PRs). The content is identical to the corresponding files of #8713.
## Purpose
Follow-up of #8707 (catalog-managed Format Table partitions, REST contract +
server side).
This PR makes the engine-independent client side actually use the managed
partition
metadata: scans plan from the catalog instead of walking the directory tree,
and batch
writes register the partitions they touched as part of the commit.
## Main changes
- `Catalog#supportsManagedFormatTablePartitions` capability (default false;
REST catalog
returns true), mirroring the existing capability-method pattern.
- `ManagedFormatTableScan`: partitions come from paged catalog listing with a
partition-name prefix pattern pushed down; no filesystem fallback. A
registered
partition whose directory is missing reads as **empty with a WARN** (Hive
semantics —
an `ADD PARTITION` before the first insert is a legitimate empty
partition), while a
catalog with no partitions but a non-empty table directory warns that a
metadata sync
is needed. Catalog metadata is validated defensively (corrupt/duplicate
specs).
- `FormatTableCommit`: batch writes register touched partitions with the
idempotent
create contract, in batches of the catalog page size (one backfill commit
can touch
tens of thousands of partitions). Registration failure fails the job; data
files are
never silently dropped.
- Create/alter validation: the managed option is only accepted on internal
Format Tables
on catalogs with the capability; existing tables carrying the option on
other catalogs
(e.g. Hive) still load, downgraded to unmanaged with a WARN, so they can
be repaired
with `ALTER TABLE ... RESET/UNSET`. `CREATE TABLE IF NOT EXISTS` stays
idempotent.
- `PartitionPathUtils` hardening shared by the scan/discovery paths.
## Behavior notes reviewers should be aware of
1. **Tightened shared utils**: `generatePartitionPathUtil` now rejects
null/empty
partition values (they cannot form a path component), and value-only spec
extraction
unescapes directory names so specs round-trip with what writers register.
Existing
tables with `%xx`-escaped value-only directories will surface unescaped
values.
2. **In-process listing cache**: partition listings are cached for up to 30s
per JVM
(invalidated on same-process mutations, including attempted-but-ambiguous
ones).
Cross-JVM readers may observe metadata up to 30s stale; same-process
reads are
immediately consistent.
3. **Reads-empty drift semantics** (vs failing the scan) was chosen
deliberately: an
empty registered partition is indistinguishable from an out-of-band
directory
deletion, and Hive treats both as empty. Repair paths stay fail-closed
(see the
administration part).
## Tests
Core: managed scan (19), commit registration (7), catalog
validation/fallback (core +
FileSystem/Jdbc/Hive/REST catalog suites, 354 tests in paimon-core plus 53 in
HiveCatalogTest), compatibility test pinning the Catalog binary surface,
path-utils
tests. Spark write glue: batch write test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
