sundapeng opened a new pull request, #8750:
URL: https://github.com/apache/paimon/pull/8750
## What
A partitioned Format Table finds its partitions by walking the table
directory.
That is how Hive works and it stays the default, but it means a directory
appearing under the table is a partition whether or not anyone meant it to
be,
and listing a large table costs a full traversal of object storage.
This adds a second answer, for an internal Format Table in a REST catalog:
```sql
CREATE TABLE t (id INT, dt STRING) USING CSV PARTITIONED BY (dt)
TBLPROPERTIES ('format-table.partition-source' = 'rest');
```
The catalog then owns the partitions. A scan reads the ones registered there
and a batch write registers the ones it wrote, so a directory nobody
registered
is not part of the table, and an empty catalog means an empty table. Reads
never
fall back to the filesystem — that would defeat the point. A registered
partition whose directory is missing reads as empty, matching Hive, where
`ADD PARTITION` before the first insert is ordinary.
Engine support for the partition DDL this makes possible (`ADD`/`DROP
PARTITION`,
`MSCK REPAIR TABLE`) follows in a separate PR.
## Why a new option rather than `metastore.partitioned-table`
`metastore.partitioned-table` means something else: it mirrors a Paimon
table's
partitions into the Hive metastore, and the docs recommend setting it.
Setting
it as a catalog-level `table-default.*` is a reasonable thing to do — and it
would then land on every Format Table created in that catalog and silently
decide where their partitions come from.
`format-table.partition-source` is scoped to Format Tables, joins the
existing
`format-table.*` family, and is an enum so the two answers name themselves.
`metastore.partitioned-table` is left exactly as it was.
## Failing rather than degrading
The option names one source and one only. Asking for it where it cannot be
served — outside a REST catalog, on an external table, under
`format-table.implementation = engine` — fails the load. The alternative,
handing back a table whose options say one thing and whose partitions come
from
another, is worse: it cannot be noticed. Nothing inherits the option by
accident, so there is no table this breaks by surprise.
Whether a table's partitions come from the catalog is answered in exactly one
place — whether loading it produced a partition manager — so core and the
engines cannot disagree about it.
## Shape
`FormatTable` carries a `FormatTablePartitionManager`: list by a leading
prefix
of raw partition values, look partitions up by name, register, unregister.
Paging, per-request batching and pattern encoding are the implementation's
business, not the caller's. It holds a `CatalogLoader`, not a catalog, and
creates one catalog per operation and closes it, so the table stays
serializable without owning a client — the same shape as `SnapshotLoader` and
`BranchManager`. A strict create is never split across requests, since that
would break the whole batch being rejected together.
## Server side
Nothing in the REST protocol changes: the partition endpoints, DTOs and paths
added by #8707 are untouched. A server deciding whether a Format Table's
partitions are catalog-owned should read `format-table.partition-source` from
the table properties instead of `metastore.partitioned-table`.
## Tests
paimon-core: 273 in the partition, commit, scan, catalog assembly, path utils
and REST suites, including the partition manager's paging, repeated-token
detection, prefix pushdown and client-side re-filtering, per-request
batching,
catalog close on both the success and failure paths, and Java serialization.
## Relationship
Splits #8713. Supersedes #8728, #8729 and #8730, which reviewed an earlier
shape
of this work; the comments there are addressed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
