sundapeng opened a new pull request, #9295:
URL: https://github.com/apache/paimon/pull/9295
### Purpose
A catalog-managed format table registers the partitions a commit wrote. It
has no snapshot, so the
channel data tables use to report statistics, `commitSnapshot`, does not
exist for it, and there is
nowhere else for the numbers to go. This adds them to the registration
itself.
`CreatePartitionsRequest` gains an optional `partitionStatistics` list and a
`statisticsMode`.
Registration and statistics then land in one request and one server-side
transaction, which is the
point: "the statistics failed but the partition registered" is not a state
anyone has to handle, and
no extra round trip is paid for the numbers. The alternative, a separate
endpoint, buys only the
ability to report statistics without registering, which is exactly the state
worth not having.
The mode says how a report combines with what the server holds, per field:
`ADD` sums the three
counts and takes the later creation time, `SET` replaces. Reporting a field
as unknown leaves the
stored value alone under either mode, so a reporter that can measure only
some fields never erases
the others. Statistics never create or remove a partition row, whatever the
numbers say. The mode
travels as a string rather than an enum so a value the server has no name
for deserializes instead
of failing the request it arrived in: such a report is dropped and the
partitions are still
registered.
A non-empty `ADD` report declares itself unsafe to retry (#9122), so an
automatically replayed POST
cannot count the same increment twice.
This is the first of four PRs. The next three add a reporter each: a format
table commit (S4), MSCK
REPAIR TABLE (S5) and ANALYZE TABLE (S6). This one only opens the channel;
nothing in this PR sends
statistics.
### Two positions worth stating
`ADD` is not idempotent, and this does not pretend otherwise. The largest
source of double counting
is transport-level replay, and that one is closed rather than tolerated:
only a non-empty `ADD`
answers `isRetrySafe()` with false. What is left is job-level retry, which
rewrites the data files
too, so duplicated statistics come with duplicated data. A dedup token in
front of every commit
would give duplicated data non-duplicated statistics, which is a new
inconsistency rather than a
fix. Convergence is a later full `SET` over the same partition, which S5 and
S6 produce.
The response carries no acknowledgement. A missing observation is something
consumers can handle, a
wrong one is not, and an acknowledgement would be a second contract to keep
in step with the first.
`CreatePartitionsResponse` is byte for byte what it was before this PR.
### Compatibility
Both directions work and neither errors. A new client against an old server:
the fields are ignored,
the statistics are dropped and the columns stay unknown. An old client
against a new server: no
statistics arrive and the columns stay unknown. The existing three-argument
`createPartitions` keeps
working and keeps sending exactly the request it sends today, so a caller
that reports nothing does
not change the shape the server sees. The Python client mirrors these DTOs
since #9198 and sends
neither new field, which is the old-client case.
### Tests
- `HttpClientRetrySafetyTest`: which requests declare themselves unsafe to
retry, that only a
non-empty `ADD` report does, and that the flag never reaches the wire.
- `RESTApiJsonTest`: the request shape, including a mode the server has no
name for surviving
deserialization.
- `MockRESTCatalogTest`: reporting against the mock server end to end. `ADD`
accumulating, `SET`
replacing, unknown fields leaving the stored value alone, statistics for a
partition the request
does not register being dropped, a report that only partly matches not
being applied at all, and
an unreadable mode still registering. These assertions are the contract
the modes are described
by.
- `CatalogFormatTablePartitionManagerTest`: what the manager sends.
Statistics riding in the request
of their own partitions, a batch that reports nothing sending an empty
list rather than null, and
the rejections of malformed reports.
- `CachingCatalogTest`, `DelegateCatalogTest`: the new overload forwarding
through and invalidating
the partition cache.
### API and Format
`Catalog.createPartitions` gains an overload taking the statistics and the
mode, and the existing
signature forwards to it. `CreatePartitionsRequest` gains two optional
fields. No format change.
### Documentation
`docs/static/rest-catalog-open-api.yaml` carries both fields, including what
happens to a mode the
server does not know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
