sundapeng opened a new pull request, #8785:
URL: https://github.com/apache/paimon/pull/8785
### Purpose
Partitions of a Format Table managed by a REST catalog (#8750) are currently
listed with only a partition name prefix pattern pushed down; any other
partition predicate (ranges, non-leading equalities, IN) falls back to listing
all partitions and filtering on the client, so the number of round trips grows
linearly with the partition count.
This PR introduces the REST API definition for filtered partition listing:
```
POST
/v1/{prefix}/databases/{database}/tables/{table}/partitions/list-by-filter
```
Design notes:
- The request carries a structured JSON predicate (`PartitionFilter`):
`column` / `literal` leaves combined by `and` / `or` / `not` / `isNull` /
`equal` / comparison / `in` nodes. Literal values are transported as strings
and cast by `valueType` on the server. The shape follows the JSON predicates of
the Delta Sharing protocol (plus an `in` op, so a large values list does not
need an `or`-of-`equal` expansion), rather than an SQL-like filter string: the
server side has no SQL parser, and string filters have well-known escaping and
typing pitfalls.
- **The server filters in a best-effort way**: a node the server does not
understand counts as always-true, so the response is a superset of the matching
partitions and never misses one, and the wire format can evolve without version
negotiation. Clients keep applying the predicate locally. Note this is a
different contract from the auth row filter (`AuthTableQueryResponse#filter`):
that one is a mandate the engine must fully enforce, while this filter is only
a pushdown hint, so a server ignoring part of it stays correct.
- Pages may be sparse: a page may carry fewer elements than `maxResults`, or
even none, while a non-empty `nextPageToken` means there are more partitions to
check.
- The op set covers everything Spark pushes down for partition pruning (see
`SparkV2FilterConverter`): equality, comparisons, IN, IS (NOT) NULL, AND / OR /
NOT; null-safe equality is already expanded by the converter into `isNull` /
`isNotNull + equal`. String functions (startsWith etc.) are not part of the
wire format; the existing `partitionNamePattern` (also accepted by this
endpoint, combined as a conjunction) continues to serve leading-key prefixes.
- Existing routes are unchanged. `Catalog#listPartitionsByFilterPaged` is a
default method throwing `UnsupportedOperationException`; only `RESTCatalog`
implements it, and callers can fall back to the plain listing against servers
without this endpoint.
Marked as WIP: wiring the format table scan (translating
`PartitionPredicate` into the wire filter, with fallback for older servers)
will be a follow-up.
### Tests
- `RESTCatalogTest#testListPartitionsByFilterPaged`: range / `in` / pattern
conjunction, and the superset behavior for an unknown op, against the mock REST
server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
