SteNicholas opened a new pull request, #165:
URL: https://github.com/apache/paimon-cpp/pull/165
### Purpose
Linked issue: #164
Add a REST catalog implementation, selected via the `metastore=rest` option,
that talks the Paimon REST catalog open API.
**New components**
- `RestHttpClient` — blocking libcurl client with exponential-backoff
retries:
- 429/503 responses are retried for all methods, transport errors only for
idempotent methods via an explicit allowlist of transient failures
(name-resolution and connect failures, timeouts, TLS failures and a server
closing the connection without responding are never retried).
- A `Retry-After` response header — delta-seconds or HTTP-date form,
parsed locale-independently — takes precedence over the backoff when it yields
a positive delay. Every retry sleep is bounded by a per-sleep cap and an
overall retry budget; a `Retry-After` beyond either bound stops retrying rather
than sleeping less than the server requested.
- Redirects keep the method and body of POST/DELETE requests
(`CURLOPT_POSTREDIR`) and are restricted to http(s) targets.
- Header names must be valid HTTP tokens and header values must not
contain CR/LF/NUL; a violating header fails the request before anything is sent.
- Easy handles are pooled and reset between requests, so a request reuses
the connection opened by the previous one instead of paying a TCP and TLS
handshake every time; TLS below 1.2 is refused and requests carry a default
`paimon-cpp` user agent that a `header.User-Agent` option overrides.
- Per-request logging carries the request id (debug on success, warn on a
retry or a final failure).
- `RestApi` — HTTP + JSON layer:
- `/v1/config` option merging (overrides > client options > defaults),
`header.` options sent as request headers, and paged listing.
- Error mapping to `Status` (404 → `NotExist`, 409 → `Exist`, 400 →
`Invalid`, 501 → `NotImplemented`) that prefers the code of the parsed error
body over the http status.
- Sensitive server messages are redacted, and response bodies are never
echoed into error messages (they may carry credentials), so a body that is not
an error object is reported as unparsable, distinct from an error object
carrying no message.
- Errors carry the request id (`x-request-id`, falling back to any
`request-id` header) and attach a `RestErrorDetail` with the mapped code, so
callers can distinguish e.g. authentication failures programmatically.
- `RestCatalog` — database/table create/drop/rename/list/get, snapshot
listing, `table-default.` option defaults on table creation, and system/branch
table checks:
- The server table response is converted into a `TableSchema` with the
computed `highestFieldId`: ids at or above
`SpecialFieldIds::SYSTEM_FIELD_ID_START` are excluded, a duplicated field id at
any nesting level is rejected, and a missing or wrong-typed
`partitionKeys`/`primaryKeys`/`options` member fails instead of silently
defaulting.
- Branch identifiers are passed through to the server, which resolves the
branch and returns the branch's own schema (the default branch `main` maps
case-insensitively to the bare table); snapshots of a branch are listed under
the branch object name.
- The `sys` database serves the local global system tables like
`FileSystemCatalog`.
- Bear token authentication provider (`token.provider=bear`, the protocol's
historical spelling of "bearer").
**Shared code**
- The pieces overlapping with the object-store file systems are unified
under `common/utils`: the generic HTTP client moved there from `common/fs` and
is shared by the S3 file system and the REST catalog, `UrlUtils` carries both
URL-encoding flavors (form-urlencoded for the REST api, RFC 3986 for S3) with
the S3 client reusing it, and libcurl detection in CMake is a single
`find_package(CURL)` gated on `PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST`.
- `CatalogUtils` holds the system-database and system/branch table checks
shared by `FileSystemCatalog` and `RestCatalog`, with the check order and
messages aligned with the Java `CatalogUtils` (`Cannot 'createTable' for system
table ...`). `FileSystemCatalog` now also rejects branch identifiers on
create/drop/rename — a branch identifier resolves to the main table directory,
so dropping `t$branch_b` would otherwise delete the whole table.
- The new `SensitiveConfigUtils` centralizes credential redaction with the
Java `SensitiveConfigUtils` key markers and masking rule (matched on the key
lower-cased with separators removed; a key naming a true secret is masked as a
whole, while an identifier-like key such as `dlf.access-key-id` keeps at most a
four-character tail of a long enough value). `sys.catalog_options` masks every
row through it so credentials such as the bearer token never surface to users
who can query that table, and a server error message carrying a secret marker
is redacted as a whole.
- `RestUtil::ExtractRequestId` and `RestAuthParameter::Create` likewise hold
the request-id lookup and the query-parameter encoding in one place, so the log
line and the error message report the same id and no call site can sign an
unencoded parameter.
**Build**
libcurl is used from the system rather than bundled, so the new CMake option
`PAIMON_ENABLE_REST` defaults to `OFF`, consistent with the other optional
components that need something outside the bundled third-party toolchain
(jindo, lance, lumina, lucene, tantivy). CI enables it explicitly in
`ci/scripts/build_paimon.sh` and installs the libcurl development package, so
the REST code and its tests are still built and run on every job. With the
option off, `Catalog::Create` reports that `metastore=rest` requires a build
with `PAIMON_ENABLE_REST=ON`.
Follow-ups (not in this PR): DLF signing, data-token file IO, paged listing
parameters (`maxResults`/name patterns), and the remaining endpoints
(alter/commit/views/partitions/tags/functions).
### Tests
New `rest_test` binary (62 cases, registered under the `unittest` label so
it runs in CI), all against an in-process mock HTTP/REST server:
- `RestHttpClientTest`: uri normalization, query encoding, a pooled handle
carrying no method or body over from the previous request, retry classification
(429/503 retried, 404 and non-allowlisted transport errors such as a refused
connection or a redirect loop not, truncated responses retried only for
idempotent methods), `Retry-After` precedence in both forms plus the per-sleep
and overall budget bounds, retry exhaustion, redirect following with POST
keeping its method and body, rejection of invalid header names/values, and
transport errors omitting the url and query.
- `RestUtilTest` / `ResourcePathsTest` / `RestMessagesTest`, each next to
its implementation: prefix extraction and the request-id lookup including the
gateway-header fallback, resource path building with url-encoded segments, JSON
round trips of all request/response messages, and config merge with null-value
filtering (a key present in defaults, client options and overrides at once,
plus the null-override and null-default paths).
- `RestCatalogTest`: client-side option validation (missing uri/token,
unsupported token provider), end-to-end catalog operations including
pagination, `ignore_if_exists` / `ignore_if_not_exists` paths, client/server
headers and the json content type, `table-default.` defaults, system tables,
branch tables resolved by the server, broken-schema rejection (missing and
wrong-typed `partitionKeys`/`primaryKeys`/`options` members), a non-empty
partition-key round trip, server errors surfacing as errors instead of "does
not exist", and snapshot listing with sorting.
- `RestApiErrorTest`: http-status-to-`Status` mapping including the
request-id fallback, redaction and the `RestErrorDetail` code.
The shared pieces are covered next to their implementations:
- `UrlUtilsTest`: both encoding flavors, including that a literal `%` is
encoded even when `/` is preserved, which is what the S3 client relies on.
- `SensitiveConfigUtilsTest`: key classification, whole-value and
tail-preserving masking, and free-form text redaction.
- `HttpClientUtilTest` in the common utils suite.
- `TableSchemaTest.TestComputeHighestFieldId`: the highest-field-id
computation.
- `SnapshotTest.TestToSnapshotInfo`: the snapshot-to-info conversion.
- `FileSystemCatalogTest.TestBranchIdentifierRejectedForTableOperations`:
the branch rejection.
- The `sys.catalog_options` integration test: credential-carrying options
(token, access-key secret, account key, credential, SAS) are masked in every
emitted row, and an access-key id keeps only its four-character tail.
### API and Format
- New public header `include/paimon/catalog_options.h` with a
`PAIMON_EXPORT`ed `CatalogOptions` struct holding the catalog-level option keys
`METASTORE`, `URI`, `TOKEN`, `TOKEN_PROVIDER` and the
`TABLE_DEFAULT_OPTION_PREFIX` (`table-default.`) key prefix; table-level keys
stay in `Options` (`defs.h`), mirroring the CatalogOptions/CoreOptions
separation of the Java implementation.
- `RestCatalog` implements the `Catalog` interface including `GetOptions()`,
which returns the client options merged with the server-side `/v1/config`.
- `Catalog::Create` now dispatches on the `metastore` option (`filesystem`
remains the default; unknown values are rejected). For `metastore=rest`,
`root_path` is not a filesystem path but the warehouse (instance) name
registered on the REST server, documented on the API.
- No storage format changes. The wire protocol is the Paimon REST catalog
open API.
### Documentation
- `docs/source/user_guide/catalog.rst` now documents both metastores: the
existing filesystem metastore section and a new REST Catalog section covering
the `PAIMON_ENABLE_REST` build requirement, the `root_path` semantics
(warehouse/instance name), the `CatalogOptions` keys (`metastore`, `uri`,
`token.provider`, `token`, `table-default.<key>`) and a configuration example,
replacing the note that claimed REST catalog support is future work. It also
states which parts of the REST catalog are not implemented yet (altering a
database or a table, views, functions, partitions, tags, branch management,
consumers and the `dlf` token provider), so the guide is not read as promising
them.
- `docs/source/building.rst` lists `-DPAIMON_ENABLE_REST=ON` under the
optional components, including its libcurl requirement.
### Generative AI tooling
Generated-by: Claude Code (claude-fable-5, claude-opus-5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
