GitHub user yuqi1129 added a comment to the discussion: [DISCUSS] Is a Paimon
REST Catalog Server still necessary when Gravitino is used as the unified
access layer?
Thanks for the thoughtful questions. I think the Paimon REST Catalog Server
and Gravitino address different layers of the architecture.
One clarification first: the request path shown above is conceptually correct
for catalog and metadata operations, but not every Paimon operation or data I/O
request necessarily passes through the Gravitino server. The current Gravitino
Flink and Spark connectors route catalog/DDL operations through Gravitino,
while native Paimon catalog and table implementations are still used for
format-specific behavior and data I/O.
> 1. What additional value does a Paimon REST Catalog Server provide?
Its primary value is protocol compatibility. It exposes the standard Paimon
REST API, allowing Paimon-native clients to connect without depending on a
Gravitino-specific connector or API.
It may also expose Paimon-specific capabilities that do not fit naturally into
Gravitino's unified metadata model, such as format-specific metadata, snapshot,
branch, partition, and commit operations. The Paimon REST specification also
allows such technology-specific logic to remain on the server side, while
clients remain relatively lightweight and decoupled from the backend
implementation.
Therefore, it is an additional access protocol rather than a replacement for
Gravitino.
> 2. Which deployment model should be recommended?
There is no single deployment model for every use case. If all engines can use
the Gravitino connectors and the supported Gravitino metadata model is
sufficient, the simpler deployment is:
```text
Flink / Spark
-> Gravitino connector
-> Gravitino
-> Paimon filesystem / Hive / JDBC backend
```
In that case, a separate Paimon REST server is not required. If users need
standard Paimon REST compatibility, another valid deployment is:
```text
Flink / Spark / other Paimon REST clients
-> Paimon REST endpoint integrated with Gravitino
-> Paimon backend
```
In the second model, Gravitino can remain the catalog and governance control
plane, while the Paimon REST endpoint provides the format-specific protocol.
> 3. Is the Paimon REST server mainly useful for clients that do not integrate
> with Gravitino directly?
Yes, that is probably its clearest use case. It allows clients that already
support the Paimon REST protocol to access Paimon catalogs without adopting a
Gravitino-specific connector.
Other benefits include language-independent access, decoupling client and
server versions, keeping Paimon-specific logic on the server side, and
potentially applying Gravitino-managed authorization, auditing, and credential
vending to standard Paimon clients.
> 4. Should a Paimon REST server behind Gravitino be treated only as an
> internal backend?
Not necessarily. We should distinguish two directions:
- Existing rest backend: Gravitino -> external Paimon REST server
- Proposed Gravitino-backed endpoint: Paimon REST client -> Paimon REST
endpoint integrated with Gravitino
In the first case, the external Paimon REST server can be treated as an
internal backend. If Gravitino is intended to be the mandatory governance entry
point, clients should not bypass it unless the external server enforces
equivalent authorization and auditing policies.
In the second case, the Paimon REST endpoint is intentionally a public,
standard-compatible entry point. It should authenticate Paimon REST clients and
integrate their identities and operations with Gravitino's catalog management,
authorization, and auditing capabilities. Thus, the endpoint can be parallel to
the Gravitino REST API from the client's perspective, while still being
governed by Gravitino internally.
> 5. Is there a community plan to implement a Gravitino-backed Paimon REST
> server?
As far as the current code and discussions show, there is no accepted
implementation plan or roadmap yet. There is an open feature discussion in
#7663 (https://github.com/apache/gravitino/issues/7663), but the community has
also raised concerns about the maturity of the Paimon REST specification,
actual user demand, and the long-term cost of maintaining another
format-specific REST implementation. Before implementation, I think we should
first agree on:
- The concrete clients and use cases that cannot be covered by the existing
Gravitino connectors.
- The minimum Paimon REST API scope that should be supported.
- Whether the service is standalone, an auxiliary Gravitino service, or both.
- How authentication identities and Paimon operations map to Gravitino
authorization and audit events.
- Which operations go through Gravitino's unified metadata APIs and which must
operate directly on the Paimon backend.
- The expected compatibility and maintenance policy as the Paimon REST
specification evolves.
In summary, a Paimon REST server is not a mandatory extra hop when using
Gravitino. It is an optional, format-specific protocol endpoint whose main
value is enabling standard Paimon REST clients to benefit from
Gravitino-managed catalogs and governance.
GitHub link:
https://github.com/apache/gravitino/discussions/12161#discussioncomment-17791993
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
