FANNG1 opened a new issue, #11108:
URL: https://github.com/apache/gravitino/issues/11108
### Describe the feature
I would like to propose adding a **commit API** to the Gravitino Lance REST
server, and preferably also **table version APIs**.
This would make the Lance write path namespace-aware on the server side, so
Gravitino can:
1. enforce authorization on commits / writes,
2. validate and observe the latest committed table metadata,
3. retrieve the latest table schema from committed Lance manifests,
4. unblock Gravitino Web UI from showing Lance table schema reliably.
### Background
In Lance, the actual data write path is split between object storage and
namespace-managed version registration.
For namespace-managed tables, the flow is roughly:
1. the client resolves the table through the namespace (`declare_table` /
`describe_table`),
2. data files and manifest are written to object storage,
3. the final table version is committed / registered through
namespace-managed manifest APIs.
Relevant Lance code paths:
- `rust/lance/src/dataset.rs`
- `write_into_namespace(...)`
- for `Create`, Lance calls `declare_table(...)`
- for `Append` / `Overwrite`, Lance calls `describe_table(...)`
- `rust/lance/src/dataset/builder.rs`
- `DatasetBuilder::from_namespace(...)`
- when `managed_versioning == true`, Lance installs an
`ExternalManifestCommitHandler`
- `rust/lance/src/io/commit/namespace_manifest.rs`
- namespace-backed external manifest store
- `get_latest_version(...)` -> `list_table_versions(...)`
- `get(...)` -> `describe_table_version(...)`
- `put(...)` -> `create_table_version(...)`
- `rust/lance-namespace/src/namespace.rs`
- `create_table_version(...)`
- `list_table_versions(...)`
- `describe_table_version(...)`
So from Lance’s perspective, namespace-managed commit/version APIs are part
of the expected integration model.
### Current problem
Without a server-side commit API in Gravitino Lance REST server:
- the write path cannot be fully governed by Gravitino,
- authorization is effectively limited because the final commit/version
registration is not mediated by the REST server,
- Gravitino cannot reliably know the latest committed Lance table version,
- Gravitino Web UI cannot easily resolve and display the latest Lance table
schema from the committed manifest.
### Proposal
Add a **commit-oriented API** to the Gravitino Lance REST server, ideally
aligned with Lance namespace-managed versioning.
At minimum, a commit API should allow the client to submit:
- table identifier
- target version
- manifest path (or staged manifest path)
- manifest size
- optional ETag
- manifest naming scheme
The server should then:
1. authorize the operation,
2. validate the target table and commit request,
3. atomically register the new table version,
4. return the committed version metadata,
5. make the latest version discoverable for readers / UI.
### Suggested APIs
#### 1. Commit API
Either of the following shapes would work:
- `POST /api/.../tables/{table}/commit`
- or `POST /api/.../tables/{table}/versions`
The semantics should be similar to Lance namespace
`create_table_version(...)` with conflict detection / put-if-not-exists
behavior.
#### 2. Table version APIs
Please also consider exposing:
- `GET /api/.../tables/{table}/versions`
- `GET /api/.../tables/{table}/versions/{version}`
- optionally `GET /api/.../tables/{table}/versions/latest`
These APIs would allow Gravitino services and Web UI to:
- inspect the latest committed version,
- locate the latest manifest,
- derive the latest Lance schema,
- support debugging and lineage/version browsing.
### Why this matters
This is useful not only for correctness, but also for product capabilities:
- **Security / governance**: enforce write authorization at commit time
- **Consistency**: always resolve the latest committed table version
- **Observability**: expose manifest/version metadata for diagnosis
- **UI support**: Gravitino Web UI can finally display Lance table schema
from the latest committed manifest
- **Better Lance integration**: aligns Gravitino Lance REST server with
Lance namespace-managed commit flow
### Additional note
If adding a single “commit” endpoint is too narrow, another acceptable
direction would be to implement the Lance namespace-managed version APIs in the
REST server, namely:
- `create_table_version`
- `list_table_versions`
- `describe_table_version`
This would align very naturally with Lance’s current integration model.
### References
Lance namespace/write flow references:
- `rust/lance/src/dataset.rs`
- `rust/lance/src/dataset/builder.rs`
- `rust/lance/src/io/commit/namespace_manifest.rs`
- `rust/lance-namespace/src/namespace.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
