markhoerth opened a new pull request, #12285:
URL: https://github.com/apache/gravitino/pull/12285
### What changes were proposed in this pull request?
Rewrites the five Lance and generic lakehouse catalog documentation pages
around the model they never stated, and corrects claims that turned out to be
wrong when checked against source. 681 insertions, 1351 deletions.
| File | Lines | Was |
|------------------------------------|-------|-----|
| `lance-rest-service.md` | 299 | 425 |
| `lakehouse-generic-lance-table.md` | 219 | 369 |
| `lance-rest-integration.md` | 194 | 299 |
| `lakehouse-generic-catalog.md` | 169 | 194 |
| `lakehouse-generic-delta-table.md` | 134 | 314 |
#### Corrections verified against source
- Alter is partially supported (drop column, rename column, add index), not
"Not support now" (`LanceTableOperations.handleLanceTableChange`)
- Index creation is supported through alter; the page listed indexes under
"Feature Limitations" (same)
- Name `lance.schema-refresh-mode`, which the refresh section described at
length but never identified (`LanceConstants`)
- Add `lance.declared` to the table properties, which was missing
(`LanceTableDelegator.tablePropertyEntries`)
- `Map` is supported by Lance; the page said it was not (lance
`docs/src/format/table/schema.md`)
- `Union` is not supported by Lance; the page implied it was (same)
- `Interval_year` is not supported because Lance has no interval type at
all; `Interval_day` is fine because it maps to an Arrow `Duration` (same)
- Vector columns need a fixed-size list. Gravitino `List` produces a
variable-length Arrow `List`, which cannot carry a vector index
(`LanceDataTypeConverter`, lance schema spec)
- Mark `GET /v1/namespace/list` as a Gravitino extension, absent from the
specification and previously listed alongside spec endpoints with no
distinction (lance-namespace `v0.7.5` `openapi.yaml`)
- Add three endpoints missing from the operations table:
`/v1/namespace/list`, `drop_columns`, `alter_columns`
(`LanceNamespaceOperations`, `LanceTableOperations`)
- Delta `location` inherits from schema and catalog; resolution happens
before the format delegator runs
(`GenericCatalogOperations.calculateTableLocation`)
- Gravitino performs no Delta type conversion; the mapping table is guidance
for declaring metadata, not a conversion it applies (no converter exists in the
`delta` package)
- Document three Delta create-time rejections that were undocumented:
distributions, sort orders, indexes (`DeltaTableOperations.createTable`)
- Fix the `lance-ray` example, which used `write_lance(namespace=...)`, a
signature the page's own compatibility notes call unsupported
(`clients/client-python/tests/integration/test_lance_ray.py`)
- `provider` is a field on the create request, not a catalog property as the
page listed it (`GenericCatalogPropertiesMetadata`)
- Add `lance.storage.*` to the catalog properties table, although three
pages tell readers to set it there (same)
- The generic catalog supports Lance and Delta, not "Lance, Iceberg, Hudi"
as the page claimed (`META-INF/services`)
#### `lance-rest-service.md`
- Delete the Deployment Modes table; state the recommended deployment, then
the alternative and its limitations
- New `Lance REST API` section: endpoint table, identifiers, calling
details, limitations, storage options
- Fold `Namespace Model` into the API section as `Identifiers`; hierarchy
and delimiter exist only to let a reader build a request
- Dissolve the orphan `Authentication` and `Storage Configuration` sections
into `Configuration` and the API section
- State where configuration properties go, and that a separate process uses
the same names in a different file
- Structure configuration by outcome rather than by property
- Cut `Engine Integration`; which engines can read Lance is the integration
page's subject
- Cut the Iceberg REST counterpart paragraph
- Add a Quick Start, correct the specification links, consolidate the
configuration tables
- Move the endpoint examples to the Lance tables page so both APIs get equal
treatment
- Reorder the Quick Start so the server starts once; the previous ordering
implied start, stop, edit config, start again
- Correct readiness. `NamespaceWrapper` initializes lazily, so
`/health/ready` returns 503 until the first namespace or table call. The Quick
Start previously checked readiness before the call that triggers
initialization, so the check failed as written
- Create the metalake in the Quick Start, which the service requires but the
page never mentioned
- Pass `location` and `lance.storage.*` in the Quick Start namespace create;
it previously used an `s3://` path with no credentials, so it could not work
- Split `Gravitino-Specific Behavior` into `Calling the Operations` and
`Limitations`; the old heading claimed everything under it was a deviation, but
several rows were plain specification mechanics
- State that Gravitino implements a subset. The `v0.7.5` specification
defines 29 table endpoints; Gravitino serves 9, with the entire data plane
absent
- Document that `DropTable` purges. Every table this service creates is
marked `external`, but `DropTable` removes the dataset files anyway
- Rewrite storage options to say the prefix is stripped:
`lance.storage.region` reaches the client as `region`
- Note that health endpoints bypass the authentication filter
#### `lance-rest-integration.md`
- Reorganize per engine; version information now sits with the engine it
applies to, instead of a shared matrix that implied the two libraries were
coupled
- Promote the compatibility content out of a `:::note` block into real
sections
- Cut `Rationale`, which claimed incompatible versions cause data corruption
- Cut `General Integration Pattern`, which was contentless
- Remove the duplicated MinIO catalog example
- Keep Table Location on the Lance path; the old page reached for a
Gravitino API call to set properties the Lance `CreateNamespace` call accepts
directly
- Correct the claim that an engine without Lance REST support needs its own
credentials; `DescribeTable` returns storage options alongside the location
- Drop the Quick Start; with only two engines it privileged Spark
arbitrarily, and each engine section is its own quick start
- Add a credentials section pointing at the service page
#### `lakehouse-generic-catalog.md`
- Rewrite the Overview around what distinguishes this catalog: Lance and
Delta define no catalog of their own, so Gravitino holds the metadata directly
rather than federating. Explains the word "generic", which the page previously
asserted without grounding
- Split catalog creation per format; a Lance catalog carries `location` and
`lance.storage.*`, a Delta catalog needs no properties at all
- Reorganize properties so format-neutral ones come first and Lance-only
ones are separated
- Move location resolution next to the `location` rows it explains, and
convert a 20-line ASCII block into a table plus one worked paragraph
- Document the `lance.storage.*` option names; four pages told readers to
set them and none listed them
- Cut the Benefits list and both bullet lists of "Supported Operations"
- Remove `"owner": "sales-team"` from the schema example, which implied a
property that does not exist
#### `lakehouse-generic-lance-table.md`
- Restructure to be path-neutral; the page is reachable from both APIs and
previously read as Gravitino-API-only, giving a Lance REST reader instructions
that do not apply
- Replace `Supported Operations` with a `Capabilities` table giving both
APIs side by side
- Add a column to the table properties showing how the Lance REST service
supplies each one, instead of marking five of eight as irrelevant
- Cover both APIs symmetrically in Examples, with the Lance examples moved
here from the service page
- Add Troubleshooting with ten error strings taken verbatim from source
- Cut the Migration Guide, a bash `for` loop registering tables
- Preserve `#data-type-mappings`, which
`manage-relational-metadata-using-gravitino.md` links into
#### `lakehouse-generic-delta-table.md`
- Lead the Overview with the constraint that matters: every Delta table is
external and a create is a registration
- State the consequence the old page only implied: Gravitino and the
`_delta_log` are two records of the same table, and Gravitino never reconciles
them
- Align the `Capabilities` table with the Lance page's phrasing so the two
formats can be compared
- Add Troubleshooting with eight verbatim error strings, three of which the
old page did not mention
- Cut Planned Enhancements, which speculated about unreleased features
- Cut the 20-line Spark modification example and the read-via-Spark snippet,
both of which belong to Delta Lake's documentation
#### Conventions applied across all five pages
- No `:::` admonition blocks, no `<Tabs>` / `<TabItem>`, no ✅ / ❌ in tables,
no em dashes
- No `Since Version` columns or "Introduced in X" sentences, since the docs
are version-selected
- No raw HTML in table cells; `<sup>` renders literally, so footnote markers
use Unicode superscripts
- Title Case headings, `{snake_case}` placeholders, aligned table pipes
- Shell variables declared once per code block so each block stands alone
- Code lines kept short enough to avoid horizontal scroll
- No one-column tables and no sections with a single child
### Why are the changes needed?
Five pages document Lance in Gravitino, and none of them states the model
that makes the set coherent: Lance tables live in a `lakehouse-generic`
catalog, and two APIs reach the same tables. Configure the Lance REST service
and it creates the catalog when a client first connects, or create the catalog
through the Gravitino REST API and skip the service entirely.
Because that was never written down, the pages were organized by artifact
rather than by the model. Authentication appeared on two pages. Storage
configuration lived on one page and engine configuration on another, with no
cross-reference. A Lance REST user landing on the tables page got operations
and properties that do not apply on their path. The word "generic" was never
explained. Following the service page Quick Start in order did not work,
because it checked readiness before the call that initializes the service.
The five files cannot be split into smaller PRs. Ten cross-page anchors
point at headings that only exist after this rewrite, running in every
direction between the files, so no ordering of smaller PRs avoids shipping
broken links.
`lance-rest-server-chart.md` is deliberately not included. Its most useful
content requires a chart change rather than a doc change, and its doc-only
fixes will follow separately.
### Does this PR introduce _any_ user-facing change?
Documentation only. No API, property, or behavior changes.
The pages themselves change substantially, including headings. All inbound
anchors from other documentation pages were checked and preserved, including
`#data-type-mappings`, which `manage-relational-metadata-using-gravitino.md`
links into.
### How was this patch tested?
Documentation only, so no code tests. Claims were checked as follows:
- Endpoint names, methods, and paths against lance-namespace `v0.7.5`
`openapi.yaml`
- Type mappings against `LanceDataTypeConverter` and the Lance format schema
specification
- Property names and defaults against `LanceConstants`,
`GenericCatalogPropertiesMetadata`, `GenericTablePropertiesMetadata`, and
`LanceTableDelegator`
- Operation support against `LanceTableOperations` and `DeltaTableOperations`
- Error strings copied verbatim from source
- The `lance-ray` example against
`clients/client-python/tests/integration/test_lance_ray.py`
- All cross-page and in-page anchors verified to resolve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
