This is an automated email from the ASF dual-hosted git repository.
yuqi4733 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 5af9f93c13 [#9169] improvement(docs): Add docs about Lance REST
service and Generic Lakehouse catalog (#9173)
5af9f93c13 is described below
commit 5af9f93c13bd6c525e45f13b8a43895377b06165
Author: Mini Yu <[email protected]>
AuthorDate: Thu Dec 11 15:23:49 2025 +0800
[#9169] improvement(docs): Add docs about Lance REST service and Generic
Lakehouse catalog (#9173)
### What changes were proposed in this pull request?
Add docs about Lance REST service.
### Why are the changes needed?
Let users deploy the Lance REST service easily.
Fix: #9169
Fix: #9405
### Does this PR introduce _any_ user-facing change?
N/A.
### How was this patch tested?
N/A.
---
conf/gravitino.conf.template | 2 +-
docs/docker-image-details.md | 4 +-
docs/index.md | 6 +
docs/lakehouse-generic-catalog.md | 202 +++++++++++
docs/lakehouse-generic-lance-table.md | 304 ++++++++++++++++
docs/lance-rest-service.md | 401 +++++++++++++++++++++
docs/manage-relational-metadata-using-gravitino.md | 50 +--
7 files changed, 944 insertions(+), 25 deletions(-)
diff --git a/conf/gravitino.conf.template b/conf/gravitino.conf.template
index 7867576a0b..029e1d2121 100644
--- a/conf/gravitino.conf.template
+++ b/conf/gravitino.conf.template
@@ -115,4 +115,4 @@ gravitino.lance-rest.namespace-backend = gravitino
# The uri of the Lance REST service gravitino namespace backend
gravitino.lance-rest.gravitino-uri = http://localhost:8090
# The metalake name used for Lance REST service gravitino namespace backend,
please create the metalake first before using it, and configure the metalake
name here.
-# gravitino.lance-rest.gravitino.metalake = metalake
+# gravitino.lance-rest.gravitino-metalake = metalake
diff --git a/docs/docker-image-details.md b/docs/docker-image-details.md
index a98f845a14..0976325fa6 100644
--- a/docs/docker-image-details.md
+++ b/docs/docker-image-details.md
@@ -149,7 +149,7 @@ You can deploy the standalone Gravitino Lance REST server
with the Docker image.
```shell
-docker run --rm -d -p 9102:9102 -e LANCE_REST_GRAVITINO_METALAKE_NAME=test -e
LANCE_REST_PORT=9102 apache/gravitino-lance-rest:latest
+docker run --rm -d -p 9102:9102 -e LANCE_REST_GRAVITINO_METALAKE_NAME=test -e
LANCE_REST_GRAVITINO_URI=http://gravitino-host:port -e LANCE_REST_PORT=9102
apache/gravitino-lance-rest:latest
```
Memory settings
@@ -159,7 +159,7 @@ Use `GRAVITINO_MEM` to size the JVM (default `-Xms1024m
-Xmx1024m -XX:MaxMetaspa
Currently, Gravitino Lance REST server supports setting the following
environment variables
- LANCE_REST_GRAVITINO_METALAKE_NAME: It will overwrite the configuration
"gravitino.lance-rest.gravitino-metalake" in configuration file
`conf/gravitino-lance-rest-server.conf`. **You should set it to your Gravitino
metalake name.**
- LANCE_REST_NAMESPACE_BACKEND: It will overwrite the configuration
"gravitino.lance-rest.namespace-backend" in configuration file
`conf/gravitino-lance-rest-server.conf`. The default value is "gravitino" and
you should not change it as of now.
-- LANCE_REST_GRAVITINO_URI: It will overwrite the configuration
"gravitino.lance-rest.gravitino-uri" in configuration file
`conf/gravitino-lance-rest-server.conf`. The default value is
"http://localhost:8090"; and you can change it to your Gravitino server address.
+- LANCE_REST_GRAVITINO_URI: It will overwrite the configuration
"gravitino.lance-rest.gravitino-uri" in configuration file
`conf/gravitino-lance-rest-server.conf`. The default value is
"http://localhost:8090"; and you can change it to your Gravitino server address.
**Be care that Gravitino server URI `http://localhost:8090` is a docker
container internal address, if your Gravitino server is running outside the
docker container, you should set it to your host IP address like `http://host-i
[...]
- LANCE_REST_HOST: It will overwrite the configuration
"gravitino.lance-rest.host" in configuration file
`conf/gravitino-lance-rest-server.conf`. The default value is `0.0.0.0`.
- LANCE_REST_PORT: It will overwrite the configuration
"gravitino.lance-rest.httpPort" in configuration file
`conf/gravitino-lance-rest-server.conf`. The default value is `9101`.
diff --git a/docs/index.md b/docs/index.md
index 8283ae089c..0871384270 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -85,6 +85,7 @@ Gravitino currently supports the following catalogs:
* [**PostgreSQL catalog**](./jdbc-postgresql-catalog.md)
* [**OceanBase catalog**](./jdbc-oceanbase-catalog.md)
* [**StarRocks catalog**](./jdbc-starrocks-catalog.md)
+* [**Lakehouse generic catalog**](./lakehouse-generic-catalog.md)
If you want to operate table and partition statistics, you can refer to the
[document](./manage-statistics-in-gravitino.md).
@@ -134,6 +135,7 @@ Gravitino supports different catalogs to manage the
metadata in different source
* [Paimon catalog](./lakehouse-paimon-catalog.md): a complete guide to using
Gravitino to manage Apache Paimon data.
* [PostgreSQL catalog](./jdbc-postgresql-catalog.md): a complete guide to
using Gravitino to manage PostgreSQL data.
* [OceanBase catalog](./jdbc-oceanbase-catalog.md): a complete guide to using
Gravitino to manage OceanBase data.
+* [Lakehouse generic catalog](./lakehouse-generic-catalog.md): a complete
guide to using Gravitino to manage lakehouse data sources.
### Governance
@@ -151,6 +153,10 @@ Gravitino provides governance features to manage metadata
in a unified way. See:
* [Iceberg REST catalog service](./iceberg-rest-service.md): a guide to using
Gravitino
as an Apache Iceberg REST catalog service.
+### Gravitino Lance REST catalog service
+* [Lance REST catalog service](./lance-rest-service.md): a guide to using
Gravitino
+ as a Lance REST catalog service.
+
### Connectors
#### Trino connector
diff --git a/docs/lakehouse-generic-catalog.md
b/docs/lakehouse-generic-catalog.md
new file mode 100644
index 0000000000..3abc474ac9
--- /dev/null
+++ b/docs/lakehouse-generic-catalog.md
@@ -0,0 +1,202 @@
+---
+title: "Generic Lakehouse Catalog"
+slug: /lakehouse-generic-catalog
+keywords:
+ - lakehouse
+ - lance
+ - metadata
+ - generic catalog
+ - file system
+license: "This software is licensed under the Apache License version 2."
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Overview
+
+The Generic Lakehouse Catalog is a Gravitino catalog implementation designed
to seamlessly integrate with lakehouse storage systems built on file
system-based architectures. This catalog enables unified metadata management
for lakehouse tables stored on various storage backends, providing a consistent
interface for data discovery, governance, and access control.
+
+Currently, Gravitino fully supports the **Lance** lakehouse format, with plans
to extend support to additional formats in the future.
+
+### Why Use Generic Lakehouse Catalog?
+
+1. **Unified Metadata Management**: Single source of truth for table metadata
across multiple storage backends
+2. **Multi-Format Support**: Extensible architecture to support various
lakehouse table formats such as Lance, Iceberg, Hudi, etc.
+3. **Storage Flexibility**: Work with any file system, local, or cloud object
stores
+4. **Gravitino Integration**: Leverage Gravitino's metadata management, access
control, lineage tracking, and data discovery
+5. **Easy Migration**: Register existing lakehouse tables without data movement
+
+## Catalog Management
+
+### Capabilities
+
+The Generic Lakehouse Catalog provides comprehensive relational metadata
management capabilities equivalent to standard relational catalogs:
+
+**Supported Operations:**
+- ✅ Create, read, update, and delete catalogs
+- ✅ List all catalogs in a metalake
+- ✅ Manage catalog properties and metadata
+- ✅ Set and modify catalog locations
+- ✅ Configure storage backend credentials
+
+For detailed information on available operations, see [Manage Relational
Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md).
+
+### Catalog Properties
+
+| Property | Description | Example
| Required | Since Version |
+|------------|----------------------------------------------|-------------------------|----------|---------------|
+| `provider` | Catalog provider type |
`lakehouse-generic` | Yes | 1.1.0 |
+| `location` | Root storage path for all schemas and tables |
`s3://bucket/lakehouse` | No | 1.1.0 |
+
+#### Key Property: `location`
+
+The `location` property specifies the root directory for the lakehouse table.
All schemas and tables are stored under this location unless explicitly
overridden at the schema or table level.
+
+**Location Resolution Hierarchy:**
+1. Table-level `location` (highest priority)
+2. Schema-level `location`, then the location of the table will be
`{schema_location}/{table_name}`
+3. Catalog-level `location` (fallback), then the location of the table will be
`{catalog_location}/{schema_name}/{table_name}`
+
+**Example Location Hierarchy:**
+```
+Case1: only catalog location is set
+Catalog location: hdfs://namenode:9000/lakehouse
+└── Schema: sales
+ ├── Table: orders. Final location of table:
hdfs://namenode:9000/lakehouse/sales/orders
+ └── Table: customers. Final location of table:
hdfs://namenode:9000/lakehouse/sales/customers
+
+case2: schema location is set, overriding catalog location and table location
is not set
+Catalog location: hdfs://namenode:9000/lakehouse
+└── Schema: sales: s3://sales-bucket/data
+ ├── Table: orders. Final location of table: s3://sales-bucket/data/orders
+ └── Table: customers. Final location of table:
s3://sales-bucket/data/customers
+
+case3: table location is set, overriding both schema and catalog locations
+Catalog location: hdfs://namenode:9000/lakehouse
+└── Schema: sales: s3://sales-bucket/data
+ ├── Table: orders. Table location: s3://sales-bucket/my_orders, Final
location of table: s3://sales-bucket/my_orders
+ └── Table: customers. Table location: s3://sales-bucket/my_customers,
Final location of table: s3://sales-bucket/my_customers
+
+```
+
+### Creating a Catalog
+
+Use `provider: "lakehouse-generic"` when creating a generic lakehouse catalog.
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d '{
+ "name": "generic_lakehouse_catalog",
+ "type": "RELATIONAL",
+ "comment": "Generic lakehouse catalog for Lance datasets",
+ "provider": "lakehouse-generic",
+ "properties": {
+ "location": "hdfs://localhost:9000/user/lakehouse"
+ }
+}' http://localhost:8090/api/metalakes/metalake/catalogs
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+GravitinoClient gravitinoClient = GravitinoClient
+ .builder("http://127.0.0.1:8090";)
+ .withMetalake("metalake")
+ .build();
+
+Map<String, String> catalogProperties = ImmutableMap.<String, String>builder()
+ .put("location", "hdfs://localhost:9000/user/lakehouse")
+ .build();
+
+Catalog catalog = gravitinoClient.createCatalog(
+ "generic_lakehouse_catalog",
+ Type.RELATIONAL,
+ "lakehouse-generic",
+ "Generic lakehouse catalog for Lance datasets",
+ catalogProperties
+);
+```
+
+</TabItem>
+</Tabs>
+
+Other catalog operations are general with relational catalogs. See [Catalog
Operations](./manage-relational-metadata-using-gravitino.md#catalog-operations)
for detailed documentation.
+
+## Schema Management
+
+### Capabilities
+
+Schema operations follow the same patterns as relational catalogs:
+
+**Supported Operations:**
+- ✅ Create schemas with custom properties
+- ✅ List all schemas in a catalog
+- ✅ Load schema metadata and properties
+- ✅ Update schema properties
+- ✅ Delete schemas
+- ✅ Check schema existence
+
+See [Schema
Operations](./manage-relational-metadata-using-gravitino.md#schema-operations)
for detailed documentation.
+
+### Schema Properties
+
+Schemas inherit catalog properties and can override specific settings:
+
+| Property | Description |
Example | Required | Since version |
+|------------|----------------------------------------------------------|------------------------------|----------|---------------|
+| `location` | Custom storage root path for all tables under the schema |
's3://bucket/path_to_schema' | No | 1.1.0 |
+
+About location resolution hierarchy, please see [Key Property:
`location`](#key-property-location) in the Catalog Management section for more
details.
+
+### Schema Operations
+
+**Creating a Schema:**
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d '{
+ "name": "sales",
+ "comment": "Sales department data",
+ "properties": {
+ "location": "s3://sales-bucket/data",
+ "owner": "sales-team"
+ }
+}'
http://localhost:8090/api/metalakes/metalake/catalogs/lakehouse_catalog/schemas
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+Map<String, String> schemaProperties = ImmutableMap.<String, String>builder()
+ .put("location", "s3://sales-bucket/data")
+ .put("owner", "sales-team")
+ .build();
+
+catalog.asSchemas().createSchema(
+ "sales",
+ "Sales department data",
+ schemaProperties
+);
+```
+
+</TabItem>
+</Tabs>
+
+For additional operations, refer to [Schema Operations
documentation](./manage-relational-metadata-using-gravitino.md#schema-operations).
+
+## Table management
+
+### Supported Operations
+
+Since different lakehouse table formats have varying capabilities, table
operation support may differ. The following are table operations for different
lakehouse formats.
+
+- [Lance Format Support](./lakehouse-generic-lance-table.md)
\ No newline at end of file
diff --git a/docs/lakehouse-generic-lance-table.md
b/docs/lakehouse-generic-lance-table.md
new file mode 100644
index 0000000000..174e5f9a8c
--- /dev/null
+++ b/docs/lakehouse-generic-lance-table.md
@@ -0,0 +1,304 @@
+---
+title: "Lance table support"
+slug: /lance-table-support
+keywords:
+- lakehouse
+- lance
+- metadata
+- generic catalog
+license: "This software is licensed under the Apache License version 2."
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+## Overview
+
+This document describes how to use Apache Gravitino to manage a generic
lakehouse catalog using Lance as the underlying table format.
+
+
+## Table Management
+
+### Supported Operations
+
+For Lance tables in a Generic Lakehouse Catalog, the following table
summarizes supported operations:
+
+| Operation | Support Status |
+|-----------|-----------------|
+| List | ✅ Full |
+| Load | ✅ Full |
+| Alter | Not support now |
+| Create | ✅ Full |
+| Register | ✅ Full |
+| Drop | ✅ Full |
+| Purge | ✅ Full |
+
+:::note Feature Limitations
+- **Partitioning:** Not currently supported
+- **Sort Orders:** Not currently supported
+- **Distributions:** Not currently supported
+- **Indexes:** Not currently supported
+:::
+
+### Data Type Mappings
+
+Lance uses Apache Arrow for table schemas. The following table shows type
mappings between Gravitino and Arrow:
+
+| Gravitino Type | Arrow Type |
+|----------------------------------|-----------------------------------------|
+| `Struct` | `Struct` |
+| `Map` | `Map` |
+| `List` | `Array` |
+| `Boolean` | `Boolean` |
+| `Byte` | `Int8` |
+| `Short` | `Int16` |
+| `Integer` | `Int32` |
+| `Long` | `Int64` |
+| `Float` | `Float` |
+| `Double` | `Double` |
+| `String` | `Utf8` |
+| `Binary` | `Binary` |
+| `Decimal(p, s)` | `Decimal(p, s)` (128-bit) |
+| `Date` | `Date` |
+| `Timestamp`/`Timestamp(6)` | `TimestampType withoutZone` |
+| `Timestamp(0)` | `TimestampType Second withoutZone` |
+| `Timestamp(3)` | `TimestampType Millisecond withoutZone` |
+| `Timestamp(9)` | `TimestampType Nanosecond withoutZone` |
+| `Timestamp_tz`/`Timestamp_tz(6)` | `TimestampType Microsecond withUtc` |
+| `Timestamp_tz(0)` | `TimestampType Second withUtc` |
+| `Timestamp_tz(3)` | `TimestampType Millisecond withUtc` |
+| `Timestamp_tz(9)` | `TimestampType Nanosecond withUtc` |
+| `Time`/`Time(9)` | `Time Nanosecond` |
+| `Null` | `Null` |
+| `Fixed(n)` | `Fixed-Size Binary(n)` |
+| `Interval_year` | `Interval(YearMonth)` |
+| `Interval_day` | `Duration(Microsecond)` |
+| `External(arrow_field_json_str)` | Any Arrow Field |
+
+### External Type Support
+
+For Arrow types not natively mapped in Gravitino, use the
`External(arrow_field_json_str)` type, which accepts a JSON string
representation of an Arrow `Field`.
+
+**Requirements:**
+- JSON must conform to Apache Arrow [Field
specification](https://github.com/apache/arrow-java/blob/ed81e5981a2bee40584b3a411ed755cb4cc5b91f/vector/src/main/java/org/apache/arrow/vector/types/pojo/Field.java#L80C1-L86C68)
+- `name` attribute must match column name exactly
+- `nullable` attribute must match column nullability
+- `children` array:
+ - Empty for primitive types
+ - Contains child field definitions for complex types (Struct, List)
+
+**Examples:**
+
+| Arrow Type | External Type Definition
|
+|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `Large Utf8` |
`External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largeutf8\"},\"children\":[]}")`
|
+| `Large Binary` |
`External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largebinary\"},\"children\":[]}")`
|
+| `Large List` |
`External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largelist\"},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\",\"bitWidth\":32,\"isSigned\":true},\"children\":[]}]}")`
|
+| `Fixed-Size List` |
`External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"fixedsizelist\",\"listSize\":10},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\",\"bitWidth\":32,\"isSigned\":true},\"children\":[]}]}")`
|
+
+### Table Properties
+
+Required and optional properties for tables in a Generic Lakehouse Catalog:
+
+| Property | Description
| Default | Required | Since
Version |
+|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------------|---------------|
+| `format` | Table format: `lance`, currently only `lance` is
fully supported.
| (none) | Yes |
1.1.0 |
+| `location` | Storage path for table metadata and data, Lance
currently supports: S3, GCS, OSS, AZ, File, Memory and file-object-store.
| (none) | Conditional* |
1.1.0 |
+| `external` | Whether the data directory is an external location.
If it's `true`, dropping a table will only remove metadata in Gravitino and
will not delete the data directory, and purge table will delete both. For a
non-external table, dropping will drop both.
| false | No |
1.1.0 |
+| `lance.creation-mode` | Create mode: for create table, it can be `CREATE`,
`EXIST_OK` or `OVERWRITE`. and it should be `CREATE` or `OVERWRITE` for
registering tables
| `CREATE` | No
| 1.1.0 |
+| `lance.register` | Whether it is a register table operation. If it's
`true`, This API will not create data directory actually and it's the user's
responsibility to create and manage the data directory. `false` it will
actually create a table.
| false | No
| 1.1.0 |
+| `lance.storage.xxxx` | Any additional storage-specific properties required
by Lance format (e.g., S3 credentials, HDFS configs). Replace `xxxx` with
actual property names. For example, we can use
`lance.storage.aws_access_key_id` to set S3 aws_access_key_id when using a S3
location, for detail, please refer to
https://lancedb.com/docs/storage/integrations/ | (none) | No |
1.1.0 |
+
+- `CREATE`: Create a new table, fail if the table already exists.
+- `EXIST_OK`: Create a new table if it does not exist, otherwise do nothing.
+- `OVERWRITE`: Create a new table, overwrite if the table already exists, it
will delete the existing data directory first if the table is not a registered
table and then create a new one.
+
+**Location Requirement:** Must be specified at catalog, schema, or table
level. See [Location
Resolution](./lakehouse-generic-catalog.md#key-property-location).
+
+You may also set additional properties specific to your lakehouse format or
custom requirements.
+
+### Table Operations
+
+Table operations follow standard relational catalog patterns. See [Table
Operations](./manage-relational-metadata-using-gravitino.md#table-operations)
for comprehensive documentation.
+
+The following sections provide examples and important details for working with
Lance tables.
+
+#### Creating a Lance Table
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d '{
+ "name": "lance_table",
+ "comment": "Example Lance table",
+ "columns": [
+ {
+ "name": "id",
+ "type": "integer",
+ "comment": "Primary identifier",
+ "nullable": false
+ }
+ ],
+ "properties": {
+ "format": "lance",
+ "location": "/tmp/lance_catalog/schema/lance_table"
+ }
+}'
http://localhost:8090/api/metalakes/test/catalogs/generic_lakehouse_lance_catalog/schemas/schema/tables
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+Catalog catalog =
gravitinoClient.loadCatalog("generic_lakehouse_lance_catalog");
+TableCatalog tableCatalog = catalog.asTableCatalog();
+
+Map<String, String> tableProperties = ImmutableMap.<String, String>builder()
+ .put("format", "lance")
+ .put("location", "/tmp/lance_catalog/schema/example_table")
+ .build();
+
+tableCatalog.createTable(
+ NameIdentifier.of("schema", "lance_table"),
+ new Column[] {
+ Column.of("id", Types.IntegerType.get(), "Primary identifier",
+ true, false, null)
+ },
+ "Example Lance table",
+ tableProperties,
+ null, // partitions
+ null, // distributions
+ null, // sortOrders
+ null // indexes
+);
+```
+
+</TabItem>
+</Tabs>
+
+#### Registering External Tables
+
+Register existing Lance tables without moving or copying data:
+
+<Tabs groupId='language' queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d '{
+ "name": "register_lance_table",
+ "comment": "Registered existing Lance table",
+ "columns": [],
+ "properties": {
+ "format": "lance",
+ "lance.register": "true",
+ "location": "/tmp/lance_catalog/schema/existing_lance_table"
+ }
+}'
http://localhost:8090/api/metalakes/test/catalogs/generic_lakehouse_lance_catalog/schemas/schema/tables
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+Catalog catalog =
gravitinoClient.loadCatalog("generic_lakehouse_lance_catalog");
+TableCatalog tableCatalog = catalog.asTableCatalog();
+
+Map<String, String> registerProperties = ImmutableMap.<String, String>builder()
+ .put("format", "lance")
+ .put("lance.register", "true")
+ .put("location", "/tmp/lance_catalog/schema/existing_lance_table")
+ .build();
+
+tableCatalog.createTable(
+ NameIdentifier.of("schema", "register_lance_table"),
+ new Column[] {}, // Schema auto-detected from existing table
+ "Registered existing Lance table",
+ registerProperties,
+ null, null, null, null
+);
+```
+
+</TabItem>
+</Tabs>
+
+:::tip Registration vs Creation
+- **Registration** (`lance.register: true`):
+ - Links to existing Lance dataset or a path placeholder
+ - Schema automatically detected from Lance metadata
+ - Useful for importing existing datasets
+
+- **Creation** (default):
+ - Creates new Lance table from scratch
+ - Requires column schema definition
+ - Initializes new Lance dataset files
+:::
+
+## Advanced Topics
+
+### Troubleshooting
+
+#### Common Issues
+
+**Issue: "Location not specified" error**
+```
+Solution: Ensure at least one level (catalog/schema/table) specifies the
location property
+```
+
+**Issue: Permission denied errors**
+```
+Solution: Check file system permissions and credentials for the storage backend
+```
+
+**Issue: Table not found after registration**
+```
+Solution: Verify the location path points to a valid Lance dataset directory
+```
+
+### Migration Guide
+
+#### Migrating Existing Lance Tables
+
+1. **Inventory**: List all existing Lance table locations
+2. **Create Catalog**: Create Generic Lakehouse catalog pointing to root
location
+3. **Register Tables**: Use register operation for each table
+4. **Verify**: Confirm all tables are accessible through Gravitino
+5. **Update Clients**: Point applications to Gravitino metadata instead of
direct Lance access
+
+**Example Migration Script:**
+
+```shell
+# List of existing Lance tables to register
+tables_to_migrate=(
+ "sales orders /data/sales/orders"
+ "sales customers /data/sales/customers"
+ "inventory products /data/inventory/products"
+)
+
+# Register each table
+for entry in "${tables_to_migrate[@]}"; do
+ read -r schema table location <<< "$entry"
+ echo ${schema}
+ echo ${table}
+
+ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d "{
+ \"name\": \"${table}\",
+ \"comment\": \"Registered existing Lance table\",
+ \"columns\": [],
+ \"properties\": {
+ \"format\": \"lance\",
+ \"lance.register\": \"true\",
+ \"location\": \"${location}\"
+ }
+ }"
http://localhost:8090/api/metalakes/test/catalogs/generic_lakehouse_lance_catalog/schemas/$schema/tables
+
+ echo "Registered ${schema}.${table}"
+done
+```
+
+Other table operations (load, alter, drop, truncate) follow standard
relational catalog patterns. See [Table
Operations](./manage-relational-metadata-using-gravitino.md#table-operations)
for details.
+
diff --git a/docs/lance-rest-service.md b/docs/lance-rest-service.md
new file mode 100644
index 0000000000..4b0db579f1
--- /dev/null
+++ b/docs/lance-rest-service.md
@@ -0,0 +1,401 @@
+---
+title: "Lance REST service"
+slug: /lance-rest-service
+keywords:
+ - Lance REST
+ - Lance datasets
+ - REST API
+license: "This software is licensed under the Apache License version 2."
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+## Overview
+
+The Lance REST service provides a RESTful interface for managing Lance
datasets through HTTP endpoints. Introduced in Gravitino version 1.1.0, this
service enables seamless interaction with Lance datasets for data operations
and metadata management.
+
+The service implements the [Lance REST API
specification](https://docs.lancedb.com/api-reference/introduction). For
detailed specification documentation, see the [official Lance REST
documentation](https://lance.org/format/namespace/rest/catalog-spec/).
+
+### What is Lance?
+
+[Lance](https://lance.org/format/) is a modern columnar data format designed
for AI/ML workloads. It provides:
+
+- **High-performance vector search**: Native support for similarity search on
high-dimensional embeddings
+- **Columnar storage**: Optimized for analytical queries and machine learning
pipelines
+- **Fast random access**: Efficient row-level operations unlike traditional
columnar formats
+- **Version control**: Built-in dataset versioning and time-travel capabilities
+- **Incremental updates**: Append and update data without full rewrites
+
+### Architecture
+
+The Lance REST service acts as a bridge between Lance datasets and
applications:
+
+```
+┌─────────────────┐
+│ Applications │
+│ (Python/Java) │
+└────────┬────────┘
+ │ HTTP/REST
+ ▼
+┌─────────────────┐
+│ Lance REST │
+│ Service │
+└────────┬────────┘
+ │
+ ▼ Gravitino Client API
+┌─────────────────┐
+│ Gravitino Server │
+│(Metadata Backend)│
+└────────┬────────┘
+ │ File System Operations
+ ▼
+┌─────────────────┐
+│ Lance Datasets │
+│ (S3/GCS/Local) │
+└─────────────────┘
+```
+
+**Key Features:**
+- Full compliance with Lance REST API specification
+- Can run standalone or integrated with Gravitino server
+- Support for namespace and table management
+- Index creation and management capabilities (Index operations are not
supported in version 1.1.0)
+- Metadata stored in Gravitino for unified governance
+
+## Supported Operations
+
+The Lance REST service provides comprehensive support for namespace
management, table management, and index operations. The table below lists all
supported operations:
+
+| Operation | Description
| HTTP Method | Endpoint Pattern | Since Version |
+|-------------------|-------------------------------------------------------------------|-------------|---------------------------------------|---------------|
+| CreateNamespace | Create a new Lance namespace
| POST | `/lance/v1/namespace/{id}/create` | 1.1.0 |
+| ListNamespaces | List all namespaces under a parent namespace
| GET | `/lance/v1/namespace/{parent}/list` | 1.1.0 |
+| DescribeNamespace | Retrieve detailed information about a specific namespace
| POST | `/lance/v1/namespace/{id}/describe` | 1.1.0 |
+| DropNamespace | Delete a namespace
| POST | `/lance/v1/namespace/{id}/drop` | 1.1.0 |
+| NamespaceExists | Check whether a namespace exists
| POST | `/lance/v1/namespace/{id}/exists` | 1.1.0 |
+| ListTables | List all tables in a namespace
| GET | `/lance/v1/namespace/{id}/table/list` | 1.1.0 |
+| CreateTable | Create a new table in a namespace
| POST | `/lance/v1/table/{id}/create` | 1.1.0 |
+| DropTable | Delete a table including both metadata and data
| POST | `/lance/v1/table/{id}/drop` | 1.1.0 |
+| TableExists | Check whether a table exists
| POST | `/lance/v1/table/{id}/exists` | 1.1.0 |
+| RegisterTable | Register an existing Lance table to a namespace
| POST | `/lance/v1/table/{id}/register` | 1.1.0 |
+| DeregisterTable | Unregister a table from a namespace (metadata only, data
remains) | POST | `/lance/v1/table/{id}/deregister` | 1.1.0 |
+
+More details, please refer to the [Lance REST API
specification](https://lance.org/format/namespace/rest/catalog-spec/)
+
+### Operation Details
+
+Some operations have specific behaviors and modes. Below are important details
to consider:
+
+#### Namespace Operations
+
+**CreateNamespace** supports three modes:
+- `create`: Fails if namespace already exists
+- `exist_ok`: Succeeds even if namespace exists
+- `overwrite`: Replaces existing namespace
+
+**DropNamespace** behavior:
+- Recursively deletes all child namespaces and tables
+- Deletes both metadata and Lance data files
+- Operation is irreversible
+
+#### Table Operations
+
+**RegisterTable vs CreateTable**:
+- **RegisterTable**: Links existing Lance datasets into Gravitino catalog
without data movement
+- **CreateTable**: Creates new Lance table with schema and write metadata files
+:::note
+The `version` field of `CreateTable` response is always null, which stands for
the latest version.
+:::
+
+**DropTable vs DeregisterTable**:
+- **DropTable**: Permanently deletes metadata and data files from storage
+- **DeregisterTable**: Removes metadata from Gravitino but preserves Lance
data files
+
+
+## Deployment
+
+### Running with Gravitino Server
+
+To enable the Lance REST service within Gravitino server, configure the
following properties in your Gravitino configuration file
`${GRAVITINO_HOME}/conf/gravitino.conf`:
+
+| Configuration Property | Description
| Default Value |
Required | Since Version |
+|-------------------------------------------|------------------------------------------------------------------------------|-------------------------|----------|---------------|
+| `gravitino.auxService.names` | Auxiliary services to run.
Include `lance-rest` to enable Lance REST service | iceberg-rest,lance-rest |
Yes | 0.2.0 |
+| `gravitino.lance-rest.classpath` | Classpath for Lance REST
service, relative to Gravitino home directory | lance-rest-server/libs |
Yes | 1.1.0 |
+| `gravitino.lance-rest.httpPort` | Port number for Lance REST
service | 9101 |
No | 1.1.0 |
+| `gravitino.lance-rest.host` | Hostname for Lance REST service
| 0.0.0.0 | No
| 1.1.0 |
+| `gravitino.lance-rest.namespace-backend` | Namespace metadata backend
(currently only `gravitino` is supported) | gravitino |
Yes | 1.1.0 |
+| `gravitino.lance-rest.gravitino-uri` | Gravitino server URI (required
when namespace-backend is `gravitino`) | http://localhost:8090 | Yes
| 1.1.0 |
+| `gravitino.lance-rest.gravitino-metalake` | Gravitino metalake name
(required when namespace-backend is `gravitino`) | (none)
| Yes | 1.1.0 |
+
+**Example Configuration:**
+
+```properties
+gravitino.auxService.names = lance-rest
+gravitino.lance-rest.httpPort = 9101
+gravitino.lance-rest.host = 0.0.0.0
+gravitino.lance-rest.namespace-backend = gravitino
+gravitino.lance-rest.gravitino-uri = http://localhost:8090
+gravitino.lance-rest.gravitino-metalake = my_metalake
+```
+
+### Running Standalone
+
+To run Lance REST service independently without Gravitino server (You need to
start Gravitino server first):
+
+```shell
+{GRAVITINO_HOME}/bin/gravitino-lance-rest-server.sh start
+```
+
+Configure the service by editing
`{GRAVITINO_HOME}/conf/gravitino-lance-rest-server.conf` or passing
command-line arguments:
+
+| Configuration Property | Description |
Default Value | Required | Since Version |
+|-------------------------------------------|-----------------------------|-----------------------|----------|---------------|
+| `gravitino.lance-rest.namespace-backend` | Namespace metadata backend |
gravitino | Yes | 1.1.0 |
+| `gravitino.lance-rest.gravitino-uri` | Gravitino server URI |
http://localhost:8090 | Yes | 1.1.0 |
+| `gravitino.lance-rest.gravitino-metalake` | Gravitino metalake name |
(none) | Yes | 1.1.0 |
+| `gravitino.lance-rest.httpPort` | Service port number |
9101 | No | 1.1.0 |
+| `gravitino.lance-rest.host` | Service hostname |
0.0.0.0 | No | 1.1.0 |
+
+:::tip
+In most cases, you only need to configure
`gravitino.lance-rest.gravitino-metalake` and other properties can use their
default values.
+:::
+
+
+### Running with Docker
+
+Launch Lance REST service using Docker(You need to start Gravitino server
first):
+
+```shell
+docker run -d --name lance-rest-service -p 9101:9101 \
+ -e LANCE_REST_GRAVITINO_URI=http://gravitino-host:8090 \
+ -e LANCE_REST_GRAVITINO_METALAKE_NAME=your_metalake_name \
+ -e LANCE_REST_GRAVITINO_URI=http://gravitino-host:port \
+ apache/gravitino-lance-rest:latest
+```
+
+Access the service at `http://localhost:9101`.
+
+**Environment Variables:**
+
+| Environment Variable | Configuration Property
| Required | Default Value | Since Version |
+|--------------------------------------|-------------------------------------------|----------|-------------------------|---------------|
+| `LANCE_REST_NAMESPACE_BACKEND` |
`gravitino.lance-rest.namespace-backend` | Yes | `gravitino`
| 1.1.0 |
+| `LANCE_REST_GRAVITINO_METALAKE_NAME` |
`gravitino.lance-rest.gravitino-metalake` | Yes | (none)
| 1.1.0 |
+| `LANCE_REST_GRAVITINO_URI` | `gravitino.lance-rest.gravitino-uri`
| Yes | `http://localhost:8090` | 1.1.0 |
+| `LANCE_REST_HOST` | `gravitino.lance-rest.host`
| No | `0.0.0.0` | 1.1.0 |
+| `LANCE_REST_PORT` | `gravitino.lance-rest.httpPort`
| No | `9101` | 1.1.0 |
+
+:::tip Configuration Tips
+- **Required:** Set `LANCE_REST_GRAVITINO_METALAKE_NAME` to your Gravitino
metalake name
+- **Conditional:** Update `LANCE_REST_GRAVITINO_URI` if Gravitino server is
not on `localhost` in the docker instance.
+- **Optional:** Other variables can use default values unless you have
specific requirements
+:::
+
+## Usage Guidelines
+
+When using Lance REST service with Gravitino backend, keep the following
considerations in mind:
+
+### Prerequisites
+- A running Gravitino server with a created metalake
+
+### Namespace Hierarchy
+Gravitino follows a three-level hierarchy: **catalog → schema → table**. When
creating namespaces or tables:
+
+1. **Parent must exist:** Before creating `lance_catalog/schema`, ensure
`lance_catalog` catalog exists in Gravitino metalake.
+2. **Two-level limit:** You can create namespace `lance_catalog/schema`, but
**not** `lance_catalog/schema/sub_schema`.
+3. **Table placement:** Tables can only be created under
`lance_catalog/schema`, not at catalog level.
+
+**Example Hierarchy:**
+```
+metalake
+└── lance_catalog (catalog - create via REST)
+ └── schema (namespace - create via REST)
+ └── table01 (table - create via REST)
+```
+
+### Delimiter Convention
+
+The Lance REST API uses `$` as the default delimiter to separate namespace
levels in URIs. When making HTTP requests:
+
+- **URL Encoding Required**: `$` must be URL-encoded as `%24`
+- **Example**: `lance_catalog$schema$table01` becomes
`lance_catalog%24schema%24table01` in URLs
+
+**Common Delimiters:**
+```
+Namespace path: lance_catalog.schema.table01
+URI representation: lance_catalog$schema$table01
+URL encoded: lance_catalog%24schema%24table01
+```
+
+:::caution Important Limitations
+- Currently supports only **two levels of namespaces** before tables
+- Tables **cannot** be nested deeper than schema level
+- Parent catalog must be created in Gravitino before using Lance REST API
+- Metadata operations require Gravitino server to be available
+- Namespace deletion is recursive and irreversible
+:::
+
+## Examples
+
+The following examples demonstrate how to interact with Lance REST service
using different programming languages and tools.
+
+**Prerequisites:**
+- Gravitino server is running with Lance REST service enabled.
+- A metalake has been created in Gravitino.
+
+<Tabs groupId="language" queryString>
+<TabItem value="shell" label="Shell">
+
+```shell
+# Create a catalog-level namespace
+# mode: "create" | "exist_ok" | "overwrite" for create namespace/table; mode:
"create" | "overwrite" for register table
+curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog/create \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "id": ["lance_catalog"],
+ "mode": "create"
+ }'
+
+# Create a schema namespace
+# Note: %24 is URL-encoded '$' character used as delimiter
+curl -X POST
http://localhost:9101/lance/v1/namespace/lance_catalog%24schema/create \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "id": ["lance_catalog", "schema"],
+ "mode": "create"
+ }'
+
+# Register an existing table
+curl -X POST
http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table01/register \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "id": ["lance_catalog", "schema", "table01"],
+ "location": "/tmp/lance_catalog/schema/table01",
+ "mode": "CREATE"
+ }'
+
+# Create a new empty table
+curl -X POST
http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table02/create-empty
\
+ -H 'Content-Type: application/json' \
+ -d '{
+ "id": ["lance_catalog", "schema", "table02"],
+ "location": "/tmp/lance_catalog/schema/table02",
+ "properties": { "description": "This is table02" }
+ }'
+
+# Create a table with schema, the schema is inferred from the Arrow IPC file
+curl -X POST \
+
"http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table03/create"; \
+ -H 'Content-Type: application/vnd.apache.arrow.stream' \
+ -H "x-lance-table-location: /tmp/lance_catalog/schema/table03" \
+ -H "x-lance-table-properties: {}" \
+ --data-binary "@${ARROW_FILE}"
+```
+
+</TabItem>
+<TabItem value="java" label="Java">
+
+```java
+// Add dependency: implementation("com.lancedb:lance-namespace-core:0.0.20")
+
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import java.util.HashMap;
+import java.util.Map;
+
+// Initialize allocator and namespace connection
+private final BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
+
+Map<String, String> props = new HashMap<>();
+props.put(RestNamespaceConfig.URI, "http://localhost:9101/lance";);
+props.put(RestNamespaceConfig.DELIMITER,
RestNamespaceConfig.DELIMITER_DEFAULT);
+
+LanceNamespace ns = LanceNamespaces.connect("rest", props, null, allocator);
+
+// Create catalog namespace
+CreateNamespaceRequest createCatalogNsRequest = new CreateNamespaceRequest();
+createCatalogNsRequest.addIdItem("lance_catalog");
+createCatalogNsRequest.setMode(CreateNamespaceRequest.ModeEnum.CREATE);
+ns.createNamespace(createCatalogNsRequest);
+
+// Create schema namespace
+CreateNamespaceRequest createSchemaNsRequest = new CreateNamespaceRequest();
+createSchemaNsRequest.addIdItem("lance_catalog");
+createSchemaNsRequest.addIdItem("schema");
+createSchemaNsRequest.setMode(CreateNamespaceRequest.ModeEnum.CREATE);
+ns.createNamespace(createSchemaNsRequest);
+
+// Register a table
+RegisterTableRequest registerTableRequest = new RegisterTableRequest();
+registerTableRequest.setLocation("/tmp/lance_catalog/schema/table01");
+registerTableRequest.setId(Lists.newArrayList("lance_catalog", "schema",
"table01"));
+registerTableRequest.setMode(RegisterTableRequest.ModeEnum.CREATE);
+ns.registerTable(registerTableRequest);
+
+// Create an empty table
+CreateEmptyTableRequest createEmptyTableRequest = new
CreateEmptyTableRequest();
+createEmptyTableRequest.setLocation("/tmp/lance_catalog/schema/table02");
+createEmptyTableRequest.setId(Lists.newArrayList("lance_catalog", "schema",
"table02"));
+ns.createEmptyTable(createEmptyTableRequest);
+
+// Create a table with schema inferred from Arrow IPC file
+CreateTableRequest createTableRequest = new CreateTableRequest();
+createTableRequest.setIds(Lists.newArrayList("lance_catalog", "schema",
"table03"));
+createTableRequest.setLocation("/tmp/lance_catalog/schema/table03");
+org.apache.arrow.vector.types.pojo.Schema schema =
+ new org.apache.arrow.vector.types.pojo.Schema(
+ Arrays.asList(
+ Field.nullable("id", new ArrowType.Int(32, true)),
+ Field.nullable("value", new ArrowType.Utf8())));
+byte[] body = ArrowUtils.generateIpcStream(schema);
+ns.createTable(createTableRequest, body);
+
+```
+
+</TabItem>
+<TabItem value="python" label="Python">
+
+```python
+# Install: pip install lance-namespace==0.0.20
+
+import lance_namespace as ln
+
+# Connect to Lance REST service
+ns = ln.connect("rest", {"uri": "http://your_lance_rest:9101/lance"})
+
+# Create catalog namespace
+create_catalog_ns_request = ln.CreateNamespaceRequest(id=["lance_catalog"])
+catalog = ns.create_namespace(create_catalog_ns_request)
+
+# Create schema namespace
+create_schema_ns_request = ln.CreateNamespaceRequest(id=["lance_catalog",
"schema"])
+schema = ns.create_namespace(create_schema_ns_request)
+
+# Register a table
+register_table_request = ln.RegisterTableRequest(
+ id=['lance_catalog', 'schema', 'table01'],
+ location='/tmp/lance_catalog/schema/table01'
+)
+ns.register_table(register_table_request)
+
+# Create an empty table
+create_empty_table_request = ln.CreateEmptyTableRequest(
+ id=['lance_catalog', 'schema', 'table02'],
+ location='/tmp/lance_catalog/schema/table02'
+)
+
+# Create a table with schema inferred from Arrow IPC file
+create_table_request = ln.CreateTableRequest(
+ id=['lance_catalog', 'schema', 'table03'],
+ location='/tmp/lance_catalog/schema/table03'
+)
+with open('schema.ipc', 'rb') as f:
+ body = f.read()
+
+ns.create_table(create_table_request, body)
+```
+
+</TabItem>
+</Tabs>
diff --git a/docs/manage-relational-metadata-using-gravitino.md
b/docs/manage-relational-metadata-using-gravitino.md
index a05d9393f1..a1fdfa87f5 100644
--- a/docs/manage-relational-metadata-using-gravitino.md
+++ b/docs/manage-relational-metadata-using-gravitino.md
@@ -27,6 +27,7 @@ For more details, please refer to the related doc.
- [**Apache Iceberg**](./lakehouse-iceberg-catalog.md)
- [**Apache Paimon**](./lakehouse-paimon-catalog.md)
- [**Apache Hudi**](./lakehouse-hudi-catalog.md)
+- [**Lakehouse generic**](./lakehouse-generic-catalog.md)
If you want to operate table and partition statistics, you can refer to the
[document](./manage-statistics-in-gravitino.md).
@@ -111,17 +112,18 @@ gravitino_client.create_catalog(name="catalog",
Currently, Gravitino supports the following catalog providers:
-| Catalog provider | Catalog property
|
-|---------------------|--------------------------------------------------------------------------------|
-| `hive` | [Hive catalog
property](./apache-hive-catalog.md#catalog-properties) |
-| `lakehouse-iceberg` | [Iceberg catalog
property](./lakehouse-iceberg-catalog.md#catalog-properties) |
-| `lakehouse-paimon` | [Paimon catalog
property](./lakehouse-paimon-catalog.md#catalog-properties) |
-| `lakehouse-hudi` | [Hudi catalog
property](./lakehouse-hudi-catalog.md#catalog-properties) |
-| `jdbc-mysql` | [MySQL catalog
property](./jdbc-mysql-catalog.md#catalog-properties) |
-| `jdbc-postgresql` | [PostgreSQL catalog
property](./jdbc-postgresql-catalog.md#catalog-properties) |
-| `jdbc-doris` | [Doris catalog
property](./jdbc-doris-catalog.md#catalog-properties) |
-| `jdbc-oceanbase` | [OceanBase catalog
property](./jdbc-oceanbase-catalog.md#catalog-properties) |
-| `jdbc-starrocks` | [StarRocks catalog
property](./jdbc-starrocks-catalog.md#catalog-properties) |
+| Catalog provider | Catalog property
|
+|---------------------|-----------------------------------------------------------------------------------------|
+| `hive` | [Hive catalog
property](./apache-hive-catalog.md#catalog-properties) |
+| `lakehouse-iceberg` | [Iceberg catalog
property](./lakehouse-iceberg-catalog.md#catalog-properties) |
+| `lakehouse-paimon` | [Paimon catalog
property](./lakehouse-paimon-catalog.md#catalog-properties) |
+| `lakehouse-hudi` | [Hudi catalog
property](./lakehouse-hudi-catalog.md#catalog-properties) |
+| `jdbc-mysql` | [MySQL catalog
property](./jdbc-mysql-catalog.md#catalog-properties) |
+| `jdbc-postgresql` | [PostgreSQL catalog
property](./jdbc-postgresql-catalog.md#catalog-properties) |
+| `jdbc-doris` | [Doris catalog
property](./jdbc-doris-catalog.md#catalog-properties) |
+| `jdbc-oceanbase` | [OceanBase catalog
property](./jdbc-oceanbase-catalog.md#catalog-properties) |
+| `jdbc-starrocks` | [StarRocks catalog
property](./jdbc-starrocks-catalog.md#catalog-properties) |
+| `lakehouse-generic` | [Lakehouse generic catalog
property](./lakehouse-generic-catalog.md#catalog-properties) |
### Load a catalog
@@ -506,6 +508,7 @@ Currently, Gravitino supports the following schema property:
| `jdbc-doris` | [Doris schema
property](./jdbc-doris-catalog.md#schema-properties) |
| `jdbc-oceanbase` | [OceanBase schema
property](./jdbc-oceanbase-catalog.md#schema-properties) |
| `jdbc-starrocks` | [StarRocks schema
property](./jdbc-starrocks-catalog.md#schema-properties) |
+| `lakehouse-generic` | [Lakehouse generic schema
property](./lakehouse-generic-catalog.md#schema-properties) |
### Load a schema
@@ -990,6 +993,7 @@ The following is a table of the column default value that
Gravitino supports for
| `jdbc-doris` | ✔ |
| `jdbc-oceanbase` | ✔ |
| `jdbc-starrocks` | ✔ |
+| `lakehouse-generic` | ✘ |
#### Table column auto-increment
@@ -1007,22 +1011,24 @@ The following table shows the column auto-increment
that Gravitino supports for
| `jdbc-doris` | ✘
|
| `jdbc-oceanbase` |
✔([limitations](./jdbc-oceanbase-catalog.md#table-column-auto-increment))
|
| `jdbc-starrocks` | ✔
|
+| `lakehouse-generic` | ✘
|
#### Table property and type mapping
The following is the table property that Gravitino supports:
-| Catalog provider | Table property
| Type mapping
|
-|---------------------|-----------------------------------------------------------------------------|----------------------------------------------------------------------------|
-| `hive` | [Hive table
property](./apache-hive-catalog.md#table-properties) | [Hive type
mapping](./apache-hive-catalog.md#table-column-types) |
-| `lakehouse-iceberg` | [Iceberg table
property](./lakehouse-iceberg-catalog.md#table-properties) | [Iceberg type
mapping](./lakehouse-iceberg-catalog.md#table-column-types) |
-| `lakehouse-paimon` | [Paimon table
property](./lakehouse-paimon-catalog.md#table-properties) | [Paimon type
mapping](./lakehouse-paimon-catalog.md#table-column-types) |
-| `lakehouse-hudi` | [Hudi table
property](./lakehouse-hudi-catalog.md#table-properties) | [Hudi type
mapping](./lakehouse-hudi-catalog.md#table-column-types) |
-| `jdbc-mysql` | [MySQL table
property](./jdbc-mysql-catalog.md#table-properties) | [MySQL type
mapping](./jdbc-mysql-catalog.md#table-column-types) |
-| `jdbc-postgresql` | [PostgreSQL table
property](./jdbc-postgresql-catalog.md#table-properties) | [PostgreSQL type
mapping](./jdbc-postgresql-catalog.md#table-column-types) |
-| `jdbc-doris` | [Doris table
property](./jdbc-doris-catalog.md#table-properties) | [Doris type
mapping](./jdbc-doris-catalog.md#table-column-types) |
-| `jdbc-oceanbase` | [OceanBase table
property](./jdbc-oceanbase-catalog.md#table-properties) | [OceanBase type
mapping](./jdbc-oceanbase-catalog.md#table-column-types) |
-| `jdbc-starrocks` | [StarRocks table
property](./jdbc-starrocks-catalog.md#table-properties) | [StarRocks type
mapping](./jdbc-starrocks-catalog.md#table-column-types) |
+| Catalog provider | Table property
| Type mapping
|
+|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `hive` | [Hive table
property](./apache-hive-catalog.md#table-properties)
| [Hive type
mapping](./apache-hive-catalog.md#table-column-types)
|
+| `lakehouse-iceberg` | [Iceberg table
property](./lakehouse-iceberg-catalog.md#table-properties)
| [Iceberg type
mapping](./lakehouse-iceberg-catalog.md#table-column-types)
|
+| `lakehouse-paimon` | [Paimon table
property](./lakehouse-paimon-catalog.md#table-properties)
| [Paimon type
mapping](./lakehouse-paimon-catalog.md#table-column-types)
|
+| `lakehouse-hudi` | [Hudi table
property](./lakehouse-hudi-catalog.md#table-properties)
| [Hudi type
mapping](./lakehouse-hudi-catalog.md#table-column-types)
|
+| `jdbc-mysql` | [MySQL table
property](./jdbc-mysql-catalog.md#table-properties)
| [MySQL type
mapping](./jdbc-mysql-catalog.md#table-column-types)
|
+| `jdbc-postgresql` | [PostgreSQL table
property](./jdbc-postgresql-catalog.md#table-properties)
| [PostgreSQL type
mapping](./jdbc-postgresql-catalog.md#table-column-types)
|
+| `jdbc-doris` | [Doris table
property](./jdbc-doris-catalog.md#table-properties)
| [Doris type
mapping](./jdbc-doris-catalog.md#table-column-types)
|
+| `jdbc-oceanbase` | [OceanBase table
property](./jdbc-oceanbase-catalog.md#table-properties)
| [OceanBase type
mapping](./jdbc-oceanbase-catalog.md#table-column-types)
|
+| `jdbc-starrocks` | [StarRocks table
property](./jdbc-starrocks-catalog.md#table-properties)
| [StarRocks type
mapping](./jdbc-starrocks-catalog.md#table-column-types)
|
+| `lakehouse-generic` | Lakehouse generic table property depends on specific
table implementation, for Lance table, please refer to
[doc](./lakehouse-generic-lance-table.md#table-properties), other table format,
please refer to related docs. | Lakehouse generic type mapping. Similar to
table properties, for Lance table, please refer to
[docs](./lakehouse-generic-lance-table.md#data-type-mappings) |
#### Table partitioning, distribution, sort ordering and indexes
