Copilot commented on code in PR #9173: URL: https://github.com/apache/gravitino/pull/9173#discussion_r2562391228
########## docs/lance-rest-service.md: ########## @@ -0,0 +1,204 @@ +--- +title: "Lance REST service" +slug: /lance-rest-service +keywords: + - Lance REST +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Background + +Since version 1.1.0, Gravitino includes a REST service for Lance datasets. The Lance REST service is a web service that allows you to interact with Lance datasets over HTTP. It provides endpoints for querying, inserting, updating, and deleting data in Lance datasets. +It abides by the [Lance REST API specification](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/lancedb/lance-namespace/refs/heads/main/docs/src/rest.yaml). More details about the specification, please refer to docs [here](https://lance.org/format/namespace/impls/rest/) + +Besides, Lance REST service can be run standalone without Gravitino server. + +## Capabilities + +The Lance REST service supports the APIs defined in the Lance REST API specification. The following are some of the key capabilities of the Lance REST service: +- Namespace management including creating namespace, listing namespaces, describing, deleting namespace, namespace exists check. +- Table management including creating tables including creating empty tables, dropping tables, registering tables and unregistering tables. +- Index management including creating index, listing indexes. Dropping index is not supported in 1.1.0. + +Full capabilities are listed in the following table: + +| Operation ID | Description | Since version | +|-------------------|-----------------------------------------------------------------------------------------------------|---------------| +| CreateNamespace | Create a Lance namespace | 1.1.0 | +| ListNamespaces | List all namespaces under a specific namespace | 1.1.0 | +| DescribeNamespace | Get details of a specific namespace | 1.1.0 | +| DropNamespace | Delete a specific namespace | 1.1.0 | +| NamespaceExists | Check if a namespace exists | 1.1.0 | +| ListTables | List all tables in a specific namespace | 1.1.0 | +| CreateTable | Create a new table in a specific namespace | 1.1.0 | +| DropTable | Delete a specific table from a namespace, drop table will drop metadata and Lance data all together | 1.1.0 | +| TableExists | Check if a specific table exists in a namespace | 1.1.0 | +| RegisterTable | Register an existing Lance table to a specific namespace | 1.1.0 | +| DeregisterTable | Unregister a specific table from a namespace, it will only remove metadata, Lance data will be kept | 1.1.0 | +| CreateIndex | Create an index on a specific table | 1.1.0 | +| ListIndexes | List all indexes on a specific table | 1.1.0 | + +## Getting started + +### Running Lance REST service with Gravitino + +To use the Lance REST service, you need to have Gravitino server running with Lance REST service enabled. The following are configurations to enable Lance REST service in Gravitino server. + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|--------------------------------------------|---------------| +| `gravitino.auxService.names` | Auxiliary service that runs Lance REST service, currently it supports `iceberg-rest` and `lance-rest`. It should include `lance-rest` if you want to start the Lance REST service like `lance-rest`, or `lance-rest, iceberg-rest` | iceberg-rest,lance-rest | Yes if Lance REST service is going to run | 0.2.0 | Review Comment: The description uses 'like' incorrectly. It should read 'such as' instead of 'like' for formal technical documentation. ```suggestion | `gravitino.auxService.names` | Auxiliary service that runs Lance REST service, currently it supports `iceberg-rest` and `lance-rest`. It should include `lance-rest` if you want to start the Lance REST service such as `lance-rest`, or `lance-rest, iceberg-rest` | iceberg-rest,lance-rest | Yes if Lance REST service is going to run | 0.2.0 | ``` ########## docs/lance-rest-service.md: ########## @@ -0,0 +1,204 @@ +--- +title: "Lance REST service" +slug: /lance-rest-service +keywords: + - Lance REST +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Background + +Since version 1.1.0, Gravitino includes a REST service for Lance datasets. The Lance REST service is a web service that allows you to interact with Lance datasets over HTTP. It provides endpoints for querying, inserting, updating, and deleting data in Lance datasets. +It abides by the [Lance REST API specification](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/lancedb/lance-namespace/refs/heads/main/docs/src/rest.yaml). More details about the specification, please refer to docs [here](https://lance.org/format/namespace/impls/rest/) + +Besides, Lance REST service can be run standalone without Gravitino server. + +## Capabilities + +The Lance REST service supports the APIs defined in the Lance REST API specification. The following are some of the key capabilities of the Lance REST service: +- Namespace management including creating namespace, listing namespaces, describing, deleting namespace, namespace exists check. +- Table management including creating tables including creating empty tables, dropping tables, registering tables and unregistering tables. +- Index management including creating index, listing indexes. Dropping index is not supported in 1.1.0. + +Full capabilities are listed in the following table: + +| Operation ID | Description | Since version | +|-------------------|-----------------------------------------------------------------------------------------------------|---------------| +| CreateNamespace | Create a Lance namespace | 1.1.0 | +| ListNamespaces | List all namespaces under a specific namespace | 1.1.0 | +| DescribeNamespace | Get details of a specific namespace | 1.1.0 | +| DropNamespace | Delete a specific namespace | 1.1.0 | +| NamespaceExists | Check if a namespace exists | 1.1.0 | +| ListTables | List all tables in a specific namespace | 1.1.0 | +| CreateTable | Create a new table in a specific namespace | 1.1.0 | +| DropTable | Delete a specific table from a namespace, drop table will drop metadata and Lance data all together | 1.1.0 | +| TableExists | Check if a specific table exists in a namespace | 1.1.0 | +| RegisterTable | Register an existing Lance table to a specific namespace | 1.1.0 | +| DeregisterTable | Unregister a specific table from a namespace, it will only remove metadata, Lance data will be kept | 1.1.0 | +| CreateIndex | Create an index on a specific table | 1.1.0 | +| ListIndexes | List all indexes on a specific table | 1.1.0 | + +## Getting started + +### Running Lance REST service with Gravitino + +To use the Lance REST service, you need to have Gravitino server running with Lance REST service enabled. The following are configurations to enable Lance REST service in Gravitino server. + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|--------------------------------------------|---------------| +| `gravitino.auxService.names` | Auxiliary service that runs Lance REST service, currently it supports `iceberg-rest` and `lance-rest`. It should include `lance-rest` if you want to start the Lance REST service like `lance-rest`, or `lance-rest, iceberg-rest` | iceberg-rest,lance-rest | Yes if Lance REST service is going to run | 0.2.0 | +| `gravitino.lance-rest.classpath` | The classpath of Lance REST service, relative to the Gravitino home directory. | lance-rest-server/libs | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The hostname that the Lance REST service run on. | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.namespace-backend` | backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | + +### Running Lance REST service standalone + +To run Lance REST service standalone without Gravitino server, you can use the following command: + +```shell +{GRAVITINO_HOME}/bin/gravitino-lance-rest-server.sh start +``` + +The following configurations are required to run Lance REST service standalone, you can set them in `gravitino-lance-rest-server.conf` file or pass them as command line arguments. +Typically, you only need to change the following configurations: + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------|--------------------------------------------|---------------| +| `gravitino.lance-rest.namespace-backend` | Backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The host name that Lance REST service run in | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | + +`namespace-backend`, `uri`, `port` and `host` have the same meaning as described in the previous section, and they have the default values. In most cases you only need to change `metalake-name` to your Gravitino metalake name. +For other configurations listed in the file, just keep their default values. + +## Using Lance REST service + +Currently, as the Lance REST service only supports Gravitino backend, there are some limitations when using Lance REST service: +- You need to have a running Gravitino server with a metalake created. +- As Gravitino has three hierarchies: catalog -> schema -> table, so when you create namespaces or a table via Lance REST service, you need to make sure the parent hierarchy exists. For example, when you create a namespace `lance_catalog/schema`, you need to make sure the catalog `lance_catalog` already exists in Gravitino metalake. If not, you need to create the namespace(catalog) `lance_catalog` first. +- Currently, we can only support two layers of namespaces and then tables, that is to say, you can create namespace like `lance_catalog/schema`, but you cannot create namespace like `lance_catalog/schema/sub_schema`. Tables can only be created under the namespace `lance_catalog/schema`. + +## Example + +When Gravitino server is started with Lance REST service starts successfully, and a `generic-lakehouse` catalog named `lance_catalog` is created in Gravitino metalake, you can use the following Python code to interact with Lance REST service: + + + +<Tabs groupId="language" queryString> +<TabItem value="shell" label="Shell"> + +```shell +# Create a namespace +# mode can be create or exist_ok or overwrite +curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog/create -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog"], + "mode": "create" +}' + +# Create a schema namespace +# %24 is the URL encoded character for $ +curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog%24schema/create -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog", "schema"], + "mode": "create" +}' + +# register a table +curl -X POST http://localhost:9101/lance/v1/table/lance_catalog2%24schema%24table01/register -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog","schema","table01"], + "location": "/tmp/lance_catalog/schema/table01" +}' + +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java +// implementation("com.lancedb:lance-namespace-core:0.0.19") Review Comment: The comment shows version 0.0.19, but the Python example on line 158 uses version 0.0.20. These versions should be consistent across all language examples for clarity. ```suggestion // implementation("com.lancedb:lance-namespace-core:0.0.20") ``` ########## docs/generic-lakehouse-catalog.md: ########## @@ -0,0 +1,281 @@ +--- +title: "Lakehouse catalog" +slug: /lakehouse-catalog +keywords: + - lakehouse + - lance + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Introduction + +Generic Lakehouse Catalog is a Gravitino catalog implementation that enables Gravitino to interact with Lakehouse storage systems that use file system for storing tabular data. Such lakehouse system could be built on top of object stores like Amazon S3, Azure Blob Storage, Google Cloud Storage, or HDFS. +Theoretically, it can work with any lakehouse storage system that supports standard file system operations such as Apache Iceberg, Lance, Delta Lake, and Apache Hudi. However, currently Gravitino only provides native support for Lance-based lakehouse storage systems. + +### Requirements and limitations + +- The lakehouse storage system must support standard file system operations such as listing directories, reading files, and writing files. + +## Catalog + +### Catalog capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Catalog properties + +The only property that need to be noted for a generic lakehouse catalog is `location`. This property specifies the root location of the lakehouse storage system. All schemas and tables will be stored under this location if not +specified otherwise in schema or table properties. + + +### Catalog operations + +All operations are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +One thing needs to be noted is that the provider will be `generic_lakehouse` when creating a generic lakehouse catalog. +That is: + +<Tabs groupId='language' queryString> +<TabItem value="shell" label="Shell"> + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "generic_lakehouse_catalog", + "type": "RELATIONAL", + "comment": "comment", + "provider": "generic-lakehouse", + "properties": { + } +}' http://localhost:8090/api/metalakes/metalake/catalogs +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java + +// Assuming you have just created a metalake named `metalake` +GravitinoClient gravitinoClient = GravitinoClient + .builder("http://127.0.0.1:8090";) + .withMetalake("metalake") + .build(); + +Map<String, String> genericCatalogProperties = ImmutableMap.<String, String>builder() + .put("location", "hdfs://localhost:9000/user/lakehouse") // The root location of the lakehouse storage system + .build(); + +Catalog catalog = gravitinoClient.createCatalog("generic_lakehouse_catalog", + Type.RELATIONAL, + "generic_lakehouse", + "This is a generic lakehouse catalog", + genericCatalogProperties); // Please change the properties according to the value of the provider. +// ... +``` + +</TabItem> +</Tabs> + + +## Schema + +### Schema capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Schema properties + +The same as catalog properties, please refer to [Catalog properties](#catalog-properties) section for more details. Schema `location` property can be used to specify the location to store all tables under this schema. + +### Schema operations + +Please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations) for more details. + +## Table + +### Table capabilities + +Currently, for a lance table, Gravitino supports the following capabilities: +- List +- Load +- Alter (partial supported) +- Create/register +- Drop and truncate + +### Table partitions + +Not support now + +### Table sort orders + +Not support now. + +### Table distributions + +Not support now. + +### Table column types + +Since Lance uses Apache Arrow as the table schema, the following table shows the mapping between Gravitino types and Arrow types: + +| Gravitino Type | Arrow Type | +|----------------------------------|-----------------------------------------| +| `Struct` | `Struct` | +| `Map` | `Map` | +| `List` | `Array` | +| `Boolean` | `Boolean` | +| `Byte` | `Int8` | +| `Short` | `Int16` | +| `Integer` | `Int32` | +| `Long` | `Int64` | +| `Float` | `Float` | +| `Double` | `Double` | +| `String` | `Utf8` | +| `Binary` | `Binary` | +| `Decimal(p, s)` | `Decimal(p, s)` (128-bit) | +| `Date` | `Date` | +| `Timestamp`/`Timestamp(6)` | `TimestampType withoutZone` | +| `Timestamp(0)` | `TimestampType Second withoutZone` | +| `Timestamp(3)` | `TimestampType Millisecond withoutZone` | +| `Timestamp(9)` | `TimestampType Nanosecond withoutZone` | +| `Timestamp_tz`/`Timestamp_tz(6)` | `TimestampType Microsecond withUtc` | +| `Timestamp_tz(0)` | `TimestampType Second withUtc` | +| `Timestamp_tz(3)` | `TimestampType Millisecond withUtc` | +| `Timestamp_tz(9)` | `TimestampType Nanosecond withUtc` | +| `Time`/`Time(9)` | `Time Nanosecond` | +| `Null` | `Null` | +| `Fixed(n)` | `Fixed-Size Binary(n)` | +| `Interval_year` | `Interval(YearMonth)` | +| `Interval_day` | `Duration(Microsecond)` | +| `External(arrow_field_json_str)` | Any Arrow Field (see note below) | + +`External(arrow_field_json_str)`: + +As the table above shows, Gravitino provides mappings for most common data types. However, +in some cases, you may need to use an Arrow data type that is not directly supported by Gravitino. + +To address this, Gravitino introduces the `External(arrow_field_json_str)` type, +which allows you to define any Arrow data type by providing the JSON string of an Arrow `Field`. + +The JSON string must conform to the Apache Arrow `Field` [specification](https://github.com/apache/arrow-java/blob/ed81e5981a2bee40584b3a411ed755cb4cc5b91f/vector/src/main/java/org/apache/arrow/vector/types/pojo/Field.java#L80C1-L86C68), +including details such as the field name, data type, and nullability. +Here are some examples of how to use `External` type for various Arrow types that are not natively supported by Gravitino: + +| Arrow Type | External type | +|-------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Large Utf8` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largeutf8\"},\"children\":[]}")` | +| `Large Binary` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largebinary\"},\"children\":[]}")` | +| `Large List` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largelist\"},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\", \"bitWidth\":32, \"isSigned\": true},\"children\":[]}]}")` | +| `Fixed-Size List` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"fixedsizelist\", \"listSize\":10},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\", \"bitWidth\":32, \"isSigned\": true},\"children\":[]}]}")` | + +**Important considerations:** +- The `name` attribute and `nullable` attribute in the JSON string must exactly match the corresponding column name and nullable in the Gravitino table. +- The `children` array should be empty for primitive types. For complex types like `Struct` or `List`, it must contain the definitions of the child fields. + +### Table properties + +Currently, the following properties are required for a table in a generic lakehouse catalog + +| Configuration item | Description | Default value | Required | Since version | +|--------------------|-----------------------------------------------------------------------------------------------|---------------|-----------------------------------------------------------------------------|---------------| +| `format` | The format for a table, it can be `lance`, `iceberg`,..., currently, it only supports `lance` | (none) | Yes | 1.1.0 | +| `location` | The location to storage the table meta and data. | (none) | No, but if this is not set in catalog or schema, then it's a required value | 1.1.0 | Review Comment: Grammatical error: 'storage' should be 'store' (verb form). 'Storage' is a noun, but this context requires the verb 'to store'. ```suggestion | `location` | The location to store the table meta and data. | (none) | No, but if this is not set in catalog or schema, then it's a required value | 1.1.0 | ``` ########## docs/generic-lakehouse-catalog.md: ########## @@ -0,0 +1,281 @@ +--- +title: "Lakehouse catalog" +slug: /lakehouse-catalog +keywords: + - lakehouse + - lance + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Introduction + +Generic Lakehouse Catalog is a Gravitino catalog implementation that enables Gravitino to interact with Lakehouse storage systems that use file system for storing tabular data. Such lakehouse system could be built on top of object stores like Amazon S3, Azure Blob Storage, Google Cloud Storage, or HDFS. +Theoretically, it can work with any lakehouse storage system that supports standard file system operations such as Apache Iceberg, Lance, Delta Lake, and Apache Hudi. However, currently Gravitino only provides native support for Lance-based lakehouse storage systems. + +### Requirements and limitations + +- The lakehouse storage system must support standard file system operations such as listing directories, reading files, and writing files. + +## Catalog + +### Catalog capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Catalog properties + +The only property that need to be noted for a generic lakehouse catalog is `location`. This property specifies the root location of the lakehouse storage system. All schemas and tables will be stored under this location if not +specified otherwise in schema or table properties. + + +### Catalog operations + +All operations are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +One thing needs to be noted is that the provider will be `generic_lakehouse` when creating a generic lakehouse catalog. +That is: + +<Tabs groupId='language' queryString> +<TabItem value="shell" label="Shell"> + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "generic_lakehouse_catalog", + "type": "RELATIONAL", + "comment": "comment", + "provider": "generic-lakehouse", + "properties": { + } +}' http://localhost:8090/api/metalakes/metalake/catalogs +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java + +// Assuming you have just created a metalake named `metalake` +GravitinoClient gravitinoClient = GravitinoClient + .builder("http://127.0.0.1:8090";) + .withMetalake("metalake") + .build(); + +Map<String, String> genericCatalogProperties = ImmutableMap.<String, String>builder() + .put("location", "hdfs://localhost:9000/user/lakehouse") // The root location of the lakehouse storage system + .build(); + +Catalog catalog = gravitinoClient.createCatalog("generic_lakehouse_catalog", + Type.RELATIONAL, + "generic_lakehouse", + "This is a generic lakehouse catalog", + genericCatalogProperties); // Please change the properties according to the value of the provider. +// ... +``` + +</TabItem> +</Tabs> + + +## Schema + +### Schema capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Schema properties + +The same as catalog properties, please refer to [Catalog properties](#catalog-properties) section for more details. Schema `location` property can be used to specify the location to store all tables under this schema. + +### Schema operations + +Please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations) for more details. + +## Table + +### Table capabilities + +Currently, for a lance table, Gravitino supports the following capabilities: +- List +- Load +- Alter (partial supported) +- Create/register +- Drop and truncate + +### Table partitions + +Not support now + +### Table sort orders + +Not support now. + +### Table distributions + +Not support now. Review Comment: Grammatical error: Should be 'Not supported now.' to use the correct passive voice form. ```suggestion Not supported now. ### Table sort orders Not supported now. ### Table distributions Not supported now. ``` ########## docs/generic-lakehouse-catalog.md: ########## @@ -0,0 +1,281 @@ +--- +title: "Lakehouse catalog" +slug: /lakehouse-catalog +keywords: + - lakehouse + - lance + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Introduction + +Generic Lakehouse Catalog is a Gravitino catalog implementation that enables Gravitino to interact with Lakehouse storage systems that use file system for storing tabular data. Such lakehouse system could be built on top of object stores like Amazon S3, Azure Blob Storage, Google Cloud Storage, or HDFS. +Theoretically, it can work with any lakehouse storage system that supports standard file system operations such as Apache Iceberg, Lance, Delta Lake, and Apache Hudi. However, currently Gravitino only provides native support for Lance-based lakehouse storage systems. + +### Requirements and limitations + +- The lakehouse storage system must support standard file system operations such as listing directories, reading files, and writing files. + +## Catalog + +### Catalog capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Catalog properties + +The only property that need to be noted for a generic lakehouse catalog is `location`. This property specifies the root location of the lakehouse storage system. All schemas and tables will be stored under this location if not +specified otherwise in schema or table properties. + + +### Catalog operations + +All operations are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +One thing needs to be noted is that the provider will be `generic_lakehouse` when creating a generic lakehouse catalog. +That is: + +<Tabs groupId='language' queryString> +<TabItem value="shell" label="Shell"> + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "generic_lakehouse_catalog", + "type": "RELATIONAL", + "comment": "comment", + "provider": "generic-lakehouse", + "properties": { + } +}' http://localhost:8090/api/metalakes/metalake/catalogs +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java + +// Assuming you have just created a metalake named `metalake` +GravitinoClient gravitinoClient = GravitinoClient + .builder("http://127.0.0.1:8090";) + .withMetalake("metalake") + .build(); + +Map<String, String> genericCatalogProperties = ImmutableMap.<String, String>builder() + .put("location", "hdfs://localhost:9000/user/lakehouse") // The root location of the lakehouse storage system + .build(); + +Catalog catalog = gravitinoClient.createCatalog("generic_lakehouse_catalog", + Type.RELATIONAL, + "generic_lakehouse", + "This is a generic lakehouse catalog", + genericCatalogProperties); // Please change the properties according to the value of the provider. +// ... +``` + +</TabItem> +</Tabs> + + +## Schema + +### Schema capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Schema properties + +The same as catalog properties, please refer to [Catalog properties](#catalog-properties) section for more details. Schema `location` property can be used to specify the location to store all tables under this schema. + +### Schema operations + +Please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations) for more details. + +## Table + +### Table capabilities + +Currently, for a lance table, Gravitino supports the following capabilities: +- List +- Load +- Alter (partial supported) +- Create/register +- Drop and truncate + +### Table partitions + +Not support now + +### Table sort orders + +Not support now. + +### Table distributions + +Not support now. Review Comment: Grammatical error: Should be 'Not supported now' to use the correct passive voice form. ```suggestion Not supported now ### Table sort orders Not supported now. ### Table distributions Not supported now. ``` ########## docs/lance-rest-service.md: ########## @@ -0,0 +1,204 @@ +--- +title: "Lance REST service" +slug: /lance-rest-service +keywords: + - Lance REST +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Background + +Since version 1.1.0, Gravitino includes a REST service for Lance datasets. The Lance REST service is a web service that allows you to interact with Lance datasets over HTTP. It provides endpoints for querying, inserting, updating, and deleting data in Lance datasets. +It abides by the [Lance REST API specification](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/lancedb/lance-namespace/refs/heads/main/docs/src/rest.yaml). More details about the specification, please refer to docs [here](https://lance.org/format/namespace/impls/rest/) + +Besides, Lance REST service can be run standalone without Gravitino server. + +## Capabilities + +The Lance REST service supports the APIs defined in the Lance REST API specification. The following are some of the key capabilities of the Lance REST service: +- Namespace management including creating namespace, listing namespaces, describing, deleting namespace, namespace exists check. +- Table management including creating tables including creating empty tables, dropping tables, registering tables and unregistering tables. +- Index management including creating index, listing indexes. Dropping index is not supported in 1.1.0. + +Full capabilities are listed in the following table: + +| Operation ID | Description | Since version | +|-------------------|-----------------------------------------------------------------------------------------------------|---------------| +| CreateNamespace | Create a Lance namespace | 1.1.0 | +| ListNamespaces | List all namespaces under a specific namespace | 1.1.0 | +| DescribeNamespace | Get details of a specific namespace | 1.1.0 | +| DropNamespace | Delete a specific namespace | 1.1.0 | +| NamespaceExists | Check if a namespace exists | 1.1.0 | +| ListTables | List all tables in a specific namespace | 1.1.0 | +| CreateTable | Create a new table in a specific namespace | 1.1.0 | +| DropTable | Delete a specific table from a namespace, drop table will drop metadata and Lance data all together | 1.1.0 | +| TableExists | Check if a specific table exists in a namespace | 1.1.0 | +| RegisterTable | Register an existing Lance table to a specific namespace | 1.1.0 | +| DeregisterTable | Unregister a specific table from a namespace, it will only remove metadata, Lance data will be kept | 1.1.0 | +| CreateIndex | Create an index on a specific table | 1.1.0 | +| ListIndexes | List all indexes on a specific table | 1.1.0 | + +## Getting started + +### Running Lance REST service with Gravitino + +To use the Lance REST service, you need to have Gravitino server running with Lance REST service enabled. The following are configurations to enable Lance REST service in Gravitino server. + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|--------------------------------------------|---------------| +| `gravitino.auxService.names` | Auxiliary service that runs Lance REST service, currently it supports `iceberg-rest` and `lance-rest`. It should include `lance-rest` if you want to start the Lance REST service like `lance-rest`, or `lance-rest, iceberg-rest` | iceberg-rest,lance-rest | Yes if Lance REST service is going to run | 0.2.0 | +| `gravitino.lance-rest.classpath` | The classpath of Lance REST service, relative to the Gravitino home directory. | lance-rest-server/libs | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The hostname that the Lance REST service run on. | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.namespace-backend` | backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | + +### Running Lance REST service standalone + +To run Lance REST service standalone without Gravitino server, you can use the following command: + +```shell +{GRAVITINO_HOME}/bin/gravitino-lance-rest-server.sh start +``` + +The following configurations are required to run Lance REST service standalone, you can set them in `gravitino-lance-rest-server.conf` file or pass them as command line arguments. +Typically, you only need to change the following configurations: + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------|--------------------------------------------|---------------| +| `gravitino.lance-rest.namespace-backend` | Backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The host name that Lance REST service run in | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | Review Comment: Grammatical error: 'run in' should be 'runs on' to match the description on line 55. ```suggestion | `gravitino.lance-rest.host` | The host name that Lance REST service runs on | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | ``` ########## docs/lance-rest-service.md: ########## @@ -0,0 +1,204 @@ +--- +title: "Lance REST service" +slug: /lance-rest-service +keywords: + - Lance REST +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Background + +Since version 1.1.0, Gravitino includes a REST service for Lance datasets. The Lance REST service is a web service that allows you to interact with Lance datasets over HTTP. It provides endpoints for querying, inserting, updating, and deleting data in Lance datasets. +It abides by the [Lance REST API specification](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/lancedb/lance-namespace/refs/heads/main/docs/src/rest.yaml). More details about the specification, please refer to docs [here](https://lance.org/format/namespace/impls/rest/) + +Besides, Lance REST service can be run standalone without Gravitino server. + +## Capabilities + +The Lance REST service supports the APIs defined in the Lance REST API specification. The following are some of the key capabilities of the Lance REST service: +- Namespace management including creating namespace, listing namespaces, describing, deleting namespace, namespace exists check. +- Table management including creating tables including creating empty tables, dropping tables, registering tables and unregistering tables. +- Index management including creating index, listing indexes. Dropping index is not supported in 1.1.0. + +Full capabilities are listed in the following table: + +| Operation ID | Description | Since version | +|-------------------|-----------------------------------------------------------------------------------------------------|---------------| +| CreateNamespace | Create a Lance namespace | 1.1.0 | +| ListNamespaces | List all namespaces under a specific namespace | 1.1.0 | +| DescribeNamespace | Get details of a specific namespace | 1.1.0 | +| DropNamespace | Delete a specific namespace | 1.1.0 | +| NamespaceExists | Check if a namespace exists | 1.1.0 | +| ListTables | List all tables in a specific namespace | 1.1.0 | +| CreateTable | Create a new table in a specific namespace | 1.1.0 | +| DropTable | Delete a specific table from a namespace, drop table will drop metadata and Lance data all together | 1.1.0 | +| TableExists | Check if a specific table exists in a namespace | 1.1.0 | +| RegisterTable | Register an existing Lance table to a specific namespace | 1.1.0 | +| DeregisterTable | Unregister a specific table from a namespace, it will only remove metadata, Lance data will be kept | 1.1.0 | +| CreateIndex | Create an index on a specific table | 1.1.0 | +| ListIndexes | List all indexes on a specific table | 1.1.0 | + +## Getting started + +### Running Lance REST service with Gravitino + +To use the Lance REST service, you need to have Gravitino server running with Lance REST service enabled. The following are configurations to enable Lance REST service in Gravitino server. + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|--------------------------------------------|---------------| +| `gravitino.auxService.names` | Auxiliary service that runs Lance REST service, currently it supports `iceberg-rest` and `lance-rest`. It should include `lance-rest` if you want to start the Lance REST service like `lance-rest`, or `lance-rest, iceberg-rest` | iceberg-rest,lance-rest | Yes if Lance REST service is going to run | 0.2.0 | +| `gravitino.lance-rest.classpath` | The classpath of Lance REST service, relative to the Gravitino home directory. | lance-rest-server/libs | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The hostname that the Lance REST service run on. | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.namespace-backend` | backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | + +### Running Lance REST service standalone + +To run Lance REST service standalone without Gravitino server, you can use the following command: + +```shell +{GRAVITINO_HOME}/bin/gravitino-lance-rest-server.sh start +``` + +The following configurations are required to run Lance REST service standalone, you can set them in `gravitino-lance-rest-server.conf` file or pass them as command line arguments. +Typically, you only need to change the following configurations: + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------|--------------------------------------------|---------------| +| `gravitino.lance-rest.namespace-backend` | Backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The host name that Lance REST service run in | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | + +`namespace-backend`, `uri`, `port` and `host` have the same meaning as described in the previous section, and they have the default values. In most cases you only need to change `metalake-name` to your Gravitino metalake name. +For other configurations listed in the file, just keep their default values. + +## Using Lance REST service + +Currently, as the Lance REST service only supports Gravitino backend, there are some limitations when using Lance REST service: +- You need to have a running Gravitino server with a metalake created. +- As Gravitino has three hierarchies: catalog -> schema -> table, so when you create namespaces or a table via Lance REST service, you need to make sure the parent hierarchy exists. For example, when you create a namespace `lance_catalog/schema`, you need to make sure the catalog `lance_catalog` already exists in Gravitino metalake. If not, you need to create the namespace(catalog) `lance_catalog` first. +- Currently, we can only support two layers of namespaces and then tables, that is to say, you can create namespace like `lance_catalog/schema`, but you cannot create namespace like `lance_catalog/schema/sub_schema`. Tables can only be created under the namespace `lance_catalog/schema`. + +## Example + +When Gravitino server is started with Lance REST service starts successfully, and a `generic-lakehouse` catalog named `lance_catalog` is created in Gravitino metalake, you can use the following Python code to interact with Lance REST service: + + + +<Tabs groupId="language" queryString> +<TabItem value="shell" label="Shell"> + +```shell +# Create a namespace +# mode can be create or exist_ok or overwrite +curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog/create -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog"], + "mode": "create" +}' + +# Create a schema namespace +# %24 is the URL encoded character for $ +curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog%24schema/create -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog", "schema"], + "mode": "create" +}' + +# register a table +curl -X POST http://localhost:9101/lance/v1/table/lance_catalog2%24schema%24table01/register -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog","schema","table01"], + "location": "/tmp/lance_catalog/schema/table01" +}' + +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java +// implementation("com.lancedb:lance-namespace-core:0.0.19") + +private final BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); +LanceNamespace ns = LanceNamespace.connect("rest", Map.of("uri", "http://localhost:9101/lance";)); +HashMap<String, String> props = Maps.newHashMap(); +props.put(RestNamespaceConfig.URI, getLanceRestServiceUrl()); +props.put(RestNamespaceConfig.DELIMITER, RestNamespaceConfig.DELIMITER_DEFAULT); +LanceNamespace ns = LanceNamespaces.connect("rest", props, null, allocator); + +// Create a namespace +CreateNamespaceRequest createCatalogNsRequest = new CreateNamespaceRequest(); +createCatalogNsRequest.addIdItem("lance_catalog"); +createCatalogNsRequest.setMode(CreateNamespaceRequest.ModeEnum.CREATE); +ns.createNamespace(createCatalogNsRequest); + +// Create a schema namespace +CreateNamespaceRequest createSchemaNsRequest = new CreateNamespaceRequest(); +createSchemaNsRequest.addIdItem("lance_catalog"); +createSchemaNsRequest.addIdItem("schema"); +createSchemaNsRequest.setMode(CreateNamespaceRequest.ModeEnum.CREATE); +ns.createNamespace(createSchemaNsRequest); + +// register a table +RegisterTableRequest registerTableRequest = new RegisterTableRequest(); +registerTableRequest.setLocation(location); +registerTableRequest.setId(Lists.newArrayList("lance_catalog", "schema", "table01")); +ns.registerTable(registerTableRequest); +``` + +</TabItem> +<TabItem value="python" label="Python"> + +```python +# you need to install lance-namespace package first by 'pip install lance-namespace==0.0.20' +import lance_namespace as ln +ns = ln.connect("rest", {"uri": "http://localhost:9101/lance"}) +# Create a namespace +create_catalog_ns_request = ln.CreateNamespaceRequest(id=["lance_catalog"]) +catalog = ns.create_namespace(create_catalog_ns_request) +create_schema_ns_request = ln.CreateNamespaceRequest(id=["lance_catalog", "schema"]) +schema = ns.create_namespace(create_schema_ns_request) + +# register a table +register_table_request=ln.RegisterTableRequest(id=['lance_catalog','schema','table01'], location='/tmp/lance_catalog/schema/table01') +ns.register_table(register_table_request) +... Review Comment: The trailing '...' suggests incomplete code example. Either provide the complete example or remove this placeholder to avoid confusion. ```suggestion ``` ########## docs/lance-rest-service.md: ########## @@ -0,0 +1,204 @@ +--- +title: "Lance REST service" +slug: /lance-rest-service +keywords: + - Lance REST +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Background + +Since version 1.1.0, Gravitino includes a REST service for Lance datasets. The Lance REST service is a web service that allows you to interact with Lance datasets over HTTP. It provides endpoints for querying, inserting, updating, and deleting data in Lance datasets. +It abides by the [Lance REST API specification](https://editor-next.swagger.io/?url=https://raw.githubusercontent.com/lancedb/lance-namespace/refs/heads/main/docs/src/rest.yaml). More details about the specification, please refer to docs [here](https://lance.org/format/namespace/impls/rest/) + +Besides, Lance REST service can be run standalone without Gravitino server. + +## Capabilities + +The Lance REST service supports the APIs defined in the Lance REST API specification. The following are some of the key capabilities of the Lance REST service: +- Namespace management including creating namespace, listing namespaces, describing, deleting namespace, namespace exists check. +- Table management including creating tables including creating empty tables, dropping tables, registering tables and unregistering tables. +- Index management including creating index, listing indexes. Dropping index is not supported in 1.1.0. + +Full capabilities are listed in the following table: + +| Operation ID | Description | Since version | +|-------------------|-----------------------------------------------------------------------------------------------------|---------------| +| CreateNamespace | Create a Lance namespace | 1.1.0 | +| ListNamespaces | List all namespaces under a specific namespace | 1.1.0 | +| DescribeNamespace | Get details of a specific namespace | 1.1.0 | +| DropNamespace | Delete a specific namespace | 1.1.0 | +| NamespaceExists | Check if a namespace exists | 1.1.0 | +| ListTables | List all tables in a specific namespace | 1.1.0 | +| CreateTable | Create a new table in a specific namespace | 1.1.0 | +| DropTable | Delete a specific table from a namespace, drop table will drop metadata and Lance data all together | 1.1.0 | +| TableExists | Check if a specific table exists in a namespace | 1.1.0 | +| RegisterTable | Register an existing Lance table to a specific namespace | 1.1.0 | +| DeregisterTable | Unregister a specific table from a namespace, it will only remove metadata, Lance data will be kept | 1.1.0 | +| CreateIndex | Create an index on a specific table | 1.1.0 | +| ListIndexes | List all indexes on a specific table | 1.1.0 | + +## Getting started + +### Running Lance REST service with Gravitino + +To use the Lance REST service, you need to have Gravitino server running with Lance REST service enabled. The following are configurations to enable Lance REST service in Gravitino server. + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|--------------------------------------------|---------------| +| `gravitino.auxService.names` | Auxiliary service that runs Lance REST service, currently it supports `iceberg-rest` and `lance-rest`. It should include `lance-rest` if you want to start the Lance REST service like `lance-rest`, or `lance-rest, iceberg-rest` | iceberg-rest,lance-rest | Yes if Lance REST service is going to run | 0.2.0 | +| `gravitino.lance-rest.classpath` | The classpath of Lance REST service, relative to the Gravitino home directory. | lance-rest-server/libs | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The hostname that the Lance REST service run on. | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.namespace-backend` | backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | + +### Running Lance REST service standalone + +To run Lance REST service standalone without Gravitino server, you can use the following command: + +```shell +{GRAVITINO_HOME}/bin/gravitino-lance-rest-server.sh start +``` + +The following configurations are required to run Lance REST service standalone, you can set them in `gravitino-lance-rest-server.conf` file or pass them as command line arguments. +Typically, you only need to change the following configurations: + +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------|--------------------------------------------|---------------| +| `gravitino.lance-rest.namespace-backend` | Backend to store namespace metadata, currently it only supports `gravitino` | gravitino | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.uri` | Gravitino server URI, it should be set when `namespace-backend` is `gravitino` | http://localhost:8090 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.gravitino.metalake-name` | Gravitino metalake name, it should be set when `namespace-backend` is `gravitino` | (none) | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.port` | The port number that Lance REST service listens on. | 9101 | Yes if Lance REST service is going to run | 1.1.0 | +| `gravitino.lance-rest.host` | The host name that Lance REST service run in | 0.0.0.0 | Yes if Lance REST service is going to run | 1.1.0 | + +`namespace-backend`, `uri`, `port` and `host` have the same meaning as described in the previous section, and they have the default values. In most cases you only need to change `metalake-name` to your Gravitino metalake name. +For other configurations listed in the file, just keep their default values. + +## Using Lance REST service + +Currently, as the Lance REST service only supports Gravitino backend, there are some limitations when using Lance REST service: +- You need to have a running Gravitino server with a metalake created. +- As Gravitino has three hierarchies: catalog -> schema -> table, so when you create namespaces or a table via Lance REST service, you need to make sure the parent hierarchy exists. For example, when you create a namespace `lance_catalog/schema`, you need to make sure the catalog `lance_catalog` already exists in Gravitino metalake. If not, you need to create the namespace(catalog) `lance_catalog` first. +- Currently, we can only support two layers of namespaces and then tables, that is to say, you can create namespace like `lance_catalog/schema`, but you cannot create namespace like `lance_catalog/schema/sub_schema`. Tables can only be created under the namespace `lance_catalog/schema`. + +## Example + +When Gravitino server is started with Lance REST service starts successfully, and a `generic-lakehouse` catalog named `lance_catalog` is created in Gravitino metalake, you can use the following Python code to interact with Lance REST service: + + + +<Tabs groupId="language" queryString> +<TabItem value="shell" label="Shell"> + +```shell +# Create a namespace +# mode can be create or exist_ok or overwrite +curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog/create -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog"], + "mode": "create" +}' + +# Create a schema namespace +# %24 is the URL encoded character for $ +curl -X POST http://localhost:9101/lance/v1/namespace/lance_catalog%24schema/create -H 'Content-Type: application/json' -d '{ + "id": ["lance_catalog", "schema"], + "mode": "create" +}' + +# register a table +curl -X POST http://localhost:9101/lance/v1/table/lance_catalog2%24schema%24table01/register -H 'Content-Type: application/json' -d '{ Review Comment: The catalog name 'lance_catalog2' appears to be inconsistent with the previous examples which use 'lance_catalog'. This could confuse users following the examples sequentially. ```suggestion curl -X POST http://localhost:9101/lance/v1/table/lance_catalog%24schema%24table01/register -H 'Content-Type: application/json' -d '{ ``` ########## docs/generic-lakehouse-catalog.md: ########## @@ -0,0 +1,281 @@ +--- +title: "Lakehouse catalog" +slug: /lakehouse-catalog +keywords: + - lakehouse + - lance + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Introduction + +Generic Lakehouse Catalog is a Gravitino catalog implementation that enables Gravitino to interact with Lakehouse storage systems that use file system for storing tabular data. Such lakehouse system could be built on top of object stores like Amazon S3, Azure Blob Storage, Google Cloud Storage, or HDFS. +Theoretically, it can work with any lakehouse storage system that supports standard file system operations such as Apache Iceberg, Lance, Delta Lake, and Apache Hudi. However, currently Gravitino only provides native support for Lance-based lakehouse storage systems. + +### Requirements and limitations + +- The lakehouse storage system must support standard file system operations such as listing directories, reading files, and writing files. + +## Catalog + +### Catalog capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Catalog properties + +The only property that need to be noted for a generic lakehouse catalog is `location`. This property specifies the root location of the lakehouse storage system. All schemas and tables will be stored under this location if not +specified otherwise in schema or table properties. + + +### Catalog operations + +All operations are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +One thing needs to be noted is that the provider will be `generic_lakehouse` when creating a generic lakehouse catalog. +That is: + +<Tabs groupId='language' queryString> +<TabItem value="shell" label="Shell"> + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "generic_lakehouse_catalog", + "type": "RELATIONAL", + "comment": "comment", + "provider": "generic-lakehouse", + "properties": { + } +}' http://localhost:8090/api/metalakes/metalake/catalogs +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java + +// Assuming you have just created a metalake named `metalake` +GravitinoClient gravitinoClient = GravitinoClient + .builder("http://127.0.0.1:8090";) + .withMetalake("metalake") + .build(); + +Map<String, String> genericCatalogProperties = ImmutableMap.<String, String>builder() + .put("location", "hdfs://localhost:9000/user/lakehouse") // The root location of the lakehouse storage system + .build(); + +Catalog catalog = gravitinoClient.createCatalog("generic_lakehouse_catalog", + Type.RELATIONAL, + "generic_lakehouse", + "This is a generic lakehouse catalog", + genericCatalogProperties); // Please change the properties according to the value of the provider. +// ... +``` + +</TabItem> +</Tabs> + + +## Schema + +### Schema capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Schema properties + +The same as catalog properties, please refer to [Catalog properties](#catalog-properties) section for more details. Schema `location` property can be used to specify the location to store all tables under this schema. + +### Schema operations + +Please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations) for more details. + +## Table + +### Table capabilities + +Currently, for a lance table, Gravitino supports the following capabilities: +- List +- Load +- Alter (partial supported) +- Create/register +- Drop and truncate + +### Table partitions + +Not support now + +### Table sort orders + +Not support now. + +### Table distributions + +Not support now. Review Comment: Grammatical error: Should be 'Not supported now.' to use the correct passive voice form. ```suggestion Not supported now. ### Table sort orders Not supported now. ### Table distributions Not supported now. ``` ########## docs/generic-lakehouse-catalog.md: ########## @@ -0,0 +1,281 @@ +--- +title: "Lakehouse catalog" +slug: /lakehouse-catalog +keywords: + - lakehouse + - lance + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Introduction + +Generic Lakehouse Catalog is a Gravitino catalog implementation that enables Gravitino to interact with Lakehouse storage systems that use file system for storing tabular data. Such lakehouse system could be built on top of object stores like Amazon S3, Azure Blob Storage, Google Cloud Storage, or HDFS. +Theoretically, it can work with any lakehouse storage system that supports standard file system operations such as Apache Iceberg, Lance, Delta Lake, and Apache Hudi. However, currently Gravitino only provides native support for Lance-based lakehouse storage systems. + +### Requirements and limitations + +- The lakehouse storage system must support standard file system operations such as listing directories, reading files, and writing files. + +## Catalog + +### Catalog capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Catalog properties + +The only property that need to be noted for a generic lakehouse catalog is `location`. This property specifies the root location of the lakehouse storage system. All schemas and tables will be stored under this location if not +specified otherwise in schema or table properties. + + +### Catalog operations + +All operations are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +One thing needs to be noted is that the provider will be `generic_lakehouse` when creating a generic lakehouse catalog. +That is: + +<Tabs groupId='language' queryString> +<TabItem value="shell" label="Shell"> + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "generic_lakehouse_catalog", + "type": "RELATIONAL", + "comment": "comment", + "provider": "generic-lakehouse", + "properties": { + } +}' http://localhost:8090/api/metalakes/metalake/catalogs +``` + +</TabItem> +<TabItem value="java" label="Java"> + +```java + +// Assuming you have just created a metalake named `metalake` +GravitinoClient gravitinoClient = GravitinoClient + .builder("http://127.0.0.1:8090";) + .withMetalake("metalake") + .build(); + +Map<String, String> genericCatalogProperties = ImmutableMap.<String, String>builder() + .put("location", "hdfs://localhost:9000/user/lakehouse") // The root location of the lakehouse storage system + .build(); + +Catalog catalog = gravitinoClient.createCatalog("generic_lakehouse_catalog", + Type.RELATIONAL, + "generic_lakehouse", + "This is a generic lakehouse catalog", + genericCatalogProperties); // Please change the properties according to the value of the provider. +// ... +``` + +</TabItem> +</Tabs> + + +## Schema + +### Schema capabilities + +All capabilities are the same as relational catalog, please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md) for more details. + +### Schema properties + +The same as catalog properties, please refer to [Catalog properties](#catalog-properties) section for more details. Schema `location` property can be used to specify the location to store all tables under this schema. + +### Schema operations + +Please refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md#schema-operations) for more details. + +## Table + +### Table capabilities + +Currently, for a lance table, Gravitino supports the following capabilities: +- List +- Load +- Alter (partial supported) +- Create/register +- Drop and truncate + +### Table partitions + +Not support now + +### Table sort orders + +Not support now. + +### Table distributions + +Not support now. + +### Table column types + +Since Lance uses Apache Arrow as the table schema, the following table shows the mapping between Gravitino types and Arrow types: + +| Gravitino Type | Arrow Type | +|----------------------------------|-----------------------------------------| +| `Struct` | `Struct` | +| `Map` | `Map` | +| `List` | `Array` | +| `Boolean` | `Boolean` | +| `Byte` | `Int8` | +| `Short` | `Int16` | +| `Integer` | `Int32` | +| `Long` | `Int64` | +| `Float` | `Float` | +| `Double` | `Double` | +| `String` | `Utf8` | +| `Binary` | `Binary` | +| `Decimal(p, s)` | `Decimal(p, s)` (128-bit) | +| `Date` | `Date` | +| `Timestamp`/`Timestamp(6)` | `TimestampType withoutZone` | +| `Timestamp(0)` | `TimestampType Second withoutZone` | +| `Timestamp(3)` | `TimestampType Millisecond withoutZone` | +| `Timestamp(9)` | `TimestampType Nanosecond withoutZone` | +| `Timestamp_tz`/`Timestamp_tz(6)` | `TimestampType Microsecond withUtc` | +| `Timestamp_tz(0)` | `TimestampType Second withUtc` | +| `Timestamp_tz(3)` | `TimestampType Millisecond withUtc` | +| `Timestamp_tz(9)` | `TimestampType Nanosecond withUtc` | +| `Time`/`Time(9)` | `Time Nanosecond` | +| `Null` | `Null` | +| `Fixed(n)` | `Fixed-Size Binary(n)` | +| `Interval_year` | `Interval(YearMonth)` | +| `Interval_day` | `Duration(Microsecond)` | +| `External(arrow_field_json_str)` | Any Arrow Field (see note below) | + +`External(arrow_field_json_str)`: + +As the table above shows, Gravitino provides mappings for most common data types. However, +in some cases, you may need to use an Arrow data type that is not directly supported by Gravitino. + +To address this, Gravitino introduces the `External(arrow_field_json_str)` type, +which allows you to define any Arrow data type by providing the JSON string of an Arrow `Field`. + +The JSON string must conform to the Apache Arrow `Field` [specification](https://github.com/apache/arrow-java/blob/ed81e5981a2bee40584b3a411ed755cb4cc5b91f/vector/src/main/java/org/apache/arrow/vector/types/pojo/Field.java#L80C1-L86C68), +including details such as the field name, data type, and nullability. +Here are some examples of how to use `External` type for various Arrow types that are not natively supported by Gravitino: + +| Arrow Type | External type | +|-------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Large Utf8` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largeutf8\"},\"children\":[]}")` | +| `Large Binary` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largebinary\"},\"children\":[]}")` | +| `Large List` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"largelist\"},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\", \"bitWidth\":32, \"isSigned\": true},\"children\":[]}]}")` | +| `Fixed-Size List` | `External("{\"name\":\"col_name\",\"nullable\":true,\"type\":{\"name\":\"fixedsizelist\", \"listSize\":10},\"children\":[{\"name\":\"element\",\"nullable\":true,\"type\":{\"name\":\"int\", \"bitWidth\":32, \"isSigned\": true},\"children\":[]}]}")` | + +**Important considerations:** +- The `name` attribute and `nullable` attribute in the JSON string must exactly match the corresponding column name and nullable in the Gravitino table. Review Comment: The phrase 'column name and nullable' is unclear. It should read 'column name and nullable status' or 'column name and nullability' for clarity. ```suggestion - The `name` attribute and `nullable` attribute in the JSON string must exactly match the corresponding column name and nullability in the Gravitino table. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
