Copilot commented on code in PR #9623: URL: https://github.com/apache/gravitino/pull/9623#discussion_r2675916599
########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. +::: + +## Prerequisites + +- Gravitino server running with Lance REST service enabled (default endpoint: `http://localhost:9101/lance`). +- A Lance catalog created in Gravitino, for example `lance_catalog`. +- Downloaded `lance-spark` bundle JAR that matches your Spark version (set the absolute path in the examples below). +- Python environments with required packages: + - Spark: `pyspark` + - Ray: `ray`, `lance-namespace`, `lance-ray` + +## Using Lance REST with Spark + +The example below starts a local PySpark session that talks to Lance REST and creates a table through Spark SQL. + +```python +from pyspark.sql import SparkSession +import os +import logging +logging.basicConfig(level=logging.INFO) + +# Point to your downloaded lance-spark bundle. +os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/lance-spark-bundle-3.5_2.12-0.0.15.jar --conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --master local[1] pyspark-shell" + +# Create the Lance catalog named "lance_catalog" in Gravitino beforehand. +spark = SparkSession.builder \ + .appName("lance_rest_example") \ + .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \ + .config("spark.sql.catalog.lance.impl", "rest") \ + .config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \ + .config("spark.sql.catalog.lance.parent", "lance_catalog") \ + .config("spark.sql.defaultCatalog", "lance") \ + .getOrCreate() + +spark.sparkContext.setLogLevel("DEBUG") + +# Create schema and table, write, then read data. +spark.sql("create database schema") +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION '/tmp/schema/sample.lance/' +TBLPROPERTIES ('format' = 'lance') +""") +spark.sql(""" +insert into schema.sample values(1, 1.1) +""") +spark.sql("select * from schema.sample").show() +``` + +:::note +- Keep the Lance REST service reachable from Spark executors. +- Replace the JAR path with the actual location on your machine or cluster. +- Add your own JVM debugging flags only when needed. +::: + +The storage location in the example above is local path, if you want to use cloud storage, please refer to the following Minio example: Review Comment: Inconsistent capitalization of "MinIO". Throughout the documentation and in the other file, "MinIO" is used (capital I, capital O), but here it's written as "Minio". Please use the correct capitalization "MinIO" to match the product name and maintain consistency. ```suggestion The storage location in the example above is local path, if you want to use cloud storage, please refer to the following MinIO example: ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. +::: + +## Prerequisites + +- Gravitino server running with Lance REST service enabled (default endpoint: `http://localhost:9101/lance`). +- A Lance catalog created in Gravitino, for example `lance_catalog`. +- Downloaded `lance-spark` bundle JAR that matches your Spark version (set the absolute path in the examples below). +- Python environments with required packages: + - Spark: `pyspark` + - Ray: `ray`, `lance-namespace`, `lance-ray` + +## Using Lance REST with Spark + +The example below starts a local PySpark session that talks to Lance REST and creates a table through Spark SQL. + +```python +from pyspark.sql import SparkSession +import os +import logging +logging.basicConfig(level=logging.INFO) + +# Point to your downloaded lance-spark bundle. +os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/lance-spark-bundle-3.5_2.12-0.0.15.jar --conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --master local[1] pyspark-shell" + +# Create the Lance catalog named "lance_catalog" in Gravitino beforehand. +spark = SparkSession.builder \ + .appName("lance_rest_example") \ + .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \ + .config("spark.sql.catalog.lance.impl", "rest") \ + .config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \ + .config("spark.sql.catalog.lance.parent", "lance_catalog") \ + .config("spark.sql.defaultCatalog", "lance") \ + .getOrCreate() + +spark.sparkContext.setLogLevel("DEBUG") + +# Create schema and table, write, then read data. +spark.sql("create database schema") +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION '/tmp/schema/sample.lance/' +TBLPROPERTIES ('format' = 'lance') +""") +spark.sql(""" +insert into schema.sample values(1, 1.1) +""") +spark.sql("select * from schema.sample").show() +``` + +:::note +- Keep the Lance REST service reachable from Spark executors. +- Replace the JAR path with the actual location on your machine or cluster. +- Add your own JVM debugging flags only when needed. +::: + +The storage location in the example above is local path, if you want to use cloud storage, please refer to the following Minio example: Review Comment: Grammar issue: The sentence should read "The storage location in the example above is a local path" (add article "a" before "local path"). ```suggestion The storage location in the example above is a local path, if you want to use cloud storage, please refer to the following Minio example: ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. +::: + +## Prerequisites + +- Gravitino server running with Lance REST service enabled (default endpoint: `http://localhost:9101/lance`). +- A Lance catalog created in Gravitino, for example `lance_catalog`. +- Downloaded `lance-spark` bundle JAR that matches your Spark version (set the absolute path in the examples below). +- Python environments with required packages: + - Spark: `pyspark` + - Ray: `ray`, `lance-namespace`, `lance-ray` + +## Using Lance REST with Spark + +The example below starts a local PySpark session that talks to Lance REST and creates a table through Spark SQL. + +```python +from pyspark.sql import SparkSession +import os +import logging +logging.basicConfig(level=logging.INFO) + +# Point to your downloaded lance-spark bundle. +os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/lance-spark-bundle-3.5_2.12-0.0.15.jar --conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --master local[1] pyspark-shell" + +# Create the Lance catalog named "lance_catalog" in Gravitino beforehand. +spark = SparkSession.builder \ + .appName("lance_rest_example") \ + .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \ + .config("spark.sql.catalog.lance.impl", "rest") \ + .config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \ + .config("spark.sql.catalog.lance.parent", "lance_catalog") \ + .config("spark.sql.defaultCatalog", "lance") \ + .getOrCreate() + +spark.sparkContext.setLogLevel("DEBUG") + +# Create schema and table, write, then read data. +spark.sql("create database schema") +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION '/tmp/schema/sample.lance/' +TBLPROPERTIES ('format' = 'lance') +""") +spark.sql(""" +insert into schema.sample values(1, 1.1) +""") +spark.sql("select * from schema.sample").show() +``` + +:::note +- Keep the Lance REST service reachable from Spark executors. +- Replace the JAR path with the actual location on your machine or cluster. +- Add your own JVM debugging flags only when needed. +::: + +The storage location in the example above is local path, if you want to use cloud storage, please refer to the following Minio example: + +```python +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION 's3://bucket/tmp/schema/sample.lance/' +TBLPROPERTIES ( + 'format' = 'lance', + 'lance.storage.access_key_id' = 'ak', + 'lance.storage.endpoint' = 'http://minio:9000', + 'lance.storage.secret_access_key' = 'sk', + 'lance.storage.allow_http'= 'true' Review Comment: Spacing inconsistency in property assignment. Line 94 has no space before the equals sign ('allow_http'= 'true'), while all other properties on lines 91-93 have consistent spacing around the equals sign. Please add a space before the equals sign to maintain consistency. ```suggestion 'lance.storage.allow_http' = 'true' ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. +::: + +## Prerequisites + +- Gravitino server running with Lance REST service enabled (default endpoint: `http://localhost:9101/lance`). +- A Lance catalog created in Gravitino, for example `lance_catalog`. +- Downloaded `lance-spark` bundle JAR that matches your Spark version (set the absolute path in the examples below). +- Python environments with required packages: + - Spark: `pyspark` + - Ray: `ray`, `lance-namespace`, `lance-ray` + +## Using Lance REST with Spark + +The example below starts a local PySpark session that talks to Lance REST and creates a table through Spark SQL. + +```python +from pyspark.sql import SparkSession +import os +import logging +logging.basicConfig(level=logging.INFO) + +# Point to your downloaded lance-spark bundle. +os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/lance-spark-bundle-3.5_2.12-0.0.15.jar --conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --master local[1] pyspark-shell" + +# Create the Lance catalog named "lance_catalog" in Gravitino beforehand. +spark = SparkSession.builder \ + .appName("lance_rest_example") \ + .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \ + .config("spark.sql.catalog.lance.impl", "rest") \ + .config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \ + .config("spark.sql.catalog.lance.parent", "lance_catalog") \ + .config("spark.sql.defaultCatalog", "lance") \ + .getOrCreate() + +spark.sparkContext.setLogLevel("DEBUG") + +# Create schema and table, write, then read data. +spark.sql("create database schema") +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION '/tmp/schema/sample.lance/' +TBLPROPERTIES ('format' = 'lance') +""") +spark.sql(""" +insert into schema.sample values(1, 1.1) +""") +spark.sql("select * from schema.sample").show() +``` + +:::note +- Keep the Lance REST service reachable from Spark executors. +- Replace the JAR path with the actual location on your machine or cluster. +- Add your own JVM debugging flags only when needed. +::: + +The storage location in the example above is local path, if you want to use cloud storage, please refer to the following Minio example: + +```python +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION 's3://bucket/tmp/schema/sample.lance/' +TBLPROPERTIES ( + 'format' = 'lance', + 'lance.storage.access_key_id' = 'ak', + 'lance.storage.endpoint' = 'http://minio:9000', + 'lance.storage.secret_access_key' = 'sk', + 'lance.storage.allow_http'= 'true' + ) +``` + +## Using Lance REST with Ray + +The snippet below writes and reads a Lance dataset through the Lance REST namespace. + +```shell +pip install lance-ray +``` +Please note that ray will also be installed if not already present. Currently lance-ray only tested with ray version 2.50.0, please +ensure ray version compatibility in your environment. + Review Comment: Multiple grammar issues in this paragraph: 1) "ray" should be capitalized to "Ray" (product name), 2) Missing verb - "only tested" should be "is only tested" or "has only been tested", 3) Inconsistent capitalization - "ray version" should be "Ray version" for consistency. The corrected version should read: "Please note that Ray will also be installed if not already present. Currently lance-ray is only tested with Ray version 2.50.0, please ensure Ray version compatibility in your environment." ```suggestion Please note that Ray will also be installed if not already present. Currently lance-ray is only tested with Ray version 2.50.0, please ensure Ray version compatibility in your environment. ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. +::: + +## Prerequisites + +- Gravitino server running with Lance REST service enabled (default endpoint: `http://localhost:9101/lance`). +- A Lance catalog created in Gravitino, for example `lance_catalog`. +- Downloaded `lance-spark` bundle JAR that matches your Spark version (set the absolute path in the examples below). +- Python environments with required packages: + - Spark: `pyspark` + - Ray: `ray`, `lance-namespace`, `lance-ray` + +## Using Lance REST with Spark + +The example below starts a local PySpark session that talks to Lance REST and creates a table through Spark SQL. + +```python +from pyspark.sql import SparkSession +import os +import logging +logging.basicConfig(level=logging.INFO) + +# Point to your downloaded lance-spark bundle. +os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/lance-spark-bundle-3.5_2.12-0.0.15.jar --conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --master local[1] pyspark-shell" + +# Create the Lance catalog named "lance_catalog" in Gravitino beforehand. +spark = SparkSession.builder \ + .appName("lance_rest_example") \ + .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \ + .config("spark.sql.catalog.lance.impl", "rest") \ + .config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \ + .config("spark.sql.catalog.lance.parent", "lance_catalog") \ + .config("spark.sql.defaultCatalog", "lance") \ + .getOrCreate() + +spark.sparkContext.setLogLevel("DEBUG") + +# Create schema and table, write, then read data. +spark.sql("create database schema") +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION '/tmp/schema/sample.lance/' +TBLPROPERTIES ('format' = 'lance') +""") +spark.sql(""" +insert into schema.sample values(1, 1.1) +""") +spark.sql("select * from schema.sample").show() +``` + +:::note +- Keep the Lance REST service reachable from Spark executors. +- Replace the JAR path with the actual location on your machine or cluster. +- Add your own JVM debugging flags only when needed. +::: + +The storage location in the example above is local path, if you want to use cloud storage, please refer to the following Minio example: + +```python +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION 's3://bucket/tmp/schema/sample.lance/' +TBLPROPERTIES ( + 'format' = 'lance', + 'lance.storage.access_key_id' = 'ak', + 'lance.storage.endpoint' = 'http://minio:9000', + 'lance.storage.secret_access_key' = 'sk', + 'lance.storage.allow_http'= 'true' + ) +``` + +## Using Lance REST with Ray + +The snippet below writes and reads a Lance dataset through the Lance REST namespace. + +```shell +pip install lance-ray +``` +Please note that ray will also be installed if not already present. Currently lance-ray only tested with ray version 2.50.0, please +ensure ray version compatibility in your environment. + Review Comment: Missing code block language identifier. The shell command on line 102-103 is in a shell code block, but the subsequent Python code starting at line 108 is not clearly separated or introduced. Consider adding a clearer introduction between the pip install command and the Python code example, or moving the pip install into the prerequisites section for better organization. ```suggestion After installing `lance-ray`, you can run the following Ray script: ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. +::: + +## Prerequisites + +- Gravitino server running with Lance REST service enabled (default endpoint: `http://localhost:9101/lance`). +- A Lance catalog created in Gravitino, for example `lance_catalog`. +- Downloaded `lance-spark` bundle JAR that matches your Spark version (set the absolute path in the examples below). +- Python environments with required packages: + - Spark: `pyspark` + - Ray: `ray`, `lance-namespace`, `lance-ray` + +## Using Lance REST with Spark + +The example below starts a local PySpark session that talks to Lance REST and creates a table through Spark SQL. + +```python +from pyspark.sql import SparkSession +import os +import logging +logging.basicConfig(level=logging.INFO) + +# Point to your downloaded lance-spark bundle. +os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/lance-spark-bundle-3.5_2.12-0.0.15.jar --conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --master local[1] pyspark-shell" + +# Create the Lance catalog named "lance_catalog" in Gravitino beforehand. +spark = SparkSession.builder \ + .appName("lance_rest_example") \ + .config("spark.sql.catalog.lance", "com.lancedb.lance.spark.LanceNamespaceSparkCatalog") \ + .config("spark.sql.catalog.lance.impl", "rest") \ + .config("spark.sql.catalog.lance.uri", "http://localhost:9101/lance";) \ + .config("spark.sql.catalog.lance.parent", "lance_catalog") \ + .config("spark.sql.defaultCatalog", "lance") \ + .getOrCreate() + +spark.sparkContext.setLogLevel("DEBUG") + +# Create schema and table, write, then read data. +spark.sql("create database schema") +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION '/tmp/schema/sample.lance/' +TBLPROPERTIES ('format' = 'lance') +""") +spark.sql(""" +insert into schema.sample values(1, 1.1) +""") +spark.sql("select * from schema.sample").show() +``` + +:::note +- Keep the Lance REST service reachable from Spark executors. +- Replace the JAR path with the actual location on your machine or cluster. +- Add your own JVM debugging flags only when needed. +::: + +The storage location in the example above is local path, if you want to use cloud storage, please refer to the following Minio example: + +```python +spark.sql(""" +create table schema.sample(id int, score float) +USING lance +LOCATION 's3://bucket/tmp/schema/sample.lance/' +TBLPROPERTIES ( + 'format' = 'lance', + 'lance.storage.access_key_id' = 'ak', + 'lance.storage.endpoint' = 'http://minio:9000', + 'lance.storage.secret_access_key' = 'sk', + 'lance.storage.allow_http'= 'true' + ) Review Comment: Missing closing parenthesis. The Python code snippet is incomplete - line 96 should have a closing parenthesis to properly close the SQL statement that starts on line 85. ```suggestion ) """) ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- + +## Overview + +This guide shows how to use the Lance REST service from Apache Gravitino with the Lance Spark connector (`lance-spark`) and the Lance Ray connector (`lance-ray`). It builds on the Lance REST service setup described in [Lance REST service](./lance-rest-service). + +## Compatibility matrix + +| Gravitino version (Lance REST) | Supported lance-spark versions | Supported lance-ray versions | +|--------------------------------|--------------------------------|------------------------------| +| 1.1.1 | 0.0.10 – 0.0.15 | 0.0.6 – 0.0.8 | + +:::note +- Update this matrix when newer Gravitino versions (for example 1.2.0) are released. +- Align connector versions with the Lance REST service bundled in the target release. Review Comment: Documentation concern: The compatibility matrix includes version ranges (e.g., "0.0.10 – 0.0.15") for connectors that may not have been verified to work with Gravitino 1.1.1. Consider clarifying which specific versions have been tested, or add a note that these are expected-compatible versions that should be verified by users. Additionally, the note on lines 24-25 suggests this is a template that needs updating, which may not be appropriate for production documentation. ```suggestion These version ranges represent combinations that are expected to be compatible. Only a subset of versions within each range may have been explicitly tested, so you should verify a specific connector version in your own environment. :::note The compatibility information in this section applies to Gravitino 1.1.1. For newer Gravitino versions, refer to that release's documentation and ensure that your `lance-spark` and `lance-ray` versions are compatible with the Lance REST service bundled with your Gravitino distribution. ``` ########## docs/lance-rest-spark-ray-integration.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Lance REST integration with Spark and Ray" +slug: /lance-rest-spark-ray-integration +keywords: + - lance + - lance-rest + - spark + - ray + - integration +license: "This software is licensed under the Apache License version 2." +--- Review Comment: Missing Apache License header comment block. All documentation files in the Apache Gravitino project should include the standard Apache License 2.0 header comment block at the top of the file, after the front matter. Please add the complete license header comment block as seen in other documentation files in the project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
