Copilot commented on code in PR #11523: URL: https://github.com/apache/gravitino/pull/11523#discussion_r3378873754
########## docs/spark-connector/spark-catalog-glue.md: ########## @@ -0,0 +1,153 @@ +--- +title: "Spark connector Glue catalog" +slug: /spark-connector/spark-catalog-glue +keyword: spark connector glue catalog aws +license: "This software is licensed under the Apache License version 2." +--- + +With the Apache Gravitino Spark connector, accessing data or managing metadata in AWS Glue Data Catalog becomes straightforward, enabling seamless federation queries across Glue catalogs. + +## Capabilities + +Supports most DDL and DML operations in SparkSQL, except such operations: + +- Function operations (Gravitino UDFs are supported, see [Spark connector - User-defined functions](spark-connector-udf.md)) +- Partition operations +- View operations +- `LOAD` clause +- `CREATE TABLE LIKE` clause +- `TRUNCATE TABLE` clause + +## Table format support + +The Glue catalog supports mixed table formats within a single database: + +- **Hive-format tables** (PARQUET, ORC, TEXTFILE, etc.): routed to HiveTableCatalog backed by AWS Glue Data Catalog Hive client +- **Iceberg-format tables**: routed to Iceberg's SparkCatalog (GlueCatalog) for I/O + +Table routing is based on the `table-format` property in Glue table parameters. Tables with `table-format=ICEBERG` are delegated to the Iceberg backend. + +:::note +HiveTableCatalog is configured with `hive.imetastoreclient.factory.class` set to `AWSGlueDataCatalogHiveClientFactory`, which replaces the embedded Derby metastore with a direct AWS Glue API connection. This means no local Derby sync is required and the catalog works correctly across multiple concurrent Spark applications sharing the same Glue catalog. Review Comment: The Hive configuration key is misspelled: `hive.imetastoreclient.factory.class` is not a valid Hive/Spark property. The correct key for using `AWSGlueDataCatalogHiveClientFactory` is `hive.metastore.client.factory.class`. ########## docs/spark-connector/spark-connector.md: ########## @@ -11,7 +11,7 @@ The Apache Gravitino Spark connector leverages the Spark DataSourceV2 interface ## Capabilities -1. Supports [Hive catalog](spark-catalog-hive.md), [Iceberg catalog](spark-catalog-iceberg.md), [Paimon catalog](spark-catalog-paimon.md) and [Jdbc catalog](spark-catalog-jdbc.md). +1. Supports [Hive catalog](spark-catalog-hive.md), [Iceberg catalog](spark-catalog-iceberg.md), [Paimon catalog](spark-catalog-paimon.md), [Jdbc catalog](spark-catalog-jdbc.md) and [AWS Glue catalog](spark-catalog-glue.md). Review Comment: The Spark connector codebase does not currently support a Gravitino catalog with provider `glue` (there is no `glue-*` entry in `CatalogNameAdaptor`, and there is no Spark catalog implementation/converter for it). Advertising AWS Glue here will mislead users into thinking Spark can load `provider=glue` catalogs. ########## docs/aws-glue-catalog.md: ########## @@ -0,0 +1,319 @@ +--- +title: "AWS Glue catalog" +slug: /aws-glue-catalog +keywords: + - glue + - aws + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +Apache Gravitino offers the capability to utilize [AWS Glue Data Catalog](https://aws.amazon.com/glue/) as a catalog for metadata management. + +### Requirements and limitations + +* The Glue catalog requires network access to the AWS Glue API. +* Gravitino uses the AWS SDK v2 to communicate with Glue. + +:::note +The Glue catalog is case-insensitive for schema and table names. AWS Glue folds database and table names to lowercase on storage. +::: + +## Catalog + +### Catalog capabilities + +The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog. + +- Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default. +- Supports Hive-format table partitioning, bucketing, and sort orders. +- Does not support views. Glue views (tables with `TableType=VIRTUAL_VIEW`) are filtered out. + +### Catalog properties + +Besides the [common catalog properties](./gravitino-server-config.md#apache-gravitino-catalog-properties-configuration), the Glue catalog has the following properties: + +| Property Name | Description | Default Value | Required | Immutable | Since Version | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|-----------|---------------| +| `aws-region` | AWS region for the Glue Data Catalog (e.g. `us-east-1`). | (none) | Yes | Yes | 1.3.0 | +| `aws-glue-catalog-id` | The 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID. | (none) | No | Yes | 1.3.0 | +| `aws-access-key-id` | AWS access key ID for static credential authentication. When omitted, the default credential chain is used. | (none) | No | No | 1.3.0 | +| `aws-secret-access-key` | AWS secret access key paired with `aws-access-key-id`. When omitted, the default credential chain is used. | (none) | No | No | 1.3.0 | +| `aws-glue-endpoint` | Custom Glue endpoint URL for VPC endpoints or LocalStack testing (e.g. `http://localhost:4566`). | (none) | No | No | 1.3.0 | +| `warehouse` | Base storage path used as the warehouse when no explicit `location` is specified at table creation time (e.g. `s3://my-bucket/warehouse`). Table location is derived as `warehouse/database/table`. Required for Iceberg table creation. | (none) | Yes | No | 1.3.0 | +| `default-table-format` | Default format for tables created via Gravitino's `createTable()` API. Accepted values: `iceberg`, `hive`. | `hive` | No | No | 1.3.0 | +| `table-format-filter` | Comma-separated list of table formats exposed by `listTables()` and `loadTable()`. Accepted values: `all`, `hive`, `iceberg`, `delta`, `parquet`. Use to restrict visible table types. | `all` | No | No | 1.3.0 | + +:::note +**Authentication priority**: Static credentials (`aws-access-key-id` + `aws-secret-access-key`) take precedence over the default credential chain (environment variables, instance profile, container credentials). +::: + +When you use Gravitino with Trino, you can pass the Trino connector configuration using prefix `trino.bypass.`. For example, using `trino.bypass.hive.metastore.glue.region` to pass the `hive.metastore.glue.region` to the Gravitino Glue catalog in Trino runtime. + +When you use Gravitino with Spark, you can pass the Spark connector configuration using prefix `spark.bypass.`. + +### Catalog operations + +Refer to [Manage Relational Metadata Using Gravitino](./manage-relational-metadata-using-gravitino.md#catalog-operations) for more details. + +## Schema + +### Schema capabilities + +The Glue catalog supports creating, updating, and deleting databases in the AWS Glue Data Catalog. + +### Schema properties + +The Glue catalog does not define predefined schema properties beyond `comment`. You can define your own key-value pair properties and transmit them to the underlying Glue database. Review Comment: The Glue connector does handle a schema/database `location` property: `location` is read from the schema properties and mapped to Glue `DatabaseInput.locationUri`. Saying there are no predefined schema properties beyond `comment` is therefore inaccurate. ########## docs/aws-glue-catalog.md: ########## @@ -0,0 +1,319 @@ +--- +title: "AWS Glue catalog" +slug: /aws-glue-catalog +keywords: + - glue + - aws + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +Apache Gravitino offers the capability to utilize [AWS Glue Data Catalog](https://aws.amazon.com/glue/) as a catalog for metadata management. + +### Requirements and limitations + +* The Glue catalog requires network access to the AWS Glue API. +* Gravitino uses the AWS SDK v2 to communicate with Glue. + +:::note +The Glue catalog is case-insensitive for schema and table names. AWS Glue folds database and table names to lowercase on storage. +::: + +## Catalog + +### Catalog capabilities + +The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog. + +- Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default. +- Supports Hive-format table partitioning, bucketing, and sort orders. +- Does not support views. Glue views (tables with `TableType=VIRTUAL_VIEW`) are filtered out. + +### Catalog properties + +Besides the [common catalog properties](./gravitino-server-config.md#apache-gravitino-catalog-properties-configuration), the Glue catalog has the following properties: + +| Property Name | Description | Default Value | Required | Immutable | Since Version | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|-----------|---------------| +| `aws-region` | AWS region for the Glue Data Catalog (e.g. `us-east-1`). | (none) | Yes | Yes | 1.3.0 | +| `aws-glue-catalog-id` | The 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID. | (none) | No | Yes | 1.3.0 | +| `aws-access-key-id` | AWS access key ID for static credential authentication. When omitted, the default credential chain is used. | (none) | No | No | 1.3.0 | +| `aws-secret-access-key` | AWS secret access key paired with `aws-access-key-id`. When omitted, the default credential chain is used. | (none) | No | No | 1.3.0 | +| `aws-glue-endpoint` | Custom Glue endpoint URL for VPC endpoints or LocalStack testing (e.g. `http://localhost:4566`). | (none) | No | No | 1.3.0 | +| `warehouse` | Base storage path used as the warehouse when no explicit `location` is specified at table creation time (e.g. `s3://my-bucket/warehouse`). Table location is derived as `warehouse/database/table`. Required for Iceberg table creation. | (none) | Yes | No | 1.3.0 | +| `default-table-format` | Default format for tables created via Gravitino's `createTable()` API. Accepted values: `iceberg`, `hive`. | `hive` | No | No | 1.3.0 | +| `table-format-filter` | Comma-separated list of table formats exposed by `listTables()` and `loadTable()`. Accepted values: `all`, `hive`, `iceberg`, `delta`, `parquet`. Use to restrict visible table types. | `all` | No | No | 1.3.0 | Review Comment: `table-format-filter` is described as supporting `delta` and `parquet`, but the Glue connector's filtering logic keys off the `table-format` table parameter (and a fallback for Iceberg via `table_type`/`metadata_location`). Delta/Parquet tables are not detected as such unless `table-format` is explicitly set accordingly, so listing them as supported values is misleading. ########## docs/spark-connector/spark-catalog-glue.md: ########## @@ -0,0 +1,153 @@ +--- +title: "Spark connector Glue catalog" +slug: /spark-connector/spark-catalog-glue +keyword: spark connector glue catalog aws +license: "This software is licensed under the Apache License version 2." +--- + +With the Apache Gravitino Spark connector, accessing data or managing metadata in AWS Glue Data Catalog becomes straightforward, enabling seamless federation queries across Glue catalogs. + +## Capabilities + +Supports most DDL and DML operations in SparkSQL, except such operations: + +- Function operations (Gravitino UDFs are supported, see [Spark connector - User-defined functions](spark-connector-udf.md)) +- Partition operations +- View operations +- `LOAD` clause +- `CREATE TABLE LIKE` clause +- `TRUNCATE TABLE` clause + +## Table format support + +The Glue catalog supports mixed table formats within a single database: + +- **Hive-format tables** (PARQUET, ORC, TEXTFILE, etc.): routed to HiveTableCatalog backed by AWS Glue Data Catalog Hive client +- **Iceberg-format tables**: routed to Iceberg's SparkCatalog (GlueCatalog) for I/O + +Table routing is based on the `table-format` property in Glue table parameters. Tables with `table-format=ICEBERG` are delegated to the Iceberg backend. + +:::note +HiveTableCatalog is configured with `hive.imetastoreclient.factory.class` set to `AWSGlueDataCatalogHiveClientFactory`, which replaces the embedded Derby metastore with a direct AWS Glue API connection. This means no local Derby sync is required and the catalog works correctly across multiple concurrent Spark applications sharing the same Glue catalog. +::: + +## Requirements + +- Network access to the AWS Glue API and Amazon S3 +- AWS IAM credentials with necessary Glue and S3 permissions + See [AWS IAM permissions](../aws-glue-catalog.md#aws-iam-permissions) for the required policy +- Apache Spark 3.3+, 3.4, or 3.5 + +## SQL example + +```sql +// Suppose glue_catalog is the Glue catalog name managed by Gravitino +USE glue_catalog; + +CREATE DATABASE IF NOT EXISTS mydatabase; +USE mydatabase; + +// Create a Hive-format Parquet table +CREATE TABLE IF NOT EXISTS employees ( + id INT, + name STRING, + age INT +) +PARTITIONED BY (department STRING) +STORED AS PARQUET; + +DESC TABLE EXTENDED employees; + +INSERT OVERWRITE TABLE employees PARTITION(department='Engineering') +VALUES (1, 'John Doe', 30), (2, 'Jane Smith', 28); + +SELECT * FROM employees WHERE department = 'Engineering'; + +// Create an Iceberg table +CREATE TABLE IF NOT EXISTS orders ( + order_id BIGINT, + customer STRING, + amount DECIMAL(10, 2), + order_ts TIMESTAMP +) +USING iceberg +PARTITIONED BY (days(order_ts)); + +INSERT INTO orders VALUES (1, 'alice', 99.99, TIMESTAMP '2024-01-01 00:00:00'); + +SELECT * FROM orders WHERE order_ts >= DATE '2024-01-01'; +``` + +## Catalog properties + +Gravitino Spark connector transforms the following property names defined in catalog properties to Spark Hive/Iceberg connector configuration. + +| Property name in Gravitino catalog properties | Spark Hive connector configuration | Description | Since Version | +|-----------------------------------------------|-------------------------------------------|---------------------------------------|---------------| +| `aws-region` | `aws.region` | AWS region for Glue Data Catalog | 0.7.0 | +| `aws-glue-catalog-id` | `aws.glue.catalog.id` | 12-digit AWS account ID owning Glue catalog | 0.7.0 | +| `aws-glue-endpoint` | `aws.glue.endpoint` | Custom Glue endpoint URL | 0.7.0 | +| `warehouse` | (Iceberg) `warehouse` | Base storage path for Iceberg tables | 0.7.0 | + +For Iceberg tables, Gravitino properties are mapped to Iceberg GlueCatalog configuration: + +| Gravitino property | Iceberg GlueCatalog property | Description | +|-------------------------|--------------------------------------|------------------------------------------| +| `aws-region` | `client.region` | AWS region | +| `aws-glue-catalog-id` | `glue.id` | Glue catalog ID | +| `aws-glue-endpoint` | `glue.endpoint` | Glue endpoint URL | +| `aws-access-key-id` | `client.credentials-provider.*` | AWS access key (via custom provider) | +| `aws-secret-access-key` | `client.credentials-provider.*` | AWS secret key (via custom provider) | + +Gravitino catalog property names with the prefix `spark.bypass.` are passed directly to the Spark Hive connector. For example, using `spark.bypass.hive.exec.dynamic.partition.mode` to pass `hive.exec.dynamic.partition.mode` to the Spark Hive connector. + +## Creating a catalog + +Use Gravitino REST API or the Gravitino CLI to create a Glue catalog: + +```bash +curl -X POST -H "Content-Type: application/json" \ +-d '{ + "name": "glue_catalog", + "type": "RELATIONAL", + "provider": "glue", + "properties": { + "aws-region": "us-east-1", + "aws-access-key-id": "<your-access-key-id>", + "aws-secret-access-key": "<your-secret-access-key>", + "warehouse": "s3://my-bucket/warehouse" + } +}' http://gravitino-host:8090/api/metalakes/{metalake}/catalogs +``` + +For more information about the Glue catalog, refer to [AWS Glue catalog](../aws-glue-catalog.md). + +## S3 storage + +When using AWS S3 with the Glue catalog, configure S3 credentials in the Spark configuration: + +```bash +spark-submit \ + --conf spark.sql.catalog.glue_catalog=com.gravitino.spark.connector.GlueCatalog \ + --conf spark.sql.catalog.glue_catalog.warehouse=s3://my-bucket/warehouse \ + --conf spark.sql.catalog.glue_catalog.fs.s3a.access.key=<your-access-key> \ + --conf spark.sql.catalog.glue_catalog.fs.s3a.secret.key=<your-secret-key> \ + ... Review Comment: The example uses `spark.sql.catalog.glue_catalog=com.gravitino.spark.connector.GlueCatalog`, but there is no such class in the repository, and the Spark connector does not currently register a catalog implementation for `provider=glue`. As written, this `spark-submit` snippet will not work. ########## docs/aws-glue-catalog.md: ########## @@ -0,0 +1,319 @@ +--- +title: "AWS Glue catalog" +slug: /aws-glue-catalog +keywords: + - glue + - aws + - metadata +license: "This software is licensed under the Apache License version 2." +--- + +## Introduction + +Apache Gravitino offers the capability to utilize [AWS Glue Data Catalog](https://aws.amazon.com/glue/) as a catalog for metadata management. + +### Requirements and limitations + +* The Glue catalog requires network access to the AWS Glue API. +* Gravitino uses the AWS SDK v2 to communicate with Glue. + +:::note +The Glue catalog is case-insensitive for schema and table names. AWS Glue folds database and table names to lowercase on storage. +::: + +## Catalog + +### Catalog capabilities + +The Glue catalog supports creating, updating, and deleting databases and tables in the AWS Glue Data Catalog. + +- Supports all table types stored in Glue (Hive, Iceberg, Delta, Parquet, and others) by default. +- Supports Hive-format table partitioning, bucketing, and sort orders. +- Does not support views. Glue views (tables with `TableType=VIRTUAL_VIEW`) are filtered out. + +### Catalog properties + +Besides the [common catalog properties](./gravitino-server-config.md#apache-gravitino-catalog-properties-configuration), the Glue catalog has the following properties: + +| Property Name | Description | Default Value | Required | Immutable | Since Version | +|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------|-----------|---------------| +| `aws-region` | AWS region for the Glue Data Catalog (e.g. `us-east-1`). | (none) | Yes | Yes | 1.3.0 | +| `aws-glue-catalog-id` | The 12-digit AWS account ID that owns the Glue catalog. When omitted, defaults to the caller's AWS account ID. | (none) | No | Yes | 1.3.0 | +| `aws-access-key-id` | AWS access key ID for static credential authentication. When omitted, the default credential chain is used. | (none) | No | No | 1.3.0 | +| `aws-secret-access-key` | AWS secret access key paired with `aws-access-key-id`. When omitted, the default credential chain is used. | (none) | No | No | 1.3.0 | +| `aws-glue-endpoint` | Custom Glue endpoint URL for VPC endpoints or LocalStack testing (e.g. `http://localhost:4566`). | (none) | No | No | 1.3.0 | +| `warehouse` | Base storage path used as the warehouse when no explicit `location` is specified at table creation time (e.g. `s3://my-bucket/warehouse`). Table location is derived as `warehouse/database/table`. Required for Iceberg table creation. | (none) | Yes | No | 1.3.0 | Review Comment: The `warehouse` property is marked as required, but the description says it is only required for Iceberg table creation. In the connector implementation, `warehouse` is a required catalog property in general (used for deriving default locations), so the description should not imply it is optional outside Iceberg. ########## docs/spark-connector/spark-catalog-glue.md: ########## @@ -0,0 +1,153 @@ +--- +title: "Spark connector Glue catalog" +slug: /spark-connector/spark-catalog-glue +keyword: spark connector glue catalog aws +license: "This software is licensed under the Apache License version 2." +--- + +With the Apache Gravitino Spark connector, accessing data or managing metadata in AWS Glue Data Catalog becomes straightforward, enabling seamless federation queries across Glue catalogs. + +## Capabilities + +Supports most DDL and DML operations in SparkSQL, except such operations: + +- Function operations (Gravitino UDFs are supported, see [Spark connector - User-defined functions](spark-connector-udf.md)) +- Partition operations +- View operations +- `LOAD` clause +- `CREATE TABLE LIKE` clause +- `TRUNCATE TABLE` clause + +## Table format support + +The Glue catalog supports mixed table formats within a single database: + +- **Hive-format tables** (PARQUET, ORC, TEXTFILE, etc.): routed to HiveTableCatalog backed by AWS Glue Data Catalog Hive client +- **Iceberg-format tables**: routed to Iceberg's SparkCatalog (GlueCatalog) for I/O + +Table routing is based on the `table-format` property in Glue table parameters. Tables with `table-format=ICEBERG` are delegated to the Iceberg backend. + +:::note +HiveTableCatalog is configured with `hive.imetastoreclient.factory.class` set to `AWSGlueDataCatalogHiveClientFactory`, which replaces the embedded Derby metastore with a direct AWS Glue API connection. This means no local Derby sync is required and the catalog works correctly across multiple concurrent Spark applications sharing the same Glue catalog. +::: + +## Requirements + +- Network access to the AWS Glue API and Amazon S3 +- AWS IAM credentials with necessary Glue and S3 permissions + See [AWS IAM permissions](../aws-glue-catalog.md#aws-iam-permissions) for the required policy +- Apache Spark 3.3+, 3.4, or 3.5 + +## SQL example + +```sql +// Suppose glue_catalog is the Glue catalog name managed by Gravitino +USE glue_catalog; + +CREATE DATABASE IF NOT EXISTS mydatabase; +USE mydatabase; + +// Create a Hive-format Parquet table +CREATE TABLE IF NOT EXISTS employees ( + id INT, + name STRING, + age INT +) +PARTITIONED BY (department STRING) +STORED AS PARQUET; + +DESC TABLE EXTENDED employees; + +INSERT OVERWRITE TABLE employees PARTITION(department='Engineering') +VALUES (1, 'John Doe', 30), (2, 'Jane Smith', 28); + +SELECT * FROM employees WHERE department = 'Engineering'; + +// Create an Iceberg table +CREATE TABLE IF NOT EXISTS orders ( + order_id BIGINT, + customer STRING, + amount DECIMAL(10, 2), + order_ts TIMESTAMP +) +USING iceberg +PARTITIONED BY (days(order_ts)); + +INSERT INTO orders VALUES (1, 'alice', 99.99, TIMESTAMP '2024-01-01 00:00:00'); + +SELECT * FROM orders WHERE order_ts >= DATE '2024-01-01'; +``` + +## Catalog properties + +Gravitino Spark connector transforms the following property names defined in catalog properties to Spark Hive/Iceberg connector configuration. + +| Property name in Gravitino catalog properties | Spark Hive connector configuration | Description | Since Version | +|-----------------------------------------------|-------------------------------------------|---------------------------------------|---------------| +| `aws-region` | `aws.region` | AWS region for Glue Data Catalog | 0.7.0 | +| `aws-glue-catalog-id` | `aws.glue.catalog.id` | 12-digit AWS account ID owning Glue catalog | 0.7.0 | +| `aws-glue-endpoint` | `aws.glue.endpoint` | Custom Glue endpoint URL | 0.7.0 | +| `warehouse` | (Iceberg) `warehouse` | Base storage path for Iceberg tables | 0.7.0 | Review Comment: This section claims the Spark connector will transform Glue catalog properties (e.g., `aws-region` -> `aws.region`), but the Spark connector only has property converters for Hive/Iceberg/JDBC/Paimon catalogs. There is no Glue-specific `PropertiesConverter`, and `CatalogNameAdaptor` does not map `provider=glue` to any Spark catalog implementation, so these mappings are not applied today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
