qzyu999 commented on code in PR #3424: URL: https://github.com/apache/fluss/pull/3424#discussion_r3598389102
########## website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/glue.md: ########## @@ -0,0 +1,411 @@ +--- +title: AWS Glue +sidebar_position: 2 +--- + +# AWS Glue + +## Introduction + +[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service that includes a central metadata repository known as the AWS Glue Data Catalog. The Glue Data Catalog is compatible with Apache Iceberg, making it a convenient option for managing Iceberg table metadata on AWS. + +This guide explains how to configure Fluss to use AWS Glue as its Iceberg catalog. For general Iceberg integration details (table mapping, data types, limitations), see [Iceberg](../formats/iceberg.md). + +## How It Works + +When Fluss is configured with AWS Glue as its Iceberg catalog: + +1. **Data ingestion**: Applications write data to Fluss tables using the Fluss client (Java/Python), Flink SQL, or any Kafka-compatible producer. Fluss stores this data in its real-time log (similar to Kafka topics). +2. **Tiering to Iceberg**: A separate Flink job (the [tiering service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service)) periodically reads accumulated data from Fluss, converts it to Parquet format, writes the files to S3, and commits an Iceberg snapshot to the Glue Data Catalog. +3. **Query via Athena/Spark/Trino**: Any Iceberg-compatible engine can discover and query the tiered tables through AWS Glue — no additional configuration needed. + +``` +┌─ Data Sources ─────────┐ ┌─ Fluss Cluster ──────┐ ┌─ AWS ────────────────────┐ +│ │ │ │ │ │ +│ App (Fluss client) ─┼────▶│ Coordinator │ │ Glue Data Catalog │ +│ Flink CDC job ─┼────▶│ TabletServer(s) │ │ (table metadata) │ +│ Kafka producer ─┼────▶│ ZooKeeper │ │ │ +│ │ │ │ │ S3 Bucket │ +└─────────────────────────┘ └──────────┬───────────┘ │ (Parquet data files) │ + │ │ │ + ┌──────────▼───────────┐ │ Athena / Spark / Trino │ + │ Flink Tiering Job ├────▶│ (query engine) │ + │ (reads Fluss log, │ │ │ + │ writes Iceberg) │ └──────────────────────────┘ + └──────────────────────┘ +``` + +> **KEY CONCEPT**: Flink is used here for the **tiering service** (Fluss → Iceberg/S3/Glue) and optionally for data ingestion via SQL. But Flink is NOT the only way to write data to Fluss. Applications can write directly using the Fluss client library (Java, Python) or any Kafka-compatible producer. The tiering service is what bridges Fluss's real-time log to the Glue Data Catalog. + +## Prerequisites + +### Java Version + +Fluss 0.9.x requires **Java 17 or later** for the Tablet Server. Java 11 will fail with `NoSuchMethodError: MappedByteBuffer.duplicate()`. + +### AWS IAM Permissions + +The processes running Fluss servers and the Flink tiering service must have IAM permissions for both Glue and S3. Below is a minimal IAM policy: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "glue:CreateDatabase", + "glue:GetDatabase", + "glue:GetDatabases", + "glue:DeleteDatabase", + "glue:CreateTable", + "glue:GetTable", + "glue:GetTables", + "glue:UpdateTable", + "glue:DeleteTable" + ], + "Resource": [ + "arn:aws:glue:<region>:<account-id>:catalog", + "arn:aws:glue:<region>:<account-id>:database/*", + "arn:aws:glue:<region>:<account-id>:table/*" + ] + }, + { + "Effect": "Allow", + "Action": [ + "s3:GetObject", + "s3:PutObject", + "s3:DeleteObject", + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::<your-bucket>", + "arn:aws:s3:::<your-bucket>/*" + ] + } + ] +} +``` + +> **NOTE**: If your account uses **AWS Lake Formation**, the IAM role must also have Lake Formation permissions (Create Table, Describe, Alter, Insert, Select, Delete, Drop) on the target database. Standard IAM `glue:*` permissions alone are not sufficient when Lake Formation governance is enabled. + +### Prepare Required JARs + +Fluss bundles `iceberg-core` but does **not** bundle the Glue catalog implementation or the AWS SDK. You must supply additional JARs. + +#### For Fluss Servers (Coordinator & Tablet Servers) + +Place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/` directory: + +- **Iceberg AWS Bundle**: [iceberg-aws-bundle-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.10.1/iceberg-aws-bundle-1.10.1.jar) +- **Iceberg AWS**: [iceberg-aws-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws/1.10.1/iceberg-aws-1.10.1.jar) +- **Failsafe**: [failsafe-3.3.2.jar](https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar) — required by `iceberg-aws` for Glue API retry logic + +Both Iceberg JARs are required. The bundle provides AWS SDK v2 dependencies, while `iceberg-aws` provides the `GlueCatalog` class that Iceberg loads via reflection. The plugin classloader cannot find `GlueCatalog` from the bundle alone. The `failsafe` library is a transitive dependency of `iceberg-aws` that is not bundled. Review Comment: Hi @leekeiabstraction, correct, confirmed via E2E testing. The bundle contains the AWS SDK v2 dependencies but the `GlueCatalog` class itself lives in `iceberg-aws`. Both JARs are required. The doc explains this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
