qzyu999 commented on code in PR #3424: URL: https://github.com/apache/fluss/pull/3424#discussion_r3598231434
########## website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/glue.md: ########## @@ -0,0 +1,411 @@ +--- +title: AWS Glue +sidebar_position: 2 +--- + +# AWS Glue + +## Introduction + +[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration service that includes a central metadata repository known as the AWS Glue Data Catalog. The Glue Data Catalog is compatible with Apache Iceberg, making it a convenient option for managing Iceberg table metadata on AWS. + +This guide explains how to configure Fluss to use AWS Glue as its Iceberg catalog. For general Iceberg integration details (table mapping, data types, limitations), see [Iceberg](../formats/iceberg.md). + +## How It Works + +When Fluss is configured with AWS Glue as its Iceberg catalog: + +1. **Data ingestion**: Applications write data to Fluss tables using the Fluss client (Java/Python), Flink SQL, or any Kafka-compatible producer. Fluss stores this data in its real-time log (similar to Kafka topics). +2. **Tiering to Iceberg**: A separate Flink job (the [tiering service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service)) periodically reads accumulated data from Fluss, converts it to Parquet format, writes the files to S3, and commits an Iceberg snapshot to the Glue Data Catalog. +3. **Query via Athena/Spark/Trino**: Any Iceberg-compatible engine can discover and query the tiered tables through AWS Glue — no additional configuration needed. Review Comment: Hi @leekeiabstraction, added union read section in the Usage Example (see "Query Data with Flink (Union Read)"). Also added a NOTE clarifying that PK table union reads are not yet supported with Iceberg per @fresh-borzoni's feedback (c0af314f). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
