fresh-borzoni commented on code in PR #3424:
URL: https://github.com/apache/fluss/pull/3424#discussion_r3570117890
##########
website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/hive.md:
##########
@@ -0,0 +1,452 @@
+---
+title: Hive Metastore
+sidebar_position: 3
+---
+
+# Hive Metastore
+
+## Introduction
+
+The **Hive Metastore (HMS)** is a central metadata repository commonly used in
Apache Hadoop and other big data ecosystems to store schema and metadata
information for tables. Apache Iceberg provides native integration with Hive
Metastore, storing Iceberg table names and metadata locations directly within
HMS.
+
+This guide explains how to configure Fluss to use Hive Metastore as its
Iceberg catalog. For general Iceberg integration details (table mapping, data
types, limitations), see [Iceberg](../formats/iceberg.md).
+
+## How It Works
+
+When Fluss is configured with Hive Metastore as its Iceberg catalog:
+
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss
client (Java/Python), Flink SQL, or any Kafka-compatible producer. Fluss stores
this data in its real-time log.
+2. **Tiering to Iceberg**: A separate Flink job (the [tiering
service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service))
periodically reads accumulated data from Fluss, converts it to Parquet format,
writes the files to HDFS (or S3/OSS), and commits an Iceberg snapshot to the
Hive Metastore.
+3. **Query via Spark/Trino/Flink**: Any Iceberg-compatible engine configured
with Hive catalog can discover and query the tiered tables through HMS.
+
+```mermaid
+flowchart LR
+ subgraph Sources["Data Sources"]
+ A1[App - Fluss Client]
+ A2[Flink CDC Job]
+ end
+
+ subgraph Fluss["Fluss Cluster"]
+ B1[Coordinator]
+ B2[TabletServer]
+ B3[ZooKeeper]
+ end
+
+ subgraph Hadoop["Hadoop / On-Prem"]
+ C1[Hive Metastore]
+ C2[HDFS / S3 / OSS]
+ C3[Spark / Trino / StarRocks]
+ end
+
+ A1 -->|write| B2
+ A2 -->|write| B2
+ B1 -->|register tables| C1
+
+ subgraph Tiering["Flink Tiering Job"]
+ D1[Reads Fluss log]
+ D2[Writes Parquet]
+ D3[Commits Iceberg snapshot]
+ end
+
+ B2 -->|read log| D1
+ D2 -->|Parquet files| C2
+ D3 -->|snapshot metadata| C1
+ C1 -->|discover tables| C3
+ C2 -->|read data| C3
+```
+
+> **Note**: The tiering service is a Flink job that bridges Fluss's real-time
log to Iceberg tables in Hive Metastore. Flink is also commonly used for data
ingestion (via SQL), but applications can write directly to Fluss using the
client library.
+
+## Prerequisites
+
+### Java Version
+
+Fluss 0.9.x requires **Java 17 or later** for the Tablet Server.
+
+### Running Hive Metastore
+
+Ensure you have a running Hive Metastore service (version **2.x or 3.x**). By
default, HMS listens on thrift port `9083` (e.g.,
`thrift://<metastore-host>:9083`).
+
+:::warning Hive Metastore Version Compatibility
+Iceberg 1.10.1 compiles its Hive catalog client against **Hive 2.3.10**. This
client is compatible with HMS **2.x and 3.x** servers. HMS **4.x** uses an
incompatible thrift protocol and will fail with `TApplicationException: Invalid
method name: 'get_table'`. Use HMS 3.x or earlier.
+:::
+
+### Prepare Required JARs
+
+Iceberg's Hive catalog class (`HiveCatalog`) is distributed separately from
`iceberg-core` and requires the Hive metastore client, Hadoop, and several
transitive dependencies. You must supply all of them.
+
+#### For Fluss Servers (Coordinator & Tablet Servers)
+
+Download and place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/`
directory:
+
+| # | JAR | Size | Purpose |
+|---|-----|------|---------|
+| 1 |
[iceberg-hive-metastore-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar)
| 92 KB | Contains `HiveCatalog` class |
+| 2 |
[hive-exec-2.3.10.jar](https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar)
| 46 MB | Uber JAR: bundles `HiveConf`, `HiveMetaStoreClient`, Thrift, Guava |
+| 3 |
[hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar)
| 20 MB | Hadoop public API (`Configuration`, `FileSystem`) |
+| 4 |
[hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar)
| 30 MB | Hadoop runtime implementations |
+| 5 |
[commons-logging-1.2.jar](https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar)
| 62 KB | Logging facade required by Hadoop and Hive |
+| 6 |
[jackson-core-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar)
| 571 KB | JSON processing |
+| 7 |
[jackson-databind-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar)
| 1.5 MB | JSON data binding |
+| 8 |
[jackson-annotations-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar)
| 75 KB | JSON annotations |
+
+> **NOTE**: The Fluss binary distribution already includes
`fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need
to download it separately — only add the 8 JARs above.
+
+> **TIP**: `hive-exec-2.3.10.jar` is a shaded uber JAR that bundles the Hive
metastore client, `HiveConf`, Apache Thrift, libfb303, and Guava. Using the
full `hive-exec` (not the `-core` variant) avoids needing to track down each
transitive dependency individually. Despite its size, it is the simplest way to
satisfy all Hive client requirements.
+
+#### For the Flink Tiering Service
+
+Place the following JARs in `${FLINK_HOME}/lib`:
+
+1. **All 8 JARs listed above** (iceberg-hive-metastore, hive-exec,
hadoop-client-api, hadoop-client-runtime, commons-logging, jackson-core,
jackson-databind, jackson-annotations)
+2. **Fluss Flink Connector**:
[fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar)
(pick the version matching your Flink runtime)
+3. **Fluss Lake Iceberg**:
[fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
+4. **Fluss Flink Tiering**:
[fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar)
— the tiering job JAR itself
+
+### Hadoop Classpath Configuration (HDFS Only)
+
+If your warehouse is on **HDFS**, both Fluss and Flink must be able to resolve
HDFS paths. This requires Hadoop configuration files on the classpath.
+
+**Option 1: Export Hadoop Classpath (Recommended)**
+
+```bash
+export HADOOP_CLASSPATH=`hadoop classpath`
+```
+
+**Option 2: Copy Hadoop XML Configs**
+
+Copy `core-site.xml` and `hdfs-site.xml` to the configuration directories of
both Fluss (`${FLUSS_HOME}/conf/`) and Flink (`${FLINK_HOME}/conf/`).
+
+> **NOTE**: If your warehouse uses a local filesystem path (for testing) or
S3/OSS (with the appropriate Fluss filesystem plugin), you do not need
`HADOOP_CLASSPATH`.
+
+## Configure Fluss with Hive Metastore
+
+### Cluster Configuration
+
+Add the following to your `server.yaml`:
+
+```yaml
+datalake.format: iceberg
+datalake.iceberg.type: hive
+datalake.iceberg.uri: thrift://<hive-metastore-host>:9083
+datalake.iceberg.warehouse: hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+Fluss strips the `datalake.iceberg.` prefix and passes the remaining
properties directly to Iceberg's Hive catalog. The properties above become
`type=hive`, `uri=thrift://...`, and `warehouse=hdfs://...` when initializing
the catalog.
+
+:::note
+If your Hive warehouse is on cloud object storage, set
`datalake.iceberg.warehouse` to the corresponding URI (e.g.,
`s3://<your-bucket>/warehouse`) and configure the required filesystem plugin.
See [AWS Glue](glue.md) for S3 credential setup.
+:::
+
+### Start Tiering Service
+
+Follow the [Iceberg tiering service
setup](../formats/iceberg.md#start-tiering-service-to-iceberg) instructions to
prepare the environment. Launch the Flink tiering job:
+
+```bash
+${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers <coordinator-host>:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hive \
+ --datalake.iceberg.uri thrift://<hive-metastore-host>:9083 \
+ --datalake.iceberg.warehouse
hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+## Quick Start (Docker Compose)
+
+This section provides a complete docker-compose setup that runs the entire
Hive Metastore integration end-to-end. It starts HMS, ZooKeeper, Fluss, Flink,
creates a table, inserts data, tiers it to Iceberg via HMS, and reads it back.
No cloud account or Hadoop cluster needed — everything runs locally.
+
+**Prerequisites**: Docker and Docker Compose installed.
+
+Create a `docker-compose.yml`:
+
+```yaml
+services:
+ # Hive Metastore (Thrift on port 9083, embedded Derby)
+ metastore:
+ image: apache/hive:3.1.3
+ environment:
+ SERVICE_NAME: metastore
+ ports:
+ - "9083:9083"
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9083'
2>/dev/null"]
+ interval: 10s
+ timeout: 10s
+ retries: 15
+ start_period: 90s
+
+ zookeeper:
+ image: zookeeper:3.8.4
+ healthcheck:
+ test: ["CMD", "zkServer.sh", "status"]
+ interval: 5s
+ timeout: 5s
+ retries: 5
+
+ # Downloads plugin JARs into a shared volume
+ init-plugins:
+ image: amazoncorretto:17
+ entrypoint: ["/bin/bash", "-c"]
+ command:
+ - |
+ set -e
+ yum install -y curl 2>/dev/null
+ mkdir -p /plugins/iceberg
+ curl -sSfL -o /plugins/iceberg/iceberg-hive-metastore-1.10.1.jar \
+
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar
+ curl -sSfL -o /plugins/iceberg/hive-exec-2.3.10.jar \
+
https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-api-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-runtime-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/commons-logging-1.2.jar \
+
https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-core-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-databind-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-annotations-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar
+ echo 'Plugins ready:' && ls -la /plugins/iceberg/
+ volumes:
+ - fluss-plugins:/plugins
+
+ coordinator:
+ image: apache/fluss:0.9.0-incubating
+ command: coordinatorServer
+ depends_on:
+ zookeeper: { condition: service_healthy }
+ metastore: { condition: service_healthy }
+ init-plugins: { condition: service_completed_successfully }
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ coordinator.host: coordinator
+ coordinator.port: 9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hive
+ datalake.iceberg.uri: thrift://metastore:9083
+ datalake.iceberg.warehouse: /tmp/fluss/warehouse
+ volumes:
+ - fluss-data:/tmp/fluss
+ - fluss-plugins:/opt/fluss/plugins
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/coordinator/9123'
2>/dev/null"]
+ interval: 10s
+ timeout: 5s
+ retries: 12
+ start_period: 30s
+
+ tablet-server:
+ image: apache/fluss:0.9.0-incubating
Review Comment:
ditto
##########
website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/glue.md:
##########
@@ -0,0 +1,422 @@
+---
+title: AWS Glue
+sidebar_position: 2
+---
+
+# AWS Glue
+
+## Introduction
+
+[AWS Glue](https://aws.amazon.com/glue/) is a serverless data integration
service that includes a central metadata repository known as the AWS Glue Data
Catalog. The Glue Data Catalog is compatible with Apache Iceberg, making it a
convenient option for managing Iceberg table metadata on AWS.
+
+This guide explains how to configure Fluss to use AWS Glue as its Iceberg
catalog. For general Iceberg integration details (table mapping, data types,
limitations), see [Iceberg](../formats/iceberg.md).
+
+## How It Works
+
+When Fluss is configured with AWS Glue as its Iceberg catalog:
+
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss
client (Java/Python) or Flink SQL. Fluss stores this data in its real-time log.
+2. **Tiering to Iceberg**: A separate Flink job (the tiering service)
periodically reads accumulated data from Fluss, converts it to Parquet format,
writes the files to S3, and commits an Iceberg snapshot to the Glue Data
Catalog.
+3. **Query via Athena/Spark/Trino**: Any Iceberg-compatible engine can
discover and query the tiered tables through AWS Glue — no additional
configuration needed.
+
+```mermaid
+flowchart LR
+ subgraph Sources["Data Sources"]
+ A1[App - Fluss Client]
+ A2[Flink CDC Job]
+ end
+
+ subgraph Fluss["Fluss Cluster"]
+ B1[Coordinator]
+ B2[TabletServer]
+ B3[ZooKeeper]
+ end
+
+ subgraph AWS["AWS"]
+ C1[Glue Data Catalog]
+ C2[S3 Bucket]
+ C3[Athena / Spark / Trino]
+ end
+
+ A1 -->|write| B2
+ A2 -->|write| B2
+ B1 -->|register tables| C1
+
+ subgraph Tiering["Flink Tiering Job"]
+ D1[Reads Fluss log]
+ D2[Writes Parquet to S3]
+ D3[Commits Iceberg snapshot]
+ end
+
+ B2 -->|read log| D1
+ D2 -->|Parquet files| C2
+ D3 -->|snapshot metadata| C1
+ C1 -->|discover tables| C3
+ C2 -->|read data| C3
+```
+
+> **Note**: Flink is used here for the tiering service (Fluss →
Iceberg/S3/Glue) and optionally for data ingestion via SQL. Applications can
also write directly to Fluss using the client library (Java, Python). The
tiering service is what bridges Fluss's real-time log to the Glue Data Catalog.
+
+## Prerequisites
+
+### Java Version
+
+Fluss 0.9.x requires **Java 17 or later** for the Tablet Server. Java 11 will
fail with `NoSuchMethodError: MappedByteBuffer.duplicate()`.
+
+### AWS IAM Permissions
+
+The processes running Fluss servers and the Flink tiering service must have
IAM permissions for both Glue and S3. Below is a minimal IAM policy:
+
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Action": [
+ "glue:CreateDatabase",
+ "glue:GetDatabase",
+ "glue:GetDatabases",
+ "glue:DeleteDatabase",
+ "glue:CreateTable",
+ "glue:GetTable",
+ "glue:GetTables",
+ "glue:UpdateTable",
+ "glue:DeleteTable"
+ ],
+ "Resource": [
+ "arn:aws:glue:<region>:<account-id>:catalog",
+ "arn:aws:glue:<region>:<account-id>:database/*",
+ "arn:aws:glue:<region>:<account-id>:table/*"
+ ]
+ },
+ {
+ "Effect": "Allow",
+ "Action": [
+ "s3:GetObject",
+ "s3:PutObject",
+ "s3:DeleteObject",
+ "s3:ListBucket"
+ ],
+ "Resource": [
+ "arn:aws:s3:::<your-bucket>",
+ "arn:aws:s3:::<your-bucket>/*"
+ ]
+ }
+ ]
+}
+```
+
+> **NOTE**: If your account uses **AWS Lake Formation**, the IAM role must
also have Lake Formation permissions (Create Table, Describe, Alter, Insert,
Select, Delete, Drop) on the target database. Standard IAM `glue:*` permissions
alone are not sufficient when Lake Formation governance is enabled.
+
+### Prepare Required JARs
+
+Fluss bundles `iceberg-core` but does **not** bundle the Glue catalog
implementation or the AWS SDK. You must supply additional JARs.
+
+#### For Fluss Servers (Coordinator & Tablet Servers)
+
+Place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/` directory:
+
+- **Iceberg AWS Bundle**:
[iceberg-aws-bundle-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.10.1/iceberg-aws-bundle-1.10.1.jar)
+- **Iceberg AWS**:
[iceberg-aws-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws/1.10.1/iceberg-aws-1.10.1.jar)
+- **Failsafe**:
[failsafe-3.3.2.jar](https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar)
— required by `iceberg-aws` for Glue API retry logic
+
+Both Iceberg JARs are required. The bundle provides AWS SDK v2 dependencies,
while `iceberg-aws` provides the `GlueCatalog` class that Iceberg loads via
reflection. The plugin classloader cannot find `GlueCatalog` from the bundle
alone. The `failsafe` library is a transitive dependency of `iceberg-aws` that
is not bundled.
+
+> **NOTE**: You need **all three** JARs. Using only the bundle will result in
`ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog`. Missing
`failsafe` will cause `NoClassDefFoundError: dev/failsafe/FailsafeException`.
+
+Additionally, you need the Fluss S3 filesystem plugin for remote storage
access. Place
[fluss-fs-s3-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-fs-s3/$FLUSS_VERSION$/fluss-fs-s3-$FLUSS_VERSION$.jar)
in `${FLUSS_HOME}/plugins/s3/`. See [S3
Dependencies](../../../maintenance/filesystems/s3.md#dependencies) for details.
+
+> **TIP**: The Fluss binary distribution already includes
`fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need
to download it separately — only add the three JARs above (iceberg-aws-bundle,
iceberg-aws, failsafe).
+
+#### For the Flink Tiering Service
+
+Place the following JARs in `${FLINK_HOME}/lib`:
+
+1. **Iceberg AWS Bundle**: `iceberg-aws-bundle-1.10.1.jar` (same as above)
+2. **Iceberg AWS**: `iceberg-aws-1.10.1.jar` (same as above — provides
`GlueCatalog` class)
+3. **Failsafe**: `failsafe-3.3.2.jar` (same as above — Glue retry logic)
+4. **Fluss Flink Connector**:
[fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar)
(pick the version matching your Flink runtime)
+5. **Fluss Lake Iceberg**:
[fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
+6. **Fluss Flink Tiering**:
[fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar)
— the tiering job JAR itself
+7. **Fluss S3 Filesystem**:
[fluss-fs-s3-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-fs-s3/$FLUSS_VERSION$/fluss-fs-s3-$FLUSS_VERSION$.jar)
(if S3 is used as Fluss remote storage)
+8. **Hadoop Client**:
[hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar)
and
[hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar)
— required by the Iceberg Parquet writer
+
+> **NOTE**: Despite the Glue catalog itself not requiring Hadoop, the
**tiering service** needs Hadoop classes
(`org.apache.hadoop.conf.Configuration`) for writing Parquet files via Iceberg.
Use the `hadoop-client-api` and `hadoop-client-runtime` JARs (not the full
Hadoop distribution or `flink-shaded-hadoop-2-uber`) to avoid classpath
conflicts with Flink's bundled Avro version. Using the uber JAR causes
`NoSuchMethodError: LogicalTypes.timestampNanos()`.
+
+> **WARNING**: Do **not** add S3 credentials to Flink's configuration file
(`config.yaml` or `flink-conf.yaml`) by appending key-value pairs — this can
break Flink's memory configuration parser. The credential provider chain
configured in Fluss's `server.yaml` handles S3 access for the tiering service
automatically.
+
+## Configure Fluss with AWS Glue
+
+### Cluster Configuration
+
+Add the following to your `server.yaml`:
+
+```yaml
+datalake.format: iceberg
+datalake.iceberg.type: glue
+datalake.iceberg.warehouse: s3://<your-bucket>/<warehouse-path>
+```
+
+Fluss strips the `datalake.iceberg.` prefix and passes the remaining
properties directly to Iceberg's Glue catalog. The properties above become
`type=glue` and `warehouse=s3://...` when initializing the catalog.
+
+You can pass any [Iceberg AWS catalog
property](https://iceberg.apache.org/docs/1.10.1/aws/#glue-catalog) using the
same prefix. Common additional properties:
+
+```yaml
+# Specify the AWS region (required if not using default region from
credentials chain)
+datalake.iceberg.client.region: us-east-1
+
+# Use S3FileIO explicitly (Glue catalog defaults to this when warehouse is
s3://)
+datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
+
+# Optional: restrict catalog to a specific Glue database prefix
+datalake.iceberg.glue.catalog-id: <aws-account-id>
+```
+
+#### Authentication
+
+**Glue Catalog**: The Iceberg AWS SDK uses the [default credentials provider
chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html)
automatically. If running on AWS (EKS, ECS, EC2) with an attached IAM role, no
credential configuration is needed for the Glue catalog connection.
+
+**S3 Filesystem Plugin**: The Fluss S3 filesystem plugin (used for
`remote.data.dir`) requires explicit credential provider configuration. Add the
following to `server.yaml`:
+
+```yaml
+# Tell Hadoop S3A which credential providers to use (in order of preference)
+fs.s3a.aws.credentials.provider: <provider-class-list>
+```
+
+Choose the provider chain for your environment:
+
+| Environment | Provider Chain |
+|-------------|---------------|
+| **ECS Fargate** |
`com.amazonaws.auth.ContainerCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider`
|
+| **EKS (IRSA)** |
`com.amazonaws.auth.WebIdentityTokenCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider`
|
+| **EC2 (instance profile)** |
`com.amazonaws.auth.InstanceProfileCredentialsProvider` |
+
+Example for ECS Fargate:
+
+```yaml
+fs.s3a.aws.credentials.provider:
com.amazonaws.auth.ContainerCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider
+```
+
+The credential provider chain handles automatic credential refresh — no static
keys or manual token rotation needed.
+
+> **NOTE**: For non-standard AWS regions (GovCloud, China), you must also set
the S3 endpoint explicitly. Hadoop S3A defaults to `s3.amazonaws.com` which
does not work for these partitions:
+> ```yaml
+> fs.s3a.endpoint: s3.us-gov-west-1.amazonaws.com # GovCloud
+> fs.s3a.endpoint: s3.cn-north-1.amazonaws.com.cn # China
+> ```
+
+> **NOTE**: You may see a non-fatal warning: `"Session credentials from the
configured AWS credentials provider are not supported for Fluss S3 client-token
generation."` This is expected — the delegation token subsystem cannot generate
tokens from session credentials, but the actual S3 file operations work
correctly via the provider chain.
+
+**Static Credentials (Testing Only)**:
+
+For local development or testing environments where no IAM role is available:
+
+```yaml
+s3.access.key: <your-access-key>
+s3.secret.key: <your-secret-key>
+s3.endpoint: s3.<your-region>.amazonaws.com
+```
+
+### Start Tiering Service
+
+Follow the [Iceberg tiering service setup](../formats/iceberg.md) to prepare
the required JARs. Launch the Flink tiering job with Glue parameters:
+
+```bash
+${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers <coordinator-host>:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type glue \
+ --datalake.iceberg.warehouse s3://<your-bucket>/<warehouse-path> \
+ --datalake.iceberg.client.region <your-aws-region> \
+ --datalake.iceberg.io-impl org.apache.iceberg.aws.s3.S3FileIO
+```
+
+## Quick Start (Full Bootstrap)
+
+This section shows the complete setup from a blank Linux machine to data
queryable in Athena. Adapt paths and versions for your environment.
+
+```bash
+# ─── 1. Install prerequisites ───
+# Java 17+ required (Amazon Corretto, OpenJDK, etc.)
+java -version # Must show 17+
+
+# ─── 2. Install ZooKeeper ───
+wget -q
https://archive.apache.org/dist/zookeeper/zookeeper-3.8.4/apache-zookeeper-3.8.4-bin.tar.gz
+tar -xzf apache-zookeeper-3.8.4-bin.tar.gz -C /opt/zookeeper
--strip-components=1
+cat > /opt/zookeeper/conf/zoo.cfg <<EOF
+tickTime=2000
+dataDir=/var/lib/zookeeper
+clientPort=2181
+EOF
+
+# ─── 3. Install Fluss ───
+wget -q
https://dlcdn.apache.org/incubator/fluss/fluss-0.9.1-incubating/fluss-0.9.1-incubating-bin.tgz
+tar -xzf fluss-0.9.1-incubating-bin.tgz -C /opt/fluss --strip-components=1
+
+# ─── 4. Add Glue JARs to Fluss plugins/iceberg/ ───
+wget -O /opt/fluss/plugins/iceberg/iceberg-aws-bundle-1.10.1.jar \
+
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.10.1/iceberg-aws-bundle-1.10.1.jar
+wget -O /opt/fluss/plugins/iceberg/iceberg-aws-1.10.1.jar \
+
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws/1.10.1/iceberg-aws-1.10.1.jar
+wget -O /opt/fluss/plugins/iceberg/failsafe-3.3.2.jar \
+ https://repo1.maven.org/maven2/dev/failsafe/failsafe/3.3.2/failsafe-3.3.2.jar
+
+# ─── 5. Write server.yaml ───
+cat > /opt/fluss/conf/server.yaml <<EOF
+zookeeper.address: localhost:2181
+coordinator.host: localhost
+coordinator.port: 9123
+tablet-server.host: localhost
+tablet-server.id: 0
+tablet-server.port: 9124
+data.dir: /var/lib/fluss/data
+remote.data.dir: s3://YOUR-BUCKET/fluss-data
+
+datalake.format: iceberg
+datalake.iceberg.type: glue
+datalake.iceberg.warehouse: s3://YOUR-BUCKET/iceberg-warehouse
+datalake.iceberg.client.region: YOUR-REGION
+datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
+
+# S3 credentials — use the provider chain for your environment
+# ECS Fargate:
+fs.s3a.aws.credentials.provider:
com.amazonaws.auth.ContainerCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider,com.amazonaws.auth.InstanceProfileCredentialsProvider
+# EKS (IRSA): use WebIdentityTokenCredentialsProvider instead of
ContainerCredentialsProvider
+# EC2: use InstanceProfileCredentialsProvider alone
+
+# Required for non-standard regions (GovCloud, China):
+# fs.s3a.endpoint: s3.YOUR-REGION.amazonaws.com
+EOF
+
+# ─── 6. Start services ───
+/opt/zookeeper/bin/zkServer.sh start
+sleep 3
+/opt/fluss/bin/coordinator-server.sh start
+sleep 5
+/opt/fluss/bin/tablet-server.sh start
+sleep 5
+
+# ─── 7. Install Flink + tiering JARs ───
+wget -q
https://archive.apache.org/dist/flink/flink-1.20.1/flink-1.20.1-bin-scala_2.12.tgz
+tar -xzf flink-1.20.1-bin-scala_2.12.tgz -C /opt/flink --strip-components=1
+
+# Copy/download all required JARs to Flink lib
+cp /opt/fluss/plugins/iceberg/iceberg-aws-bundle-1.10.1.jar /opt/flink/lib/
+cp /opt/fluss/plugins/iceberg/iceberg-aws-1.10.1.jar /opt/flink/lib/
+cp /opt/fluss/plugins/iceberg/failsafe-3.3.2.jar /opt/flink/lib/
+wget -O /opt/flink/lib/fluss-flink-1.20-0.9.1-incubating.jar \
+
https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-1.20/0.9.1-incubating/fluss-flink-1.20-0.9.1-incubating.jar
+wget -O /opt/flink/lib/fluss-lake-iceberg-0.9.1-incubating.jar \
+
https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-iceberg/0.9.1-incubating/fluss-lake-iceberg-0.9.1-incubating.jar
+wget -O /opt/flink/lib/fluss-flink-tiering-0.9.1-incubating.jar \
+
https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-tiering/0.9.1-incubating/fluss-flink-tiering-0.9.1-incubating.jar
+wget -O /opt/flink/lib/hadoop-client-api-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar
+wget -O /opt/flink/lib/hadoop-client-runtime-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar
+
+# Start Flink
+/opt/flink/bin/start-cluster.sh
+sleep 5
+
+# ─── 8. Create table + insert data (single Flink SQL session) ───
+cat > /tmp/setup.sql <<EOF
+CREATE CATALOG fluss_catalog WITH ('type' = 'fluss', 'bootstrap.servers' =
'localhost:9123');
+USE CATALOG fluss_catalog;
+CREATE DATABASE IF NOT EXISTS my_database;
+USE my_database;
+CREATE TABLE test_table (id BIGINT, name STRING, PRIMARY KEY (id) NOT ENFORCED)
+ WITH ('table.datalake.enabled' = 'true', 'table.datalake.freshness' = '30s');
+INSERT INTO test_table VALUES (1, 'hello'), (2, 'world');
+EOF
+/opt/flink/bin/sql-client.sh -f /tmp/setup.sql
+
+# ─── 9. Start tiering service ───
+/opt/flink/bin/flink run -d
/opt/flink/lib/fluss-flink-tiering-0.9.1-incubating.jar \
+ --fluss.bootstrap.servers localhost:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type glue \
+ --datalake.iceberg.warehouse s3://YOUR-BUCKET/iceberg-warehouse \
+ --datalake.iceberg.client.region YOUR-REGION \
+ --datalake.iceberg.io-impl org.apache.iceberg.aws.s3.S3FileIO
+
+# ─── 10. Query in Athena ───
+# Wait ~30s for tiering to flush, then:
+# SELECT * FROM my_database.test_table;
+```
+
+## Usage Example
+
+### Create a Datalake-Enabled Table
+
+Connect to Fluss via Flink SQL and create a table with data lake tiering
enabled:
+
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = '<coordinator-host>:9123'
+);
+
+USE CATALOG fluss_catalog;
+CREATE DATABASE IF NOT EXISTS my_database;
+USE my_database;
+
+CREATE TABLE customer_orders (
+ `order_id` BIGINT,
+ `customer_name` STRING,
+ `total_amount` DECIMAL(15, 2),
+ `order_date` STRING,
+ PRIMARY KEY (`order_id`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Fluss will register the database in ZooKeeper and create a corresponding
Iceberg table in the Glue Data Catalog (using the same database name). Once
data is ingested and the tiering service commits, Parquet files appear in the
S3 warehouse path.
+
+> **NOTE**: The database must be created in the Fluss catalog first — Fluss
maintains its own database registry in ZooKeeper. The database name in Fluss
maps 1:1 to the Glue database name during tiering.
+
+### Query Data with Athena
+
+AWS Athena uses the Glue Data Catalog by default, so tiered tables are
immediately queryable:
+
+```sql title="Athena SQL"
+SELECT * FROM my_database.customer_orders LIMIT 10;
+```
+
+### Query Data with Flink (Union Read)
+
+```sql title="Flink SQL"
+SET 'execution.runtime-mode' = 'batch';
+
+-- Union read: combines fresh data in Fluss with historical data in Iceberg
+SELECT COUNT(*) FROM customer_orders;
Review Comment:
I don't think we can do union read on Iceberg PK tables, it's a known
limitation.
##########
website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/hive.md:
##########
@@ -0,0 +1,452 @@
+---
+title: Hive Metastore
+sidebar_position: 3
+---
+
+# Hive Metastore
+
+## Introduction
+
+The **Hive Metastore (HMS)** is a central metadata repository commonly used in
Apache Hadoop and other big data ecosystems to store schema and metadata
information for tables. Apache Iceberg provides native integration with Hive
Metastore, storing Iceberg table names and metadata locations directly within
HMS.
+
+This guide explains how to configure Fluss to use Hive Metastore as its
Iceberg catalog. For general Iceberg integration details (table mapping, data
types, limitations), see [Iceberg](../formats/iceberg.md).
+
+## How It Works
+
+When Fluss is configured with Hive Metastore as its Iceberg catalog:
+
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss
client (Java/Python), Flink SQL, or any Kafka-compatible producer. Fluss stores
this data in its real-time log.
+2. **Tiering to Iceberg**: A separate Flink job (the [tiering
service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service))
periodically reads accumulated data from Fluss, converts it to Parquet format,
writes the files to HDFS (or S3/OSS), and commits an Iceberg snapshot to the
Hive Metastore.
+3. **Query via Spark/Trino/Flink**: Any Iceberg-compatible engine configured
with Hive catalog can discover and query the tiered tables through HMS.
+
+```mermaid
+flowchart LR
+ subgraph Sources["Data Sources"]
+ A1[App - Fluss Client]
+ A2[Flink CDC Job]
+ end
+
+ subgraph Fluss["Fluss Cluster"]
+ B1[Coordinator]
+ B2[TabletServer]
+ B3[ZooKeeper]
+ end
+
+ subgraph Hadoop["Hadoop / On-Prem"]
+ C1[Hive Metastore]
+ C2[HDFS / S3 / OSS]
+ C3[Spark / Trino / StarRocks]
+ end
+
+ A1 -->|write| B2
+ A2 -->|write| B2
+ B1 -->|register tables| C1
+
+ subgraph Tiering["Flink Tiering Job"]
+ D1[Reads Fluss log]
+ D2[Writes Parquet]
+ D3[Commits Iceberg snapshot]
+ end
+
+ B2 -->|read log| D1
+ D2 -->|Parquet files| C2
+ D3 -->|snapshot metadata| C1
+ C1 -->|discover tables| C3
+ C2 -->|read data| C3
+```
+
+> **Note**: The tiering service is a Flink job that bridges Fluss's real-time
log to Iceberg tables in Hive Metastore. Flink is also commonly used for data
ingestion (via SQL), but applications can write directly to Fluss using the
client library.
+
+## Prerequisites
+
+### Java Version
+
+Fluss 0.9.x requires **Java 17 or later** for the Tablet Server.
+
+### Running Hive Metastore
+
+Ensure you have a running Hive Metastore service (version **2.x or 3.x**). By
default, HMS listens on thrift port `9083` (e.g.,
`thrift://<metastore-host>:9083`).
+
+:::warning Hive Metastore Version Compatibility
+Iceberg 1.10.1 compiles its Hive catalog client against **Hive 2.3.10**. This
client is compatible with HMS **2.x and 3.x** servers. HMS **4.x** uses an
incompatible thrift protocol and will fail with `TApplicationException: Invalid
method name: 'get_table'`. Use HMS 3.x or earlier.
+:::
+
+### Prepare Required JARs
+
+Iceberg's Hive catalog class (`HiveCatalog`) is distributed separately from
`iceberg-core` and requires the Hive metastore client, Hadoop, and several
transitive dependencies. You must supply all of them.
+
+#### For Fluss Servers (Coordinator & Tablet Servers)
+
+Download and place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/`
directory:
+
+| # | JAR | Size | Purpose |
+|---|-----|------|---------|
+| 1 |
[iceberg-hive-metastore-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar)
| 92 KB | Contains `HiveCatalog` class |
+| 2 |
[hive-exec-2.3.10.jar](https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar)
| 46 MB | Uber JAR: bundles `HiveConf`, `HiveMetaStoreClient`, Thrift, Guava |
+| 3 |
[hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar)
| 20 MB | Hadoop public API (`Configuration`, `FileSystem`) |
+| 4 |
[hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar)
| 30 MB | Hadoop runtime implementations |
+| 5 |
[commons-logging-1.2.jar](https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar)
| 62 KB | Logging facade required by Hadoop and Hive |
+| 6 |
[jackson-core-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar)
| 571 KB | JSON processing |
+| 7 |
[jackson-databind-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar)
| 1.5 MB | JSON data binding |
+| 8 |
[jackson-annotations-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar)
| 75 KB | JSON annotations |
+
+> **NOTE**: The Fluss binary distribution already includes
`fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need
to download it separately — only add the 8 JARs above.
+
+> **TIP**: `hive-exec-2.3.10.jar` is a shaded uber JAR that bundles the Hive
metastore client, `HiveConf`, Apache Thrift, libfb303, and Guava. Using the
full `hive-exec` (not the `-core` variant) avoids needing to track down each
transitive dependency individually. Despite its size, it is the simplest way to
satisfy all Hive client requirements.
+
+#### For the Flink Tiering Service
+
+Place the following JARs in `${FLINK_HOME}/lib`:
+
+1. **All 8 JARs listed above** (iceberg-hive-metastore, hive-exec,
hadoop-client-api, hadoop-client-runtime, commons-logging, jackson-core,
jackson-databind, jackson-annotations)
+2. **Fluss Flink Connector**:
[fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar)
(pick the version matching your Flink runtime)
+3. **Fluss Lake Iceberg**:
[fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
+4. **Fluss Flink Tiering**:
[fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar)
— the tiering job JAR itself
+
+### Hadoop Classpath Configuration (HDFS Only)
+
+If your warehouse is on **HDFS**, both Fluss and Flink must be able to resolve
HDFS paths. This requires Hadoop configuration files on the classpath.
+
+**Option 1: Export Hadoop Classpath (Recommended)**
+
+```bash
+export HADOOP_CLASSPATH=`hadoop classpath`
+```
+
+**Option 2: Copy Hadoop XML Configs**
+
+Copy `core-site.xml` and `hdfs-site.xml` to the configuration directories of
both Fluss (`${FLUSS_HOME}/conf/`) and Flink (`${FLINK_HOME}/conf/`).
+
+> **NOTE**: If your warehouse uses a local filesystem path (for testing) or
S3/OSS (with the appropriate Fluss filesystem plugin), you do not need
`HADOOP_CLASSPATH`.
+
+## Configure Fluss with Hive Metastore
+
+### Cluster Configuration
+
+Add the following to your `server.yaml`:
+
+```yaml
+datalake.format: iceberg
+datalake.iceberg.type: hive
+datalake.iceberg.uri: thrift://<hive-metastore-host>:9083
+datalake.iceberg.warehouse: hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+Fluss strips the `datalake.iceberg.` prefix and passes the remaining
properties directly to Iceberg's Hive catalog. The properties above become
`type=hive`, `uri=thrift://...`, and `warehouse=hdfs://...` when initializing
the catalog.
+
+:::note
+If your Hive warehouse is on cloud object storage, set
`datalake.iceberg.warehouse` to the corresponding URI (e.g.,
`s3://<your-bucket>/warehouse`) and configure the required filesystem plugin.
See [AWS Glue](glue.md) for S3 credential setup.
+:::
+
+### Start Tiering Service
+
+Follow the [Iceberg tiering service
setup](../formats/iceberg.md#start-tiering-service-to-iceberg) instructions to
prepare the environment. Launch the Flink tiering job:
+
+```bash
+${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers <coordinator-host>:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hive \
+ --datalake.iceberg.uri thrift://<hive-metastore-host>:9083 \
+ --datalake.iceberg.warehouse
hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+## Quick Start (Docker Compose)
+
+This section provides a complete docker-compose setup that runs the entire
Hive Metastore integration end-to-end. It starts HMS, ZooKeeper, Fluss, Flink,
creates a table, inserts data, tiers it to Iceberg via HMS, and reads it back.
No cloud account or Hadoop cluster needed — everything runs locally.
+
+**Prerequisites**: Docker and Docker Compose installed.
+
+Create a `docker-compose.yml`:
+
+```yaml
+services:
+ # Hive Metastore (Thrift on port 9083, embedded Derby)
+ metastore:
+ image: apache/hive:3.1.3
+ environment:
+ SERVICE_NAME: metastore
+ ports:
+ - "9083:9083"
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9083'
2>/dev/null"]
+ interval: 10s
+ timeout: 10s
+ retries: 15
+ start_period: 90s
+
+ zookeeper:
+ image: zookeeper:3.8.4
+ healthcheck:
+ test: ["CMD", "zkServer.sh", "status"]
+ interval: 5s
+ timeout: 5s
+ retries: 5
+
+ # Downloads plugin JARs into a shared volume
+ init-plugins:
+ image: amazoncorretto:17
+ entrypoint: ["/bin/bash", "-c"]
+ command:
+ - |
+ set -e
+ yum install -y curl 2>/dev/null
+ mkdir -p /plugins/iceberg
+ curl -sSfL -o /plugins/iceberg/iceberg-hive-metastore-1.10.1.jar \
+
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar
+ curl -sSfL -o /plugins/iceberg/hive-exec-2.3.10.jar \
+
https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-api-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-runtime-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/commons-logging-1.2.jar \
+
https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-core-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-databind-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-annotations-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar
+ echo 'Plugins ready:' && ls -la /plugins/iceberg/
+ volumes:
+ - fluss-plugins:/plugins
+
+ coordinator:
+ image: apache/fluss:0.9.0-incubating
+ command: coordinatorServer
+ depends_on:
+ zookeeper: { condition: service_healthy }
+ metastore: { condition: service_healthy }
+ init-plugins: { condition: service_completed_successfully }
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ coordinator.host: coordinator
+ coordinator.port: 9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hive
+ datalake.iceberg.uri: thrift://metastore:9083
+ datalake.iceberg.warehouse: /tmp/fluss/warehouse
+ volumes:
+ - fluss-data:/tmp/fluss
+ - fluss-plugins:/opt/fluss/plugins
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/coordinator/9123'
2>/dev/null"]
+ interval: 10s
+ timeout: 5s
+ retries: 12
+ start_period: 30s
+
+ tablet-server:
+ image: apache/fluss:0.9.0-incubating
+ command: tabletServer
+ depends_on:
+ coordinator: { condition: service_healthy }
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ tablet-server.host: tablet-server
+ tablet-server.id: 0
+ tablet-server.port: 9124
+ data.dir: /tmp/fluss/data
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hive
+ datalake.iceberg.uri: thrift://metastore:9083
+ datalake.iceberg.warehouse: /tmp/fluss/warehouse
+ volumes:
+ - fluss-data:/tmp/fluss
+ - fluss-plugins:/opt/fluss/plugins
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/tablet-server/9124'
2>/dev/null"]
+ interval: 10s
+ timeout: 5s
+ retries: 12
+ start_period: 30s
+
+ # Flink: creates table, inserts data, starts tiering, reads back
+ flink:
+ image: amazoncorretto:17
+ depends_on:
+ tablet-server: { condition: service_healthy }
+ metastore: { condition: service_healthy }
+ entrypoint: ["/bin/bash", "-c"]
+ command:
+ - |
+ set -e
+ yum install -y wget tar gzip procps findutils 2>&1 | tail -3
+
+ echo '=== Installing Flink 1.20.1 ==='
+ mkdir -p /opt/flink
+ wget -q
https://archive.apache.org/dist/flink/flink-1.20.1/flink-1.20.1-bin-scala_2.12.tgz
-O /tmp/flink.tgz
+ tar -xzf /tmp/flink.tgz -C /opt/flink --strip-components=1 && rm
/tmp/flink.tgz
+
+ echo '=== Downloading Flink JARs ==='
+ wget -q -O /opt/flink/lib/fluss-flink-1.20-0.9.1-incubating.jar
https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-1.20/0.9.1-incubating/fluss-flink-1.20-0.9.1-incubating.jar
+ wget -q -O /opt/flink/lib/fluss-lake-iceberg-0.9.1-incubating.jar
https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-iceberg/0.9.1-incubating/fluss-lake-iceberg-0.9.1-incubating.jar
+ wget -q -O /opt/flink/lib/fluss-flink-tiering-0.9.1-incubating.jar
https://repo1.maven.org/maven2/org/apache/fluss/fluss-flink-tiering/0.9.1-incubating/fluss-flink-tiering-0.9.1-incubating.jar
+ wget -q -O /opt/flink/lib/iceberg-hive-metastore-1.10.1.jar
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar
+ wget -q -O /opt/flink/lib/hive-exec-2.3.10.jar
https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar
+ wget -q -O /opt/flink/lib/hadoop-client-api-3.3.6.jar
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar
+ wget -q -O /opt/flink/lib/hadoop-client-runtime-3.3.6.jar
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar
+ wget -q -O /opt/flink/lib/commons-logging-1.2.jar
https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
+ wget -q -O /opt/flink/lib/jackson-core-2.15.2.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar
+ wget -q -O /opt/flink/lib/jackson-databind-2.15.2.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar
+ wget -q -O /opt/flink/lib/jackson-annotations-2.15.2.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar
+
+ echo '=== Starting Flink ==='
+ /opt/flink/bin/start-cluster.sh && sleep 10
+
+ echo '=== Creating table + inserting data ==='
+ cat > /tmp/setup.sql << 'SQLEOF'
+ CREATE CATALOG fluss_catalog WITH ('type' = 'fluss',
'bootstrap.servers' = 'coordinator:9123');
+ USE CATALOG fluss_catalog;
+ CREATE DATABASE IF NOT EXISTS hive_test_db;
+ USE hive_test_db;
+ CREATE TABLE hive_validation (
+ `event_id` BIGINT,
+ `event_type` STRING,
+ `severity` STRING,
+ `event_date` STRING,
+ PRIMARY KEY (`event_id`) NOT ENFORCED
+ ) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+ );
+ INSERT INTO hive_validation VALUES (1, 'login', 'INFO', '2024-01-01'),
(2, 'error', 'CRITICAL', '2024-01-02'), (3, 'logout', 'INFO', '2024-01-03');
+ SQLEOF
+ sed -i 's/^ //' /tmp/setup.sql
+ /opt/flink/bin/sql-client.sh -f /tmp/setup.sql
+
+ echo '=== Starting tiering service ==='
+ /opt/flink/bin/flink run -d
/opt/flink/lib/fluss-flink-tiering-0.9.1-incubating.jar \
+ --fluss.bootstrap.servers coordinator:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hive \
+ --datalake.iceberg.uri thrift://metastore:9083 \
+ --datalake.iceberg.warehouse /tmp/fluss/warehouse
+
+ echo '=== Waiting 60s for tiering to flush ==='
+ sleep 60
+
+ echo '=== Reading data back ==='
+ cat > /tmp/read.sql << 'SQLEOF'
+ CREATE CATALOG fluss_catalog WITH ('type' = 'fluss',
'bootstrap.servers' = 'coordinator:9123');
+ USE CATALOG fluss_catalog;
+ USE hive_test_db;
+ SET 'sql-client.execution.result-mode' = 'TABLEAU';
+ SET 'execution.runtime-mode' = 'streaming';
+ SELECT * FROM hive_validation LIMIT 3;
+ SQLEOF
+ sed -i 's/^ //' /tmp/read.sql
+ /opt/flink/bin/sql-client.sh -f /tmp/read.sql
+
+ echo '=== DONE ==='
+ sleep 3600
+ volumes:
+ - fluss-data:/tmp/fluss
+
+volumes:
+ fluss-data:
+ fluss-plugins:
+```
+
+Run it:
+
+```bash
+docker compose up
+```
+
+Watch the `flink` container logs. After about 3–4 minutes (JAR downloads +
tiering flush), you should see:
+
+```
++----+----------+------------+----------+------------+
+| op | event_id | event_type | severity | event_date |
++----+----------+------------+----------+------------+
+| +I | 1 | login | INFO | 2024-01-01 |
+| +I | 2 | error | CRITICAL | 2024-01-02 |
+| +I | 3 | logout | INFO | 2024-01-03 |
++----+----------+------------+----------+------------+
+Received a total of 3 rows
+```
+
+Clean up when done:
+
+```bash
+docker compose down -v
+```
+
+## Usage Example
+
+### Create a Datalake-Enabled Table
+
+Connect to Fluss via Flink SQL and create a table with data lake tiering
enabled:
+
+```sql title="Flink SQL"
+CREATE CATALOG fluss_catalog WITH (
+ 'type' = 'fluss',
+ 'bootstrap.servers' = '<coordinator-host>:9123'
+);
+
+USE CATALOG fluss_catalog;
+CREATE DATABASE IF NOT EXISTS my_database;
+USE my_database;
+
+CREATE TABLE daily_events (
+ `event_id` BIGINT,
+ `event_type` STRING,
+ `severity` STRING,
+ `event_date` STRING,
+ PRIMARY KEY (`event_id`) NOT ENFORCED
+) WITH (
+ 'table.datalake.enabled' = 'true',
+ 'table.datalake.freshness' = '30s'
+);
+```
+
+Fluss will create a corresponding Iceberg table in the Hive Metastore under
the database matching your Fluss namespace. Once data is ingested and the
tiering service commits a snapshot, Parquet files appear in the warehouse path
and the table becomes queryable via HMS.
+
+> **NOTE**: The database must be created in the Fluss catalog first — Fluss
maintains its own database registry in ZooKeeper. The database name in Fluss
maps 1:1 to the HMS database name during tiering.
+
+### Query Data with Spark
+
+Since HMS manages the metadata, you can register HMS as an Iceberg catalog in
Apache Spark and query the tiered table immediately:
+
+```sql title="Spark SQL"
+-- Query the tiered Iceberg table from Hive Metastore catalog
+SELECT * FROM hive_catalog.my_database.daily_events;
+```
+
+### Query Data with Flink (Union Read)
+
+```sql title="Flink SQL"
+-- Union read: combines fresh data in Fluss with historical data in Iceberg
+SET 'execution.runtime-mode' = 'streaming';
+SELECT * FROM daily_events;
Review Comment:
ditto
##########
website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/hive.md:
##########
@@ -0,0 +1,452 @@
+---
+title: Hive Metastore
+sidebar_position: 3
+---
+
+# Hive Metastore
+
+## Introduction
+
+The **Hive Metastore (HMS)** is a central metadata repository commonly used in
Apache Hadoop and other big data ecosystems to store schema and metadata
information for tables. Apache Iceberg provides native integration with Hive
Metastore, storing Iceberg table names and metadata locations directly within
HMS.
+
+This guide explains how to configure Fluss to use Hive Metastore as its
Iceberg catalog. For general Iceberg integration details (table mapping, data
types, limitations), see [Iceberg](../formats/iceberg.md).
+
+## How It Works
+
+When Fluss is configured with Hive Metastore as its Iceberg catalog:
+
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss
client (Java/Python), Flink SQL, or any Kafka-compatible producer. Fluss stores
this data in its real-time log.
+2. **Tiering to Iceberg**: A separate Flink job (the [tiering
service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service))
periodically reads accumulated data from Fluss, converts it to Parquet format,
writes the files to HDFS (or S3/OSS), and commits an Iceberg snapshot to the
Hive Metastore.
+3. **Query via Spark/Trino/Flink**: Any Iceberg-compatible engine configured
with Hive catalog can discover and query the tiered tables through HMS.
+
+```mermaid
+flowchart LR
+ subgraph Sources["Data Sources"]
+ A1[App - Fluss Client]
+ A2[Flink CDC Job]
+ end
+
+ subgraph Fluss["Fluss Cluster"]
+ B1[Coordinator]
+ B2[TabletServer]
+ B3[ZooKeeper]
+ end
+
+ subgraph Hadoop["Hadoop / On-Prem"]
+ C1[Hive Metastore]
+ C2[HDFS / S3 / OSS]
+ C3[Spark / Trino / StarRocks]
+ end
+
+ A1 -->|write| B2
+ A2 -->|write| B2
+ B1 -->|register tables| C1
+
+ subgraph Tiering["Flink Tiering Job"]
+ D1[Reads Fluss log]
+ D2[Writes Parquet]
+ D3[Commits Iceberg snapshot]
+ end
+
+ B2 -->|read log| D1
+ D2 -->|Parquet files| C2
+ D3 -->|snapshot metadata| C1
+ C1 -->|discover tables| C3
+ C2 -->|read data| C3
+```
+
+> **Note**: The tiering service is a Flink job that bridges Fluss's real-time
log to Iceberg tables in Hive Metastore. Flink is also commonly used for data
ingestion (via SQL), but applications can write directly to Fluss using the
client library.
+
+## Prerequisites
+
+### Java Version
+
+Fluss 0.9.x requires **Java 17 or later** for the Tablet Server.
+
+### Running Hive Metastore
+
+Ensure you have a running Hive Metastore service (version **2.x or 3.x**). By
default, HMS listens on thrift port `9083` (e.g.,
`thrift://<metastore-host>:9083`).
+
+:::warning Hive Metastore Version Compatibility
+Iceberg 1.10.1 compiles its Hive catalog client against **Hive 2.3.10**. This
client is compatible with HMS **2.x and 3.x** servers. HMS **4.x** uses an
incompatible thrift protocol and will fail with `TApplicationException: Invalid
method name: 'get_table'`. Use HMS 3.x or earlier.
+:::
+
+### Prepare Required JARs
+
+Iceberg's Hive catalog class (`HiveCatalog`) is distributed separately from
`iceberg-core` and requires the Hive metastore client, Hadoop, and several
transitive dependencies. You must supply all of them.
+
+#### For Fluss Servers (Coordinator & Tablet Servers)
+
+Download and place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/`
directory:
+
+| # | JAR | Size | Purpose |
+|---|-----|------|---------|
+| 1 |
[iceberg-hive-metastore-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar)
| 92 KB | Contains `HiveCatalog` class |
+| 2 |
[hive-exec-2.3.10.jar](https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar)
| 46 MB | Uber JAR: bundles `HiveConf`, `HiveMetaStoreClient`, Thrift, Guava |
+| 3 |
[hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar)
| 20 MB | Hadoop public API (`Configuration`, `FileSystem`) |
+| 4 |
[hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar)
| 30 MB | Hadoop runtime implementations |
+| 5 |
[commons-logging-1.2.jar](https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar)
| 62 KB | Logging facade required by Hadoop and Hive |
+| 6 |
[jackson-core-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar)
| 571 KB | JSON processing |
+| 7 |
[jackson-databind-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar)
| 1.5 MB | JSON data binding |
+| 8 |
[jackson-annotations-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar)
| 75 KB | JSON annotations |
+
+> **NOTE**: The Fluss binary distribution already includes
`fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need
to download it separately — only add the 8 JARs above.
+
+> **TIP**: `hive-exec-2.3.10.jar` is a shaded uber JAR that bundles the Hive
metastore client, `HiveConf`, Apache Thrift, libfb303, and Guava. Using the
full `hive-exec` (not the `-core` variant) avoids needing to track down each
transitive dependency individually. Despite its size, it is the simplest way to
satisfy all Hive client requirements.
+
+#### For the Flink Tiering Service
+
+Place the following JARs in `${FLINK_HOME}/lib`:
+
+1. **All 8 JARs listed above** (iceberg-hive-metastore, hive-exec,
hadoop-client-api, hadoop-client-runtime, commons-logging, jackson-core,
jackson-databind, jackson-annotations)
+2. **Fluss Flink Connector**:
[fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar)
(pick the version matching your Flink runtime)
+3. **Fluss Lake Iceberg**:
[fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
+4. **Fluss Flink Tiering**:
[fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar)
— the tiering job JAR itself
+
+### Hadoop Classpath Configuration (HDFS Only)
+
+If your warehouse is on **HDFS**, both Fluss and Flink must be able to resolve
HDFS paths. This requires Hadoop configuration files on the classpath.
+
+**Option 1: Export Hadoop Classpath (Recommended)**
+
+```bash
+export HADOOP_CLASSPATH=`hadoop classpath`
+```
+
+**Option 2: Copy Hadoop XML Configs**
+
+Copy `core-site.xml` and `hdfs-site.xml` to the configuration directories of
both Fluss (`${FLUSS_HOME}/conf/`) and Flink (`${FLINK_HOME}/conf/`).
+
+> **NOTE**: If your warehouse uses a local filesystem path (for testing) or
S3/OSS (with the appropriate Fluss filesystem plugin), you do not need
`HADOOP_CLASSPATH`.
+
+## Configure Fluss with Hive Metastore
+
+### Cluster Configuration
+
+Add the following to your `server.yaml`:
+
+```yaml
+datalake.format: iceberg
+datalake.iceberg.type: hive
+datalake.iceberg.uri: thrift://<hive-metastore-host>:9083
+datalake.iceberg.warehouse: hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+Fluss strips the `datalake.iceberg.` prefix and passes the remaining
properties directly to Iceberg's Hive catalog. The properties above become
`type=hive`, `uri=thrift://...`, and `warehouse=hdfs://...` when initializing
the catalog.
+
+:::note
+If your Hive warehouse is on cloud object storage, set
`datalake.iceberg.warehouse` to the corresponding URI (e.g.,
`s3://<your-bucket>/warehouse`) and configure the required filesystem plugin.
See [AWS Glue](glue.md) for S3 credential setup.
+:::
+
+### Start Tiering Service
+
+Follow the [Iceberg tiering service
setup](../formats/iceberg.md#start-tiering-service-to-iceberg) instructions to
prepare the environment. Launch the Flink tiering job:
+
+```bash
+${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers <coordinator-host>:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hive \
+ --datalake.iceberg.uri thrift://<hive-metastore-host>:9083 \
+ --datalake.iceberg.warehouse
hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+## Quick Start (Docker Compose)
+
+This section provides a complete docker-compose setup that runs the entire
Hive Metastore integration end-to-end. It starts HMS, ZooKeeper, Fluss, Flink,
creates a table, inserts data, tiers it to Iceberg via HMS, and reads it back.
No cloud account or Hadoop cluster needed — everything runs locally.
+
+**Prerequisites**: Docker and Docker Compose installed.
+
+Create a `docker-compose.yml`:
+
+```yaml
+services:
+ # Hive Metastore (Thrift on port 9083, embedded Derby)
+ metastore:
+ image: apache/hive:3.1.3
+ environment:
+ SERVICE_NAME: metastore
+ ports:
+ - "9083:9083"
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9083'
2>/dev/null"]
+ interval: 10s
+ timeout: 10s
+ retries: 15
+ start_period: 90s
+
+ zookeeper:
+ image: zookeeper:3.8.4
+ healthcheck:
+ test: ["CMD", "zkServer.sh", "status"]
+ interval: 5s
+ timeout: 5s
+ retries: 5
+
+ # Downloads plugin JARs into a shared volume
+ init-plugins:
+ image: amazoncorretto:17
+ entrypoint: ["/bin/bash", "-c"]
+ command:
+ - |
+ set -e
+ yum install -y curl 2>/dev/null
+ mkdir -p /plugins/iceberg
+ curl -sSfL -o /plugins/iceberg/iceberg-hive-metastore-1.10.1.jar \
+
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar
+ curl -sSfL -o /plugins/iceberg/hive-exec-2.3.10.jar \
+
https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-api-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-runtime-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/commons-logging-1.2.jar \
+
https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-core-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-databind-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-annotations-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar
+ echo 'Plugins ready:' && ls -la /plugins/iceberg/
+ volumes:
+ - fluss-plugins:/plugins
+
+ coordinator:
+ image: apache/fluss:0.9.0-incubating
Review Comment:
why 0.9.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
