qzyu999 commented on code in PR #3424:
URL: https://github.com/apache/fluss/pull/3424#discussion_r3598413109
##########
website/docs/streaming-lakehouse/integrate-data-lakes/catalogs/hive.md:
##########
@@ -0,0 +1,452 @@
+---
+title: Hive Metastore
+sidebar_position: 3
+---
+
+# Hive Metastore
+
+## Introduction
+
+The **Hive Metastore (HMS)** is a central metadata repository commonly used in
Apache Hadoop and other big data ecosystems to store schema and metadata
information for tables. Apache Iceberg provides native integration with Hive
Metastore, storing Iceberg table names and metadata locations directly within
HMS.
+
+This guide explains how to configure Fluss to use Hive Metastore as its
Iceberg catalog. For general Iceberg integration details (table mapping, data
types, limitations), see [Iceberg](../formats/iceberg.md).
+
+## How It Works
+
+When Fluss is configured with Hive Metastore as its Iceberg catalog:
+
+1. **Data ingestion**: Applications write data to Fluss tables using the Fluss
client (Java/Python), Flink SQL, or any Kafka-compatible producer. Fluss stores
this data in its real-time log.
+2. **Tiering to Iceberg**: A separate Flink job (the [tiering
service](maintenance/tiered-storage/lakehouse-storage.md#start-the-datalake-tiering-service))
periodically reads accumulated data from Fluss, converts it to Parquet format,
writes the files to HDFS (or S3/OSS), and commits an Iceberg snapshot to the
Hive Metastore.
+3. **Query via Spark/Trino/Flink**: Any Iceberg-compatible engine configured
with Hive catalog can discover and query the tiered tables through HMS.
+
+```mermaid
+flowchart LR
+ subgraph Sources["Data Sources"]
+ A1[App - Fluss Client]
+ A2[Flink CDC Job]
+ end
+
+ subgraph Fluss["Fluss Cluster"]
+ B1[Coordinator]
+ B2[TabletServer]
+ B3[ZooKeeper]
+ end
+
+ subgraph Hadoop["Hadoop / On-Prem"]
+ C1[Hive Metastore]
+ C2[HDFS / S3 / OSS]
+ C3[Spark / Trino / StarRocks]
+ end
+
+ A1 -->|write| B2
+ A2 -->|write| B2
+ B1 -->|register tables| C1
+
+ subgraph Tiering["Flink Tiering Job"]
+ D1[Reads Fluss log]
+ D2[Writes Parquet]
+ D3[Commits Iceberg snapshot]
+ end
+
+ B2 -->|read log| D1
+ D2 -->|Parquet files| C2
+ D3 -->|snapshot metadata| C1
+ C1 -->|discover tables| C3
+ C2 -->|read data| C3
+```
+
+> **Note**: The tiering service is a Flink job that bridges Fluss's real-time
log to Iceberg tables in Hive Metastore. Flink is also commonly used for data
ingestion (via SQL), but applications can write directly to Fluss using the
client library.
+
+## Prerequisites
+
+### Java Version
+
+Fluss 0.9.x requires **Java 17 or later** for the Tablet Server.
+
+### Running Hive Metastore
+
+Ensure you have a running Hive Metastore service (version **2.x or 3.x**). By
default, HMS listens on thrift port `9083` (e.g.,
`thrift://<metastore-host>:9083`).
+
+:::warning Hive Metastore Version Compatibility
+Iceberg 1.10.1 compiles its Hive catalog client against **Hive 2.3.10**. This
client is compatible with HMS **2.x and 3.x** servers. HMS **4.x** uses an
incompatible thrift protocol and will fail with `TApplicationException: Invalid
method name: 'get_table'`. Use HMS 3.x or earlier.
+:::
+
+### Prepare Required JARs
+
+Iceberg's Hive catalog class (`HiveCatalog`) is distributed separately from
`iceberg-core` and requires the Hive metastore client, Hadoop, and several
transitive dependencies. You must supply all of them.
+
+#### For Fluss Servers (Coordinator & Tablet Servers)
+
+Download and place the following JARs in the `${FLUSS_HOME}/plugins/iceberg/`
directory:
+
+| # | JAR | Size | Purpose |
+|---|-----|------|---------|
+| 1 |
[iceberg-hive-metastore-1.10.1.jar](https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar)
| 92 KB | Contains `HiveCatalog` class |
+| 2 |
[hive-exec-2.3.10.jar](https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar)
| 46 MB | Uber JAR: bundles `HiveConf`, `HiveMetaStoreClient`, Thrift, Guava |
+| 3 |
[hadoop-client-api-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar)
| 20 MB | Hadoop public API (`Configuration`, `FileSystem`) |
+| 4 |
[hadoop-client-runtime-3.3.6.jar](https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar)
| 30 MB | Hadoop runtime implementations |
+| 5 |
[commons-logging-1.2.jar](https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar)
| 62 KB | Logging facade required by Hadoop and Hive |
+| 6 |
[jackson-core-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar)
| 571 KB | JSON processing |
+| 7 |
[jackson-databind-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar)
| 1.5 MB | JSON data binding |
+| 8 |
[jackson-annotations-2.15.2.jar](https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar)
| 75 KB | JSON annotations |
+
+> **NOTE**: The Fluss binary distribution already includes
`fluss-lake-iceberg-$FLUSS_VERSION$.jar` in `plugins/iceberg/`. You do not need
to download it separately — only add the 8 JARs above.
+
+> **TIP**: `hive-exec-2.3.10.jar` is a shaded uber JAR that bundles the Hive
metastore client, `HiveConf`, Apache Thrift, libfb303, and Guava. Using the
full `hive-exec` (not the `-core` variant) avoids needing to track down each
transitive dependency individually. Despite its size, it is the simplest way to
satisfy all Hive client requirements.
+
+#### For the Flink Tiering Service
+
+Place the following JARs in `${FLINK_HOME}/lib`:
+
+1. **All 8 JARs listed above** (iceberg-hive-metastore, hive-exec,
hadoop-client-api, hadoop-client-runtime, commons-logging, jackson-core,
jackson-databind, jackson-annotations)
+2. **Fluss Flink Connector**:
[fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar)
(pick the version matching your Flink runtime)
+3. **Fluss Lake Iceberg**:
[fluss-lake-iceberg-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-iceberg/$FLUSS_VERSION$/fluss-lake-iceberg-$FLUSS_VERSION$.jar)
+4. **Fluss Flink Tiering**:
[fluss-flink-tiering-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-tiering/$FLUSS_VERSION$/fluss-flink-tiering-$FLUSS_VERSION$.jar)
— the tiering job JAR itself
+
+### Hadoop Classpath Configuration (HDFS Only)
+
+If your warehouse is on **HDFS**, both Fluss and Flink must be able to resolve
HDFS paths. This requires Hadoop configuration files on the classpath.
+
+**Option 1: Export Hadoop Classpath (Recommended)**
+
+```bash
+export HADOOP_CLASSPATH=`hadoop classpath`
+```
+
+**Option 2: Copy Hadoop XML Configs**
+
+Copy `core-site.xml` and `hdfs-site.xml` to the configuration directories of
both Fluss (`${FLUSS_HOME}/conf/`) and Flink (`${FLINK_HOME}/conf/`).
+
+> **NOTE**: If your warehouse uses a local filesystem path (for testing) or
S3/OSS (with the appropriate Fluss filesystem plugin), you do not need
`HADOOP_CLASSPATH`.
+
+## Configure Fluss with Hive Metastore
+
+### Cluster Configuration
+
+Add the following to your `server.yaml`:
+
+```yaml
+datalake.format: iceberg
+datalake.iceberg.type: hive
+datalake.iceberg.uri: thrift://<hive-metastore-host>:9083
+datalake.iceberg.warehouse: hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+Fluss strips the `datalake.iceberg.` prefix and passes the remaining
properties directly to Iceberg's Hive catalog. The properties above become
`type=hive`, `uri=thrift://...`, and `warehouse=hdfs://...` when initializing
the catalog.
+
+:::note
+If your Hive warehouse is on cloud object storage, set
`datalake.iceberg.warehouse` to the corresponding URI (e.g.,
`s3://<your-bucket>/warehouse`) and configure the required filesystem plugin.
See [AWS Glue](glue.md) for S3 credential setup.
+:::
+
+### Start Tiering Service
+
+Follow the [Iceberg tiering service
setup](../formats/iceberg.md#start-tiering-service-to-iceberg) instructions to
prepare the environment. Launch the Flink tiering job:
+
+```bash
+${FLINK_HOME}/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers <coordinator-host>:9123 \
+ --datalake.format iceberg \
+ --datalake.iceberg.type hive \
+ --datalake.iceberg.uri thrift://<hive-metastore-host>:9083 \
+ --datalake.iceberg.warehouse
hdfs://<namenode-host>:9000/user/hive/warehouse
+```
+
+## Quick Start (Docker Compose)
+
+This section provides a complete docker-compose setup that runs the entire
Hive Metastore integration end-to-end. It starts HMS, ZooKeeper, Fluss, Flink,
creates a table, inserts data, tiers it to Iceberg via HMS, and reads it back.
No cloud account or Hadoop cluster needed — everything runs locally.
+
+**Prerequisites**: Docker and Docker Compose installed.
+
+Create a `docker-compose.yml`:
+
+```yaml
+services:
+ # Hive Metastore (Thrift on port 9083, embedded Derby)
+ metastore:
+ image: apache/hive:3.1.3
+ environment:
+ SERVICE_NAME: metastore
+ ports:
+ - "9083:9083"
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/9083'
2>/dev/null"]
+ interval: 10s
+ timeout: 10s
+ retries: 15
+ start_period: 90s
+
+ zookeeper:
+ image: zookeeper:3.8.4
+ healthcheck:
+ test: ["CMD", "zkServer.sh", "status"]
+ interval: 5s
+ timeout: 5s
+ retries: 5
+
+ # Downloads plugin JARs into a shared volume
+ init-plugins:
+ image: amazoncorretto:17
+ entrypoint: ["/bin/bash", "-c"]
+ command:
+ - |
+ set -e
+ yum install -y curl 2>/dev/null
+ mkdir -p /plugins/iceberg
+ curl -sSfL -o /plugins/iceberg/iceberg-hive-metastore-1.10.1.jar \
+
https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-hive-metastore/1.10.1/iceberg-hive-metastore-1.10.1.jar
+ curl -sSfL -o /plugins/iceberg/hive-exec-2.3.10.jar \
+
https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.3.10/hive-exec-2.3.10.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-api-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.6/hadoop-client-api-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/hadoop-client-runtime-3.3.6.jar \
+
https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.6/hadoop-client-runtime-3.3.6.jar
+ curl -sSfL -o /plugins/iceberg/commons-logging-1.2.jar \
+
https://repo1.maven.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-core-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.15.2/jackson-core-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-databind-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.15.2/jackson-databind-2.15.2.jar
+ curl -sSfL -o /plugins/iceberg/jackson-annotations-2.15.2.jar \
+
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.15.2/jackson-annotations-2.15.2.jar
+ echo 'Plugins ready:' && ls -la /plugins/iceberg/
+ volumes:
+ - fluss-plugins:/plugins
+
+ coordinator:
+ image: apache/fluss:0.9.0-incubating
+ command: coordinatorServer
+ depends_on:
+ zookeeper: { condition: service_healthy }
+ metastore: { condition: service_healthy }
+ init-plugins: { condition: service_completed_successfully }
+ environment:
+ - |
+ FLUSS_PROPERTIES=
+ zookeeper.address: zookeeper:2181
+ coordinator.host: coordinator
+ coordinator.port: 9123
+ remote.data.dir: /tmp/fluss/remote-data
+ datalake.format: iceberg
+ datalake.iceberg.type: hive
+ datalake.iceberg.uri: thrift://metastore:9083
+ datalake.iceberg.warehouse: /tmp/fluss/warehouse
+ volumes:
+ - fluss-data:/tmp/fluss
+ - fluss-plugins:/opt/fluss/plugins
+ healthcheck:
+ test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/coordinator/9123'
2>/dev/null"]
+ interval: 10s
+ timeout: 5s
+ retries: 12
+ start_period: 30s
+
+ tablet-server:
+ image: apache/fluss:0.9.0-incubating
Review Comment:
Hi @fresh-borzoni , same fix as above (c0af314f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
