luoyuxia commented on code in PR #3549: URL: https://github.com/apache/fluss/pull/3549#discussion_r3607492795
########## website/docs/streaming-lakehouse/datalake-formats/hudi.md: ########## @@ -0,0 +1,407 @@ +--- +title: Hudi +sidebar_position: 4 +--- + +# Hudi + +## Introduction + +[Apache Hudi](https://hudi.apache.org/) is an open lakehouse table format that provides transactional writes, record-level updates, and incremental processing on data lakes. +To integrate Fluss with Hudi, you must enable lakehouse storage and configure Hudi as the lakehouse storage. For more details, see [Deploying Streaming Lakehouse](../../install-deploy/deploying-streaming-lakehouse.md). + +Fluss tiers data to standard Hudi tables. Primary-key Fluss tables are written as Hudi Merge-On-Read tables, while Fluss log tables are written as Hudi Copy-On-Write tables. + +## Dependencies + +Apache Fluss publishes the Hudi lake connector to Maven Central: + +| Artifact | Jar | +|----------|-----| +| Fluss Hudi lake connector | [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) | + +Maven coordinates: + +```xml +<dependency> + <groupId>org.apache.fluss</groupId> + <artifactId>fluss-lake-hudi</artifactId> + <version>$FLUSS_VERSION$</version> +</dependency> +``` + +Verify downloaded JARs against the [KEYS file](https://downloads.apache.org/incubator/fluss/KEYS) using the [verification instructions](/downloads#verifying-downloads). + +## Version Compatibility + +| Component | Required/Tested Versions | +|-----------|--------------------------| +| Hudi | 1.1.0 | +| Flink tiering runtime | Flink 1.20 with `hudi-flink1.20-bundle-1.1.0.jar` | + +If you run the tiering service on another Flink version, use the Hudi Flink bundle that matches your Flink runtime. + +## Configure Hudi as LakeHouse Storage + +### Configure Hudi in Cluster Configurations + +To configure Hudi as the lakehouse storage, configure the following options in `server.yaml`. +The example below uses Hudi's DFS catalog: + +```yaml +# Hudi configuration +datalake.enabled: true +datalake.format: hudi + +# Hudi catalog configuration, using DFS catalog mode +datalake.hudi.mode: dfs +datalake.hudi.catalog.path: /tmp/hudi +``` + +The directory configured by `datalake.hudi.catalog.path` must exist and be accessible to the Fluss servers. + +Fluss processes Hudi configurations by stripping the `datalake.hudi.` prefix and passing the remaining options to Hudi. +For example, `datalake.hudi.catalog.path` is passed to Hudi as `catalog.path`. + +Fluss supports the Hudi catalog modes implemented by the connector: + +- `dfs`: Hudi DFS catalog. Tables are stored under `${catalog.path}/${database_name}/${table_name}`. +- `hms`: Hudi Hive Metastore catalog. Configure Hive Metastore access through Hudi/Hadoop options, for example `hive.conf.dir` or `hive.metastore.uris`. + +Example using Hive Metastore catalog mode: + +```yaml +datalake.enabled: true +datalake.format: hudi +datalake.hudi.mode: hms +datalake.hudi.catalog.path: hdfs:///warehouse/hudi +datalake.hudi.hive.metastore.uris: thrift://<hive-metastore-host>:9083 +``` + +### Prepare Server-Side JARs + +Put the following JARs into `${FLUSS_HOME}/plugins/hudi/` before starting the Fluss servers: + +- [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) +- The Hudi Flink bundle matching your runtime. For Flink 1.20, use [hudi-flink1.20-bundle-1.1.0.jar](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.20-bundle/1.1.0/hudi-flink1.20-bundle-1.1.0.jar). Review Comment: Could we also include the Flink core and Table API/runtime JARs required by the server-side plugin? `fluss-lake-hudi` directly imports classes such as `org.apache.flink.configuration.Configuration`, `org.apache.flink.table.catalog.Catalog`, and `DataTypes`, while its `flink-core`, `flink-table-common`, `flink-table-api-java`, and `flink-table-runtime` dependencies are declared as `provided`. `hudi-flink1.20-bundle-1.1.0.jar` does not contain these classes because it assumes an existing Flink runtime. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
