XuQianJin-Stars commented on code in PR #3549: URL: https://github.com/apache/fluss/pull/3549#discussion_r3607451389
########## website/docs/streaming-lakehouse/datalake-formats/hudi.md: ########## @@ -0,0 +1,407 @@ +--- +title: Hudi +sidebar_position: 4 +--- + +# Hudi + +## Introduction + +[Apache Hudi](https://hudi.apache.org/) is an open lakehouse table format that provides transactional writes, record-level updates, and incremental processing on data lakes. +To integrate Fluss with Hudi, you must enable lakehouse storage and configure Hudi as the lakehouse storage. For more details, see [Deploying Streaming Lakehouse](../../install-deploy/deploying-streaming-lakehouse.md). + +Fluss tiers data to standard Hudi tables. Primary-key Fluss tables are written as Hudi Merge-On-Read tables, while Fluss log tables are written as Hudi Copy-On-Write tables. + +## Dependencies + +Apache Fluss publishes the Hudi lake connector to Maven Central: + +| Artifact | Jar | +|----------|-----| +| Fluss Hudi lake connector | [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) | + +Maven coordinates: + +```xml +<dependency> + <groupId>org.apache.fluss</groupId> + <artifactId>fluss-lake-hudi</artifactId> + <version>$FLUSS_VERSION$</version> +</dependency> +``` + +Verify downloaded JARs against the [KEYS file](https://downloads.apache.org/incubator/fluss/KEYS) using the [verification instructions](/downloads#verifying-downloads). + +## Version Compatibility + +| Component | Required/Tested Versions | +|-----------|--------------------------| +| Hudi | 1.1.0 | +| Flink tiering runtime | Flink 1.20 with `hudi-flink1.20-bundle-1.1.0.jar` | + +If you run the tiering service on another Flink version, use the Hudi Flink bundle that matches your Flink runtime. + +## Configure Hudi as LakeHouse Storage + +### Configure Hudi in Cluster Configurations + +To configure Hudi as the lakehouse storage, configure the following options in `server.yaml`. +The example below uses Hudi's DFS catalog: + +```yaml +# Hudi configuration +datalake.enabled: true +datalake.format: hudi + +# Hudi catalog configuration, using DFS catalog mode +datalake.hudi.mode: dfs +datalake.hudi.catalog.path: /tmp/hudi +``` + +The directory configured by `datalake.hudi.catalog.path` must exist and be accessible to the Fluss servers. + +Fluss processes Hudi configurations by stripping the `datalake.hudi.` prefix and passing the remaining options to Hudi. +For example, `datalake.hudi.catalog.path` is passed to Hudi as `catalog.path`. + +Fluss supports the Hudi catalog modes implemented by the connector: + +- `dfs`: Hudi DFS catalog. Tables are stored under `${catalog.path}/${database_name}/${table_name}`. +- `hms`: Hudi Hive Metastore catalog. Configure Hive Metastore access through Hudi/Hadoop options, for example `hive.conf.dir` or `hive.metastore.uris`. Review Comment: Hive Metastore prose vs example key form (under "Fluss supports the Hudi catalog modes implemented by the connector:", the hms bullet) The prose reads: Configure Hive Metastore access through Hudi/Hadoop options, for example `hive.conf.dir` or `hive.metastore.uris`. while the example directly above uses the prefixed key datalake.hudi.hive.metastore.uris. A reader may be confused about which literal key to put in server.yaml. Suggest aligning the prose to the prefixed form actually used in the example: Configure Hive Metastore access through Hudi/Hadoop options, for example `datalake.hudi.hive.conf.dir` or `datalake.hudi.hive.metastore.uris`. ########## fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java: ########## @@ -1674,8 +1674,8 @@ public class ConfigOptions { .enumType(DataLakeFormat.class) .noDefaultValue() .withDescription( - "The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, and `lance`. " - + "In the future, more kinds of data lake format will be supported, such as DeltaLake or Hudi. " + "The data lake format of the table specifies the tiered Lakehouse storage format. Currently, supported formats are `paimon`, `iceberg`, `hudi`, and `lance`. " + + "In the future, more kinds of data lake format will be supported, such as DeltaLake. " Review Comment: "more kinds of data lake format will be supported". Since "kinds" is plural, "format" should be "formats": ``` ... more kinds of data lake format will be supported ... ^^^^^^^ -> formats ``` ########## website/docs/streaming-lakehouse/datalake-formats/hudi.md: ########## @@ -0,0 +1,407 @@ +--- +title: Hudi +sidebar_position: 4 +--- + +# Hudi + +## Introduction + +[Apache Hudi](https://hudi.apache.org/) is an open lakehouse table format that provides transactional writes, record-level updates, and incremental processing on data lakes. +To integrate Fluss with Hudi, you must enable lakehouse storage and configure Hudi as the lakehouse storage. For more details, see [Deploying Streaming Lakehouse](../../install-deploy/deploying-streaming-lakehouse.md). + +Fluss tiers data to standard Hudi tables. Primary-key Fluss tables are written as Hudi Merge-On-Read tables, while Fluss log tables are written as Hudi Copy-On-Write tables. + +## Dependencies + +Apache Fluss publishes the Hudi lake connector to Maven Central: + +| Artifact | Jar | +|----------|-----| +| Fluss Hudi lake connector | [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) | + +Maven coordinates: + +```xml +<dependency> + <groupId>org.apache.fluss</groupId> + <artifactId>fluss-lake-hudi</artifactId> + <version>$FLUSS_VERSION$</version> +</dependency> +``` + +Verify downloaded JARs against the [KEYS file](https://downloads.apache.org/incubator/fluss/KEYS) using the [verification instructions](/downloads#verifying-downloads). + +## Version Compatibility + +| Component | Required/Tested Versions | +|-----------|--------------------------| +| Hudi | 1.1.0 | +| Flink tiering runtime | Flink 1.20 with `hudi-flink1.20-bundle-1.1.0.jar` | + +If you run the tiering service on another Flink version, use the Hudi Flink bundle that matches your Flink runtime. + +## Configure Hudi as LakeHouse Storage + +### Configure Hudi in Cluster Configurations + +To configure Hudi as the lakehouse storage, configure the following options in `server.yaml`. +The example below uses Hudi's DFS catalog: + +```yaml +# Hudi configuration +datalake.enabled: true +datalake.format: hudi + +# Hudi catalog configuration, using DFS catalog mode +datalake.hudi.mode: dfs +datalake.hudi.catalog.path: /tmp/hudi +``` + +The directory configured by `datalake.hudi.catalog.path` must exist and be accessible to the Fluss servers. + +Fluss processes Hudi configurations by stripping the `datalake.hudi.` prefix and passing the remaining options to Hudi. +For example, `datalake.hudi.catalog.path` is passed to Hudi as `catalog.path`. + +Fluss supports the Hudi catalog modes implemented by the connector: + +- `dfs`: Hudi DFS catalog. Tables are stored under `${catalog.path}/${database_name}/${table_name}`. +- `hms`: Hudi Hive Metastore catalog. Configure Hive Metastore access through Hudi/Hadoop options, for example `hive.conf.dir` or `hive.metastore.uris`. + +Example using Hive Metastore catalog mode: + +```yaml +datalake.enabled: true +datalake.format: hudi +datalake.hudi.mode: hms +datalake.hudi.catalog.path: hdfs:///warehouse/hudi +datalake.hudi.hive.metastore.uris: thrift://<hive-metastore-host>:9083 +``` + +### Prepare Server-Side JARs + +Put the following JARs into `${FLUSS_HOME}/plugins/hudi/` before starting the Fluss servers: + +- [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) +- The Hudi Flink bundle matching your runtime. For Flink 1.20, use [hudi-flink1.20-bundle-1.1.0.jar](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.20-bundle/1.1.0/hudi-flink1.20-bundle-1.1.0.jar). +- Any Hadoop, Hive, or filesystem dependencies required by your Hudi catalog and table storage. + +Restart Fluss after changing the plugin directory. + +### Start Tiering Service to Hudi + +Then, start the datalake tiering service to tier Fluss data to Hudi. For the general process, see [Deploying Streaming Lakehouse](../../install-deploy/deploying-streaming-lakehouse.md). + +For Hudi, prepare the following JARs in `${FLINK_HOME}/lib`: + +- Put the Fluss Flink connector JAR into `${FLINK_HOME}/lib`; pick the connector matching your Flink version (see [Dependencies](../../engine-flink/getting-started.md#dependencies)). For Flink 1.20, use [fluss-flink-1.20-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-flink-1.20/$FLUSS_VERSION$/fluss-flink-1.20-$FLUSS_VERSION$.jar). +- If you are using [Amazon S3](http://aws.amazon.com/s3/), [Aliyun OSS](https://www.aliyun.com/product/oss), or [HDFS](https://hadoop.apache.org/docs/stable/) as Fluss's [remote storage](../../maintenance/tiered-storage/remote-storage.md), download the corresponding Fluss filesystem JAR (see the [Filesystems](../../maintenance/tiered-storage/filesystems/overview.md) section) and put it into `${FLINK_HOME}/lib`. +- Put [fluss-lake-hudi-$FLUSS_VERSION$.jar]($FLUSS_MAVEN_REPO_URL$/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) into `${FLINK_HOME}/lib`. +- Put the Hudi Flink bundle into `${FLINK_HOME}/lib`. For Flink 1.20, use [hudi-flink1.20-bundle-1.1.0.jar](https://repo.maven.apache.org/maven2/org/apache/hudi/hudi-flink1.20-bundle/1.1.0/hudi-flink1.20-bundle-1.1.0.jar). +- Put any Hadoop, Hive, or filesystem dependencies required by your Hudi catalog and table storage into `${FLINK_HOME}/lib`. + +Start the Flink tiering job with Hudi-specific configurations: + +```shell +<FLINK_HOME>/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \ + --fluss.bootstrap.servers localhost:9123 \ + --datalake.format hudi \ + --datalake.hudi.mode dfs \ + --datalake.hudi.catalog.path /tmp/hudi +``` + +For Hive Metastore catalog mode: + +```shell +<FLINK_HOME>/bin/flink run /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \ + --fluss.bootstrap.servers localhost:9123 \ + --datalake.format hudi \ + --datalake.hudi.mode hms \ + --datalake.hudi.catalog.path hdfs:///warehouse/hudi \ + --datalake.hudi.hive.metastore.uris thrift://<hive-metastore-host>:9083 +``` + +Then, the datalake tiering service continuously tiers data from Fluss to Hudi. The table option `table.datalake.freshness` controls the target freshness of Hudi tables. By default, the data freshness is 3 minutes. + +## Table Mapping Between Fluss and Hudi + +When a Fluss table is created with the option `'table.datalake.enabled' = 'true'` and configured with Hudi as the datalake format, Fluss automatically creates a corresponding Hudi table with the same database and table name. + +For DFS catalog mode, the Hudi table path is `${catalog.path}/${database_name}/${table_name}` unless the Hudi table path is explicitly set by Hudi options. +For Hive Metastore catalog mode, the table path follows Hudi Hive catalog path inference. + +The schema of the Hudi table matches the Fluss table schema, except for three system columns appended by Fluss: + +| Column | Type | Description | +|---------------|--------------|-----------------------------------------------| +| `__bucket` | INT | Fluss bucket identifier for data distribution | +| `__offset` | BIGINT | Fluss log offset for ordering and seeking | +| `__timestamp` | TIMESTAMP(6) | Fluss log timestamp | + +Do not use user columns named `__bucket`, `__offset`, or `__timestamp`. Hudi metadata column names starting with `_hoodie_` are also reserved. + +### Primary Key Tables + +Primary-key Fluss tables are mapped to Hudi Merge-On-Read tables: + +- Hudi table type is set to `MERGE_ON_READ`. +- Hudi record key fields are derived from the Fluss primary key. +- Hudi bucket index is enabled and uses the Fluss bucket number. +- If Fluss bucket keys are configured, Hudi bucket index key fields are derived from the Fluss bucket keys. Otherwise Hudi uses the record key fields for bucket indexing. +- Writes use Hudi upsert semantics. + +Example: + +```sql title="Flink SQL" +USE CATALOG fluss_catalog; + +CREATE TABLE user_profiles ( + `user_id` BIGINT, + `username` STRING, + `email` STRING, + `last_login` TIMESTAMP, + `profile_data` STRING, + PRIMARY KEY (`user_id`) NOT ENFORCED +) WITH ( + 'table.datalake.enabled' = 'true', + 'table.datalake.freshness' = '30s', + 'bucket.num' = '4', + 'bucket.key' = 'user_id', + 'hudi.precombine.field' = 'last_login' +); +``` + +`hudi.precombine.field` is a Hudi table option used by Hudi to order records with the same record key during upsert. Choose a field that reflects your update ordering. + +Conceptually, Fluss creates a Hudi table with properties equivalent to: + +```sql title="Hudi table properties" Review Comment: cluster/server options use datalake.hudi.* (stripped by Fluss before passing to the catalog), while table-level Fluss options use hudi.* (stripped before passing to Hudi). This distinction is implicit and easy to miss. A one-line note such as "Cluster options use the datalake.hudi. prefix; table-level Hudi options use the hudi. prefix — both are stripped before being forwarded" would prevent misconfiguration. Minor, but worth it given Hudi has many options. ########## website/docs/install-deploy/deploying-streaming-lakehouse.md: ########## @@ -122,6 +141,15 @@ Add JARs to `${FLUSS_HOME}/plugins/<format>/` based on your configuration: | Iceberg with S3 | [`iceberg-aws-<version>.jar`, `iceberg-aws-bundle-<version>.jar`](https://iceberg.apache.org/docs/1.10.1/aws/), matching your Iceberg version | | Iceberg JDBC catalog | PostgreSQL/MySQL JDBC driver | +</TabItem> +<TabItem value="hudi" label="Hudi"> + +| Scenario | Required JAR | +|----------|--------------| +| Hudi lake connector | [fluss-lake-hudi-$FLUSS_VERSION$.jar](https://repo1.maven.org/maven2/org/apache/fluss/fluss-lake-hudi/$FLUSS_VERSION$/fluss-lake-hudi-$FLUSS_VERSION$.jar) | Review Comment: URL base style The Hudi connector JAR link uses a hard-coded https://repo1.maven.org/maven2/.../fluss-lake-hudi/$FLUSS_VERSION$/..., which is consistent with the sibling Paimon/Iceberg/Lance rows in the same file (lines ~149, ~158, ~167). No change needed; just noting it is internally consistent. (hudi.md itself uses the $FLUSS_MAVEN_REPO_URL$ variable, which is the other documented convention — also fine.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
