luoyuxia commented on code in PR #2699:
URL: https://github.com/apache/fluss/pull/2699#discussion_r3523170753
##########
website/docs/streaming-lakehouse/integrate-data-lakes/_category_.json:
##########
@@ -1,4 +0,0 @@
-{
Review Comment:
This PR renames existing docs paths, but does not preserve redirects for the
old URLs.
For example:
- `/docs/maintenance/filesystems/s3`
- `/docs/maintenance/filesystems/hdfs`
- `/docs/streaming-lakehouse/integrate-data-lakes/formats/paimon`
- `/docs/streaming-lakehouse/integrate-data-lakes/catalogs/lakekeeper`
These are published documentation URLs and will become 404 after the rename.
The current `createRedirects` logic only creates version aliases for routes
that still exist; it does not create redirects for deleted old routes.
Please add explicit redirects for all renamed leaf pages.
##########
website/build_versioned_docs.sh:
##########
@@ -122,6 +122,14 @@ for branch in $branches; do
# Copy the website/docs directory to the target directory
$CP_CMD "$version_dir/" || { echo "Failed to copy for branch:
$clean_branch_name"; continue; }
echo "Copied documentation for branch '$clean_branch_name' to
'$version_dir'"
+
+ # Fix absolute links in versioned docs (convert
/docs/maintenance/filesystems/hdfs.md to relative path)
+
lakehouse_file="$version_dir/maintenance/tiered-storage/lakehouse-storage.md"
+ if [ -f "$lakehouse_file" ]; then
+ sed -i.bak
's|/docs/maintenance/filesystems/hdfs\.md|../../filesystems/hdfs|g'
"$lakehouse_file"
Review Comment:
The replacement path here looks wrong.
`lakehouse_file` is under
`version-$version/maintenance/tiered-storage/lakehouse-storage.md`, so
replacing `/docs/maintenance/filesystems/hdfs.md` with `../../filesystems/hdfs`
resolves to `version-$version/filesystems/hdfs`.
For release branches, the copied old filesystem docs are still under
`version-$version/maintenance/filesystems/hdfs.md`, so this should be
`../filesystems/hdfs` instead.
##########
website/docs/install-deploy/deploying-streaming-lakehouse.md:
##########
@@ -0,0 +1,237 @@
+---
+title: "Deploying Streaming Lakehouse"
+sidebar_position: 6
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Deploying Streaming Lakehouse
+
+This guide covers how to deploy a Fluss cluster with Streaming Lakehouse
capabilities. For conceptual overview, see [Lakehouse
Overview](../streaming-lakehouse/overview.md).
+
+## Prerequisites
+
+1. A running Fluss cluster (see [Deploying Distributed
Cluster](deploying-distributed-cluster.md))
+2. A running Flink cluster (for the Tiering Service)
+3. Access to a data lake storage system (S3, HDFS, OSS, etc.)
+
+## Cluster Configuration
+
+You can enable Lakehouse storage through:
+1. **Static configuration**: Configure in `server.yaml` before starting the
cluster
+2. **Dynamic configuration**: Enable at runtime using the
`set_cluster_configs` procedure
+
+### Method 1: Static Configuration
+
+Configure lakehouse settings in `server.yaml` on all Fluss servers
(CoordinatorServer and TabletServer).
Review Comment:
For “Method 1: Static Configuration”, could we add a short explanation of
the configuration convention before the Paimon/Iceberg/Lance examples?
Right now users can copy the examples, but it is not clear how to derive the
keys when using Hive/REST/Glue/JDBC catalogs.
Suggested points:
- `datalake.enabled: true` explicitly enables lakehouse capability for the
cluster.
- `datalake.format` selects the default lake format.
- Format-specific options use the `datalake.<format>.*` prefix.
- Fluss strips the `datalake.<format>.` prefix and passes the remaining keys
to the corresponding lake catalog/client. For example,
`datalake.paimon.metastore` becomes `metastore`, and `datalake.iceberg.type`
becomes `type`.
- Only configure options for the selected `datalake.format`.
- The tiering service also needs the same `datalake.*` options because it is
an independent Flink job.
##########
website/docs/install-deploy/deploying-streaming-lakehouse.md:
##########
@@ -0,0 +1,237 @@
+---
+title: "Deploying Streaming Lakehouse"
+sidebar_position: 6
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Deploying Streaming Lakehouse
+
+This guide covers how to deploy a Fluss cluster with Streaming Lakehouse
capabilities. For conceptual overview, see [Lakehouse
Overview](../streaming-lakehouse/overview.md).
+
+## Prerequisites
+
+1. A running Fluss cluster (see [Deploying Distributed
Cluster](deploying-distributed-cluster.md))
+2. A running Flink cluster (for the Tiering Service)
+3. Access to a data lake storage system (S3, HDFS, OSS, etc.)
+
+## Cluster Configuration
+
+You can enable Lakehouse storage through:
+1. **Static configuration**: Configure in `server.yaml` before starting the
cluster
+2. **Dynamic configuration**: Enable at runtime using the
`set_cluster_configs` procedure
+
+### Method 1: Static Configuration
+
+Configure lakehouse settings in `server.yaml` on all Fluss servers
(CoordinatorServer and TabletServer).
+
+<Tabs groupId="datalake-format">
+<TabItem value="paimon" label="Paimon" default>
+
+```yaml title="server.yaml"
+datalake.format: paimon
+datalake.paimon.metastore: filesystem
+datalake.paimon.warehouse: /path/to/paimon/warehouse
+```
+
+For Hive catalog:
+```yaml title="server.yaml"
+datalake.format: paimon
+datalake.paimon.metastore: hive
+datalake.paimon.uri: thrift://<hive-metastore-host>:<port>
+datalake.paimon.warehouse: hdfs:///path/to/warehouse
+```
+
+</TabItem>
+<TabItem value="iceberg" label="Iceberg">
+
+```yaml title="server.yaml"
+datalake.format: iceberg
+datalake.iceberg.catalog-impl: org.apache.iceberg.jdbc.JdbcCatalog
+datalake.iceberg.name: fluss_catalog
+datalake.iceberg.uri: jdbc:postgresql://postgres-host:5432/iceberg
+datalake.iceberg.jdbc.user: iceberg
+datalake.iceberg.jdbc.password: iceberg
+datalake.iceberg.warehouse: s3://bucket/iceberg
+datalake.iceberg.io-impl: org.apache.iceberg.aws.s3.S3FileIO
+```
+
+</TabItem>
+<TabItem value="lance" label="Lance">
+
+```yaml title="server.yaml"
+datalake.format: lance
+datalake.lance.warehouse: s3://bucket/lance
+```
+
+</TabItem>
+</Tabs>
+
+### Method 2: Dynamic Configuration
+
+Enable lakehouse settings at runtime using Flink SQL:
+
+```sql title="Flink SQL"
+USE fluss_catalog;
+
+CALL sys.set_cluster_configs(
+ config_pairs => 'datalake.format', 'paimon',
+ 'datalake.paimon.metastore', 'filesystem',
+ 'datalake.paimon.warehouse', '/path/to/warehouse'
+);
+```
+
+See [set_cluster_configs](../engine-flink/procedures.md#set_cluster_configs)
for more details.
+
+## Adding Required JARs
+
+### Fluss Server JARs
+
+Add JARs to `${FLUSS_HOME}/plugins/<format>/` based on your configuration:
Review Comment:
This dependency table weakens some download/configuration guidance that
existed in the old docs.
Suggested changes:
- For `Paimon with S3/OSS`, link to the Paimon download/filesystem JAR docs
and mention that users should choose `paimon-s3` / `paimon-oss` matching their
Paimon version.
- For `Paimon Hive catalog`, keep the old link to Flink Hive connector
bundled Hive jar docs.
- For `Iceberg with S3`, link to Iceberg AWS integration or Maven artifacts,
and mention `iceberg-aws` / `iceberg-aws-bundle` should match the Iceberg
version.
- For `Hadoop client JARs`, avoid only using the generic name. The old
Iceberg docs gave three options: use `HADOOP_CLASSPATH`, download
`hadoop-apache-3.3.5-2.jar`, or download a full Hadoop package. Please keep
these entry points or link back to the Iceberg Hadoop Dependencies section.
- `For HDFS, see the HDFS setup guide.` feels misplaced here. HDFS is a
storage backend dependency, not a lake format dependency. I’d remove it from
this table section.
##########
website/docs/streaming-lakehouse/tiering-service.md:
##########
@@ -0,0 +1,79 @@
+---
+title: "Tiering Service"
+sidebar_position: 2
+---
+
+# Tiering Service
+
+The Tiering Service continuously compacts real-time data from Fluss (stored in
Arrow format) into data lake formats (Parquet) for cost-efficient long-term
storage and analytics.
Review Comment:
`data lake formats (Parquet)` is not quite accurate.
Paimon/Iceberg/Lance are lake/table formats. Parquet is only an underlying
file format for some cases; Paimon can also use ORC, and Lance should not be
described as simply Parquet.
Could we make this more general, for example:
"The Tiering Service continuously compacts real-time data from Fluss into
the configured lake format for cost-efficient long-term storage and analytics."
##########
website/docs/streaming-lakehouse/tiering-service.md:
##########
@@ -0,0 +1,79 @@
+---
+title: "Tiering Service"
+sidebar_position: 2
+---
+
+# Tiering Service
+
+The Tiering Service continuously compacts real-time data from Fluss (stored in
Arrow format) into data lake formats (Parquet) for cost-efficient long-term
storage and analytics.
+
+## Overview
+
+The Tiering Service is implemented as an Apache Flink job that:
+- Reads records from Fluss tables with lakehouse storage enabled
+- Writes data to the configured data lake format (Paimon, Iceberg, or Lance)
+- Maintains exactly-once semantics between Fluss and the data lake
+- Operates incrementally, syncing only missing data segments
+
+For deployment instructions, see [Deploying Streaming
Lakehouse](../install-deploy/deploying-streaming-lakehouse.md).
+
+## Architecture
+
+The Tiering Service consists of three Flink operators:
+
+| Operator | Description |
+|----------|-------------|
+| **TieringSource** | Reads records from Fluss and writes to the data lake via
LakeWriter |
+| **TieringCommitter** | Commits batches and advances offsets in both Fluss
and lake |
+| **No-Op Sink** | Dummy sink required by Flink's topology |
+
+### How It Works
+
+1. **Enumerator** sends heartbeat to Fluss CoordinatorService, receives table
metadata
+2. **SplitGenerator** calculates data delta between Fluss offsets and lake
snapshot offsets
+3. **Splits** are created for missing data ranges and assigned to readers
+4. **Readers** fetch records from Fluss tablet servers, write via LakeWriter
+5. **Committer** performs two-phase commit: first to lake, then updates Fluss
coordinator
+6. Cycle repeats based on `table.datalake.freshness` interval
+
+### Split Types
+
+| Table Type | Split Type | Description |
+|------------|------------|-------------|
+| Log Table (append-only) | TieringLogSplit | Defines starting and stopping
offsets for a contiguous range |
+| Primary Key Table | TieringSnapshotSplit | References snapshot ID and log
offset for CDC replay |
+
+## Configuration Options
+
+### Tiering Service Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `lake.tiering.auto-expire-snapshot` | Boolean | false | Auto-trigger
snapshot expiration on commit |
+
+### Table-Level Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `table.datalake.enabled` | Boolean | false | Enable lakehouse storage for
this table |
+| `table.datalake.freshness` | Duration | 3min | Maximum lag between Fluss and
lake data |
+| `table.datalake.auto-compaction` | Boolean | false | Auto-trigger compaction
in the data lake |
+| `table.datalake.auto-expire-snapshot` | Boolean | false | Auto-expire
snapshots in the data lake |
+
+## Scaling
+
+The Tiering Service is stateless and can be scaled by adjusting Flink
parallelism:
+
+```shell
+${FLINK_HOME}/bin/flink run \
+ -Dparallelism.default=6 \
+ /path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
+ --fluss.bootstrap.servers localhost:9123 \
+ ...
+```
+
+Multiple tiering service jobs can run simultaneously. They are coordinated by
the Fluss cluster to ensure exactly-once semantics and automatic load balancing.
+
+## Monitoring
+
+Key metrics for monitoring the Tiering Service are available through Flink's
metrics system. See [Monitoring
Metrics](../maintenance/observability/monitor-metrics.md) for details on
lakehouse tiering metrics.
Review Comment:
The “Monitoring Metrics” link should point directly to the tiering service
metrics section.
From this page, the link should be:
`../maintenance/observability/monitor-metrics.md#tiering-service-metrics`
This will route to
`/docs/maintenance/observability/monitor-metrics/#tiering-service-metrics`.
##########
website/docs/streaming-lakehouse/tiering-service.md:
##########
@@ -0,0 +1,79 @@
+---
+title: "Tiering Service"
+sidebar_position: 2
+---
+
+# Tiering Service
+
+The Tiering Service continuously compacts real-time data from Fluss (stored in
Arrow format) into data lake formats (Parquet) for cost-efficient long-term
storage and analytics.
+
+## Overview
+
+The Tiering Service is implemented as an Apache Flink job that:
+- Reads records from Fluss tables with lakehouse storage enabled
+- Writes data to the configured data lake format (Paimon, Iceberg, or Lance)
+- Maintains exactly-once semantics between Fluss and the data lake
+- Operates incrementally, syncing only missing data segments
+
+For deployment instructions, see [Deploying Streaming
Lakehouse](../install-deploy/deploying-streaming-lakehouse.md).
+
+## Architecture
+
+The Tiering Service consists of three Flink operators:
+
+| Operator | Description |
+|----------|-------------|
+| **TieringSource** | Reads records from Fluss and writes to the data lake via
LakeWriter |
+| **TieringCommitter** | Commits batches and advances offsets in both Fluss
and lake |
+| **No-Op Sink** | Dummy sink required by Flink's topology |
+
+### How It Works
+
+1. **Enumerator** sends heartbeat to Fluss CoordinatorService, receives table
metadata
+2. **SplitGenerator** calculates data delta between Fluss offsets and lake
snapshot offsets
+3. **Splits** are created for missing data ranges and assigned to readers
+4. **Readers** fetch records from Fluss tablet servers, write via LakeWriter
+5. **Committer** performs two-phase commit: first to lake, then updates Fluss
coordinator
+6. Cycle repeats based on `table.datalake.freshness` interval
+
+### Split Types
+
+| Table Type | Split Type | Description |
+|------------|------------|-------------|
+| Log Table (append-only) | TieringLogSplit | Defines starting and stopping
offsets for a contiguous range |
+| Primary Key Table | TieringSnapshotSplit | References snapshot ID and log
offset for CDC replay |
+
+## Configuration Options
+
+### Tiering Service Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `lake.tiering.auto-expire-snapshot` | Boolean | false | Auto-trigger
snapshot expiration on commit |
Review Comment:
Could we add a short explanation for how Tiering Service Options are
configured?
Right now the table lists `lake.tiering.auto-expire-snapshot`, but users
cannot tell where to set it.
The tiering job arguments are split into three groups:
- `--fluss.*`: Fluss client config, for example `--fluss.bootstrap.servers`
- `--datalake.<format>.*`: lake catalog/storage config; Fluss strips the
`datalake.<format>.` prefix before passing it to the lake connector
- `--lake.tiering.*`: tiering service job-level config, for example
`--lake.tiering.auto-expire-snapshot true`
It would help to add an example:
```shell
${FLINK_HOME}/bin/flink run \
-Dparallelism.default=3 \
/path/to/fluss-flink-tiering-$FLUSS_VERSION$.jar \
--fluss.bootstrap.servers localhost:9123 \
--datalake.format paimon \
--datalake.paimon.metastore filesystem \
--datalake.paimon.warehouse /tmp/paimon \
--lake.tiering.auto-expire-snapshot true
```
Also worth clarifying:
- `table.datalake.*` is configured per table when creating/altering tables.
- `lake.tiering.*` is configured when starting the tiering service job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
