This is an automated email from the ASF dual-hosted git repository.
MartijnVisser pushed a commit to branch release-2.3
in repository https://gitbox.apache.org/repos/asf/flink.git
The following commit(s) were added to refs/heads/release-2.3 by this push:
new a71b14645ec [FLINK-41202][docs] Make connector integrations easier to
find from the Flink docs (#29609)
a71b14645ec is described below
commit a71b14645ecbb8c58f23b825ac9d39f6e83cdd65
Author: Martijn Visser <[email protected]>
AuthorDate: Wed Oct 28 15:17:50 2026 +0100
[FLINK-41202][docs] Make connector integrations easier to find from the
Flink docs (#29609)
- Sort the connector menus by title, because the weights set in the
connector repositories collide
- Sort and complete the connector lists in both overviews, list Kafka as
a bounded source too, and drop the stale Version column from the SQL
overview
- Move the fault tolerance guarantees into both overviews and keep the
old page as a hidden pointer, because the Cassandra connector docs
still link to it
- Point to Flink CDC from both overviews and from the Debezium, Canal,
Maxwell and Ogg format pages
- Link the Flink CDC, Kubernetes Operator and Flink Agents docs from the
menu
- Add a "Maintained by Other Projects" section with a page each for
Doris, Fluss, Hudi, Iceberg and Paimon, replacing the Apache Bahir
section in the DataStream overview
- Drop the Incubating suffix from Fluss
- Correct which catalogs support materialized tables
On this branch the SQL connector overview leaves out Pulsar, because
flink-connector-pulsar v4.0 has no Table API page, and the DataStream
connector overview keeps JDBC as a sink, because flink-connector-jdbc
v3.1 only documents JdbcSink.
Generated-by: Claude Code (Claude Opus 5.5)
(cherry picked from commit 3e87ae6b12e8fd4c05a3fcc88e339dcabc070981)
---
docs/config.toml | 3 +
.../docs/concepts/stateful-stream-processing.md | 7 +-
.../docs/connectors/datastream/_index.md | 1 +
.../docs/connectors/datastream/filesystem.md | 1 +
.../docs/connectors/datastream/guarantees.md | 128 +-------------------
.../docs/connectors/datastream/overview.md | 51 +++++---
.../connectors/{table => other-projects}/_index.md | 7 +-
.../{table/_index.md => other-projects/doris.md} | 24 +++-
.../docs/connectors/other-projects/fluss.md | 40 +++++++
.../{table/_index.md => other-projects/hudi.md} | 23 +++-
.../docs/connectors/other-projects/iceberg.md | 44 +++++++
.../docs/connectors/other-projects/overview.md | 48 ++++++++
.../{table/_index.md => other-projects/paimon.md} | 24 +++-
docs/content.zh/docs/connectors/table/_index.md | 1 +
.../content.zh/docs/connectors/table/filesystem.md | 1 +
.../docs/connectors/table/formats/canal.md | 3 +
.../docs/connectors/table/formats/debezium.md | 3 +
.../docs/connectors/table/formats/maxwell.md | 3 +
.../docs/connectors/table/formats/ogg.md | 3 +
docs/content.zh/docs/connectors/table/overview.md | 115 +++++++++++++-----
docs/content.zh/docs/dev/table/tuning.md | 4 +-
.../content.zh/docs/learn-flink/fault_tolerance.md | 2 +-
.../docs/sql/materialized-table/deployment.md | 6 +-
.../docs/concepts/stateful-stream-processing.md | 7 +-
docs/content/docs/connectors/datastream/_index.md | 1 +
.../docs/connectors/datastream/guarantees.md | 133 +--------------------
.../content/docs/connectors/datastream/overview.md | 49 +++++---
.../docs/connectors/other-projects}/_index.md | 7 +-
.../docs/connectors/other-projects/doris.md} | 24 +++-
.../docs/connectors/other-projects/fluss.md | 40 +++++++
.../docs/connectors/other-projects/hudi.md} | 23 +++-
.../docs/connectors/other-projects/iceberg.md | 44 +++++++
.../docs/connectors/other-projects/overview.md | 48 ++++++++
.../docs/connectors/other-projects/paimon.md} | 24 +++-
docs/content/docs/connectors/table/_index.md | 1 +
.../content/docs/connectors/table/formats/canal.md | 3 +
.../docs/connectors/table/formats/debezium.md | 3 +
.../docs/connectors/table/formats/maxwell.md | 3 +
docs/content/docs/connectors/table/formats/ogg.md | 3 +
docs/content/docs/connectors/table/overview.md | 115 +++++++++++++-----
docs/content/docs/dev/table/tuning.md | 4 +-
docs/content/docs/learn-flink/fault_tolerance.md | 2 +-
.../docs/sql/materialized-table/deployment.md | 6 +-
docs/layouts/partials/docs/menu-filetree.html | 12 +-
44 files changed, 707 insertions(+), 387 deletions(-)
diff --git a/docs/config.toml b/docs/config.toml
index 37e5aa769c5..21fe8c61073 100644
--- a/docs/config.toml
+++ b/docs/config.toml
@@ -68,6 +68,9 @@ pygmentsUseClasses = true
# of the menu
MenuLinks = [
["Project Homepage", "//flink.apache.org"],
+ ["Flink CDC", "//nightlies.apache.org/flink/flink-cdc-docs-stable/"],
+ ["Flink Kubernetes Operator",
"//nightlies.apache.org/flink/flink-kubernetes-operator-docs-stable/"],
+ ["Flink Agents", "//nightlies.apache.org/flink/flink-agents-docs-latest/"],
["JavaDocs",
"//nightlies.apache.org/flink/flink-docs-release-2.3/api/java/"],
["PyDocs",
"//nightlies.apache.org/flink/flink-docs-release-2.3/api/python/"]
]
diff --git a/docs/content.zh/docs/concepts/stateful-stream-processing.md
b/docs/content.zh/docs/concepts/stateful-stream-processing.md
index 7be685dba3f..b2c536a5a6f 100644
--- a/docs/content.zh/docs/concepts/stateful-stream-processing.md
+++ b/docs/content.zh/docs/concepts/stateful-stream-processing.md
@@ -109,9 +109,10 @@ By default, checkpointing is disabled. See
[Checkpointing]({{< ref "docs/dev/dat
For this mechanism to realize its full guarantees, the data
stream source (such as message queue or broker) needs to be able to rewind the
stream to a defined recent point. [Apache Kafka](http://kafka.apache.org) has
-this ability and Flink's connector to Kafka exploits this. See [Fault
-Tolerance Guarantees of Data Sources and Sinks]({{< ref
"docs/connectors/datastream/guarantees" >}}) for more information about the
guarantees
-provided by Flink's connectors.
+this ability and Flink's connector to Kafka exploits this. See the fault
tolerance guarantees for
+[SQL & Table API connectors]({{< ref "docs/connectors/table/overview"
>}}#fault-tolerance-guarantees) and
+[DataStream connectors]({{< ref "docs/connectors/datastream/overview"
>}}#fault-tolerance-guarantees)
+for more information about the guarantees provided by Flink's connectors.
{{< /hint >}}
{{< hint info >}}
diff --git a/docs/content.zh/docs/connectors/datastream/_index.md
b/docs/content.zh/docs/connectors/datastream/_index.md
index fd6e50a01c2..b362045eb47 100644
--- a/docs/content.zh/docs/connectors/datastream/_index.md
+++ b/docs/content.zh/docs/connectors/datastream/_index.md
@@ -1,6 +1,7 @@
---
title: DataStream Connectors
bookCollapseSection: true
+bookSortByTitle: true
weight: 4
---
<!--
diff --git a/docs/content.zh/docs/connectors/datastream/filesystem.md
b/docs/content.zh/docs/connectors/datastream/filesystem.md
index 1a40fbbceaa..ca5b09ea84d 100644
--- a/docs/content.zh/docs/connectors/datastream/filesystem.md
+++ b/docs/content.zh/docs/connectors/datastream/filesystem.md
@@ -1,5 +1,6 @@
---
title: 文件系统
+linkTitle: FileSystem
weight: 6
type: docs
aliases:
diff --git a/docs/content.zh/docs/connectors/datastream/guarantees.md
b/docs/content.zh/docs/connectors/datastream/guarantees.md
index ea3fc7db6f8..71093dc5ac8 100644
--- a/docs/content.zh/docs/connectors/datastream/guarantees.md
+++ b/docs/content.zh/docs/connectors/datastream/guarantees.md
@@ -2,6 +2,7 @@
title: 容错保证
weight: 2
type: docs
+bookHidden: true
aliases:
- /zh/dev/connectors/guarantees.html
---
@@ -26,126 +27,7 @@ under the License.
# Data Source 和 Sink 的容错保证
-当程序出现错误的时候，Flink 的容错机制能恢复并继续运行程序。这种错误包括机器硬件故障、网络故障、瞬态程序故障等等。
-
-只有当 source 参与了快照机制的时候，Flink 才能保证对自定义状态的精确一次更新。下表列举了 Flink 与其自带连接器的状态更新的保证。
-
-请阅读各个连接器的文档来了解容错保证的细节。
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 25%">Source</th>
- <th class="text-left" style="width: 25%">Guarantees</th>
- <th class="text-left">Notes</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>Apache Kafka</td>
- <td>精确一次</td>
- <td>根据你的版本用恰当的 Kafka 连接器</td>
- </tr>
- <tr>
- <td>Amazon Kinesis Data Streams</td>
- <td>精确一次</td>
- <td></td>
- </tr>
- <tr>
- <td>RabbitMQ</td>
- <td>至多一次 (v 0.10) / 精确一次 (v 1.0) </td>
- <td></td>
- </tr>
- <tr>
- <td>Google PubSub</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Collections</td>
- <td>精确一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Files</td>
- <td>精确一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Sockets</td>
- <td>至多一次</td>
- <td></td>
- </tr>
- </tbody>
-</table>
-
-为了保证端到端精确一次的数据交付（在精确一次的状态语义上更进一步），sink需要参与 checkpointing 机制。下表列举了 Flink 与其自带
sink 的交付保证（假设精确一次状态更新）。
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 25%">Sink</th>
- <th class="text-left" style="width: 25%">Guarantees</th>
- <th class="text-left">Notes</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>Elasticsearch</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Opensearch</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Kafka producer</td>
- <td>至少一次 / 精确一次</td>
- <td>当使用事务生产者时，保证精确一次 (v 0.11+)</td>
- </tr>
- <tr>
- <td>Cassandra sink</td>
- <td>至少一次 / 精确一次</td>
- <td>只有当更新是幂等时，保证精确一次</td>
- </tr>
- <tr>
- <td>Amazon DynamoDB</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Amazon Kinesis Data Streams</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Amazon Kinesis Data Firehose</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>File sinks</td>
- <td>精确一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Socket sinks</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Standard output</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- <tr>
- <td>Redis sink</td>
- <td>至少一次</td>
- <td></td>
- </tr>
- </tbody>
-</table>
-
-{{< top >}}
+This page has moved into the connector overviews: see
+[SQL & Table API connectors]({{< ref "docs/connectors/table/overview"
>}}#fault-tolerance-guarantees)
+and [DataStream connectors]({{< ref "docs/connectors/datastream/overview"
>}}#fault-tolerance-guarantees).
+Each connector's page documents the guarantees it provides.
diff --git a/docs/content.zh/docs/connectors/datastream/overview.md
b/docs/content.zh/docs/connectors/datastream/overview.md
index 044fe73dee4..bfcce17dd34 100644
--- a/docs/content.zh/docs/connectors/datastream/overview.md
+++ b/docs/content.zh/docs/connectors/datastream/overview.md
@@ -38,22 +38,25 @@ under the License.
连接器可以和多种多样的第三方系统进行交互。目前支持以下系统。
Currently these systems are supported as part of the Apache Flink project:
- * [Apache Kafka]({{< ref "docs/connectors/datastream/kafka" >}}) (source/sink)
- * [Apache Cassandra]({{< ref "docs/connectors/datastream/cassandra" >}})
(source/sink)
- * [Amazon DynamoDB]({{< ref "docs/connectors/datastream/dynamodb" >}}) (sink)
- * [Amazon Kinesis Data Streams]({{< ref "docs/connectors/datastream/kinesis"
>}}) (source/sink)
- * [Amazon Kinesis Data Firehose]({{< ref
"docs/connectors/datastream/firehose" >}}) (sink)
+ * [Cassandra]({{< ref "docs/connectors/datastream/cassandra" >}})
(source/sink)
* [DataGen]({{< ref "docs/connectors/datastream/datagen" >}}) (source)
+ * [Dynamic Kafka]({{< ref "docs/connectors/datastream/dynamic-kafka" >}})
(source)
+ * [DynamoDB]({{< ref "docs/connectors/datastream/dynamodb" >}}) (sink)
* [Elasticsearch]({{< ref "docs/connectors/datastream/elasticsearch" >}})
(sink)
- * [Opensearch]({{< ref "docs/connectors/datastream/opensearch" >}}) (sink)
- * [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}}) (sink)
- * [RabbitMQ]({{< ref "docs/connectors/datastream/rabbitmq" >}}) (source/sink)
- * [Google PubSub]({{< ref "docs/connectors/datastream/pubsub" >}})
(source/sink)
+ * [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}})
(source/sink)
+ * [Firehose]({{< ref "docs/connectors/datastream/firehose" >}}) (sink)
+ * [Google Cloud PubSub]({{< ref "docs/connectors/datastream/pubsub" >}})
(source/sink)
+ * [HTTP]({{< ref "docs/connectors/datastream/http" >}}) (sink)
* [Hybrid Source]({{< ref "docs/connectors/datastream/hybridsource" >}})
(source)
- * [Apache Pulsar]({{< ref "docs/connectors/datastream/pulsar" >}}) (source)
* [JDBC]({{< ref "docs/connectors/datastream/jdbc" >}}) (sink)
+ * [Kafka]({{< ref "docs/connectors/datastream/kafka" >}}) (source/sink)
+ * [Kinesis]({{< ref "docs/connectors/datastream/kinesis" >}}) (source/sink)
* [MongoDB]({{< ref "docs/connectors/datastream/mongodb" >}}) (source/sink)
+ * [Opensearch]({{< ref "docs/connectors/datastream/opensearch" >}}) (sink)
* [Prometheus]({{< ref "docs/connectors/datastream/prometheus" >}}) (sink)
+ * [Pulsar]({{< ref "docs/connectors/datastream/pulsar" >}}) (source/sink)
+ * [RabbitMQ]({{< ref "docs/connectors/datastream/rabbitmq" >}}) (source/sink)
+ * [SQS]({{< ref "docs/connectors/datastream/sqs" >}}) (sink)
请记住，在使用一种连接器时，通常需要额外的第三方组件，比如：数据存储服务器或者消息队列。
要注意这些列举的连接器是 Flink 工程的一部分，包含在发布的源码中，但是不包含在二进制发行版中。
@@ -66,15 +69,29 @@ Currently these systems are supported as part of the Apache
Flink project:
如果需要在本地环境测试运行，请确保 flink-connector-base 依赖被正确的提供，而且能在 classpath 下找到。
{{< /hint >}}
-## Apache Bahir 中的连接器
+## Flink CDC
+
+[Flink CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/) is an
Apache Flink sub-project with its own releases. Its source connectors read
+changes directly from databases such as MySQL, PostgreSQL, Oracle, SQL Server,
MongoDB and Db2. See
+the [Flink CDC
sources](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for the supported databases and Flink versions.
+
+## Fault Tolerance Guarantees
+
+Flink recovers from failures by restarting from the last [checkpoint]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}}). Checkpointing is
+disabled by default. For exactly-once results, the source must be able to
rewind to the position
+stored in the checkpoint, and the sink must take part in checkpointing, either
by committing its
+writes transactionally or by writing idempotently. See [Stateful Stream
Processing]({{< ref "docs/concepts/stateful-stream-processing"
>}}#checkpointing) for how checkpoints work.
+
+Sinks built on the [Sink API]({{< ref "docs/internals/sinks" >}}) that support
more than one
+guarantee take a {{< javadoc
file="org/apache/flink/connector/base/DeliveryGuarantee.html"
name="DeliveryGuarantee" >}}
+(`NONE`, `AT_LEAST_ONCE` or `EXACTLY_ONCE`) in their builder, for example the
+[Kafka sink]({{< ref "docs/connectors/datastream/kafka" >}}). Each connector's
page documents the
+guarantees it provides.
-Flink 还有些一些额外的连接器通过 [Apache Bahir](https://bahir.apache.org/) 发布, 包括:
+## Integrations Maintained by Other Projects
- * [Apache
ActiveMQ](https://bahir.apache.org/docs/flink/current/flink-streaming-activemq/)
(source/sink)
- * [Apache
Flume](https://bahir.apache.org/docs/flink/current/flink-streaming-flume/)
(sink)
- * [Redis](https://bahir.apache.org/docs/flink/current/flink-streaming-redis/)
(sink)
- * [Akka](https://bahir.apache.org/docs/flink/current/flink-streaming-akka/)
(sink)
- * [Netty](https://bahir.apache.org/docs/flink/current/flink-streaming-netty/)
(source)
+Some projects maintain their own Flink integration, such as Apache Iceberg and
Apache Paimon. See
+[Integrations Maintained by Other Projects]({{< ref
"docs/connectors/other-projects/overview" >}}).
## 连接Flink的其他方法
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content.zh/docs/connectors/other-projects/_index.md
similarity index 91%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content.zh/docs/connectors/other-projects/_index.md
index a117e9fa887..acd631dd9f2 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content.zh/docs/connectors/other-projects/_index.md
@@ -1,7 +1,8 @@
---
-title: SQL & Table API Connectors
+title: Maintained by Other Projects
bookCollapseSection: true
-weight: 3
+bookSortByTitle: true
+weight: 6
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +21,4 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content.zh/docs/connectors/other-projects/doris.md
similarity index 51%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content.zh/docs/connectors/other-projects/doris.md
index a117e9fa887..38e5a4aec70 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content.zh/docs/connectors/other-projects/doris.md
@@ -1,7 +1,7 @@
---
-title: SQL & Table API Connectors
-bookCollapseSection: true
-weight: 3
+title: Apache Doris
+weight: 10
+type: docs
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,20 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
+
+# Apache Doris
+
+{{< hint info >}}
+The Apache Doris community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Doris](https://doris.apache.org/) is a real-time analytical database.
Its Flink connector
+reads from and writes to Doris from SQL and the DataStream API, supports
lookup joins, and can
+synchronize whole databases into Doris using Flink CDC.
+
+- [Flink
integration](https://doris.apache.org/docs/ecosystem/flink-doris-connector)
+- [Supported Flink
versions](https://doris.apache.org/docs/ecosystem/flink-doris-connector#version-description)
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/other-projects/fluss.md
b/docs/content.zh/docs/connectors/other-projects/fluss.md
new file mode 100644
index 00000000000..946f2190f09
--- /dev/null
+++ b/docs/content.zh/docs/connectors/other-projects/fluss.md
@@ -0,0 +1,40 @@
+---
+title: Apache Fluss
+weight: 10
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Fluss
+
+{{< hint info >}}
+The Apache Fluss community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Fluss](https://fluss.apache.org/) is streaming storage for real-time
analytics. Its Flink
+integration provides a catalog, reads and writes Fluss tables in streaming and
batch mode, and
+supports lookup joins. Fluss tables can be used as sources for
+[delta joins]({{< ref "docs/dev/table/tuning" >}}#delta-joins).
+
+- [Flink
integration](https://fluss.apache.org/docs/engine-flink/getting-started/)
+- [Supported Flink
versions](https://fluss.apache.org/docs/engine-flink/getting-started/#supported-flink-versions)
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content.zh/docs/connectors/other-projects/hudi.md
similarity index 53%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content.zh/docs/connectors/other-projects/hudi.md
index a117e9fa887..65cade9adf4 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content.zh/docs/connectors/other-projects/hudi.md
@@ -1,7 +1,7 @@
---
-title: SQL & Table API Connectors
-bookCollapseSection: true
-weight: 3
+title: Apache Hudi
+weight: 10
+type: docs
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,19 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
+
+# Apache Hudi
+
+{{< hint info >}}
+The Apache Hudi community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Hudi](https://hudi.apache.org/) is an open data lakehouse platform.
Its Flink integration provides a catalog and reads and writes Hudi tables in
streaming and
+batch mode from SQL and the DataStream API.
+
+- [Flink integration](https://hudi.apache.org/docs/flink-quick-start-guide/)
+- [Supported Flink
versions](https://hudi.apache.org/docs/flink-quick-start-guide/#flink-support-matrix)
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/other-projects/iceberg.md
b/docs/content.zh/docs/connectors/other-projects/iceberg.md
new file mode 100644
index 00000000000..7689f821b15
--- /dev/null
+++ b/docs/content.zh/docs/connectors/other-projects/iceberg.md
@@ -0,0 +1,44 @@
+---
+title: Apache Iceberg
+weight: 10
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Iceberg
+
+{{< hint info >}}
+The Apache Iceberg community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Iceberg](https://iceberg.apache.org/) is an open table format for
large analytic tables.
+Its Flink integration provides a catalog and reads and writes Iceberg tables
in streaming and batch
+mode from SQL, the Table API and the DataStream API.
+
+Iceberg tables are often managed by a catalog service that implements the
Iceberg REST catalog API,
+such as [Apache Polaris](https://polaris.apache.org/guides/flink/). Flink
connects to such a service
+through Iceberg's REST catalog.
+
+- [Flink integration](https://iceberg.apache.org/docs/latest/flink/)
+- [Supported Flink
versions](https://iceberg.apache.org/multi-engine-support/#apache-flink)
+- [REST catalog](https://iceberg.apache.org/docs/latest/flink/#rest-catalog)
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/other-projects/overview.md
b/docs/content.zh/docs/connectors/other-projects/overview.md
new file mode 100644
index 00000000000..ad604bbdd9d
--- /dev/null
+++ b/docs/content.zh/docs/connectors/other-projects/overview.md
@@ -0,0 +1,48 @@
+---
+title: Overview
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Integrations Maintained by Other Projects
+
+The projects listed here maintain their own Flink integration. Each
integration is developed,
+released and supported by that project's community, not by the Apache Flink
community. That
+project's documentation covers which artifacts to use and how to configure it.
Report issues and ask
+questions in that project's channels.
+
+A project is listed here when it is an Apache Software Foundation project and
has released a Flink
+integration for Flink 2.x.
+
+{{< hint info >}}
+These integrations follow their own release schedules and may not support this
Flink version yet.
+Check each project's supported Flink versions before you choose one.
+{{< /hint >}}
+
+| Project | Flink integration |
+|---------|-------------------|
+| [Apache Doris]({{< ref "docs/connectors/other-projects/doris" >}}) | Source,
sink, lookup joins and database synchronization through Flink CDC |
+| [Apache Fluss]({{< ref "docs/connectors/other-projects/fluss" >}}) |
Catalog, source, sink, lookup joins and delta joins |
+| [Apache Hudi]({{< ref "docs/connectors/other-projects/hudi" >}}) | Catalog,
source and sink |
+| [Apache Iceberg]({{< ref "docs/connectors/other-projects/iceberg" >}}) |
Catalog, source and sink; REST catalogs such as Apache Polaris |
+| [Apache Paimon]({{< ref "docs/connectors/other-projects/paimon" >}}) |
Catalog, source, sink and lookup joins |
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content.zh/docs/connectors/other-projects/paimon.md
similarity index 52%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content.zh/docs/connectors/other-projects/paimon.md
index a117e9fa887..cc14994b193 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content.zh/docs/connectors/other-projects/paimon.md
@@ -1,7 +1,7 @@
---
-title: SQL & Table API Connectors
-bookCollapseSection: true
-weight: 3
+title: Apache Paimon
+weight: 10
+type: docs
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,20 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
+
+# Apache Paimon
+
+{{< hint info >}}
+The Apache Paimon community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Paimon](https://paimon.apache.org/) is a lake format for streaming and
batch processing.
+Its Flink integration provides a catalog, reads and writes Paimon tables in
streaming and batch
+mode, and supports lookup joins.
+
+- [Flink integration](https://paimon.apache.org/docs/master/flink/quick-start/)
+- [Supported Flink
versions](https://paimon.apache.org/docs/master/flink/installation/)
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content.zh/docs/connectors/table/_index.md
index a117e9fa887..eb52eca6531 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content.zh/docs/connectors/table/_index.md
@@ -1,6 +1,7 @@
---
title: SQL & Table API Connectors
bookCollapseSection: true
+bookSortByTitle: true
weight: 3
---
<!--
diff --git a/docs/content.zh/docs/connectors/table/filesystem.md
b/docs/content.zh/docs/connectors/table/filesystem.md
index 20514387427..4794d058d48 100644
--- a/docs/content.zh/docs/connectors/table/filesystem.md
+++ b/docs/content.zh/docs/connectors/table/filesystem.md
@@ -1,5 +1,6 @@
---
title: 文件系统
+linkTitle: FileSystem
weight: 8
type: docs
aliases:
diff --git a/docs/content.zh/docs/connectors/table/formats/canal.md
b/docs/content.zh/docs/connectors/table/formats/canal.md
index 9ceb3ccc53d..9cea9f10e5e 100644
--- a/docs/content.zh/docs/connectors/table/formats/canal.md
+++ b/docs/content.zh/docs/connectors/table/formats/canal.md
@@ -32,6 +32,9 @@ under the License.
[Canal](https://github.com/alibaba/canal/wiki) 是一个 CDC（ChangeLog Data
Capture，变更日志数据捕获）工具，可以实时地将 MySQL 变更传输到其他系统。Canal 为变更日志提供了统一的数据格式，并支持使用 JSON 或
[protobuf](https://developers.google.com/protocol-buffers) 序列化消息（Canal 默认使用
protobuf）。
+This format reads change events that Canal has already written to a system
such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink 支持将 Canal 的 JSON 消息解析为 INSERT / UPDATE / DELETE 消息到 Flink SQL
系统中。在很多情况下，利用这个特性非常的有用，例如
- 将增量数据从数据库同步到其他系统
- 日志审计
diff --git a/docs/content.zh/docs/connectors/table/formats/debezium.md
b/docs/content.zh/docs/connectors/table/formats/debezium.md
index 91a9a88d9f9..1eff05fb9c5 100644
--- a/docs/content.zh/docs/connectors/table/formats/debezium.md
+++ b/docs/content.zh/docs/connectors/table/formats/debezium.md
@@ -32,6 +32,9 @@ under the License.
[Debezium](https://debezium.io/) 是一个 CDC（Changelog Data
Capture，变更数据捕获）的工具，可以把来自 MySQL、PostgreSQL、Oracle、Microsoft SQL Server
和许多其他数据库的更改实时流式传输到 Kafka 中。 Debezium 为变更日志提供了统一的格式结构，并支持使用 JSON 和 Apache Avro
序列化消息。
+This format reads change events that Debezium has already written to a system
such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink 支持将 Debezium JSON 和 Avro 消息解析为 INSERT / UPDATE / DELETE 消息到 Flink SQL
系统中。在很多情况下，利用这个特性非常的有用，例如
- 将增量数据从数据库同步到其他系统
- 日志审计
diff --git a/docs/content.zh/docs/connectors/table/formats/maxwell.md
b/docs/content.zh/docs/connectors/table/formats/maxwell.md
index 0bdedeac682..7bc3e266ff0 100644
--- a/docs/content.zh/docs/connectors/table/formats/maxwell.md
+++ b/docs/content.zh/docs/connectors/table/formats/maxwell.md
@@ -32,6 +32,9 @@ under the License.
[Maxwell](https://maxwells-daemon.io/) is a CDC (Changelog Data Capture) tool
that can stream changes in real-time from MySQL into Kafka, Kinesis and other
streaming connectors. Maxwell provides a unified format schema for changelog
and supports to serialize messages using JSON.
+This format reads change events that Maxwell has already written to a system
such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink supports to interpret Maxwell JSON messages as INSERT/UPDATE/DELETE
messages into Flink SQL system. This is useful in many cases to leverage this
feature, such as
- synchronizing incremental data from databases to other systems
- auditing logs
diff --git a/docs/content.zh/docs/connectors/table/formats/ogg.md
b/docs/content.zh/docs/connectors/table/formats/ogg.md
index 61ec97b60fd..d11175a7b3d 100644
--- a/docs/content.zh/docs/connectors/table/formats/ogg.md
+++ b/docs/content.zh/docs/connectors/table/formats/ogg.md
@@ -32,6 +32,9 @@ label "Format: Deserialization Schema" >}}
[Oracle GoldenGate](https://www.oracle.com/integration/goldengate/) (a.k.a
ogg) 是一个实现异构 IT 环境间数据实时数据集成和复制的综合软件包。
该产品集支持高可用性解决方案、实时数据集成、事务更改数据捕获、运营和分析企业系统之间的数据复制、转换和验证。Ogg
为变更日志提供了统一的格式结构，并支持使用 JSON 序列化消息。
+This format reads change events that Oracle GoldenGate has already written to
a system such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink 支持将 Ogg JSON 消息解析为 INSERT/UPDATE/DELETE 消息到 Flink SQL
系统中。在很多情况下，利用这个特性非常有用，例如
- 将增量数据从数据库同步到其他系统
diff --git a/docs/content.zh/docs/connectors/table/overview.md
b/docs/content.zh/docs/connectors/table/overview.md
index c7c447a4707..8c24511f8b6 100644
--- a/docs/content.zh/docs/connectors/table/overview.md
+++ b/docs/content.zh/docs/connectors/table/overview.md
@@ -42,85 +42,142 @@ Flink natively support various connectors. The following
tables list all availab
<thead>
<tr>
<th class="text-left">Name</th>
- <th class="text-center">Version</th>
<th class="text-center">Source</th>
<th class="text-center">Sink</th>
</tr>
</thead>
<tbody>
<tr>
- <td><a href="{{< ref "docs/connectors/table/filesystem"
>}}">Filesystem</a></td>
- <td></td>
- <td>Bounded and Unbounded Scan</td>
+ <td><a href="{{< ref "docs/connectors/table/blackhole"
>}}">BlackHole</a></td>
+ <td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/elasticsearch"
>}}">Elasticsearch</a></td>
- <td>6.x & 7.x</td>
+ <td><a href="{{< ref "docs/connectors/table/datagen"
>}}">DataGen</a></td>
+ <td>Bounded and Unbounded Scan</td>
+ <td>Not supported</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/dynamic-kafka" >}}">Dynamic
Kafka</a></td>
+ <td>Unbounded Scan</td>
+ <td>Not supported</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/dynamodb"
>}}">DynamoDB</a></td>
<td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/opensearch"
>}}">Opensearch</a></td>
- <td>1.x & 2.x</td>
+ <td><a href="{{< ref "docs/connectors/table/elasticsearch"
>}}">Elasticsearch</a></td>
<td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/kafka" >}}">Apache
Kafka</a></td>
- <td>0.10+</td>
- <td>Unbounded Scan</td>
+ <td><a href="{{< ref "docs/connectors/table/filesystem"
>}}">FileSystem</a></td>
+ <td>Bounded and Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/dynamodb" >}}">Amazon
DynamoDB</a></td>
- <td></td>
+ <td><a href="{{< ref "docs/connectors/table/firehose"
>}}">Firehose</a></td>
<td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/kinesis" >}}">Amazon Kinesis
Data Streams</a></td>
- <td></td>
- <td>Unbounded Scan</td>
- <td>Streaming Sink</td>
+ <td><a href="{{< ref "docs/connectors/table/hbase" >}}">HBase</a></td>
+ <td>Bounded Scan, Lookup</td>
+ <td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/firehose" >}}">Amazon
Kinesis Data Firehose</a></td>
- <td></td>
- <td>Not supported</td>
- <td>Streaming Sink</td>
+ <td><a href="{{< ref "docs/connectors/table/hive/overview"
>}}">Hive</a></td>
+ <td>Unbounded Scan, Bounded Scan, Lookup</td>
+ <td>Streaming Sink, Batch Sink</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/http" >}}">HTTP</a></td>
+ <td>Lookup</td>
+ <td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
<td><a href="{{< ref "docs/connectors/table/jdbc" >}}">JDBC</a></td>
- <td></td>
<td>Bounded Scan, Lookup</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/hbase" >}}">Apache
HBase</a></td>
- <td>1.4.x & 2.2.x</td>
- <td>Bounded Scan, Lookup</td>
+ <td><a href="{{< ref "docs/connectors/table/kafka" >}}">Kafka</a></td>
+ <td>Bounded and Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/hive/overview" >}}">Apache
Hive</a></td>
- <td><a href="{{< ref "docs/connectors/table/hive/overview"
>}}#supported-hive-versions">Supported Versions</a></td>
- <td>Unbounded Scan, Bounded Scan, Lookup</td>
+ <td><a href="{{< ref "docs/connectors/table/kinesis"
>}}">Kinesis</a></td>
+ <td>Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
<td><a href="{{< ref "docs/connectors/table/mongodb"
>}}">MongoDB</a></td>
- <td>3.6.x & 4.x & 5.x & 6.0.x</td>
<td>Bounded Scan, Lookup</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/opensearch"
>}}">Opensearch</a></td>
+ <td>Not supported</td>
+ <td>Streaming Sink, Batch Sink</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/print" >}}">Print</a></td>
+ <td>Not supported</td>
+ <td>Streaming Sink, Batch Sink</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/upsert-kafka" >}}">Upsert
Kafka</a></td>
+ <td>Unbounded Scan</td>
+ <td>Streaming Sink</td>
+ </tr>
</tbody>
</table>
+The following integrations are maintained by other projects, not by the Apache
Flink community. See
+[Integrations Maintained by Other Projects]({{< ref
"docs/connectors/other-projects/overview" >}}).
+
+ * [Apache Doris]({{< ref "docs/connectors/other-projects/doris" >}})
+ * [Apache Fluss]({{< ref "docs/connectors/other-projects/fluss" >}})
+ * [Apache Hudi]({{< ref "docs/connectors/other-projects/hudi" >}})
+ * [Apache Iceberg]({{< ref "docs/connectors/other-projects/iceberg" >}})
+ * [Apache Paimon]({{< ref "docs/connectors/other-projects/paimon" >}})
+
{{< top >}}
请查阅[配置]({{< ref "docs/dev/configuration/connector" >}})小节了解如何添加连接器依赖。
+Flink CDC
+------------
+
+[Flink CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/) is an
Apache Flink sub-project with its own releases. Its source connectors read
+changes directly from databases such as MySQL, PostgreSQL, Oracle, SQL Server,
MongoDB and Db2, and
+it can run complete data integration pipelines defined in YAML. See the
+[Flink CDC
sources](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for the supported databases and Flink versions.
+
+{{< top >}}
+
+Fault Tolerance Guarantees
+------------
+
+Flink recovers from failures by restarting from the last [checkpoint]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}}). Checkpointing is
+disabled by default. For exactly-once results, the source must be able to
rewind to the position
+stored in the checkpoint, and the sink must take part in checkpointing, either
by committing its
+writes transactionally or by writing idempotently. See
+[Stateful Stream Processing]({{< ref
"docs/concepts/stateful-stream-processing" >}}#checkpointing) for how
checkpoints work.
+
+Connectors that support more than one guarantee expose it as the
`sink.delivery-guarantee` option,
+for example [Kafka]({{< ref "docs/connectors/table/kafka" >}}). Exactly-once
on Kafka uses Kafka
+transactions and also requires `sink.transactional-id-prefix`. Sinks that
write by primary key, such
+as [JDBC]({{< ref "docs/connectors/table/jdbc" >}}) with a primary key
defined, overwrite the same
+rows when records are replayed after a failure; while the replay runs, readers
can briefly see older
+values for those rows.
+
+Each connector's page documents the guarantees it provides.
+
+{{< top >}}
+
How to use connectors
--------
diff --git a/docs/content.zh/docs/dev/table/tuning.md
b/docs/content.zh/docs/dev/table/tuning.md
index 8a23468b84b..dcfac187b1d 100644
--- a/docs/content.zh/docs/dev/table/tuning.md
+++ b/docs/content.zh/docs/dev/table/tuning.md
@@ -405,7 +405,7 @@ FROM TenantKafka t
该功能默认启用。当满足以下所有条件时， regular join 将自动优化为 delta join。
1. 作业拓扑结构满足优化条件。具体可以查看[支持的功能和限制]({{< ref "docs/dev/table/tuning"
>}}#supported-features-and-limitations)。
-2. 源表所在的外部存储系统提供了可供 delta join 快速查询的索引信息。目前 [Apache Fluss
(Incubating)](https://fluss.apache.org/blog/fluss-open-source/) 已支持在 Flink
中提供表级别的索引信息，其上的表可作为 delta join 的源表。具体可参考 [Fluss
文档](https://fluss.apache.org/docs/engine-flink/delta-joins/#flink-version-support)。
+2. 源表所在的外部存储系统提供了可供 delta join 快速查询的索引信息。目前 [Apache
Fluss](https://fluss.apache.org/blog/fluss-open-source/) 已支持在 Flink
中提供表级别的索引信息，其上的表可作为 delta join 的源表。具体可参考 [Fluss
文档](https://fluss.apache.org/docs/engine-flink/delta-joins/#flink-version-support)。
<a name="working-principle"></a>
@@ -461,7 +461,7 @@ SET 'table.optimizer.delta-join.strategy' = 'NONE';
6. 源表和 delta join 之间的 projection 或 filter 不能包含**非确定性函数**。
{{< hint info >}}
-[1] Flink 支持定义表级别的**不可变列**（immutable columns）约束来丰富 upsert key，从而使更多场景能够被优化为
delta join。不可变列约束声明某些列一旦为给定主键设置后便不可修改，不可变列会联合主键，作为一组新的 upsert key
传播给下游。该信息由外部存储系统提供。[Apache Fluss (Incubating)](https://fluss.apache.org/)
已在未来计划支持表级别的不可变列约束。
+[1] Flink 支持定义表级别的**不可变列**（immutable columns）约束来丰富 upsert key，从而使更多场景能够被优化为
delta join。不可变列约束声明某些列一旦为给定主键设置后便不可修改，不可变列会联合主键，作为一组新的 upsert key
传播给下游。该信息由外部存储系统提供。[Apache Fluss](https://fluss.apache.org/) 已在未来计划支持表级别的不可变列约束。
{{< /hint >}}
{{< top >}}
diff --git a/docs/content.zh/docs/learn-flink/fault_tolerance.md
b/docs/content.zh/docs/learn-flink/fault_tolerance.md
index 671558c4970..f2c65b9797c 100644
--- a/docs/content.zh/docs/learn-flink/fault_tolerance.md
+++ b/docs/content.zh/docs/learn-flink/fault_tolerance.md
@@ -198,7 +198,7 @@ Barrier 只有在需要提供精确一次的语义保证时需要进行对齐（
- [Stateful Stream Processing]({{< ref
"docs/concepts/stateful-stream-processing" >}})
- [State Backends]({{< ref "docs/ops/state/state_backends" >}})
-- [Data Sources 和 Sinks 的容错保证]({{< ref "docs/connectors/datastream/guarantees"
>}})
+- [Data Sources 和 Sinks 的容错保证]({{< ref "docs/connectors/datastream/overview"
>}}#fault-tolerance-guarantees)
- [开启和配置 Checkpointing]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
- [Checkpoints]({{< ref "docs/ops/state/checkpoints" >}})
- [Savepoints]({{< ref "docs/ops/state/savepoints" >}})
diff --git a/docs/content.zh/docs/sql/materialized-table/deployment.md
b/docs/content.zh/docs/sql/materialized-table/deployment.md
index 17f833e5f7a..3b74c0d6935 100644
--- a/docs/content.zh/docs/sql/materialized-table/deployment.md
+++ b/docs/content.zh/docs/sql/materialized-table/deployment.md
@@ -81,7 +81,11 @@ workflow-scheduler:
```
<span class="label label-danger">注意</span>
-Catalog 必须支持创建物化表，目前只有 [Paimon
Catalog](https://paimon.apache.org/docs/master/concepts/table-types/#materialized-table)
支持。
+The Catalog must support creating materialized tables.
+[Apache Paimon]({{< ref "docs/connectors/other-projects/paimon" >}}) supports
them, and so does
+[Apache
Fluss](https://fluss.apache.org/docs/engine-flink/ddl/#materialized-table) for
the continuous
+refresh mode only. The test-filesystem catalog used in the
+[Quickstart]({{< ref "docs/sql/materialized-table/quickstart" >}}) is meant
for testing only.
# 操作指南
diff --git a/docs/content/docs/concepts/stateful-stream-processing.md
b/docs/content/docs/concepts/stateful-stream-processing.md
index b1ea5da412d..cf0a09bf9fe 100644
--- a/docs/content/docs/concepts/stateful-stream-processing.md
+++ b/docs/content/docs/concepts/stateful-stream-processing.md
@@ -109,9 +109,10 @@ By default, checkpointing is disabled. See
[Checkpointing]({{< ref "docs/dev/dat
For this mechanism to realize its full guarantees, the data
stream source (such as message queue or broker) needs to be able to rewind the
stream to a defined recent point. [Apache Kafka](http://kafka.apache.org) has
-this ability and Flink's connector to Kafka exploits this. See [Fault
-Tolerance Guarantees of Data Sources and Sinks]({{< ref
"docs/connectors/datastream/guarantees" >}}) for more information about the
guarantees
-provided by Flink's connectors.
+this ability and Flink's connector to Kafka exploits this. See the fault
tolerance guarantees for
+[SQL & Table API connectors]({{< ref "docs/connectors/table/overview"
>}}#fault-tolerance-guarantees) and
+[DataStream connectors]({{< ref "docs/connectors/datastream/overview"
>}}#fault-tolerance-guarantees)
+for more information about the guarantees provided by Flink's connectors.
{{< /hint >}}
{{< hint info >}}
diff --git a/docs/content/docs/connectors/datastream/_index.md
b/docs/content/docs/connectors/datastream/_index.md
index fd6e50a01c2..b362045eb47 100644
--- a/docs/content/docs/connectors/datastream/_index.md
+++ b/docs/content/docs/connectors/datastream/_index.md
@@ -1,6 +1,7 @@
---
title: DataStream Connectors
bookCollapseSection: true
+bookSortByTitle: true
weight: 4
---
<!--
diff --git a/docs/content/docs/connectors/datastream/guarantees.md
b/docs/content/docs/connectors/datastream/guarantees.md
index 810a3931ef6..acf44187dce 100644
--- a/docs/content/docs/connectors/datastream/guarantees.md
+++ b/docs/content/docs/connectors/datastream/guarantees.md
@@ -2,6 +2,7 @@
title: Fault Tolerance Guarantees
weight: 2
type: docs
+bookHidden: true
aliases:
- /dev/connectors/guarantees.html
---
@@ -26,131 +27,7 @@ under the License.
# Fault Tolerance Guarantees of Data Sources and Sinks
-Flink's fault tolerance mechanism recovers programs in the presence of
failures and
-continues to execute them. Such failures include machine hardware failures,
network failures,
-transient program failures, etc.
-
-Flink can guarantee exactly-once state updates to user-defined state only when
the source participates in the
-snapshotting mechanism. The following table lists the state update guarantees
of Flink coupled with the bundled connectors.
-
-Please read the documentation of each connector to understand the details of
the fault tolerance guarantees.
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 25%">Source</th>
- <th class="text-left" style="width: 25%">Guarantees</th>
- <th class="text-left">Notes</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>Apache Kafka</td>
- <td>exactly once</td>
- <td>Use the appropriate Kafka connector for your version</td>
- </tr>
- <tr>
- <td>AWS Kinesis Streams</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>RabbitMQ</td>
- <td>at most once (v 0.10) / exactly once (v 1.0) </td>
- <td></td>
- </tr>
- <tr>
- <td>Google PubSub</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Collections</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>Files</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>Sockets</td>
- <td>at most once</td>
- <td></td>
- </tr>
- </tbody>
-</table>
-
-To guarantee end-to-end exactly-once record delivery (in addition to
exactly-once state semantics), the data sink needs
-to take part in the checkpointing mechanism. The following table lists the
delivery guarantees (assuming exactly-once
-state updates) of Flink coupled with bundled sinks:
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style="width: 25%">Sink</th>
- <th class="text-left" style="width: 25%">Guarantees</th>
- <th class="text-left">Notes</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td>Elasticsearch</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Opensearch</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Kafka producer</td>
- <td>at least once / exactly once</td>
- <td>exactly once with transactional producers (v 0.11+)</td>
- </tr>
- <tr>
- <td>Cassandra sink</td>
- <td>at least once / exactly once</td>
- <td>exactly once only for idempotent updates</td>
- </tr>
- <tr>
- <td>Amazon DynamoDB</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Amazon Kinesis Data Streams</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Amazon Kinesis Data Firehose</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>File sinks</td>
- <td>exactly once</td>
- <td></td>
- </tr>
- <tr>
- <td>Socket sinks</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Standard output</td>
- <td>at least once</td>
- <td></td>
- </tr>
- <tr>
- <td>Redis sink</td>
- <td>at least once</td>
- <td></td>
- </tr>
- </tbody>
-</table>
-
-{{< top >}}
+This page has moved into the connector overviews: see
+[SQL & Table API connectors]({{< ref "docs/connectors/table/overview"
>}}#fault-tolerance-guarantees)
+and [DataStream connectors]({{< ref "docs/connectors/datastream/overview"
>}}#fault-tolerance-guarantees).
+Each connector's page documents the guarantees it provides.
diff --git a/docs/content/docs/connectors/datastream/overview.md
b/docs/content/docs/connectors/datastream/overview.md
index 8d100da90b7..b24d534288a 100644
--- a/docs/content/docs/connectors/datastream/overview.md
+++ b/docs/content/docs/connectors/datastream/overview.md
@@ -39,22 +39,25 @@ The [predefined data sinks]({{< ref
"docs/dev/datastream/overview" >}}#data-sink
Connectors provide code for interfacing with various third-party systems.
Currently these systems are supported as part of the Apache Flink project:
- * [Apache Kafka]({{< ref "docs/connectors/datastream/kafka" >}}) (source/sink)
- * [Apache Cassandra]({{< ref "docs/connectors/datastream/cassandra" >}})
(source/sink)
- * [Amazon DynamoDB]({{< ref "docs/connectors/datastream/dynamodb" >}}) (sink)
- * [Amazon Kinesis Data Streams]({{< ref "docs/connectors/datastream/kinesis"
>}}) (source/sink)
- * [Amazon Kinesis Data Firehose]({{< ref
"docs/connectors/datastream/firehose" >}}) (sink)
+ * [Cassandra]({{< ref "docs/connectors/datastream/cassandra" >}})
(source/sink)
* [DataGen]({{< ref "docs/connectors/datastream/datagen" >}}) (source)
+ * [Dynamic Kafka]({{< ref "docs/connectors/datastream/dynamic-kafka" >}})
(source)
+ * [DynamoDB]({{< ref "docs/connectors/datastream/dynamodb" >}}) (sink)
* [Elasticsearch]({{< ref "docs/connectors/datastream/elasticsearch" >}})
(sink)
- * [Opensearch]({{< ref "docs/connectors/datastream/opensearch" >}}) (sink)
* [FileSystem]({{< ref "docs/connectors/datastream/filesystem" >}})
(source/sink)
- * [RabbitMQ]({{< ref "docs/connectors/datastream/rabbitmq" >}}) (source/sink)
- * [Google PubSub]({{< ref "docs/connectors/datastream/pubsub" >}})
(source/sink)
+ * [Firehose]({{< ref "docs/connectors/datastream/firehose" >}}) (sink)
+ * [Google Cloud PubSub]({{< ref "docs/connectors/datastream/pubsub" >}})
(source/sink)
+ * [HTTP]({{< ref "docs/connectors/datastream/http" >}}) (sink)
* [Hybrid Source]({{< ref "docs/connectors/datastream/hybridsource" >}})
(source)
- * [Apache Pulsar]({{< ref "docs/connectors/datastream/pulsar" >}}) (source)
* [JDBC]({{< ref "docs/connectors/datastream/jdbc" >}}) (sink)
+ * [Kafka]({{< ref "docs/connectors/datastream/kafka" >}}) (source/sink)
+ * [Kinesis]({{< ref "docs/connectors/datastream/kinesis" >}}) (source/sink)
* [MongoDB]({{< ref "docs/connectors/datastream/mongodb" >}}) (source/sink)
+ * [Opensearch]({{< ref "docs/connectors/datastream/opensearch" >}}) (sink)
* [Prometheus]({{< ref "docs/connectors/datastream/prometheus" >}}) (sink)
+ * [Pulsar]({{< ref "docs/connectors/datastream/pulsar" >}}) (source/sink)
+ * [RabbitMQ]({{< ref "docs/connectors/datastream/rabbitmq" >}}) (source/sink)
+ * [SQS]({{< ref "docs/connectors/datastream/sqs" >}}) (sink)
Keep in mind that to use one of these connectors in an application, additional
third party
components are usually required, e.g. servers for the data stores or message
queues.
@@ -70,15 +73,29 @@ If you need to run examples locally, make sure that the
flink-connector-base dep
provided and can be found in your own classpath.
{{< /hint >}}
-## Connectors in Apache Bahir
+## Flink CDC
+
+[Flink CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/) is an
Apache Flink sub-project with its own releases. Its source connectors read
+changes directly from databases such as MySQL, PostgreSQL, Oracle, SQL Server,
MongoDB and Db2. See
+the [Flink CDC
sources](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for the supported databases and Flink versions.
+
+## Fault Tolerance Guarantees
+
+Flink recovers from failures by restarting from the last [checkpoint]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}}). Checkpointing is
+disabled by default. For exactly-once results, the source must be able to
rewind to the position
+stored in the checkpoint, and the sink must take part in checkpointing, either
by committing its
+writes transactionally or by writing idempotently. See [Stateful Stream
Processing]({{< ref "docs/concepts/stateful-stream-processing"
>}}#checkpointing) for how checkpoints work.
+
+Sinks built on the [Sink API]({{< ref "docs/internals/sinks" >}}) that support
more than one
+guarantee take a {{< javadoc
file="org/apache/flink/connector/base/DeliveryGuarantee.html"
name="DeliveryGuarantee" >}}
+(`NONE`, `AT_LEAST_ONCE` or `EXACTLY_ONCE`) in their builder, for example the
+[Kafka sink]({{< ref "docs/connectors/datastream/kafka" >}}). Each connector's
page documents the
+guarantees it provides.
-Additional streaming connectors for Flink are being released through [Apache
Bahir](https://bahir.apache.org/), including:
+## Integrations Maintained by Other Projects
- * [Apache
ActiveMQ](https://bahir.apache.org/docs/flink/current/flink-streaming-activemq/)
(source/sink)
- * [Apache
Flume](https://bahir.apache.org/docs/flink/current/flink-streaming-flume/)
(sink)
- * [Redis](https://bahir.apache.org/docs/flink/current/flink-streaming-redis/)
(sink)
- * [Akka](https://bahir.apache.org/docs/flink/current/flink-streaming-akka/)
(sink)
- * [Netty](https://bahir.apache.org/docs/flink/current/flink-streaming-netty/)
(source)
+Some projects maintain their own Flink integration, such as Apache Iceberg and
Apache Paimon. See
+[Integrations Maintained by Other Projects]({{< ref
"docs/connectors/other-projects/overview" >}}).
## Other Ways to Connect to Flink
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content/docs/connectors/other-projects/_index.md
similarity index 91%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content/docs/connectors/other-projects/_index.md
index a117e9fa887..acd631dd9f2 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content/docs/connectors/other-projects/_index.md
@@ -1,7 +1,8 @@
---
-title: SQL & Table API Connectors
+title: Maintained by Other Projects
bookCollapseSection: true
-weight: 3
+bookSortByTitle: true
+weight: 6
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +21,4 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content/docs/connectors/other-projects/doris.md
similarity index 51%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content/docs/connectors/other-projects/doris.md
index a117e9fa887..38e5a4aec70 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content/docs/connectors/other-projects/doris.md
@@ -1,7 +1,7 @@
---
-title: SQL & Table API Connectors
-bookCollapseSection: true
-weight: 3
+title: Apache Doris
+weight: 10
+type: docs
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,20 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
+
+# Apache Doris
+
+{{< hint info >}}
+The Apache Doris community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Doris](https://doris.apache.org/) is a real-time analytical database.
Its Flink connector
+reads from and writes to Doris from SQL and the DataStream API, supports
lookup joins, and can
+synchronize whole databases into Doris using Flink CDC.
+
+- [Flink
integration](https://doris.apache.org/docs/ecosystem/flink-doris-connector)
+- [Supported Flink
versions](https://doris.apache.org/docs/ecosystem/flink-doris-connector#version-description)
+
+{{< top >}}
diff --git a/docs/content/docs/connectors/other-projects/fluss.md
b/docs/content/docs/connectors/other-projects/fluss.md
new file mode 100644
index 00000000000..946f2190f09
--- /dev/null
+++ b/docs/content/docs/connectors/other-projects/fluss.md
@@ -0,0 +1,40 @@
+---
+title: Apache Fluss
+weight: 10
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Fluss
+
+{{< hint info >}}
+The Apache Fluss community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Fluss](https://fluss.apache.org/) is streaming storage for real-time
analytics. Its Flink
+integration provides a catalog, reads and writes Fluss tables in streaming and
batch mode, and
+supports lookup joins. Fluss tables can be used as sources for
+[delta joins]({{< ref "docs/dev/table/tuning" >}}#delta-joins).
+
+- [Flink
integration](https://fluss.apache.org/docs/engine-flink/getting-started/)
+- [Supported Flink
versions](https://fluss.apache.org/docs/engine-flink/getting-started/#supported-flink-versions)
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content/docs/connectors/other-projects/hudi.md
similarity index 53%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content/docs/connectors/other-projects/hudi.md
index a117e9fa887..65cade9adf4 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content/docs/connectors/other-projects/hudi.md
@@ -1,7 +1,7 @@
---
-title: SQL & Table API Connectors
-bookCollapseSection: true
-weight: 3
+title: Apache Hudi
+weight: 10
+type: docs
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,19 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
+
+# Apache Hudi
+
+{{< hint info >}}
+The Apache Hudi community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Hudi](https://hudi.apache.org/) is an open data lakehouse platform.
Its Flink integration provides a catalog and reads and writes Hudi tables in
streaming and
+batch mode from SQL and the DataStream API.
+
+- [Flink integration](https://hudi.apache.org/docs/flink-quick-start-guide/)
+- [Supported Flink
versions](https://hudi.apache.org/docs/flink-quick-start-guide/#flink-support-matrix)
+
+{{< top >}}
diff --git a/docs/content/docs/connectors/other-projects/iceberg.md
b/docs/content/docs/connectors/other-projects/iceberg.md
new file mode 100644
index 00000000000..7689f821b15
--- /dev/null
+++ b/docs/content/docs/connectors/other-projects/iceberg.md
@@ -0,0 +1,44 @@
+---
+title: Apache Iceberg
+weight: 10
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Apache Iceberg
+
+{{< hint info >}}
+The Apache Iceberg community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Iceberg](https://iceberg.apache.org/) is an open table format for
large analytic tables.
+Its Flink integration provides a catalog and reads and writes Iceberg tables
in streaming and batch
+mode from SQL, the Table API and the DataStream API.
+
+Iceberg tables are often managed by a catalog service that implements the
Iceberg REST catalog API,
+such as [Apache Polaris](https://polaris.apache.org/guides/flink/). Flink
connects to such a service
+through Iceberg's REST catalog.
+
+- [Flink integration](https://iceberg.apache.org/docs/latest/flink/)
+- [Supported Flink
versions](https://iceberg.apache.org/multi-engine-support/#apache-flink)
+- [REST catalog](https://iceberg.apache.org/docs/latest/flink/#rest-catalog)
+
+{{< top >}}
diff --git a/docs/content/docs/connectors/other-projects/overview.md
b/docs/content/docs/connectors/other-projects/overview.md
new file mode 100644
index 00000000000..ad604bbdd9d
--- /dev/null
+++ b/docs/content/docs/connectors/other-projects/overview.md
@@ -0,0 +1,48 @@
+---
+title: Overview
+weight: 1
+type: docs
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Integrations Maintained by Other Projects
+
+The projects listed here maintain their own Flink integration. Each
integration is developed,
+released and supported by that project's community, not by the Apache Flink
community. That
+project's documentation covers which artifacts to use and how to configure it.
Report issues and ask
+questions in that project's channels.
+
+A project is listed here when it is an Apache Software Foundation project and
has released a Flink
+integration for Flink 2.x.
+
+{{< hint info >}}
+These integrations follow their own release schedules and may not support this
Flink version yet.
+Check each project's supported Flink versions before you choose one.
+{{< /hint >}}
+
+| Project | Flink integration |
+|---------|-------------------|
+| [Apache Doris]({{< ref "docs/connectors/other-projects/doris" >}}) | Source,
sink, lookup joins and database synchronization through Flink CDC |
+| [Apache Fluss]({{< ref "docs/connectors/other-projects/fluss" >}}) |
Catalog, source, sink, lookup joins and delta joins |
+| [Apache Hudi]({{< ref "docs/connectors/other-projects/hudi" >}}) | Catalog,
source and sink |
+| [Apache Iceberg]({{< ref "docs/connectors/other-projects/iceberg" >}}) |
Catalog, source and sink; REST catalogs such as Apache Polaris |
+| [Apache Paimon]({{< ref "docs/connectors/other-projects/paimon" >}}) |
Catalog, source, sink and lookup joins |
+
+{{< top >}}
diff --git a/docs/content.zh/docs/connectors/table/_index.md
b/docs/content/docs/connectors/other-projects/paimon.md
similarity index 52%
copy from docs/content.zh/docs/connectors/table/_index.md
copy to docs/content/docs/connectors/other-projects/paimon.md
index a117e9fa887..cc14994b193 100644
--- a/docs/content.zh/docs/connectors/table/_index.md
+++ b/docs/content/docs/connectors/other-projects/paimon.md
@@ -1,7 +1,7 @@
---
-title: SQL & Table API Connectors
-bookCollapseSection: true
-weight: 3
+title: Apache Paimon
+weight: 10
+type: docs
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
@@ -20,4 +20,20 @@ software distributed under the License is distributed on an
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
--->
\ No newline at end of file
+-->
+
+# Apache Paimon
+
+{{< hint info >}}
+The Apache Paimon community maintains this integration, not the Apache Flink
community. It may not
+support this Flink version yet, so check its supported Flink versions below.
+{{< /hint >}}
+
+[Apache Paimon](https://paimon.apache.org/) is a lake format for streaming and
batch processing.
+Its Flink integration provides a catalog, reads and writes Paimon tables in
streaming and batch
+mode, and supports lookup joins.
+
+- [Flink integration](https://paimon.apache.org/docs/master/flink/quick-start/)
+- [Supported Flink
versions](https://paimon.apache.org/docs/master/flink/installation/)
+
+{{< top >}}
diff --git a/docs/content/docs/connectors/table/_index.md
b/docs/content/docs/connectors/table/_index.md
index a117e9fa887..eb52eca6531 100644
--- a/docs/content/docs/connectors/table/_index.md
+++ b/docs/content/docs/connectors/table/_index.md
@@ -1,6 +1,7 @@
---
title: SQL & Table API Connectors
bookCollapseSection: true
+bookSortByTitle: true
weight: 3
---
<!--
diff --git a/docs/content/docs/connectors/table/formats/canal.md
b/docs/content/docs/connectors/table/formats/canal.md
index 6f253125e49..29f58282379 100644
--- a/docs/content/docs/connectors/table/formats/canal.md
+++ b/docs/content/docs/connectors/table/formats/canal.md
@@ -32,6 +32,9 @@ under the License.
[Canal](https://github.com/alibaba/canal/wiki) is a CDC (Changelog Data
Capture) tool that can stream changes in real-time from MySQL into other
systems. Canal provides a unified format schema for changelog and supports to
serialize messages using JSON and
[protobuf](https://developers.google.com/protocol-buffers) (protobuf is the
default format for Canal).
+This format reads change events that Canal has already written to a system
such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink supports to interpret Canal JSON messages as INSERT/UPDATE/DELETE
messages into Flink SQL system. This is useful in many cases to leverage this
feature, such as
- synchronizing incremental data from databases to other systems
- auditing logs
diff --git a/docs/content/docs/connectors/table/formats/debezium.md
b/docs/content/docs/connectors/table/formats/debezium.md
index 9e2f60f23a6..b89530b133e 100644
--- a/docs/content/docs/connectors/table/formats/debezium.md
+++ b/docs/content/docs/connectors/table/formats/debezium.md
@@ -32,6 +32,9 @@ under the License.
[Debezium](https://debezium.io/) is a CDC (Changelog Data Capture) tool that
can stream changes in real-time from MySQL, PostgreSQL, Oracle, Microsoft SQL
Server and many other databases into Kafka. Debezium provides a unified format
schema for changelog and supports to serialize messages using JSON and [Apache
Avro](https://avro.apache.org/).
+This format reads change events that Debezium has already written to a system
such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink supports to interpret Debezium JSON and Avro messages as
INSERT/UPDATE/DELETE messages into Flink SQL system. This is useful in many
cases to leverage this feature, such as
- synchronizing incremental data from databases to other systems
- auditing logs
diff --git a/docs/content/docs/connectors/table/formats/maxwell.md
b/docs/content/docs/connectors/table/formats/maxwell.md
index 47c87442c73..f79b2300de8 100644
--- a/docs/content/docs/connectors/table/formats/maxwell.md
+++ b/docs/content/docs/connectors/table/formats/maxwell.md
@@ -32,6 +32,9 @@ under the License.
[Maxwell](https://maxwells-daemon.io/) is a CDC (Changelog Data Capture) tool
that can stream changes in real-time from MySQL into Kafka, Kinesis and other
streaming connectors. Maxwell provides a unified format schema for changelog
and supports to serialize messages using JSON.
+This format reads change events that Maxwell has already written to a system
such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink supports to interpret Maxwell JSON messages as INSERT/UPDATE/DELETE
messages into Flink SQL system. This is useful in many cases to leverage this
feature, such as
- synchronizing incremental data from databases to other systems
- auditing logs
diff --git a/docs/content/docs/connectors/table/formats/ogg.md
b/docs/content/docs/connectors/table/formats/ogg.md
index 3b53916e36d..3e5a3f091ef 100644
--- a/docs/content/docs/connectors/table/formats/ogg.md
+++ b/docs/content/docs/connectors/table/formats/ogg.md
@@ -35,6 +35,9 @@ enabling real-time analysis. Customers can design, execute,
and monitor their da
stream data processing solutions without the need to allocate or manage
compute environments. Ogg
provides a format schema for changelog and supports to serialize messages
using JSON.
+This format reads change events that Oracle GoldenGate has already written to
a system such as Kafka. To
+read changes directly from a database instead, see [Flink
CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/).
+
Flink supports to interpret Ogg JSON as INSERT/UPDATE/DELETE messages into
Flink SQL system. This is
useful in many cases to leverage this feature, such as
diff --git a/docs/content/docs/connectors/table/overview.md
b/docs/content/docs/connectors/table/overview.md
index 4cf924330e1..1bb2ff52168 100644
--- a/docs/content/docs/connectors/table/overview.md
+++ b/docs/content/docs/connectors/table/overview.md
@@ -42,85 +42,142 @@ Flink natively support various connectors. The following
tables list all availab
<thead>
<tr>
<th class="text-left">Name</th>
- <th class="text-center">Version</th>
<th class="text-center">Source</th>
<th class="text-center">Sink</th>
</tr>
</thead>
<tbody>
<tr>
- <td><a href="{{< ref "docs/connectors/table/filesystem"
>}}">Filesystem</a></td>
- <td></td>
- <td>Bounded and Unbounded Scan</td>
+ <td><a href="{{< ref "docs/connectors/table/blackhole"
>}}">BlackHole</a></td>
+ <td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/elasticsearch"
>}}">Elasticsearch</a></td>
- <td>6.x & 7.x</td>
+ <td><a href="{{< ref "docs/connectors/table/datagen"
>}}">DataGen</a></td>
+ <td>Bounded and Unbounded Scan</td>
+ <td>Not supported</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/dynamic-kafka" >}}">Dynamic
Kafka</a></td>
+ <td>Unbounded Scan</td>
+ <td>Not supported</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/dynamodb"
>}}">DynamoDB</a></td>
<td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/opensearch"
>}}">Opensearch</a></td>
- <td>1.x & 2.x</td>
+ <td><a href="{{< ref "docs/connectors/table/elasticsearch"
>}}">Elasticsearch</a></td>
<td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/kafka" >}}">Apache
Kafka</a></td>
- <td>0.10+</td>
- <td>Unbounded Scan</td>
+ <td><a href="{{< ref "docs/connectors/table/filesystem"
>}}">FileSystem</a></td>
+ <td>Bounded and Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/dynamodb" >}}">Amazon
DynamoDB</a></td>
- <td></td>
+ <td><a href="{{< ref "docs/connectors/table/firehose"
>}}">Firehose</a></td>
<td>Not supported</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/kinesis" >}}">Amazon Kinesis
Data Streams</a></td>
- <td></td>
- <td>Unbounded Scan</td>
- <td>Streaming Sink</td>
+ <td><a href="{{< ref "docs/connectors/table/hbase" >}}">HBase</a></td>
+ <td>Bounded Scan, Lookup</td>
+ <td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/firehose" >}}">Amazon
Kinesis Data Firehose</a></td>
- <td></td>
- <td>Not supported</td>
- <td>Streaming Sink</td>
+ <td><a href="{{< ref "docs/connectors/table/hive/overview"
>}}">Hive</a></td>
+ <td>Unbounded Scan, Bounded Scan, Lookup</td>
+ <td>Streaming Sink, Batch Sink</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/http" >}}">HTTP</a></td>
+ <td>Lookup</td>
+ <td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
<td><a href="{{< ref "docs/connectors/table/jdbc" >}}">JDBC</a></td>
- <td></td>
<td>Bounded Scan, Lookup</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/hbase" >}}">Apache
HBase</a></td>
- <td>1.4.x & 2.2.x</td>
- <td>Bounded Scan, Lookup</td>
+ <td><a href="{{< ref "docs/connectors/table/kafka" >}}">Kafka</a></td>
+ <td>Bounded and Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
- <td><a href="{{< ref "docs/connectors/table/hive/overview" >}}">Apache
Hive</a></td>
- <td><a href="{{< ref "docs/connectors/table/hive/overview"
>}}#supported-hive-versions">Supported Versions</a></td>
- <td>Unbounded Scan, Bounded Scan, Lookup</td>
+ <td><a href="{{< ref "docs/connectors/table/kinesis"
>}}">Kinesis</a></td>
+ <td>Unbounded Scan</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
<tr>
<td><a href="{{< ref "docs/connectors/table/mongodb"
>}}">MongoDB</a></td>
- <td>3.6.x & 4.x & 5.x & 6.x & 7.0.x </td>
<td>Bounded Scan, Lookup</td>
<td>Streaming Sink, Batch Sink</td>
</tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/opensearch"
>}}">Opensearch</a></td>
+ <td>Not supported</td>
+ <td>Streaming Sink, Batch Sink</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/print" >}}">Print</a></td>
+ <td>Not supported</td>
+ <td>Streaming Sink, Batch Sink</td>
+ </tr>
+ <tr>
+ <td><a href="{{< ref "docs/connectors/table/upsert-kafka" >}}">Upsert
Kafka</a></td>
+ <td>Unbounded Scan</td>
+ <td>Streaming Sink</td>
+ </tr>
</tbody>
</table>
+The following integrations are maintained by other projects, not by the Apache
Flink community. See
+[Integrations Maintained by Other Projects]({{< ref
"docs/connectors/other-projects/overview" >}}).
+
+ * [Apache Doris]({{< ref "docs/connectors/other-projects/doris" >}})
+ * [Apache Fluss]({{< ref "docs/connectors/other-projects/fluss" >}})
+ * [Apache Hudi]({{< ref "docs/connectors/other-projects/hudi" >}})
+ * [Apache Iceberg]({{< ref "docs/connectors/other-projects/iceberg" >}})
+ * [Apache Paimon]({{< ref "docs/connectors/other-projects/paimon" >}})
+
{{< top >}}
Please refer to the [configuration]({{< ref "docs/dev/configuration/connector"
>}}) section on how to add connectors as a dependency.
+Flink CDC
+------------
+
+[Flink CDC](https://nightlies.apache.org/flink/flink-cdc-docs-stable/) is an
Apache Flink sub-project with its own releases. Its source connectors read
+changes directly from databases such as MySQL, PostgreSQL, Oracle, SQL Server,
MongoDB and Db2, and
+it can run complete data integration pipelines defined in YAML. See the
+[Flink CDC
sources](https://nightlies.apache.org/flink/flink-cdc-docs-stable/docs/connectors/flink-sources/overview/)
for the supported databases and Flink versions.
+
+{{< top >}}
+
+Fault Tolerance Guarantees
+------------
+
+Flink recovers from failures by restarting from the last [checkpoint]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}}). Checkpointing is
+disabled by default. For exactly-once results, the source must be able to
rewind to the position
+stored in the checkpoint, and the sink must take part in checkpointing, either
by committing its
+writes transactionally or by writing idempotently. See
+[Stateful Stream Processing]({{< ref
"docs/concepts/stateful-stream-processing" >}}#checkpointing) for how
checkpoints work.
+
+Connectors that support more than one guarantee expose it as the
`sink.delivery-guarantee` option,
+for example [Kafka]({{< ref "docs/connectors/table/kafka" >}}). Exactly-once
on Kafka uses Kafka
+transactions and also requires `sink.transactional-id-prefix`. Sinks that
write by primary key, such
+as [JDBC]({{< ref "docs/connectors/table/jdbc" >}}) with a primary key
defined, overwrite the same
+rows when records are replayed after a failure; while the replay runs, readers
can briefly see older
+values for those rows.
+
+Each connector's page documents the guarantees it provides.
+
+{{< top >}}
+
How to use connectors
--------
diff --git a/docs/content/docs/dev/table/tuning.md
b/docs/content/docs/dev/table/tuning.md
index e0954e58a8a..026c9ecb382 100644
--- a/docs/content/docs/dev/table/tuning.md
+++ b/docs/content/docs/dev/table/tuning.md
@@ -395,7 +395,7 @@ To mitigate these challenges, Flink introduces the delta
join operator. The key
This feature is enabled by default. A regular join will be automatically
optimized into a delta join when all the following conditions are met:
1. The sql pattern satisfies the optimization criteria. For details, please
refer to [Supported Features and Limitations]({{< ref "docs/dev/table/tuning"
>}}#supported-features-and-limitations)
-2. The external storage system of the source table provides index information
for fast querying for delta joins. Currently, [Apache Fluss
(Incubating)](https://fluss.apache.org/blog/fluss-open-source/) has provided
index information at the table level for Flink, allowing such tables to be used
as source tables for delta joins. Please refer to the [Fluss
documentation](https://fluss.apache.org/docs/engine-flink/delta-joins/#flink-version-support)
for more details.
+2. The external storage system of the source table provides index information
for fast querying for delta joins. Currently, [Apache
Fluss](https://fluss.apache.org/blog/fluss-open-source/) has provided index
information at the table level for Flink, allowing such tables to be used as
source tables for delta joins. Please refer to the [Fluss
documentation](https://fluss.apache.org/docs/engine-flink/delta-joins/#flink-version-support)
for more details.
### Working Principle
@@ -445,5 +445,5 @@ However, Delta Joins also have several **limitations**.
Jobs containing any of t
6. **Non-deterministic functions** are not allowed in projections or filters
between the source and the delta join.
{{< hint info >}}
-[1] Flink supports defining a table-level **immutable columns** constraint to
enrich the upsert key, enabling delta join optimization in more scenarios. The
immutable columns constraint declares that certain columns, once set for a
given primary key, cannot be modified. The immutable columns are combined with
the primary key to form a new upsert key that is propagated to downstream
operators. This information is provided by the external storage system. [Apache
Fluss (Incubating)](https:/ [...]
+[1] Flink supports defining a table-level **immutable columns** constraint to
enrich the upsert key, enabling delta join optimization in more scenarios. The
immutable columns constraint declares that certain columns, once set for a
given primary key, cannot be modified. The immutable columns are combined with
the primary key to form a new upsert key that is propagated to downstream
operators. This information is provided by the external storage system. [Apache
Fluss](https://fluss.apache [...]
{{< /hint >}}
\ No newline at end of file
diff --git a/docs/content/docs/learn-flink/fault_tolerance.md
b/docs/content/docs/learn-flink/fault_tolerance.md
index c771a62493f..68c4689c6b0 100644
--- a/docs/content/docs/learn-flink/fault_tolerance.md
+++ b/docs/content/docs/learn-flink/fault_tolerance.md
@@ -216,7 +216,7 @@ The [Flink Operations Playground]({{< ref
"docs/getting-started/flink-operations
- [Stateful Stream Processing]({{< ref
"docs/concepts/stateful-stream-processing" >}})
- [State Backends]({{< ref "docs/ops/state/state_backends" >}})
-- [Fault Tolerance Guarantees of Data Sources and Sinks]({{< ref
"docs/connectors/datastream/guarantees" >}})
+- [Fault Tolerance Guarantees of Data Sources and Sinks]({{< ref
"docs/connectors/datastream/overview" >}}#fault-tolerance-guarantees)
- [Enabling and Configuring Checkpointing]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
- [Checkpoints]({{< ref "docs/ops/state/checkpoints" >}})
- [Savepoints]({{< ref "docs/ops/state/savepoints" >}})
diff --git a/docs/content/docs/sql/materialized-table/deployment.md
b/docs/content/docs/sql/materialized-table/deployment.md
index 04a0d326d55..6ec5e5edef4 100644
--- a/docs/content/docs/sql/materialized-table/deployment.md
+++ b/docs/content/docs/sql/materialized-table/deployment.md
@@ -82,7 +82,11 @@ Start the SQL Gateway using:
```
<span class="label label-danger">Note</span>
-The Catalog must support creating materialized tables, which is currently only
supported by [Paimon
Catalog](https://paimon.apache.org/docs/master/concepts/table-types/#materialized-table).
+The Catalog must support creating materialized tables.
+[Apache Paimon]({{< ref "docs/connectors/other-projects/paimon" >}}) supports
them, and so does
+[Apache
Fluss](https://fluss.apache.org/docs/engine-flink/ddl/#materialized-table) for
the continuous
+refresh mode only. The test-filesystem catalog used in the
+[Quickstart]({{< ref "docs/sql/materialized-table/quickstart" >}}) is meant
for testing only.
# Operation Guide
diff --git a/docs/layouts/partials/docs/menu-filetree.html
b/docs/layouts/partials/docs/menu-filetree.html
index b583347dd03..5e6c26c7c7a 100644
--- a/docs/layouts/partials/docs/menu-filetree.html
+++ b/docs/layouts/partials/docs/menu-filetree.html
@@ -29,8 +29,18 @@ under the License.
{{ end }}
{{ define "book-section-children" }}{{/* (dict "Section" .Section
"CurrentPage" .CurrentPage) */}}
+ {{ $pages := where .Section.Pages "Params.bookhidden" "ne" true }}
+ {{/* Connector pages come from separate repositories that each pick their
own weight. Sort them by
+ link title so no coordination is needed; weights 1 and 2 (overview,
formats) and from 100
+ (downloads) stay pinned. Translated pages set an English linkTitle to
keep the same order. */}}
+ {{ if .Section.Params.bookSortByTitle }}
+ {{ $top := where (where $pages "Weight" "ge" 1) "Weight" "le" 2 }}
+ {{ $middle := (where $pages "Weight" "lt" 1) | append (where (where $pages
"Weight" "gt" 2) "Weight" "lt" 100) }}
+ {{ $bottom := where $pages "Weight" "ge" 100 }}
+ {{ $pages = $top.ByWeight | append $middle.ByLinkTitle | append
$bottom.ByWeight }}
+ {{ end }}
<ul>
- {{ range (where .Section.Pages "Params.bookhidden" "ne" true) }}
+ {{ range $pages }}
{{ if .IsSection }}
<li {{- if .Params.BookFlatSection }} class="book-section-flat" {{ end
-}}>
{{ template "book-page-link" (dict "Page" . "CurrentPage"
$.CurrentPage) }}
