This is an automated email from the ASF dual-hosted git repository.
MartijnVisser pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-kafka.git
The following commit(s) were added to refs/heads/main by this push:
new f6c75db8 [FLINK-41203][docs] Help users pick between the Kafka SQL
connectors (#340)
f6c75db8 is described below
commit f6c75db86f4f49cd88214cfbb47806178d9b61af
Author: Martijn Visser <[email protected]>
AuthorDate: Tue Oct 27 18:54:13 2026 +0100
[FLINK-41203][docs] Help users pick between the Kafka SQL connectors (#340)
* [FLINK-41203][docs] Help users pick between the Kafka SQL connectors
The Kafka page never mentioned Upsert Kafka, and the Dynamic Kafka SQL
page read like a general alternative. Add a section to the Kafka page on
which connector to use, point the Dynamic Kafka pages to the Kafka
connector, and fix the CDC format wording.
Generated-by: Claude Code (Claude Opus 5.5)
---
.../docs/connectors/datastream/dynamic-kafka.md | 6 ++++++
.../content.zh/docs/connectors/table/dynamic-kafka.md | 10 +++++++++-
docs/content.zh/docs/connectors/table/kafka.md | 17 +++++++++++++++++
docs/content.zh/docs/connectors/table/upsert-kafka.md | 3 +++
.../docs/connectors/datastream/dynamic-kafka.md | 6 ++++++
docs/content/docs/connectors/table/dynamic-kafka.md | 10 +++++++++-
docs/content/docs/connectors/table/kafka.md | 19 ++++++++++++++++++-
docs/content/docs/connectors/table/upsert-kafka.md | 3 +++
8 files changed, 71 insertions(+), 3 deletions(-)
diff --git a/docs/content.zh/docs/connectors/datastream/dynamic-kafka.md
b/docs/content.zh/docs/connectors/datastream/dynamic-kafka.md
index c7e033e0..31f01c7d 100644
--- a/docs/content.zh/docs/connectors/datastream/dynamic-kafka.md
+++ b/docs/content.zh/docs/connectors/datastream/dynamic-kafka.md
@@ -32,6 +32,12 @@ topics and/or clusters, without requiring a job restart.
This is especially usef
an existing Kafka cluster/topic (cluster migration/failover/other
infrastructure changes) and when you need direct integration with Hybrid
Source. The solution
makes these operations automated so that they are transparent to Kafka
consumers.
+{{< hint info >}}
+Use this connector to read one stream from several Kafka clusters, or when the
topics or clusters to
+read from change while the job runs, for example during a cluster migration.
Otherwise, use the
+[Kafka connector]({{< ref "docs/connectors/datastream/kafka" >}}).
+{{< /hint >}}
+
## Dependency
For details on Kafka compatibility, please refer to the official [Kafka
documentation](https://kafka.apache.org/protocol.html#protocol_compatibility).
diff --git a/docs/content.zh/docs/connectors/table/dynamic-kafka.md
b/docs/content.zh/docs/connectors/table/dynamic-kafka.md
index c6f11778..02258231 100644
--- a/docs/content.zh/docs/connectors/table/dynamic-kafka.md
+++ b/docs/content.zh/docs/connectors/table/dynamic-kafka.md
@@ -24,13 +24,21 @@ specific language governing permissions and limitations
under the License.
-->
-# Dynamic Kafka SQL 连接器
+# Dynamic Kafka SQL 连接器 _`Experimental`_
{{< label "Scan Source: Unbounded" >}}
Dynamic Kafka 连接器允许从可跨集群变更的 Kafka 主题读取数据，无需重启作业。
流的解析由 Kafka 元数据服务完成，尤其适用于集群迁移和动态主题/集群变更场景。
+{{< hint info >}}
+Use this connector to read from several Kafka clusters, or to follow topics
that move between clusters
+while the job runs, for example during a cluster migration. Both need a Java
class that implements
+[`KafkaMetadataService`]({{< ref "docs/connectors/datastream/dynamic-kafka"
>}}#kafka-metadata-service) and maps stream ids to clusters and topics.
+The built-in `single-cluster` service reads from one fixed cluster only.
Otherwise, use the
+[Kafka connector]({{< ref "docs/connectors/table/kafka" >}}).
+{{< /hint >}}
+
依赖
----
diff --git a/docs/content.zh/docs/connectors/table/kafka.md
b/docs/content.zh/docs/connectors/table/kafka.md
index 808fc31d..3685d06f 100644
--- a/docs/content.zh/docs/connectors/table/kafka.md
+++ b/docs/content.zh/docs/connectors/table/kafka.md
@@ -31,6 +31,22 @@ under the License.
Kafka 连接器提供从 Kafka topic 中消费和写入数据的能力。
+Which Kafka connector to use
+----------------------------
+
+Three SQL connectors work with Kafka:
+
+- `kafka` (this page) reads and writes records as they are. Each record is an
insert, unless the
+ topic holds change events in a [CDC format]({{< ref
"docs/connectors/table/formats/overview" >}}) such as Debezium. Use this
connector by
+ default.
+- [`upsert-kafka`]({{< ref "docs/connectors/table/upsert-kafka" >}}) treats a
topic as a table with a primary key: each record is the latest
+ value for its key, and a record with a null value deletes the key. Use it to
read compacted
+ topics, or to write results that are updated over time, such as aggregations.
+- [`dynamic-kafka`]({{< ref "docs/connectors/table/dynamic-kafka" >}}) reads
topics from one or more Kafka clusters, resolved through a
+ Kafka metadata service. Use it to read from several clusters at once, or to
follow topics that
+ move between clusters while the job runs, for example during a cluster
migration. Both need your
+ own Java [`KafkaMetadataService`]({{< ref
"docs/connectors/datastream/dynamic-kafka" >}}#kafka-metadata-service)
implementation.
+
依赖
------------
@@ -547,6 +563,7 @@ Flink 提供了几种 CDC 格式：
* [debezium]({{< ref "docs/connectors/table/formats/debezium.md" >}})
* [canal]({{< ref "docs/connectors/table/formats/canal.md" >}})
* [maxwell]({{< ref "docs/connectors/table/formats/maxwell.md" >}})
+* [ogg]({{< ref "docs/connectors/table/formats/ogg.md" >}})
### Sink 分区
diff --git a/docs/content.zh/docs/connectors/table/upsert-kafka.md
b/docs/content.zh/docs/connectors/table/upsert-kafka.md
index 251b746c..91007216 100644
--- a/docs/content.zh/docs/connectors/table/upsert-kafka.md
+++ b/docs/content.zh/docs/connectors/table/upsert-kafka.md
@@ -35,6 +35,9 @@ Upsert Kafka 连接器支持以 upsert 方式从 Kafka topic 中读取数据并
作为 sink，upsert-kafka 连接器可以消费 changelog 流。它会将 INSERT/UPDATE_AFTER 数据作为正常的 Kafka
消息写入，并将 DELETE 数据以 value 为空的 Kafka 消息写入（表示对应 key 的消息被删除）。Flink
将根据主键列的值对数据进行分区，从而保证主键上的消息有序，因此同一主键上的更新/删除消息将落在同一分区中。
+Use the [Kafka connector]({{< ref "docs/connectors/table/kafka" >}}) for
topics where each record is an insert or a change event
+in a CDC format.
+
依赖
------------
diff --git a/docs/content/docs/connectors/datastream/dynamic-kafka.md
b/docs/content/docs/connectors/datastream/dynamic-kafka.md
index 6ef0454b..88b2908d 100644
--- a/docs/content/docs/connectors/datastream/dynamic-kafka.md
+++ b/docs/content/docs/connectors/datastream/dynamic-kafka.md
@@ -32,6 +32,12 @@ topics and/or clusters, without requiring a job restart.
This is especially usef
an existing Kafka cluster/topic (cluster migration/failover/other
infrastructure changes) and when you need direct integration with Hybrid
Source. The solution
makes these operations automated so that they are transparent to Kafka
consumers.
+{{< hint info >}}
+Use this connector to read one stream from several Kafka clusters, or when the
topics or clusters to
+read from change while the job runs, for example during a cluster migration.
Otherwise, use the
+[Kafka connector]({{< ref "docs/connectors/datastream/kafka" >}}).
+{{< /hint >}}
+
## Dependency
For details on Kafka compatibility, please refer to the official [Kafka
documentation](https://kafka.apache.org/protocol.html#protocol_compatibility).
diff --git a/docs/content/docs/connectors/table/dynamic-kafka.md
b/docs/content/docs/connectors/table/dynamic-kafka.md
index cd9a5e2a..14976c42 100644
--- a/docs/content/docs/connectors/table/dynamic-kafka.md
+++ b/docs/content/docs/connectors/table/dynamic-kafka.md
@@ -24,7 +24,7 @@ specific language governing permissions and limitations
under the License.
-->
-# Dynamic Kafka SQL Connector
+# Dynamic Kafka SQL Connector _`Experimental`_
{{< label "Scan Source: Unbounded" >}}
@@ -32,6 +32,14 @@ The Dynamic Kafka connector allows for reading data from
Kafka topics that can m
clusters without restarting the job. Streams are resolved via a Kafka metadata
service. This is
especially useful for cluster migrations and dynamic topic/cluster changes.
+{{< hint info >}}
+Use this connector to read from several Kafka clusters, or to follow topics
that move between clusters
+while the job runs, for example during a cluster migration. Both need a Java
class that implements
+[`KafkaMetadataService`]({{< ref "docs/connectors/datastream/dynamic-kafka"
>}}#kafka-metadata-service) and maps stream ids to clusters and topics.
+The built-in `single-cluster` service reads from one fixed cluster only.
Otherwise, use the
+[Kafka connector]({{< ref "docs/connectors/table/kafka" >}}).
+{{< /hint >}}
+
Dependencies
------------
diff --git a/docs/content/docs/connectors/table/kafka.md
b/docs/content/docs/connectors/table/kafka.md
index 51e61a6a..f7612fed 100644
--- a/docs/content/docs/connectors/table/kafka.md
+++ b/docs/content/docs/connectors/table/kafka.md
@@ -32,6 +32,22 @@ under the License.
The Kafka connector allows for reading data from and writing data into Kafka
topics.
+Which Kafka connector to use
+----------------------------
+
+Three SQL connectors work with Kafka:
+
+- `kafka` (this page) reads and writes records as they are. Each record is an
insert, unless the
+ topic holds change events in a [CDC format]({{< ref
"docs/connectors/table/formats/overview" >}}) such as Debezium. Use this
connector by
+ default.
+- [`upsert-kafka`]({{< ref "docs/connectors/table/upsert-kafka" >}}) treats a
topic as a table with a primary key: each record is the latest
+ value for its key, and a record with a null value deletes the key. Use it to
read compacted
+ topics, or to write results that are updated over time, such as aggregations.
+- [`dynamic-kafka`]({{< ref "docs/connectors/table/dynamic-kafka" >}}) reads
topics from one or more Kafka clusters, resolved through a
+ Kafka metadata service. Use it to read from several clusters at once, or to
follow topics that
+ move between clusters while the job runs, for example during a cluster
migration. Both need your
+ own Java [`KafkaMetadataService`]({{< ref
"docs/connectors/datastream/dynamic-kafka" >}}#kafka-metadata-service)
implementation.
+
Dependencies
------------
@@ -595,7 +611,7 @@ e.g. an option value
`partition:0,offset:42;partition:1,offset:300` indicates of
### CDC Changelog Source
-Flink natively supports Kafka as a CDC changelog source. If messages in a
Kafka topic are change event captured from other databases using a CDC tool,
you can use the corresponding Flink CDC format to interpret the messages as
INSERT/UPDATE/DELETE statements into a Flink SQL table.
+Flink natively supports Kafka as a CDC changelog source. If messages in a
Kafka topic are change event captured from other databases using a CDC tool,
you can use the corresponding CDC format to interpret the messages as
INSERT/UPDATE/DELETE statements into a Flink SQL table.
The changelog source is a very useful feature in many cases, such as
synchronizing incremental data from databases to other systems, auditing logs,
materialized views on databases, temporal join changing history of a database
table and so on.
@@ -604,6 +620,7 @@ Flink provides several CDC formats:
* [debezium]({{< ref "docs/connectors/table/formats/debezium.md" >}})
* [canal]({{< ref "docs/connectors/table/formats/canal.md" >}})
* [maxwell]({{< ref "docs/connectors/table/formats/maxwell.md" >}})
+* [ogg]({{< ref "docs/connectors/table/formats/ogg.md" >}})
### Sink Partitioning
diff --git a/docs/content/docs/connectors/table/upsert-kafka.md
b/docs/content/docs/connectors/table/upsert-kafka.md
index 974b242d..0ae447f5 100644
--- a/docs/content/docs/connectors/table/upsert-kafka.md
+++ b/docs/content/docs/connectors/table/upsert-kafka.md
@@ -44,6 +44,9 @@ data as normal Kafka messages value, and write DELETE data as
Kafka messages wit
partition data on the values of the primary key columns, so the
update/deletion messages on the same
key will fall into the same partition.
+Use the [Kafka connector]({{< ref "docs/connectors/table/kafka" >}}) for
topics where each record is an insert or a change event
+in a CDC format.
+
Dependencies
------------
