This is an automated email from the ASF dual-hosted git repository.
JNSimba pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 3aae51801ee [doc](flink) document Binlog source semantics and retry
configuration (#4135)
3aae51801ee is described below
commit 3aae51801eec510f950c2c467803dd3b2338a853
Author: wudi <[email protected]>
AuthorDate: Mon Sep 14 14:55:30 2026 +0800
[doc](flink) document Binlog source semantics and retry configuration
(#4135)
## Summary
Clarifies that `from-timestamp` includes the specified start time.
Documents how Binlog reads wait for in-flight transactions and how the
Connector retries visibility wait timeouts using
`source.binlog.visible-wait-timeout` (default `5m`). Explains the
source's at-least-once delivery and Flink's optional CDC event
deduplication setting, with a primary key in the source-table example.
## Versions
- [x] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
The 4.x connector page has no Binlog Source section; these options are
documented only on the current page.
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [ ] Test Cases Built (documentation-only change)
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
---
.../data-integration/flink-doris-connector.md | 17 ++++++++++++++---
.../data-integration/flink-doris-connector.md | 17 ++++++++++++++---
2 files changed, 28 insertions(+), 6 deletions(-)
diff --git
a/docs/connection-integration/data-integration/flink-doris-connector.md
b/docs/connection-integration/data-integration/flink-doris-connector.md
index 078ab96ffaa..b8f6665fc2c 100644
--- a/docs/connection-integration/data-integration/flink-doris-connector.md
+++ b/docs/connection-integration/data-integration/flink-doris-connector.md
@@ -284,7 +284,8 @@ SET 'execution.checkpointing.interval' = '10s';
CREATE TABLE student_binlog (
id INT,
name STRING,
- age INT
+ age INT,
+ PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'doris',
'fenodes' = '127.0.0.1:8030',
@@ -304,7 +305,7 @@ After the job starts, changes to
`test.student_binlog_source` are continuously e
| `snapshot` | Reads the current snapshot and stops. This is the default mode.
|
| `initial` | Reads the current snapshot and switches to continuous Binlog
reading when the snapshot is complete. |
| `latest` | Skips the snapshot and reads changes generated after the job
starts. |
-| `from-timestamp` | Skips the snapshot and reads changes after the exclusive
start time specified by `source.scan.timestamp` in `yyyy-MM-dd HH:mm:ss`
format. |
+| `from-timestamp` | Skips the snapshot and reads changes starting at the
inclusive time specified by `source.scan.timestamp` in `yyyy-MM-dd HH:mm:ss`
format. |
By default, the Connector emits full row changes in `detail` mode. Set
`source.binlog.increment-type` to `min_delta` for the minimal change set or
`append_only` for append events only.
@@ -313,6 +314,15 @@ Note the following:
- Incremental reading uses Arrow Flight SQL. The Connector enables it by
default and automatically discovers its port.
- Enable Flink Checkpoint.
- Configure Doris Binlog retention to cover the maximum expected job downtime.
If the required Binlog data has expired, restart from a new snapshot or specify
a new start time.
+- When performing a Binlog incremental read, Doris waits for in-flight
transactions affecting the source table within the read window to complete. If
the wait times out, the read returns an error. The Connector retries the same
window only for this error, for up to `source.binlog.visible-wait-timeout`
(default: `5m`). Set it to `0s` to disable Connector retries; other errors fail
immediately.
+
+The Doris Binlog Source currently provides at-least-once delivery, so change
events can be replayed after a failure and may affect Flink query results.
Flink's CDC event deduplication is disabled by default. To use it, set the
following option before submitting the query:
+
+```sql
+SET 'table.exec.source.cdc-events-duplicate' = 'true';
+```
+
+When this option is enabled, the source table must declare a primary key, as
in the example above. Flink uses an additional stateful operator to normalize
the changelog. See the [Flink configuration
reference](https://nightlies.apache.org/flink/flink-docs-release-2.3/zh/docs/dev/table/config/#table-exec-source-cdc-events-duplicate)
for details.
##### Publishing Consumption Progress to Doris (Optional)
@@ -902,9 +912,10 @@ After the Flink cluster is started, you can run the
corresponding command accord
| source.use-flight-sql | TRUE | N | Whether to use
Arrow Flight SQL for reading
|
| source.flight-sql-port | - | N | When using Arrow
Flight SQL for reading, the FE's `arrow_flight_sql_port`
|
| source.scan.mode | snapshot | N | Source startup
mode. Supported values: `snapshot`, `initial`, `latest`, and `from-timestamp`.
|
-| source.scan.timestamp | -- | N | Exclusive start
time in `yyyy-MM-dd HH:mm:ss` format. Required only for `from-timestamp`.
|
+| source.scan.timestamp | -- | N | Inclusive start
time in `yyyy-MM-dd HH:mm:ss` format. Required only for `from-timestamp`.
|
| source.binlog.increment-type | detail | N | Binlog change type:
`detail`, `min_delta`, or `append_only`.
|
| source.binlog.poll-interval | 10s | N | Interval for
polling new Binlog data. The minimum value is 1 second.
|
+| source.binlog.visible-wait-timeout | 5m | N | Maximum time
for the Connector to retry the same read window after Doris returns a
transaction visibility wait timeout error. Set to `0s` to disable retries;
negative values are invalid. |
| source.binlog.offset-table | -- | N | Doris table in
`database.table` format used to publish offsets covered by completed
Checkpoints. Configure with `source.binlog.consumer-id` and `jdbc-url`. |
| source.binlog.consumer-id | -- | N | Stable consumer
identifier written to `source.binlog.offset-table`.
|
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector.md
index 92b7f63493f..64052275ce1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector.md
@@ -284,7 +284,8 @@ SET 'execution.checkpointing.interval' = '10s';
CREATE TABLE student_binlog (
id INT,
name STRING,
- age INT
+ age INT,
+ PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'doris',
'fenodes' = '127.0.0.1:8030',
@@ -304,7 +305,7 @@ SELECT * FROM student_binlog;
| `snapshot` | 读取当前快照后结束，为默认模式。 |
| `initial` | 先读取当前快照，快照读取完成后切换到持续 Binlog 读取。 |
| `latest` | 跳过快照，只读取任务启动后产生的变更。 |
-| `from-timestamp` | 跳过快照，读取 `source.scan.timestamp` 指定时间之后（不含该时间点）的变更，时间格式为
`yyyy-MM-dd HH:mm:ss`。 |
+| `from-timestamp` | 跳过快照，从 `source.scan.timestamp` 指定的时间点开始读取变更（包含该时间点），时间格式为
`yyyy-MM-dd HH:mm:ss`。 |
默认以 `detail` 类型输出完整的行变更。也可以通过 `source.binlog.increment-type` 设置为
`min_delta`（最小变更集）或 `append_only`（仅追加事件）。
@@ -313,6 +314,15 @@ SELECT * FROM student_binlog;
- 增量读取使用 Arrow Flight SQL，Connector 默认启用并自动获取端口。
- 需要启用 Flink Checkpoint。
- Doris Binlog 的保留时间应覆盖任务可能停止的最长时间。如果恢复所需的 Binlog 已过期，需要重新读取快照或指定新的起始时间。
+- 执行 Binlog 增量读取时，如果读取区间内存在影响源表的未完成事务，Doris 会等待这些事务完成。若等待超时，本次读取会报错。Connector
仅针对该错误重试同一读取区间，重试时长由 `source.binlog.visible-wait-timeout` 控制（默认 `5m`）；设置为 `0s`
可关闭 Connector 重试，其他错误会立即失败。
+
+当前 Doris Binlog Source 只保证至少一次交付，任务故障恢复后可能重放变更事件，影响 Flink 查询结果。Flink 的 CDC
事件去重配置默认关闭；如需启用，可在提交查询前设置：
+
+```sql
+SET 'table.exec.source.cdc-events-duplicate' = 'true';
+```
+
+启用该配置时，源表必须像上面的示例一样声明主键。Flink 会增加一个有状态算子来规范化变更流。详情参见 [Flink
配置文档](https://nightlies.apache.org/flink/flink-docs-release-2.3/zh/docs/dev/table/config/#table-exec-source-cdc-events-duplicate)。
##### 将消费进度写入 Doris（可选）
@@ -902,9 +912,10 @@ Flink Doris Connector 集成了 [Flink
CDC](https://nightlies.apache.org/flink/f
| source.use-flight-sql | TRUE | N | 是否使用 Arrow Flight
SQL 读取
|
| source.flight-sql-port | - | N | 使用 Arrow Flight SQL
读取时，FE 的 `arrow_flight_sql_port`
|
| source.scan.mode | snapshot | N | Source 启动模式，支持
`snapshot`、`initial`、`latest` 和 `from-timestamp`
|
-| source.scan.timestamp | -- | N | `from-timestamp`
模式的开区间起始时间，格式为 `yyyy-MM-dd HH:mm:ss`
|
+| source.scan.timestamp | -- | N | `from-timestamp`
模式的起始时间（含该时间点），格式为 `yyyy-MM-dd HH:mm:ss`
|
| source.binlog.increment-type | detail | N | Binlog 变更类型，支持
`detail`、`min_delta` 和 `append_only`
|
| source.binlog.poll-interval | 10s | N | 轮询新 Binlog
数据的时间间隔，最小值为 1 秒
|
+| source.binlog.visible-wait-timeout | 5m | N | Doris
返回事务可见性等待超时错误后，Connector 重试同一读取区间的最长时间。设置为 `0s` 可关闭重试；不能为负值。 |
| source.binlog.offset-table | -- | N | 用于发布成功 Checkpoint
所覆盖 offset 的 Doris 表，格式为 `database.table`。需要同时配置 `source.binlog.consumer-id` 和
`jdbc-url` |
| source.binlog.consumer-id | -- | N | 写入
`source.binlog.offset-table` 的稳定消费者标识
|
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
