This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 8ed6d3c4efc [doc](binlog) Document ROW binlog TTL and consumer
recovery (#4205)
8ed6d3c4efc is described below
commit 8ed6d3c4efcc72a22ba369dbb1aa71b990b7ec1b
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 10 19:25:31 2026 +0800
[doc](binlog) Document ROW binlog TTL and consumer recovery (#4205)
---
.../flink-doris-connector/incremental-read.md | 33 +++++-
docs/data-operate/incremental/incremental-query.md | 8 +-
docs/data-operate/incremental/overview.md | 2 +-
docs/data-operate/incremental/row-binlog.md | 122 +++++++++++++++++++--
.../incremental/table-stream-advanced.md | 5 +-
docs/data-operate/incremental/table-stream.md | 2 +-
.../flink-doris-connector/incremental-read.md | 33 +++++-
.../data-operate/incremental/incremental-query.md | 8 +-
.../current/data-operate/incremental/overview.md | 2 +-
.../current/data-operate/incremental/row-binlog.md | 122 +++++++++++++++++++--
.../incremental/table-stream-advanced.md | 5 +-
.../data-operate/incremental/table-stream.md | 2 +-
12 files changed, 304 insertions(+), 40 deletions(-)
diff --git
a/docs/connection-integration/data-integration/flink-doris-connector/incremental-read.md
b/docs/connection-integration/data-integration/flink-doris-connector/incremental-read.md
index 9ea08eb8d40..74037898612 100644
---
a/docs/connection-integration/data-integration/flink-doris-connector/incremental-read.md
+++
b/docs/connection-integration/data-integration/flink-doris-connector/incremental-read.md
@@ -8,7 +8,7 @@
# Incremental Reading with Doris Binlog
-For a Doris table with ROW-format Binlog enabled, Flink Doris Connector can
continuously read row-level changes. In `initial` mode, the Connector first
reads the current table snapshot and then seamlessly switches to incremental
reading.
+For a Doris table with ROW-format Binlog enabled, Flink Doris Connector can
continuously read row-level changes. In `initial` mode, the Connector first
reads the current table snapshot and then switches to incremental reading.
Changes needed for this transition must remain within the source table's Binlog
retention window.
:::info Version requirements
This feature requires Flink Doris Connector 26.3.0 or later and Doris 5.0.0 or
later. Row Binlog is an experimental feature of Doris 5.0.0 and must be enabled
with `enable_feature_binlog = true` in the FE configuration. For how to enable
it on a table, the supported table models, and its limitations, see [Row
Binlog](../../../data-operate/incremental/row-binlog.md).
@@ -72,20 +72,39 @@ After the job starts, changes to
`test.student_binlog_source` are continuously e
| `latest` | Skips the snapshot and reads changes generated after the job
starts. |
| `from-timestamp` | Skips the snapshot and reads changes starting at the
inclusive time specified by `source.scan.timestamp` in `yyyy-MM-dd HH:mm:ss`
format. |
+When restoring from a Flink Checkpoint or Savepoint, the Connector resumes the
saved consumption state. Changing `source.scan.mode` or `source.scan.timestamp`
does not reset that state. Both a restored offset and a `from-timestamp` start
are subject to the source table's `binlog.ttl_seconds`; see the expiration
behavior below.
+
## Change Types {#increment-type}
By default, the Connector emits full row changes in `detail` mode. Set
`source.binlog.increment-type` to `min_delta` for the minimal change set or
`append_only` for append events only.
+The Connector reads changes through Doris `@incr` queries. If the requested
start has expired according to `binlog.ttl_seconds`, the behavior depends on
the change type:
+
+| Change type | Behavior when the start has expired |
+| ---- | ---- |
+| `detail` (default), `append_only` | Doris moves the effective start to the
first retained TSO and skips expired changes. The job can continue without an
expiration error, but those changes are missing from its output. |
+| `min_delta` | The read fails with `Row binlog offset has expired according
to binlog.ttl_seconds`, rather than returning incomplete net changes. |
+
+These rules apply during normal consumption, after the initial snapshot, and
when restoring from saved state. A running job or a successful Checkpoint does
not prove that no changes have expired. `source.binlog.visible-wait-timeout`
does not retry expiration errors.
+
## Notes
- Incremental reading uses Arrow Flight SQL. The Connector enables it by
default and automatically discovers its port.
- Enable Flink Checkpoint.
-- Configure Doris Binlog retention to cover the maximum expected job downtime.
If the required Binlog data has expired, restart from a new snapshot or specify
a new start time.
+- Doris `binlog.ttl_seconds` defaults to `86400` seconds (one day) and must be
greater than `0`. Expiration does not wait for the Flink job or its
Checkpoints. Increasing the TTL cannot restore physically cleaned-up records.
+- Set the TTL to cover the initial snapshot duration, consumption backlog, and
the time needed to recover from an outage and catch up, with additional margin.
Monitor the age of the oldest changes still needed by the job, including when
restoring an older Checkpoint or Savepoint.
+- For automatic physical cleanup, also enable `enable_feature_binlog = true`
on the BEs and keep automatic compaction enabled. Incremental queries filter
expired records even before physical cleanup finishes. See [Row Binlog
retention and
cleanup](../../../data-operate/incremental/row-binlog.md#retention-and-cleanup).
- When performing a Binlog incremental read, Doris waits for in-flight
transactions affecting the source table within the read window to complete. If
the wait times out, the read returns an error. The Connector retries the same
window only for this error, for up to `source.binlog.visible-wait-timeout`
(default: `5m`). Set it to `0s` to disable Connector retries; other errors fail
immediately.
+### Recovery after Binlog expiration {#binlog-expiration-recovery}
+
+If required changes have expired, the existing offset cannot provide complete
incremental recovery. For a job that must keep downstream data consistent with
the current source table, start a new bootstrap with `source.scan.mode =
'initial'`, without restoring the expired Checkpoint or Savepoint, and rebuild
downstream data from the new snapshot. Restoring old state while setting
`initial` still resumes the saved state instead of starting a new full snapshot.
+
+A new snapshot rebuilds the current table state; it cannot reproduce expired
historical events. Starting with `latest` or a newer `from-timestamp` also
skips the missing interval. Use those modes only when skipping that interval is
acceptable or the missing data has been recovered separately.
+
## Delivery Semantics and Deduplication {#delivery}
-The Doris Binlog Source currently provides at-least-once delivery, so change
events can be replayed after a failure and may affect Flink query results.
Flink's CDC event deduplication is disabled by default. To use it, set the
following option before submitting the query:
+The Doris Binlog Source provides at-least-once delivery while all required
change records are still retained in Doris. Change events can be replayed after
a failure and may affect Flink query results. Flink's CDC event deduplication
is disabled by default. To use it, set the following option before submitting
the query:
```sql
SET 'table.exec.source.cdc-events-duplicate' = 'true';
@@ -93,6 +112,8 @@ SET 'table.exec.source.cdc-events-duplicate' = 'true';
When this option is enabled, the source table must declare a primary key, as
in the example above. Flink uses an additional stateful operator to normalize
the changelog. See the [Flink configuration
reference](https://nightlies.apache.org/flink/flink-docs-release-2.3/zh/docs/dev/table/config/#table-exec-source-cdc-events-duplicate)
for details.
+Checkpoints save consumption state, and CDC deduplication handles replayed
events. Neither prevents Doris Binlog expiration nor recovers events that have
already been lost to expiration.
+
## Publishing Consumption Progress to Doris (Optional) {#offset-table}
Consumption progress is stored in Flink Checkpoint state. The optional offset
table is only for observing progress covered by completed Checkpoints. The
Connector does not read this table to automatically resume consumption when a
job starts or restarts.
@@ -122,7 +143,7 @@ PROPERTIES (
`source.binlog.consumer-id` identifies the consumer job in the offset table.
Keep it unchanged when the same job restarts to track its progress.
-If you cannot restore from a Flink Checkpoint or Savepoint, copy the
`offset_timestamp` from the offset table into `source.scan.timestamp` and set
`source.scan.mode` to `from-timestamp` to manually resume from that offset. For
example:
+If you cannot restore from a Flink Checkpoint or Savepoint and all changes
needed from the saved `offset_timestamp` are still retained, copy it into
`source.scan.timestamp` and set `source.scan.mode` to `from-timestamp` to
manually resume from that offset. If it has expired, follow [Recovery after
Binlog expiration](#binlog-expiration-recovery); copying an expired timestamp
does not restore the missing changes. For example, if the following timestamp
is still within the retention window:
```sql
'source.scan.mode' = 'from-timestamp',
@@ -136,8 +157,8 @@ The following options control incremental reading. The
other Source options are
| Key | Default Value | Required | Comment
|
| --------------------------- | ------------- | -------- |
------------------------------------------------------------------------------------------------------------------------------------------------------
|
| source.scan.mode | snapshot | N | Source startup
mode. Supported values: `snapshot`, `initial`, `latest`, and `from-timestamp`.
|
-| source.scan.timestamp | -- | N | Inclusive start
time in `yyyy-MM-dd HH:mm:ss` format. Required only for `from-timestamp`.
|
-| source.binlog.increment-type | detail | N | Binlog change type:
`detail`, `min_delta`, or `append_only`.
|
+| source.scan.timestamp | -- | N | Inclusive start
time in `yyyy-MM-dd HH:mm:ss` format. Required only for `from-timestamp`. An
expired start follows the behavior in [Change Types](#increment-type); it
cannot recover expired changes. |
+| source.binlog.increment-type | detail | N | Binlog change type:
`detail`, `min_delta`, or `append_only`. On an expired start, `detail` /
`append_only` skip expired changes and `min_delta` fails. |
| source.binlog.poll-interval | 10s | N | Interval for
polling new Binlog data. The minimum value is 1 second.
|
| source.binlog.visible-wait-timeout | 5m | N | Maximum time
for the Connector to retry the same read window after Doris returns a
transaction visibility wait timeout error. Set to `0s` to disable retries;
negative values are invalid. |
| source.binlog.offset-table | -- | N | Doris table in
`database.table` format used to publish offsets covered by completed
Checkpoints for observation only, not for automatic recovery. Configure with
`source.binlog.consumer-id` and `jdbc-url`. |
diff --git a/docs/data-operate/incremental/incremental-query.md
b/docs/data-operate/incremental/incremental-query.md
index 1ca8c354c40..53af40f4ddf 100644
--- a/docs/data-operate/incremental/incremental-query.md
+++ b/docs/data-operate/incremental/incremental-query.md
@@ -73,11 +73,11 @@ SELECT ... FROM <table_name>@incr(
| Parameter | Default | Description |
|---|---|---|
-| `startTimestamp` | unbounded | Optional. Start of the window, format
`yyyy-MM-dd HH:mm:ss`, parsed in the session `time_zone`. Changes whose commit
time is **greater than or equal to** the start are returned |
+| `startTimestamp` | first retained TSO | Optional. Start of the window,
format `yyyy-MM-dd HH:mm:ss`, parsed in the session `time_zone`. Changes whose
commit time is **greater than or equal to** the start are returned, subject to
the base table's `binlog.ttl_seconds` |
| `endTimestamp` | unbounded | Optional. End of the window, same format.
Changes whose commit time is **less than** the end are returned |
| `incrementType` | `MIN_DELTA` | Optional. Incremental mode, see [The three
incremental modes](#the-three-incremental-modes) |
-All three parameters can be omitted; `t@incr()` reads the entire change
history in `MIN_DELTA` mode. The window is a left-closed, right-open interval
`[startTimestamp, endTimestamp)`; a start later than the end, or a start in the
future, returns an empty result.
+All three parameters can be omitted; `t@incr()` reads retained changes in
`MIN_DELTA` mode. An explicit start outside the retention window fails in
`MIN_DELTA`; `DETAIL` and `APPEND_ONLY` move the effective start to the first
retained TSO. See [Retention and cleanup](row-binlog#retention-and-cleanup).
The window is a left-closed, right-open interval `[startTimestamp,
endTimestamp)`; a start later than the end, or a start in the future, returns
an empty result when the start passes retent [...]
### Result columns
@@ -206,6 +206,8 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__, op;
### No window: net changes since the table was created
+This example assumes that all changes since table creation are still within
the retention window. Otherwise, omitting the window reads only retained
changes.
+
```sql
SELECT order_id, status, amount, __DORIS_BINLOG_OP__ AS op
FROM orders@incr()
@@ -229,7 +231,7 @@ Counting from table creation, every key is absent at the
start, so every key tha
<!-- Knowledge type: Usage recommendations -->
<!-- Use cases: Periodic increments driven by an external scheduler -->
-- When an external scheduler drives the increments, use the previous
`endTimestamp` as the next `startTimestamp`; the left-closed, right-open window
guarantees no gaps and no duplicates. Note that windows are defined by commit
time: a transaction that is still running does not appear in the current window
and shows up in the window that contains its commit.
+- When an external scheduler drives the increments, use the previous
`endTimestamp` as the next `startTimestamp`; within the retention window, the
left-closed, right-open window guarantees no gaps and no duplicates. Keep
consumption delay below `binlog.ttl_seconds`. Note that windows are defined by
commit time: a transaction that is still running does not appear in the current
window and shows up in the window that contains its commit.
- If only the latest values matter and the values before an update are not
needed, keep only the rows with `__DORIS_BINLOG_OP__ IN (0, 1, 3)`.
- An `@incr` query reads all change records within the window; the larger the
window, the more it reads. Specify the window whenever possible.
diff --git a/docs/data-operate/incremental/overview.md
b/docs/data-operate/incremental/overview.md
index fbddde27312..747a4a91fe2 100644
--- a/docs/data-operate/incremental/overview.md
+++ b/docs/data-operate/incremental/overview.md
@@ -141,7 +141,7 @@ With Row Binlog enabled, every write additionally generates
and persists change
| Creating a Stream fails with `Table Stream is experimental. Please set
enable_table_stream=true to enable it.` | `enable_table_stream` is not enabled
on the FE; update `fe.conf` and restart the FE |
| Does it work in the compute-storage decoupled mode? | Row Binlog and Table
Stream do; use `@incr` in the integrated storage-compute mode for now |
| How does Row Binlog affect writes? | Every write additionally generates and
persists change records, and MoW tables also read the old values, so load
throughput drops noticeably. Enable it only on tables that need incremental
consumption and evaluate with a realistic workload |
-| Are change records cleaned up automatically? | Not in the current version;
reserve extra storage for tables with Row Binlog, see [Retention and
cleanup](row-binlog#retention-and-cleanup) |
+| Are change records cleaned up automatically? | Yes. `binlog.ttl_seconds`
controls retention (one day by default); incremental reads filter expired
records, and background compaction reclaims their storage. Consume changes
within the retention window; see [Retention and
cleanup](row-binlog#retention-and-cleanup) |
## Reading guide
diff --git a/docs/data-operate/incremental/row-binlog.md
b/docs/data-operate/incremental/row-binlog.md
index d8c72774122..be8729ca1fa 100644
--- a/docs/data-operate/incremental/row-binlog.md
+++ b/docs/data-operate/incremental/row-binlog.md
@@ -2,7 +2,7 @@
{
"title": "Row Binlog",
"language": "en",
- "description": "Doris Row Binlog records row-level inserts, updates and
deletes: enabling it at table creation, binlog.* properties, supported table
models, DDL limits, errors.",
+ "description": "Doris Row Binlog records row-level inserts, updates and
deletes: table properties, TTL retention, automatic cleanup, supported table
models and troubleshooting.",
"keywords": [
"Row Binlog",
"row-level binlog",
@@ -10,6 +10,8 @@
"binlog.enable",
"binlog.format ROW",
"binlog.need_historical_value",
+ "binlog.ttl_seconds",
+ "Row Binlog TTL",
"before image",
"historical value",
"commit TSO",
@@ -35,6 +37,8 @@
Row Binlog is the row-level change log of Doris internal tables. Once enabled,
every row-level change produced by a write (insert, update, delete) is
persisted together with the values before and after the change and the commit
timestamp. It is the data source of [Table Stream](table-stream), [Incremental
Query](incremental-query) and [Incremental View Maintenance
(IVM)](../../query-acceleration/materialized-view/async-materialized-view/incremental-materialized-view),
and can also be con [...]
+Change records are retained for one day by default. Set `binlog.ttl_seconds`
to cover the longest expected consumption delay; expiring change records does
not delete data from the base table.
+
:::caution Experimental feature
This feature is available since version 5.0.0 and is experimental. It requires
`enable_feature_binlog = true` in the FE configuration.
:::
@@ -45,6 +49,7 @@ This feature is available since version 5.0.0 and is
experimental. It requires `
- Doris 5.0.0 or later.
- `enable_feature_binlog = true` in `fe.conf` on the FE (not a dynamic
configuration; the FE must be restarted).
+- For automatic cleanup, also set `enable_feature_binlog = true` in `be.conf`
on the BEs (not a dynamic configuration; the BEs must be restarted).
- A Duplicate Key table, or a Unique Key Merge-on-Write (MoW) table without
cluster keys; see [Supported scope and
limitations](#supported-scope-and-limitations).
- Row Binlog can only be enabled at table creation, so evaluate it before
creating the table.
@@ -76,7 +81,8 @@ PROPERTIES (
"enable_unique_key_merge_on_write" = "true",
"binlog.enable" = "true",
"binlog.format" = "ROW",
- "binlog.need_historical_value" = "true"
+ "binlog.need_historical_value" = "true",
+ "binlog.ttl_seconds" = "86400"
);
```
@@ -89,7 +95,7 @@ PROPERTIES (
| `binlog.enable` | `true` / `false` | `false` | Cannot be disabled once
enabled | Whether binlog is enabled; must be set together with `binlog.format =
"ROW"` |
| `binlog.format` | `ROW` | - | No | Must be `ROW`, meaning row-level changes
are recorded. The value is case-sensitive; a lowercase `row` fails with
`Invalid binlog format value: row` |
| `binlog.need_historical_value` | `true` / `false` | `false` | No | Whether
the values before a change (before image) are recorded. Only Unique Key MoW
tables can set it to `true`. `min_delta` / `detail` Table Streams, `MIN_DELTA`
incremental queries, and IVMs that need to handle updates or deletes depend on
it |
-| `binlog.ttl_seconds` | integer (seconds) | `86400` | Yes | Retention period.
**Has no effect in the current version**, see [Retention and
cleanup](#retention-and-cleanup) |
+| `binlog.ttl_seconds` | positive integer (seconds) | `86400` (one day) | Yes
| Retention period for change records. Expired records are filtered from
incremental reads and reclaimed through background compaction; `0` and negative
values are not allowed. See [Retention and cleanup](#retention-and-cleanup) |
| `binlog.max_bytes` | integer (bytes) | unlimited | Yes | Retention size
limit. **Has no effect in the current version** |
| `binlog.max_history_nums` | integer | unlimited | Yes | Retention count
limit. **Has no effect in the current version** |
@@ -258,11 +264,111 @@ Table-level operations such as `TRUNCATE TABLE` and
backup / restore work as usu
## Retention and cleanup
<!-- Knowledge type: Behavior rules -->
-<!-- Use cases: Capacity planning -->
+<!-- Use cases: Setting retention / Handling expired offsets / Capacity
planning -->
+
+Since Doris 5.0.0, `binlog.ttl_seconds` controls Row Binlog retention in both
the integrated storage-compute mode and the compute-storage decoupled mode.
Retention is based on the physical time in the commit TSO, using the current
Master FE TSO as the reference. Records whose commit time is at or before the
reference time minus the TTL are expired. The retention window advances even
when the table receives no new writes.
+
+### Configuring retention
+
+The default TTL is `86400` seconds (one day). When a database explicitly sets
`binlog.ttl_seconds`, new tables inherit it unless their table properties
override it. Changing the database setting does not change existing tables.
`CREATE TABLE ... LIKE` copies the source table's TTL.
+
+To retain changes for seven days on the `orders` table created above:
+
+```sql
+ALTER TABLE orders SET ("binlog.ttl_seconds" = "604800");
+```
+
+After the statement succeeds, `SHOW CREATE TABLE orders` includes:
+
+```text
+"binlog.ttl_seconds" = "604800"
+```
+
+The TTL must be greater than `0`; neither `0` nor `-1` disables expiration.
Shortening the TTL can immediately expire existing records. Extending it can
make records that are still physically present readable again, but cannot
recover records that have already been cleaned up. If an `ALTER TABLE` fails
while publishing the setting to partitions, a shorter TTL may already have
taken effect; check `SHOW CREATE TABLE` and retry the same statement to
complete the update.
+
+### Effect on reads
+
+Incremental reads apply the retention window when the statement is planned,
before physical cleanup finishes. A statement uses one reference TSO, and each
new statement recalculates the window.
+
+| Read method | Behavior when the requested start is expired |
+|---|---|
+| `@incr` without `startTimestamp` | Reads only retained changes, including in
`MIN_DELTA` mode |
+| `@incr` with `incrementType = MIN_DELTA` and an explicit `startTimestamp` |
Fails with `Row binlog offset has expired according to binlog.ttl_seconds` to
avoid returning incomplete net changes |
+| `@incr` with `incrementType = DETAIL` or `APPEND_ONLY` | Moves the effective
start to the first retained TSO and returns only the retained part of the
requested window |
+| Table Stream with `type = min_delta` | Fails with the same error if a
partition's consumption offset is expired |
+| Table Stream with `type = detail` or `append_only` | Skips expired changes
and reads from the first retained TSO |
+| MoW snapshot reads (`FOR TIME AS OF`, `FOR VERSION AS OF`, or a Stream's
`@snapshot()`) | When reconstruction requires Row Binlog, an expired target
snapshot fails with the same error because its before images are no longer
available in the retention window |
+| `binlog()` table function | Does not apply the TTL filter; it can still show
expired raw records until physical cleanup removes them |
+
+Expiration does not wait for consumers. Consume changes within the retention
window, even if a Stream is still reported as usable by `IS_STALE` /
`STALE_REASON`: these fields do not report TTL expiration in the current
implementation. After required changes have expired, rebuild downstream data
from a full snapshot. For a Table Stream, use the [reset
procedure](table-stream-advanced#reset-reset) to reload downstream and advance
the offsets. Manage IVM's internal Streams through [material [...]
+
+### Example: expired changes with base data preserved
+
+Run this example in an existing database with `CREATE` and `LOAD` privileges,
and `SELECT` privilege for the queries:
+
+```sql
+CREATE TABLE row_binlog_ttl_demo (id INT, value INT)
+DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW",
+ "binlog.ttl_seconds" = "30"
+);
+
+INSERT INTO row_binlog_ttl_demo VALUES (1, 10);
+
+SELECT COUNT(*) AS change_count
+FROM row_binlog_ttl_demo@incr("incrementType" = "DETAIL");
+```
+
+Immediately after the insert, the query returns:
+
+```text
++--------------+
+| change_count |
++--------------+
+| 1 |
++--------------+
+```
+
+Wait more than 30 seconds, then run the incremental query again:
+
+```sql
+SELECT COUNT(*) AS change_count
+FROM row_binlog_ttl_demo@incr("incrementType" = "DETAIL");
+```
+
+```text
++--------------+
+| change_count |
++--------------+
+| 0 |
++--------------+
+```
+
+The base row is still present:
+
+```sql
+SELECT id, value FROM row_binlog_ttl_demo;
+```
+
+```text
++------+-------+
+| id | value |
++------+-------+
+| 1 | 10 |
++------+-------+
+```
+
+### Physical cleanup and capacity planning
+
+Background compaction reclaims fully expired Row Binlog Rowsets, including on
idle tables. A Rowset that also contains unexpired records is retained until
all its records expire. Cleanup preserves base-table data and version
continuity. Storage is reclaimed asynchronously, so the TTL is not a deadline
for files to disappear or a strict storage size limit.
-The current version does not clean up Row Binlog data automatically: change
records are kept for the lifetime of the table, and the three properties
`binlog.ttl_seconds`, `binlog.max_bytes`, and `binlog.max_history_nums` can be
set but have no effect yet. Automatic cleanup by time and size is under
development and will be supported in the next version.
+Automatic cleanup requires `enable_feature_binlog = true` on the BEs and
automatic compaction to be enabled. Setting the BE configuration or table
property `disable_auto_compaction = true` pauses automatic cleanup, but
incremental reads still filter expired records. After a BE restart, cleanup
waits for a valid reference TSO from a Master FE heartbeat.
-Until then, reserve extra storage for tables with Row Binlog: the volume of
change records grows with the write volume, and for MoW tables with the before
image enabled, every update stores an additional copy of the old values.
+`binlog.max_bytes` and `binlog.max_history_nums` still have no effect on Row
Binlog. Reserve storage according to write volume, TTL and cleanup delay. For
MoW tables with the before image enabled, every update stores an additional
copy of the old values. Set the TTL longer than the maximum consumption delay,
refresh interval and expected outage, with time for recovery.
## Write overhead
@@ -305,7 +411,7 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__;
| `partition` | No | Partition names separated by commas; defaults to all
partitions |
| `tablet` | No | Tablet IDs separated by commas; defaults to all tablets |
-`binlog()` reads the stored raw records without any folding or filtering, and
its `__DORIS_BINLOG_OP__` uses the raw encoding (`0` insert, `1` update, `2`
delete). The full syntax is in [BINLOG table
function](../../sql-manual/sql-functions/table-valued-functions/binlog).
+`binlog()` reads the stored raw records without change folding or TTL
filtering, and its `__DORIS_BINLOG_OP__` uses the raw encoding (`0` insert, `1`
update, `2` delete). Expired records may remain visible here until physical
cleanup finishes. The full syntax is in [BINLOG table
function](../../sql-manual/sql-functions/table-valued-functions/binlog).
## Common errors
@@ -314,6 +420,8 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__;
| Error message | Cause | Action |
|---|---|---|
| `Invalid binlog format value: row` | `binlog.format` was given in lowercase
| The value is case-sensitive; use uppercase `ROW` |
+| `ROW binlog.ttl_seconds must be greater than 0` | The TTL is `0` or negative
| Set a positive integer in seconds; the default is `86400` |
+| `Row binlog offset has expired according to binlog.ttl_seconds` | A
`MIN_DELTA` start or a MoW snapshot is outside the retention window | Rebuild
downstream from a full snapshot and realign consumption; see [Effect on
reads](#effect-on-reads) |
| `not support change binlog format from STATEMENT_AND_SNAPSHOT to ROW` |
`ALTER TABLE` was used to enable Row Binlog on an existing table | Row Binlog
can only be enabled at table creation. Create a new table with Row Binlog
enabled, load the data, and swap the two with `ALTER TABLE ... REPLACE WITH
TABLE` |
| `can't disable binlog when format is [Row]` | `binlog.enable = false` was
set on a table with Row Binlog enabled | Row Binlog cannot be disabled once
enabled |
| `not support change binlog.need_historical_value from true to false` |
`binlog.need_historical_value` was modified | The property cannot be changed
after creation |
diff --git a/docs/data-operate/incremental/table-stream-advanced.md
b/docs/data-operate/incremental/table-stream-advanced.md
index 62c1fe27eac..404bb1bfce4 100644
--- a/docs/data-operate/incremental/table-stream-advanced.md
+++ b/docs/data-operate/incremental/table-stream-advanced.md
@@ -245,14 +245,15 @@ The internal Streams created by IVM also appear in these
system tables. You can
| Watch | Source | Notes |
|---|---|---|
| Backlog | `LAG` in `information_schema.table_stream_consumption` | Query it
regularly and alert on partitions where it stays non-zero or keeps growing |
-| State | `ENABLED`, `IS_STALE`, and `STALE_REASON` in
`information_schema.table_streams` | Show whether a Stream is usable. The
current version never cleans up Row Binlog, so a Stream does not become
unusable by being left unconsumed; once automatic cleanup is available, a
Stream whose change records were cleaned up will be marked stale and must be
re-aligned with `@reset()` |
+| State | `ENABLED`, `IS_STALE`, and `STALE_REASON` in
`information_schema.table_streams` | Show the Stream's state, but do not report
TTL expiration in the current implementation. Monitor consumption delay against
the base table's `binlog.ttl_seconds` and check for expired-offset errors; see
[Retention and cleanup](row-binlog#retention-and-cleanup) |
| Consumption history | `LAST_CONSUMPTION_TIME` in
`information_schema.table_stream_consumption` | Tells whether the consumption
job runs as scheduled |
### Recovery
| Situation | Action |
|---|---|
-| The consuming statement failed | The offset is unchanged; simply rerun it
and no change is skipped |
+| The consuming statement failed | The offset is unchanged; rerun it while the
required changes are still within the retention window |
+| Required changes have expired according to `binlog.ttl_seconds` | Reload
downstream in full with `@reset()` and realign the offsets. `min_delta` reads
fail on expired offsets; `detail` and `append_only` skip expired changes |
| Consumption succeeded but the scheduler did not record it | A rerun reads
the next batch of changes and does not consume the previous batch again. To
make repeated runs completely free of side effects, use a Unique Key model for
the target table (writes by primary key are naturally idempotent) |
| Downstream data is wrong and must be rebuilt | Restore the state
corresponding to the consumption offset with `@snapshot()` and continue
incremental consumption, or reload in full with `@reset()` which also advances
the offset to the current point |
| FE restart or master switch | Offsets are persisted in the metadata and
consumption continues after a restart |
diff --git a/docs/data-operate/incremental/table-stream.md
b/docs/data-operate/incremental/table-stream.md
index 06348a4dd46..684a94e4472 100644
--- a/docs/data-operate/incremental/table-stream.md
+++ b/docs/data-operate/incremental/table-stream.md
@@ -376,7 +376,7 @@ The columns of a Stream are the current visible columns of
the base table (synch
- A Stream reads only the base index of the base table; rollups / materialized
views cannot be specified, and the `TABLET (...)` clause is not supported.
- A Stream cannot be the target of `INSERT`, `UPDATE`, or `DELETE`.
- The consumption type and `show_initial_rows` cannot be changed after
creation.
-- The current version never cleans up Row Binlog data, so a Stream that is not
consumed for a long time loses no changes. Once automatic cleanup is available,
a Stream left unconsumed for too long may be unable to continue because its
change records were cleaned up; this will be flagged through `IS_STALE` /
`STALE_REASON` in `information_schema.table_streams`.
+- Row Binlog retention is controlled by `binlog.ttl_seconds` (one day by
default), regardless of consumption progress. An expired offset makes
`min_delta` reads fail; `detail` and `append_only` skip expired changes.
`IS_STALE` / `STALE_REASON` do not report TTL expiration in the current
implementation. Consume within the retention window; see [Retention and
cleanup](row-binlog#retention-and-cleanup).
- The effect of dropping the base table, dropping or replacing partitions, and
similar operations on a Stream is described in [Table Stream
Advanced](table-stream-advanced#effect-of-base-table-changes).
## FAQ
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector/incremental-read.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector/incremental-read.md
index 90e88bd993e..713843d5b6f 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector/incremental-read.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/connection-integration/data-integration/flink-doris-connector/incremental-read.md
@@ -8,7 +8,7 @@
# 增量读取 Doris Binlog
-对于使用 Doris 开启 ROW 格式 Binlog 的表，Flink Doris Connector 可以持续读取行级数据变更。使用 `initial`
模式时，Connector 会先读取当前表快照，再无缝切换到增量变更读取。
+对于使用 Doris 开启 ROW 格式 Binlog 的表，Flink Doris Connector 可以持续读取行级数据变更。使用 `initial`
模式时，Connector 会先读取当前表快照，再切换到增量变更读取。切换所需的变更必须仍处于源表的 Binlog 保留窗口内。
:::info 版本要求
该功能需要 Flink Doris Connector 26.3.0 及以上、Doris 5.0.0 及以上版本。Row Binlog 是 Doris
5.0.0 的实验性功能，需要在 FE 配置中开启 `enable_feature_binlog = true`；表上如何开启、支持的表模型和限制见 [Row
Binlog](../../../data-operate/incremental/row-binlog.md)。
@@ -72,20 +72,39 @@ SELECT * FROM student_binlog;
| `latest` | 跳过快照，只读取任务启动后产生的变更。 |
| `from-timestamp` | 跳过快照，从 `source.scan.timestamp` 指定的时间点开始读取变更（包含该时间点），时间格式为
`yyyy-MM-dd HH:mm:ss`。 |
+从 Flink Checkpoint 或 Savepoint 恢复时，Connector 会继续使用保存的消费状态。修改
`source.scan.mode` 或 `source.scan.timestamp` 不会重置该状态。恢复的位点和 `from-timestamp`
起点均受源表 `binlog.ttl_seconds` 限制，过期后的行为见下文。
+
## 变更类型 {#increment-type}
默认以 `detail` 类型输出完整的行变更。也可以通过 `source.binlog.increment-type` 设置为
`min_delta`（最小变更集）或 `append_only`（仅追加事件）。
+Connector 通过 Doris `@incr` 查询读取变更。如果请求的起点已因 `binlog.ttl_seconds`
过期，不同变更类型的行为如下：
+
+| 变更类型 | 起点已过期时的行为 |
+| ---- | ---- |
+| `detail`（默认）、`append_only` | Doris 将实际起点推进到第一个仍保留的
TSO，跳过过期变更。任务可能不报过期错误并继续运行，但输出中会缺失这些变更。 |
+| `min_delta` | 读取报 `Row binlog offset has expired according to
binlog.ttl_seconds`，避免返回不完整的净变化。 |
+
+这些规则适用于正常消费、首次快照后的增量读取，以及从保存状态恢复的读取。任务持续运行或 Checkpoint
成功，并不能证明没有变更过期。`source.binlog.visible-wait-timeout` 不会对过期错误进行重试。
+
## 注意事项
- 增量读取使用 Arrow Flight SQL，Connector 默认启用并自动获取端口。
- 需要启用 Flink Checkpoint。
-- Doris Binlog 的保留时间应覆盖任务可能停止的最长时间。如果恢复所需的 Binlog 已过期，需要重新读取快照或指定新的起始时间。
+- Doris `binlog.ttl_seconds` 默认值为 `86400` 秒（一天），必须大于 `0`。过期机制不会等待 Flink 任务或其
Checkpoint；延长 TTL 无法恢复已被物理清理的记录。
+- TTL 应覆盖首次快照耗时、消费积压，以及故障恢复和追赶进度所需的时间，并预留余量。请监控任务仍需读取的最早变更的年龄，从较早的 Checkpoint
或 Savepoint 恢复时也需考虑这一点。
+- 如需自动物理清理，BE 也需开启 `enable_feature_binlog = true`，且自动 compaction
未被关闭。物理清理完成前，增量查询也会过滤过期记录，见 [Row Binlog
保留与清理](../../../data-operate/incremental/row-binlog.md#保留与清理)。
- 执行 Binlog 增量读取时，如果读取区间内存在影响源表的未完成事务，Doris 会等待这些事务完成。若等待超时，本次读取会报错。Connector
仅针对该错误重试同一读取区间，重试时长由 `source.binlog.visible-wait-timeout` 控制（默认 `5m`）；设置为 `0s`
可关闭 Connector 重试，其他错误会立即失败。
+### Binlog 过期后的恢复 {#binlog-expiration-recovery}
+
+如果所需变更已过期，原位点无法提供完整的增量恢复。对于需要保证下游数据与源表当前状态一致的任务，应以 `source.scan.mode =
'initial'` 重新执行初始化，不恢复已过期的 Checkpoint 或 Savepoint，并通过新快照重建下游数据。恢复旧状态时，即使设置了
`initial`，仍会继续使用保存的状态，而不会重新读取全量快照。
+
+新快照可以重建表的当前状态，但无法重现已过期的历史事件。使用 `latest` 或较新的 `from-timestamp`
启动也会跳过缺失区间；仅在允许跳过该区间，或已通过其他方式补齐缺失数据时使用。
+
## 交付语义与去重 {#delivery}
-当前 Doris Binlog Source 只保证至少一次交付，任务故障恢复后可能重放变更事件，影响 Flink 查询结果。Flink 的 CDC
事件去重配置默认关闭；如需启用，可在提交查询前设置：
+在所需变更记录仍完整保留于 Doris 的前提下，Doris Binlog Source 提供至少一次交付。任务故障恢复后可能重放变更事件，影响 Flink
查询结果。Flink 的 CDC 事件去重配置默认关闭；如需启用，可在提交查询前设置：
```sql
SET 'table.exec.source.cdc-events-duplicate' = 'true';
@@ -93,6 +112,8 @@ SET 'table.exec.source.cdc-events-duplicate' = 'true';
启用该配置时，源表必须像上面的示例一样声明主键。Flink 会增加一个有状态算子来规范化变更流。详情参见 [Flink
配置文档](https://nightlies.apache.org/flink/flink-docs-release-2.3/zh/docs/dev/table/config/#table-exec-source-cdc-events-duplicate)。
+Checkpoint 保存消费状态，CDC 去重处理重放的事件；两者都不会阻止 Doris Binlog 过期，也无法恢复因过期而丢失的事件。
+
## 将消费进度写入 Doris（可选） {#offset-table}
消费进度保存在 Flink Checkpoint 状态中。可选的 Offset 表仅用于观测已完成 Checkpoint
所覆盖的消费进度。Connector 不会在任务启动或重启时读取该表来自动恢复消费位点。
@@ -122,7 +143,7 @@ PROPERTIES (
`source.binlog.consumer-id` 用于在 Offset 表中标识当前消费任务，同一任务重启时应保持不变，便于观测消费进度。
-如果无法从 Flink Checkpoint 或 Savepoint 恢复，可以取出 Offset 表中的 `offset_timestamp`，填入
`source.scan.timestamp`，并将 `source.scan.mode` 设置为
`from-timestamp`，手动从该位点恢复消费。例如：
+如果无法从 Flink Checkpoint 或 Savepoint 恢复，且从保存的 `offset_timestamp`
开始所需的变更仍完整保留，可以取出该值，填入 `source.scan.timestamp`，并将 `source.scan.mode` 设置为
`from-timestamp`，手动从该位点恢复消费。如果该时间戳已过期，请按 [Binlog
过期后的恢复](#binlog-expiration-recovery) 处理；填入已过期的时间戳无法恢复缺失变更。例如，以下时间戳仍处于保留窗口内时：
```sql
'source.scan.mode' = 'from-timestamp',
@@ -136,8 +157,8 @@ PROPERTIES (
| Key | Default Value | Required | Comment
|
| --------------------------- | ------------- | -------- |
------------------------------------------------------------------------------------------------------------------------------------------------------
|
| source.scan.mode | snapshot | N | Source 启动模式，支持
`snapshot`、`initial`、`latest` 和 `from-timestamp`
|
-| source.scan.timestamp | -- | N | `from-timestamp`
模式的起始时间（含该时间点），格式为 `yyyy-MM-dd HH:mm:ss`
|
-| source.binlog.increment-type | detail | N | Binlog 变更类型，支持
`detail`、`min_delta` 和 `append_only`
|
+| source.scan.timestamp | -- | N | `from-timestamp`
模式的起始时间（含该时间点），格式为 `yyyy-MM-dd HH:mm:ss`。起点已过期时遵循 [变更类型](#increment-type)
中的行为，无法恢复过期变更。 |
+| source.binlog.increment-type | detail | N | Binlog 变更类型，支持
`detail`、`min_delta` 和 `append_only`。起点过期时，`detail` / `append_only`
跳过过期变更，`min_delta` 报错。 |
| source.binlog.poll-interval | 10s | N | 轮询新 Binlog
数据的时间间隔，最小值为 1 秒
|
| source.binlog.visible-wait-timeout | 5m | N | Doris
返回事务可见性等待超时错误后，Connector 重试同一读取区间的最长时间。设置为 `0s` 可关闭重试；不能为负值。 |
| source.binlog.offset-table | -- | N | 用于发布已完成 Checkpoint
所覆盖 offset 的 Doris 表，仅供观测，不用于自动恢复。格式为 `database.table`，需要同时配置
`source.binlog.consumer-id` 和 `jdbc-url`。 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/incremental-query.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/incremental-query.md
index 3a30c580996..9faa88cc48d 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/incremental-query.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/incremental-query.md
@@ -73,11 +73,11 @@ SELECT ... FROM <table_name>@incr(
| 参数 | 默认值 | 说明 |
|---|---|---|
-| `startTimestamp` | 不限 | 可选。窗口起点，格式 `yyyy-MM-dd HH:mm:ss`，按会话 `time_zone`
解析。返回提交时间 **大于等于** 起点的变更 |
+| `startTimestamp` | 第一个仍保留的 TSO | 可选。窗口起点，格式 `yyyy-MM-dd HH:mm:ss`，按会话
`time_zone` 解析。返回提交时间 **大于等于** 起点的变更，受基表 `binlog.ttl_seconds` 限制 |
| `endTimestamp` | 不限 | 可选。窗口终点，格式同上。返回提交时间 **小于** 终点的变更 |
| `incrementType` | `MIN_DELTA` | 可选。增量模式，见 [三种增量模式](#三种增量模式) |
-三个参数都可以省略，`t@incr()` 等价于按 `MIN_DELTA` 读取全部变更历史。窗口为左闭右开区间 `[startTimestamp,
endTimestamp)`，起点晚于终点或起点在未来时返回空结果。
+三个参数都可以省略，`t@incr()` 按 `MIN_DELTA` 读取保留窗口内的变更。显式起点超出保留窗口时，`MIN_DELTA`
报错，`DETAIL` 和 `APPEND_ONLY` 将实际起点推进到第一个仍保留的 TSO，见
[保留与清理](row-binlog#保留与清理)。窗口为左闭右开区间 `[startTimestamp,
endTimestamp)`；起点通过保留窗口校验后，若晚于终点或位于未来，则返回空结果。
### 结果列
@@ -206,6 +206,8 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__, op;
### 不指定窗口：从表创建至今的净变化
+本例假设建表以来的全部变更仍处于保留窗口内；否则，省略窗口仅读取尚未过期的变更。
+
```sql
SELECT order_id, status, amount, __DORIS_BINLOG_OP__ AS op
FROM orders@incr()
@@ -229,7 +231,7 @@ ORDER BY order_id;
<!-- 知识类型: 使用建议 -->
<!-- 适用场景: 用外部调度系统做周期性增量 -->
-- 用外部调度系统做增量时，把上一次的 `endTimestamp` 作为下一次的
`startTimestamp`，窗口左闭右开保证不重不漏。要注意窗口是按提交时间划分的，正在执行、尚未提交的事务不会出现在当前窗口，而会出现在它提交后的窗口里。
+- 用外部调度系统做增量时，把上一次的 `endTimestamp` 作为下一次的
`startTimestamp`，在保留窗口内，左闭右开的时间窗口保证不重不漏。请将消费延迟控制在 `binlog.ttl_seconds`
以内。要注意窗口是按提交时间划分的，正在执行、尚未提交的事务不会出现在当前窗口，而会出现在它提交后的窗口里。
- 只需要最新值、不需要更新前的值时，可以只保留 `__DORIS_BINLOG_OP__ IN (0, 1, 3)` 的行。
- `@incr` 查询会读取窗口内的全部变更记录，窗口越大、读取越多，请尽量指定窗口。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/overview.md
index a61a0486dae..36693b13b2b 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/overview.md
@@ -141,7 +141,7 @@ IVM 位于这两层能力之上。它使用 Row Binlog 记录的变化和 Doris
| 创建 Stream 报 `Table Stream is experimental. Please set
enable_table_stream=true to enable it.` | FE 未开启 `enable_table_stream`，修改
`fe.conf` 后重启 FE |
| 存算分离模式能用吗？ | Row Binlog 与 Table Stream 可以；`@incr` 请在存算一体模式下使用 |
| 开启 Row Binlog 对写入有什么影响？ | 每次写入要额外生成并持久化变更记录，MoW
表还要读取旧值，导入吞吐会有可感知的下降。只对需要增量消费的表开启，上线前用真实负载评估 |
-| 变更记录会自动清理吗？ | 当前版本不会自动清理，需要为开启 Row Binlog 的表预留存储空间，见
[保留与清理](row-binlog#保留与清理) |
+| 变更记录会自动清理吗？ | 会。`binlog.ttl_seconds` 控制保留时长（默认一天）；增量读取过滤过期记录，后台 compaction
回收其存储空间。请在保留窗口内完成消费，见 [保留与清理](row-binlog#保留与清理) |
## 文档导读
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/row-binlog.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/row-binlog.md
index 514976ad0cd..89aeb5dd8c1 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/row-binlog.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/row-binlog.md
@@ -2,7 +2,7 @@
{
"title": "Row Binlog",
"language": "zh-CN",
- "description": "Doris Row Binlog 记录内表的行级增删改：建表时如何开启、binlog.*
属性、支持的表模型与限制、before/after 镜像与 TSO 的记录模型、对 DDL 的约束、写入开销与常见报错。",
+ "description": "Doris Row Binlog 记录内表的行级增删改：建表属性、TTL
保留策略、自动清理、支持的表模型与限制、before/after 镜像与 TSO 的记录模型、对 DDL 的约束、写入开销与常见报错。",
"keywords": [
"Row Binlog",
"行级 Binlog",
@@ -10,6 +10,8 @@
"binlog.enable",
"binlog.format ROW",
"binlog.need_historical_value",
+ "binlog.ttl_seconds",
+ "Row Binlog TTL",
"before 镜像",
"historical value",
"commit TSO",
@@ -35,6 +37,8 @@
Row Binlog 是 Doris
内表的行级变更日志。开启后，每一次写入产生的行级变化（新增、更新、删除）都会连同变更前后的值、提交时间戳一起持久化，作为 [Table
Stream](table-stream)、[增量查询](incremental-query) 和
[物化视图增量维护（IVM）](../../query-acceleration/materialized-view/async-materialized-view/incremental-materialized-view)
的数据来源，也可以通过 [Flink Doris
Connector](../../connection-integration/data-integration/flink-doris-connector/incremental-read)（26.3.0
及以上版本）在 Flink 中消费。
+变更记录默认保留一天。请通过 `binlog.ttl_seconds` 设置能覆盖最长预期消费延迟的保留时长；变更记录过期不会删除基表中的数据。
+
:::caution 实验性功能
该功能自 5.0.0 版本起提供，目前处于实验阶段，需要在 FE 中开启 `enable_feature_binlog = true`。
:::
@@ -45,6 +49,7 @@ Row Binlog 是 Doris 内表的行级变更日志。开启后，每一次写入
- Doris 5.0.0 及以上版本。
- FE 已在 `fe.conf` 中开启 `enable_feature_binlog = true`（非动态配置，需重启 FE）。
+- 如需自动清理，BE 也需在 `be.conf` 中开启 `enable_feature_binlog = true`（非动态配置，需重启 BE）。
- 表模型为 Duplicate Key，或 Unique Key Merge-on-Write（MoW）且没有 cluster key，详见
[支持范围与限制](#支持范围与限制)。
- Row Binlog 只能在建表时开启，请在建表前完成评估。
@@ -76,7 +81,8 @@ PROPERTIES (
"enable_unique_key_merge_on_write" = "true",
"binlog.enable" = "true",
"binlog.format" = "ROW",
- "binlog.need_historical_value" = "true"
+ "binlog.need_historical_value" = "true",
+ "binlog.ttl_seconds" = "86400"
);
```
@@ -89,7 +95,7 @@ PROPERTIES (
| `binlog.enable` | `true` / `false` | `false` | 开启后不可关闭 | 是否开启 binlog，需与
`binlog.format = "ROW"` 同时设置 |
| `binlog.format` | `ROW` | - | 不可修改 | 必须为 `ROW`，表示记录行级变更。取值区分大小写，小写的 `row` 会报
`Invalid binlog format value: row` |
| `binlog.need_historical_value` | `true` / `false` | `false` | 不可修改 |
是否记录变更前的值（before 镜像）。仅 Unique Key MoW 表可设为 `true`；`min_delta` / `detail` 类型的
Table Stream、`MIN_DELTA` 增量查询和需要处理更新或删除的 IVM 都依赖它 |
-| `binlog.ttl_seconds` | 整数（秒） | `86400` | 可修改 | 保留时长。**当前版本不生效**，见
[保留与清理](#保留与清理) |
+| `binlog.ttl_seconds` | 正整数（秒） | `86400`（一天） | 可修改 | 变更记录的保留时长。增量读取会过滤过期记录，后台
compaction 会回收过期记录的存储空间；不允许设置为 `0` 或负数。见 [保留与清理](#保留与清理) |
| `binlog.max_bytes` | 整数（字节） | 无限制 | 可修改 | 保留大小上限。**当前版本不生效** |
| `binlog.max_history_nums` | 整数 | 无限制 | 可修改 | 保留条数上限。**当前版本不生效** |
@@ -258,11 +264,111 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__;
## 保留与清理
<!-- 知识类型: 行为规则 -->
-<!-- 适用场景: 容量规划 -->
+<!-- 适用场景: 设置保留时长 / 处理过期位点 / 容量规划 -->
+
+自 Doris 5.0.0 起，`binlog.ttl_seconds` 控制 Row Binlog
的保留时长，存算一体和存算分离模式均支持。保留时长按提交 TSO 中的物理时间计算，以当前 Master FE 的 TSO
为基准。提交时间小于等于「基准时间减去 TTL」的记录视为过期。即使表没有新写入，保留窗口也会随时间推进。
+
+### 配置保留时长
+
+TTL 默认值为 `86400` 秒（一天）。如果数据库显式设置了
`binlog.ttl_seconds`，新建表会继承该值，表属性中的显式设置优先。修改数据库配置不会改变已有表的 TTL。`CREATE TABLE ...
LIKE` 会复制源表的 TTL。
+
+例如，将上文创建的 `orders` 表的保留时长改为七天：
+
+```sql
+ALTER TABLE orders SET ("binlog.ttl_seconds" = "604800");
+```
+
+语句成功后，`SHOW CREATE TABLE orders` 的输出包含：
+
+```text
+"binlog.ttl_seconds" = "604800"
+```
+
+TTL 必须大于 `0`，不能通过 `0` 或 `-1` 关闭过期机制。缩短 TTL 可能立即使已有记录过期；延长 TTL
可以使尚未被物理清理的记录重新可读，但无法恢复已被清理的记录。如果 `ALTER TABLE` 在向分区下发配置时失败，较短的 TTL 可能已经生效，请检查
`SHOW CREATE TABLE` 并重试同一条语句，完成配置更新。
+
+### 对读取的影响
+
+增量读取在生成执行计划时应用保留窗口，不需要等待物理清理完成。同一条语句使用同一个基准 TSO，每条新语句都会重新计算窗口。
+
+| 读取方式 | 请求的起点已过期时的行为 |
+|---|---|
+| 未指定 `startTimestamp` 的 `@incr` | 仅读取保留窗口内的变更，`MIN_DELTA` 模式也适用 |
+| 显式指定 `startTimestamp` 的 `@incr`，且 `incrementType = MIN_DELTA` | 报 `Row
binlog offset has expired according to binlog.ttl_seconds`，避免返回不完整的净变化 |
+| `incrementType = DETAIL` 或 `APPEND_ONLY` 的 `@incr` | 将实际起点推进到第一个仍保留的
TSO，仅返回请求窗口内尚未过期的部分 |
+| `type = min_delta` 的 Table Stream | 任一分区的消费位点已过期时，报相同错误 |
+| `type = detail` 或 `append_only` 的 Table Stream | 跳过过期变更，从第一个仍保留的 TSO 开始读取 |
+| MoW 表的快照读取（`FOR TIME AS OF`、`FOR VERSION AS OF` 或 Stream 的 `@snapshot()`） |
重建需要读取 Row Binlog 时，已过期的目标快照报相同错误，因为所需的 before 镜像已不在保留窗口内 |
+| `binlog()` 表函数 | 不应用 TTL 过滤；物理清理前仍可能读到已过期的原始记录 |
+
+过期机制不会等待消费者。即使 `IS_STALE` / `STALE_REASON` 仍显示 Stream
可用，也应在保留窗口内完成消费：当前实现不会通过这两个字段报告 TTL 过期。所需变更过期后，需要从全量快照重建下游数据。对于 Table Stream，可按
[reset 流程](table-stream-advanced#重置-reset) 全量重载下游并推进位点。IVM 的内部 Stream 请通过
[物化视图刷新任务](../../query-acceleration/materialized-view/async-materialized-view/incremental-materialized-view#内部-table-stream)
管理。
+
+### 示例：变更过期后基表数据仍保留
+
+在已有数据库中执行以下示例。执行用户需具备建表所需的 `CREATE` 权限、写入所需的 `LOAD` 权限，以及查询所需的 `SELECT` 权限：
+
+```sql
+CREATE TABLE row_binlog_ttl_demo (id INT, value INT)
+DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1
+PROPERTIES (
+ "replication_num" = "1",
+ "binlog.enable" = "true",
+ "binlog.format" = "ROW",
+ "binlog.ttl_seconds" = "30"
+);
+
+INSERT INTO row_binlog_ttl_demo VALUES (1, 10);
+
+SELECT COUNT(*) AS change_count
+FROM row_binlog_ttl_demo@incr("incrementType" = "DETAIL");
+```
+
+写入后立即查询，结果为：
+
+```text
++--------------+
+| change_count |
++--------------+
+| 1 |
++--------------+
+```
+
+等待超过 30 秒后，再次执行增量查询：
+
+```sql
+SELECT COUNT(*) AS change_count
+FROM row_binlog_ttl_demo@incr("incrementType" = "DETAIL");
+```
+
+```text
++--------------+
+| change_count |
++--------------+
+| 0 |
++--------------+
+```
+
+基表中的行仍然存在：
+
+```sql
+SELECT id, value FROM row_binlog_ttl_demo;
+```
+
+```text
++------+-------+
+| id | value |
++------+-------+
+| 1 | 10 |
++------+-------+
+```
+
+### 物理清理与容量规划
+
+后台 compaction 会回收 Row Binlog 中已全部过期的 Rowset，即使表没有新写入也会执行。包含未过期记录的 Rowset
会保留到其中所有记录都过期。清理会保留基表数据和版本连续性。存储空间异步回收，因此 TTL 不代表文件必须在该时刻消失，也不构成严格的存储大小上限。
-当前版本不会自动清理 Row Binlog
数据，变更记录随表一直保留，`binlog.ttl_seconds`、`binlog.max_bytes`、`binlog.max_history_nums`
三个属性可以设置但暂不生效。基于时间和大小的自动清理正在开发中，将在下个版本支持。
+自动清理要求 BE 开启 `enable_feature_binlog = true`，且自动 compaction 未被关闭。将 BE 配置或表属性
`disable_auto_compaction` 设为 `true` 会暂停自动清理，但增量读取仍会过滤过期记录。BE 重启后，清理会等待 Master
FE 心跳下发有效的基准 TSO。
-在此之前，请为开启 Row Binlog 的表预留额外的存储空间：变更记录的体积与写入量成正比，对开启 before 镜像的 MoW
表，每次更新会额外记录一份旧值。
+`binlog.max_bytes` 和 `binlog.max_history_nums` 对 Row Binlog 仍不生效。请根据写入量、TTL
和清理延迟预留存储空间；对开启 before 镜像的 MoW 表，每次更新还会额外记录一份旧值。TTL
应覆盖最长消费延迟、刷新间隔和预期故障停机时间，并为恢复预留余量。
## 写入开销
@@ -305,7 +411,7 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__;
| `partition` | 否 | 分区名，多个用逗号分隔，默认全部分区 |
| `tablet` | 否 | tablet ID，多个用逗号分隔，默认全部 tablet |
-`binlog()` 直接读取存储的原始记录，不做任何折叠或过滤，返回的 `__DORIS_BINLOG_OP__` 使用原始编码（`0` 新增、`1`
更新、`2` 删除）。完整语法见 [BINLOG
表函数](../../sql-manual/sql-functions/table-valued-functions/binlog)。
+`binlog()` 直接读取存储的原始记录，不做变更折叠或 TTL 过滤，返回的 `__DORIS_BINLOG_OP__` 使用原始编码（`0`
新增、`1` 更新、`2` 删除）。物理清理完成前，这里仍可能看到过期记录。完整语法见 [BINLOG
表函数](../../sql-manual/sql-functions/table-valued-functions/binlog)。
## 常见错误对照
@@ -314,6 +420,8 @@ ORDER BY __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__;
| 错误信息 | 原因 | 处理 |
|---|---|---|
| `Invalid binlog format value: row` | `binlog.format` 使用了小写 `row` |
取值区分大小写，改为大写 `ROW` |
+| `ROW binlog.ttl_seconds must be greater than 0` | TTL 为 `0` 或负数 |
设置以秒为单位的正整数，默认值为 `86400` |
+| `Row binlog offset has expired according to binlog.ttl_seconds` |
`MIN_DELTA` 起点或 MoW 快照超出保留窗口 | 从全量快照重建下游数据并重新对齐消费位点，见 [对读取的影响](#对读取的影响) |
| `not support change binlog format from STATEMENT_AND_SNAPSHOT to ROW` |
对已有表用 `ALTER TABLE` 开启 Row Binlog | Row Binlog 只能在建表时开启。新建开启 Row Binlog
的表并导入数据，再用 `ALTER TABLE ... REPLACE WITH TABLE` 原子替换 |
| `can't disable binlog when format is [Row]` | 尝试在已开启 Row Binlog 的表上设置
`binlog.enable = false` | Row Binlog 开启后不可关闭 |
| `not support change binlog.need_historical_value from true to false` | 尝试修改
`binlog.need_historical_value` | 该属性建表后不可修改 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream-advanced.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream-advanced.md
index 53265aa98a7..d31529a0779 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream-advanced.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream-advanced.md
@@ -245,14 +245,15 @@ IVM 创建的内部 Stream 也会出现在相关系统表中。可以观察其
| 关注点 | 数据来源 | 说明 |
|---|---|---|
| 积压 | `information_schema.table_stream_consumption` 的 `LAG` | 定期查询，对长时间不为 `0`
或持续增长的分区告警 |
-| 状态 | `information_schema.table_streams` 的
`ENABLED`、`IS_STALE`、`STALE_REASON` | 反映 Stream 是否可用。当前版本 Row Binlog
不会自动清理，Stream 不会因为长期不消费而失效；自动清理能力上线后，变更记录已被清理的 Stream 会被标记为 stale，需要通过
`@reset()` 重新对齐 |
+| 状态 | `information_schema.table_streams` 的
`ENABLED`、`IS_STALE`、`STALE_REASON` | 反映 Stream 状态，但当前实现不会报告 TTL 过期。需将消费延迟与基表的
`binlog.ttl_seconds` 对比，并关注位点过期报错，见 [保留与清理](row-binlog#保留与清理) |
| 消费历史 | `information_schema.table_stream_consumption` 的
`LAST_CONSUMPTION_TIME` | 判断消费任务是否按计划运行 |
### 故障恢复
| 场景 | 处理 |
|---|---|
-| 消费语句失败 | 位点不变，直接重跑即可，不会漏掉变更 |
+| 消费语句失败 | 位点不变，请在所需变更仍处于保留窗口内时重跑 |
+| 所需变更已因 `binlog.ttl_seconds` 过期 | 用 `@reset()` 全量重载下游并重新对齐位点。位点过期时
`min_delta` 读取报错，`detail` 和 `append_only` 跳过过期变更 |
| 消费成功但调度系统没记录到 | 如果重跑，本轮读到的是下一批变更，不会重复消费上一批。要让重复执行完全无副作用，目标表建议使用 Unique Key
模型（按主键写入天然幂等） |
| 下游数据错误需要重建 | 用 `@snapshot()` 恢复到消费位点对应的状态后继续增量消费；或者用 `@reset()`
全量重刷并把位点推进到当前 |
| FE 重启、主从切换 | 位点持久化在元数据中，重启后继续消费 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream.md
index 98a3dbf14b1..cfc1e706aff 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data-operate/incremental/table-stream.md
@@ -376,7 +376,7 @@ Stream 的列与基表当前的可见列一致（基表 `ADD COLUMN` / `DROP COL
- Stream 只能读取基表的基础索引，不支持指定 rollup / 物化视图，也不支持 `TABLET (...)` 子句。
- Stream 不能作为 `INSERT`、`UPDATE`、`DELETE` 的目标。
- 消费类型和 `show_initial_rows` 创建后不可修改。
-- 当前版本 Row Binlog 数据不会自动清理，Stream 长时间不消费不会丢失变更；自动清理能力上线后，长期不消费的 Stream
可能因为变更记录被清理而无法继续消费，届时会通过 `information_schema.table_streams` 的 `IS_STALE` /
`STALE_REASON` 标识。
+- Row Binlog 由 `binlog.ttl_seconds` 控制保留时长（默认一天），不受消费进度影响。位点过期后，`min_delta`
读取报错，`detail` 和 `append_only` 跳过过期变更。当前实现不会通过 `IS_STALE` / `STALE_REASON` 报告
TTL 过期。请在保留窗口内完成消费，见 [保留与清理](row-binlog#保留与清理)。
- 基表被删除、分区被删除或替换等操作对 Stream 的影响见 [Table Stream
进阶](table-stream-advanced#基表变更的影响)。
## 常见问题
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
