yujun777 opened a new issue, #65897: URL: https://github.com/apache/doris/issues/65897
## Problem For a UNIQUE KEY Merge-on-Write table with ROW binlog, binlog entries remain readable after they are older than the configured binlog.ttl_seconds. Please confirm whether this is a bug or an intentional limitation of ROW binlog TTL. Related context: [DORIS-27284](https://issues.apache.org/jira/browse/DORIS-27284) ## Environment - Repository: apache/doris - Commit: a4dae0b64e9e7c6db5983b566656442f1ae2bc5d - Deployment: local non-cloud FE/BE cluster ## Reproduction ```sql CREATE TABLE stream_ttl_probe.base ( id BIGINT, v BIGINT ) UNIQUE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "enable_unique_key_merge_on_write" = "true", "binlog.enable" = "true", "binlog.format" = "ROW", "binlog.need_historical_value" = "true", "binlog.ttl_seconds" = "1", "binlog.max_history_nums" = "100000" ); CREATE STREAM stream_ttl_probe.s ON TABLE stream_ttl_probe.base PROPERTIES ("type" = "min_delta", "show_initial_rows" = "false"); INSERT INTO stream_ttl_probe.base VALUES (1, 10), (2, 20); ``` Waited about 45 seconds, which is well beyond the 1-second TTL and multiple FE GC intervals, then executed: ```sql SET show_hidden_columns = true; SELECT id, v, __DORIS_STREAM_CHANGE_TYPE_COL__, __DORIS_STREAM_SEQUENCE_COL__ FROM stream_ttl_probe.s ORDER BY id; ``` ## Observed result The stream still returned both rows: ```text 1 10 APPEND 467850668704530433 2 20 APPEND 467850668704530433 ``` The stream remained non-stale (`IS_STALE = 0`). Filesystem inspection of the corresponding BE tablet also showed that the `_row_binlog` files remained after more than one minute. The same behavior was observed in an IVM reproduction after waiting about 74 seconds. ## Expected result If `binlog.ttl_seconds` applies to ROW binlog, entries older than the TTL should become eligible for physical GC. A stream whose required binlog range is no longer available should report a stale/binlog-broken condition, rather than continue reading expired entries. If TTL is intentionally only supported for CCR/statement binlog and not ROW binlog, the documentation and property validation should make that limitation explicit. ## Source analysis The FE BinlogGcer path appears to process FE TableBinlog objects used by CCR and excludes ROW binlog. ROW binlog is stored under BE tablet `_row_binlog`; the current ROW binlog compaction/GC path does not appear to apply `binlog.ttl_seconds`. `Tablet::binlog_ttl_ms()` exists, but no effective ROW binlog TTL GC call path was found. Please confirm the intended contract and, if this is a bug, add ROW binlog TTL GC and the corresponding stream/binlog-broken handling. ## Related issues - [#65265](https://github.com/apache/doris/issues/65265) - [#65418](https://github.com/apache/doris/issues/65418) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
