zclllyybb commented on issue #65947: URL: https://github.com/apache/doris/issues/65947#issuecomment-5054552814
Breakwater-GitHub-Analysis-Slot: slot_2543afc8eebe This content is generated by AI for reference only. Initial triage for `87d57a84729b61fcdd2e00e4e10697b46b2a82e9`: This looks like a real row-binlog write-path amplification issue, but the current report does not include enough workload detail or profiles to pin the single dominant bottleneck yet. What the code path shows: - If the table uses `binlog.enable=true` with `binlog.format=ROW`, FE creates a separate row-binlog index/schema for streaming binlog. That schema contains the source key/visible columns, optional BEFORE columns when `binlog.need_historical_value=true`, and hidden TSO/LSN/op columns. - On the BE load path, `VTabletWriter` detects the row-binlog index and allocates one row-binlog LSN per input row through the auto-increment buffer. Those LSNs are copied through the add-block request, `TabletsChannel`, the memtable sidecar, and finally into the segment writer options at memtable flush time. - The rowset writer is wrapped as a `GroupRowsetWriter`, so each flush/build is synchronous for both the normal data rowset and the row-binlog rowset. In other words, row-binlog generation is on the import critical path, not an asynchronous post-commit task. - `RowBinlogSegmentWriter::append_block()` reconstructs and writes a second segment stream: it converts source columns again, writes the TSO/LSN/op columns, builds a key index, and, when historical values are requested or partial update needs full AFTER rows, performs historical-row retrieval/fill work before writing BEFORE/AFTER columns. - FE transaction commit also fetches a commit TSO for row-binlog tables. That is worth measuring, especially for workloads made of many small transactions, but for large bulk import the code suggests the main suspects are BE row-binlog segment construction, sidecar LSN handling, and optional historical-row lookup. So the 65% throughput drop is plausible from the current implementation, but I would not classify the exact regression boundary from the provided numbers alone. The strongest code-level suspect is that the current ROW binlog implementation synchronously writes a full secondary rowset per flush and may additionally do per-row LSN copying plus historical lookup. That can directly throttle import throughput when the workload is CPU-bound in conversion/sorting/index writing or I/O-bound on the extra segment files. Missing information needed to make this actionable: - Exact `CREATE TABLE` DDL, especially key type, number/types of columns, partitions/buckets/tablets, replica count, and the binlog properties: `binlog.enable`, `binlog.format`, and `binlog.need_historical_value`. - Exact load method and workload shape: Stream Load/Broker Load/Routine Load/Insert, total rows/bytes, file format, batch size, concurrency, transaction count, and whether both tests created the same number of rowsets/segments. - FE/BE configs that affect this path, including `enable_feature_binlog`, memtable flush settings, compaction settings, load parallelism, auto-increment prefetch settings, and disk/network layout. - Side-by-side BE load profiles or flamegraphs for enabled vs disabled runs, with attention to `VTabletWriter`, `TabletsChannel::add_block`, `MemTableFlushExecutor`, `GroupRowsetWriter::flush_memtable/build_rowsets`, `RowBinlogSegmentWriter::append_block`, `OlapBlockDataConvertor`, `build_key_index`, and historical-row retriever code if `need_historical_value=true`. - FE transaction timing for commit/visible/publish phases, especially TSO-fetch latency if the workload uses many small transactions. Recommended next checks: 1. First confirm whether this is ROW binlog. If it is `STATEMENT_AND_SNAPSHOT`, this investigation should pivot to the CCR-style binlog path instead of the row-binlog writer path above. 2. Run the same benchmark with `binlog.format=ROW` and `binlog.need_historical_value=false` versus `true`. A large gap there would point to historical-row retrieval/BEFORE-column filling rather than basic secondary-rowset writing. 3. Compare flush count, segment count, rowset count, CPU time, and disk write bytes between the two runs. If the slowdown scales with segment/flush count, the synchronous secondary rowset build is the likely limiter. 4. Add or collect timing around the row-binlog writer stages before optimizing: LSN allocation/copy, source-column conversion, binlog meta-column fill, key-index build, BEFORE/AFTER historical lookup, and normal-vs-row-binlog flush/build time. 5. If profiles confirm the BE row-binlog writer dominates, likely optimization directions are batching/reducing LSN sidecar copies, avoiding repeated column conversion where possible, bulk-filling LSN/op columns, reducing key-index build cost for append-only cases, or decoupling row-binlog materialization from the synchronous import critical path while preserving ordering and correctness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
