matanper opened a new issue, #70837: URL: https://github.com/apache/doris/issues/70837
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description Vertical compaction reads its input rowsets once per column group: the key columns first, then each group of `vertical_compaction_num_columns_per_group` value columns. Every group re-applies the delete predicates of the input rowsets: - `Merger::vertical_compact_one_group` (`be/src/storage/merger.cpp:393` on master) calls `read_source.fill_delete_predicates()` for every group. - `BetaRowsetReader::get_segment_iterators` (`be/src/storage/rowset/beta_rowset_reader.cpp:138`) adds every delete-condition column to that group's read columns and builds `delete_condition_predicates` for each segment iterator. - So every value group reads the predicate columns and evaluates the predicates again. The value groups need this only to drop the same rows that the key group dropped, so that they stay aligned with the row source buffer. With many column groups, this dominates a base compaction whose inputs include delete predicates. Take a table of a few hundred columns at the default 5 columns per group, or one with a smaller per-group setting: the predicate columns are read once per group. The predicates are evaluated once per group too, and a `DELETE ... WHERE id IN (...)` with a long IN list is not cheap. On such a table, a profile of a base compaction showed that most of the reading went to the delete-predicate columns of the value groups, not to the columns being compacted. The compaction read many times more data than it wrote. ### Solution For a duplicate-key table, a value group must read exactly the rows that the key group kept, because a duplicate-key table merges no rows. So: 1. While the key group runs, record the row ids it keeps from each input segment. `record_rowids` and `current_block_row_locations()` already provide them; merge-on-write row-id conversion uses the same mechanism. 2. In the value groups, drop the delete predicates. Give each segment iterator the bitmap of rows to read for its segment and intersect `_row_bitmap` with it in `SegmentIterator::_lazy_init`, the same way a merge-on-write delete bitmap is applied there. The value groups then read the same rows as before, but they no longer read the predicate columns or evaluate the predicates. Unique-key and aggregate-key tables can keep the current path. Their value groups also rely on agg flags, which the change does not need to touch. We have a patch with BE unit tests and can open a PR if this approach is acceptable. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
