matanper opened a new issue, #70837:
URL: https://github.com/apache/doris/issues/70837

   ### Search before asking
   
   - [x] I had searched in the 
[issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no 
similar issues.
   
   ### Description
   
   Vertical compaction reads its input rowsets once per column group: the key 
columns first, then each group of `vertical_compaction_num_columns_per_group` 
value columns. Every group re-applies the delete predicates of the input 
rowsets:
   
   - `Merger::vertical_compact_one_group` (`be/src/storage/merger.cpp:393` on 
master) calls `read_source.fill_delete_predicates()` for every group.
   - `BetaRowsetReader::get_segment_iterators` 
(`be/src/storage/rowset/beta_rowset_reader.cpp:138`) adds every 
delete-condition column to that group's read columns and builds 
`delete_condition_predicates` for each segment iterator.
   - So every value group reads the predicate columns and evaluates the 
predicates again.
   
   The value groups need this only to drop the same rows that the key group 
dropped, so that they stay aligned with the row source buffer.
   
   With many column groups, this dominates a base compaction whose inputs 
include delete predicates. Take a table of a few hundred columns at the default 
5 columns per group, or one with a smaller per-group setting: the predicate 
columns are read once per group. The predicates are evaluated once per group 
too, and a `DELETE ... WHERE id IN (...)` with a long IN list is not cheap.
   
   On such a table, a profile of a base compaction showed that most of the 
reading went to the delete-predicate columns of the value groups, not to the 
columns being compacted. The compaction read many times more data than it wrote.
   
   ### Solution
   
   For a duplicate-key table, a value group must read exactly the rows that the 
key group kept, because a duplicate-key table merges no rows. So:
   
   1. While the key group runs, record the row ids it keeps from each input 
segment. `record_rowids` and `current_block_row_locations()` already provide 
them; merge-on-write row-id conversion uses the same mechanism.
   2. In the value groups, drop the delete predicates. Give each segment 
iterator the bitmap of rows to read for its segment and intersect `_row_bitmap` 
with it in `SegmentIterator::_lazy_init`, the same way a merge-on-write delete 
bitmap is applied there.
   
   The value groups then read the same rows as before, but they no longer read 
the predicate columns or evaluate the predicates.
   
   Unique-key and aggregate-key tables can keep the current path. Their value 
groups also rely on agg flags, which the change does not need to touch.
   
   We have a patch with BE unit tests and can open a PR if this approach is 
acceptable.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to