matanper commented on issue #70771:
URL: https://github.com/apache/doris/issues/70771#issuecomment-7027415556

   Thanks for the review. Some follow-up details:
   
   **Table properties** (DUPLICATE KEY, cloud mode): `compaction_policy = 
time_series`, `time_series_compaction_goal_size_mbytes = 1024`, 
`time_series_compaction_file_count_threshold = 1000`, 
`time_series_compaction_time_threshold_seconds = 3600`, 
`time_series_compaction_empty_rowsets_threshold = 5`, 
`time_series_compaction_level_threshold = 2`. The threshold was raised from 1 
to 2 with `ALTER TABLE ... SET`. The table was under continuous load at the 
time.
   
   **One affected tablet, read from the meta-service** (cumulative point 6315):
   
   | versions | rowsets | segments | compaction_level | created |
   |---|---|---|---|---|
   | `[6315-6342]` … `[6427-6450]` | 5 | 1 each | 0 | within about a minute of 
the ALTER |
   | `[6451-…]` … `[11711-11714]` | 203 | 1 each | 1 | after the ALTER |
   | `[11715-11715]` | 1 | 0 (empty) | 0 | a late load |
   
   All rowsets above the point are NONOVERLAPPING, and none has a delete 
predicate. `/api/compaction/show` shows `last cumulative status: [E-2000]` and 
no cumulative success.
   
   **Creation history:** we have not reconstructed it. Below threshold 2 the 
cumulative point normally moves past each cumulative output, so we can't yet 
say why these level-0 outputs stayed at or above it. On other tablets of the 
same table, the level-0 merged rowsets at the point were created up to three 
days before the ALTER. The blocked state itself is consistent across tablets: 
on every affected tablet, the first rowset above the point is a merged level-0 
rowset from before the ALTER, followed by level-1 rowsets.
   
   **Patch:** we treat a merged (start != end), non-empty, stored level-0 
rowset as level 1 in the level-2 scan of `pick_input_rowsets`, in 
`calc_cumulative_compaction_score`'s initial classification and level-2 scan, 
and in `get_compaction_level`. The output is labelled one level above its 
*lowest* effective input, so a level-1 pick that starts at such a rowset and 
includes new loads still writes level 1. Conditions 2 and 3 skip a lone 
non-overlapping rowset and fall through.
   
   Unit tests cover:
   - the mixed prefix with no tail, an aged empty tail and an aged non-empty 
tail, asserting the selected versions;
   - output level 2, plus cumulative-point movement for a non-empty output and 
no movement for an empty one;
   - a mixed pick writing level 1;
   - a lone load past the time threshold;
   - condition 2 with file-count threshold 1;
   - an overlapping lone load still being picked.
   
   With the patch all 19 time-series policy tests pass; without it the new 
cases fail.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to