[
https://issues.apache.org/jira/browse/KUDU-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247722#comment-17247722
]
wangningito commented on KUDU-2726:
-----------------------------------
Either modify tablet_compaction_budget_mb or KUDU-1400 could work around on
this. But either of them has disadvantages, the first one may result in
compaction time longer, and the second one is as mentioned by William.
To a CompactRowsetOp, it has a logic like this as my understanding,
picking rowsets, and calculate the quality for selection (stage 1) -> amplify
the quality with FLAGS_maintenance_op_multiplier and '
{{kudu.table.maintenance_priority}}' and set the amplified value as score of
op (stage 2)
And to maintenance_manager, it iterate over over all registered op, and select
one with higher op score.
But to a very big tablet, it may get very low solution value and below the
FLAGS_compaction_minimum_improvement, so its' quality would be set to 0, and
there is no amplify factor can be apply to compact ops of this tablet. It will
be always 0, and always fail in 'stage 1'.
I saw now table maintenance strategy can be modified via tables' extra_config
interface. So I come up with this:
we can try pass the '{{kudu.table.maintenance_priority}}' and multiple it in
'stage 1' or just move FLAGS_compaction_minimum_improvement detect to 'stage
2', then to a very big tablet, the compaction could also work. This also has
advantage over the two work arounds mentioned before: its' scope is table, the
adjustment can make compaction strategy more flexible. And also, all tablets in
a table usually has similar size on disk.
How do you think about it? I can work on this.
> Very large tablets defeat budgeted compaction
> ---------------------------------------------
>
> Key: KUDU-2726
> URL: https://issues.apache.org/jira/browse/KUDU-2726
> Project: Kudu
> Issue Type: Improvement
> Affects Versions: 1.9.0
> Reporter: William Berkeley
> Priority: Major
> Labels: density, roadmap-candidate
>
> On very large tablets (50GB+), despite being very uncompacted with a large
> average rowset height, a default budget (128MB) worth of compaction may not
> reduce average rowset height enough to pass the minimum threshold. Thus the
> tablet stays uncompacted forever.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
