vagetablechicken commented on issue #3624:
URL:
https://github.com/apache/incubator-doris/issues/3624#issuecomment-634573432
We can add a PermitLimiter to do the limitation, and use
single-producer/multi-consumer model.
Producer will try to acquire permits, if ok, add the compaction task to
consumer pool.
So we need a PermitLimiter member function:
```
bool PermitLimiter::block_acquire(int permits);
// return false: drop this task
// return true: block waiting for permits succeed, task can start now
```
### pseudocode
Producer:
```
generate tasks
for task: tasks
if PermitLimiter.block_acquire(task.permits)
add to consumer pool
else
drop this task
// one cycle end
```
Consumers:
```
PermitLimiter.release(permits) when task finished(whether successful or not)
```
PermitLimiter:
```
bool PermitLimiter::block_acquire(int permits):
if permits > total_permits
if over_sold
wait_until used_permits == 0 // only allow this task to run
else
return false
else
wait_until (total_permits - used_permits) >= permits
used_permits += permits
return true
void PermitLimiter::release(int permits):
used_permits -= permits
```
Droping a compaction task is not a good idea, that means this tablet can't
do compaction anymore(rowsets will incr over time).
So I use `over_sold`, which can be modified in runtime. If memory usage is
too high, we can disable over_sold for a while.
And I don't use free_permtis, use (total_permits - used_permits) instread.
Thus, the total_permits can be modified in runtime too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
