keith-turner edited a comment on pull request #2330: URL: https://github.com/apache/accumulo/pull/2330#issuecomment-953853715
> To be clear, my concern is starvation and the users inability to change it. The priority is decided by the pluggable compaction planner, so it could be changed by a user. If there would be starvation, it could have happened before this change. This change slightly improved the current default way of prioritizing that has the goal of reducing the avg number of files per tablet across the tserver. I suspect any priority scheme that does not explicitly incorporate age always has some non-zero chance of starvation for a particular tablet. This is based on memory, so I may be wrong, prior to the compaction changes in 2.1 the priority mechanism may have been hard coded w/o any for a user to change it. > I'm thinking about the case where you have unbalanced ingest into a table, will you run into a case where compactions never happen for tablets that have a light amount of ingest because the tablets that have a large amount of ingest activity are constantly compacting? In addition to giving user control over priority, 2.1 gives users another way to manage this situation. Users can assign tables to different compaction services which can allows tables to be placed on different queues and thread pools. The situation of starvation can only happen when queues are constantly full. I had to limit the compaction threads to one for the test just to force the condition of the queue always being full. When the queues were not always full, I did not observe any impact from this change. One other thing to consider when thinking about starvation is that user compactions are prioritized over system compactions in the default prio mechanism. So system compactions can never starve user compactions. For system compactions, I don't think the age really matters, just compact the tablet that will yield the most impact. However for user compactions, maybe we should consider the age. That way an older user compaction is not starved by newer user compaction. May be good to open an issue about changing the default prio mechanism of user compactions to consider age. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
