[
https://issues.apache.org/jira/browse/HBASE-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Lawlor updated HBASE-6778:
-----------------------------------
Attachment: thread_dump_HMaster.local.out
I started to look into this issue this week. Initial investigation revealed
that on average the number of threads running within the HMaster process that
can be attributed to Chores is around 12 threads. I have attached a sample
thread dump above (you can see the active Chore threads by searching for
'Chore').
As discussed, the goal of a proposed solution should be to reduce the number of
running threads by allowing the running Chores to share threads that are idle
in between Chore cycles. The idea of using ScheduledThreadPoolExecutor seems
well suited for the job.
A ScheduleThreadPoolExecutor allows us to specify the maximum number of threads
that we want to allow the running Chores to use. Since there may be Chores with
tasks that take a while to execute, this value would have to be dynamic so that
it could increase/decrease during varying Chore loads. Specifically, the
executor needs to account for the worst case scenario that all running Chores
have tasks that take longer than their period (i.e. the chore finishes after
its next scheduled execution should have started). Thus the core pool size
could be set to a small initial value, and if we recognize that Chores are
missing their scheduled start times, we could increase the core pool size to a
maximum of however many active chores there are.
In summary, I am proposing that we design a service that acts as an interface
to an underlying ScheduleThreadPoolExecutor whose core pool size can vary. Any
feedback on this proposed solution and where this may fit into the HBase
workflow would be greatly appreciated.
> Deprecate Chore; its a thread per task when we should have one thread to do
> all tasks
> -------------------------------------------------------------------------------------
>
> Key: HBASE-6778
> URL: https://issues.apache.org/jira/browse/HBASE-6778
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Lawlor
> Attachments: thread_dump_HMaster.local.out
>
>
> Should use something like ScheduledThreadPoolExecutor instead (Elliott said
> this first I think; J-D said something similar just now).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
