[
https://issues.apache.org/jira/browse/HBASE-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Lawlor updated HBASE-6778:
-----------------------------------
Attachment: HBASE_6778_WIP_v2.patch
AFTER_thread_dump.txt
BEFORE_thread_dump.txt
Work In Progress Update #2:
I have attached the latest version of the solution to this issue above
(HBASE_6778_WIP_v2.patch). Since the last WIP update, the following changes
have been made:
- Made changes in response to the feedback that [~stack] provided
- Added Stoppable support to ScheduledChore (Stoppable support was needed as
the same behavior could not be replicated with cancels alone). What this means
is that now an instance of a Stoppable can be passed into the ScheduledChore
during construction and the ScheduledChore will cancel itself if
Stoppable#isStopped() becomes true.
- Added shutdown() to ChoreService to allow for convenient all-in-one cleanup
of a ChoreService
- Changed usages of Chore (the implementation we would like to get rid of) into
usages of ScheduledChore
- Expanded the test coverage of ScheduledChore and ChoreService
While converting Chores to ScheduledChores the primary goal was to maintain the
same behavior. Most changes can be understood through the following mapping
(Chore usage -> ScheduledChore usage):
- Chore.interrupt() -> ScheduledChore.cancel(mayInterruptWhileRunning = true)
- Threads.setDaemonThreadRunning(Chore) ->
ChoreService.scheduleChore(ScheduledChore)
- Chore.isAlive -> ScheduledChore.isScheduled()
- Chore.getSleeper().skipSleepCycle() -> ScheduledChore.triggerNow()
Important Notes on recent changes:
- There are two chores in the codebase that don't seem to do anything at the
moment. The first is DelayedClosing (defined within ConnectionManager.java).
The second is MovedRegionsCleaner (defined within HRegionServer). While
converting them into ScheduledChores I noticed that when they are used,
instances of the chores are created but the threads are never started. I looked
into it a bit and the issue with DelayedClosing is called out in HBASE-11354,
but I couldn't find anything that spoke about MovedRegionsCleaner. As it
stands, I have not changed this behavior (i.e. both chores still don't do
anything) but I thought I should raise the issue regardless.
- With the movement to this new implementation of Chores, it should be
understood that we will only see a reduction in the number of threads in cases
where we can schedule many chores with a common ChoreService object (e.g.
schedule 5 chores with a single instance of ChoreService). Scheduling each
chore with a different ChoreService instance will not provide a reduction in
the number of threads since each instance will always have at least one thread
in its core pool (in fact, scheduling each Chore with a separate instance of
ChoreService would be equivalent to the current implementation of Chores since
it would be a thread per Chore).
- While it would have been nice to have a single ChoreService that all
ScheduledChores could talk to, this did not seem feasible. Instead, an instance
of ChoreService has been created inside HRegionServer, and "getChoreService()"
has been added to the interfaces RegionServerServices and MasterServices. Most
of the places where Chores were used had access to an HRegionServer, so this
allowed many of the existing chores to be scheduled under a common ChoreService
and thus reduced the number of threads. However, in instances where there
wasn't access to an HRegionServer, an instance of ChoreService had to be
defined. Note that this is simply equivalent to the old implementation of
Chores (1 thread per chore).
Summary:
- If you observe the BEFORE(Chore) and AFTER(ScheduledChore) thread dumps, you
will see that the movement to the usage of ChoreService has reduced the number
of threads used for Chores from 12 to 3 which is promising.
- While the thread count may increase (due to a larger Chore load), we are
guaranteed that the new implementation's worst case thread count will never
exceed the previous implementation's thread count.
Any feedback with regards to the patch above would be greatly appreciated.
Thanks
> Deprecate Chore; its a thread per task when we should have one thread to do
> all tasks
> -------------------------------------------------------------------------------------
>
> Key: HBASE-6778
> URL: https://issues.apache.org/jira/browse/HBASE-6778
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: Jonathan Lawlor
> Attachments: AFTER_thread_dump.txt, BEFORE_thread_dump.txt,
> HBASE_6778_WIP_v1.patch, HBASE_6778_WIP_v2.patch,
> thread_dump_HMaster.local.out
>
>
> Should use something like ScheduledThreadPoolExecutor instead (Elliott said
> this first I think; J-D said something similar just now).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
