[
https://issues.apache.org/jira/browse/HADOOP-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vivek Ratan updated HADOOP-4980:
--------------------------------
Attachment: 4980.1.patch
Attaching patch 4980.1.patch. Following changes have been made:
* _tasksKilled_ member variable of _ReclaimResource_ class removed (never used
anywhere)
* Some of the member variables in _QueueSchedulingInfo_, such as the user limit
_ulMin_, applied to a queue while others, such as _guaranteedCapacityPercent_,
were applicable to both map and reduce tasks. Rather than have two separate
objects of type _QueueSchedulingInfo_, one each for map and reduce tasks, and
thus duplicating the queue-specific information, we now have a separate
_TaskSchedulingInfo_ class that encapsulates map/reduce-specific information.
Queue-specific information remains in _QueueSchedulingInfo_. This also makes it
much easier to print UI information. Each class' _toString_ method prints out
its relevant information, which can be concatenated.
* Renamed _SchedulingInfo_ class to _SchedulingDisplayInfo_. This class has
been changed to be outside of the Scheduler - it is only aware of the name of
the queue it represents. Its _toString_ method, that is called to obtain the UI
display information, now simply gets the display information from the scheduler
object. This does three things: the _SchedulingDisplayInfo_ interacts with the
scheduler object and is oblivious to its internal data structures, the display
information is created by the _QueueSchedulingInfo_ (which contains all data),
and the call to the scheduler object is synchronized.
* The _TaskLookupResult_ and _TaskLookUpStatus_ classes have been reorganized.
The latter is moved inside the former, and static factory methods have been
provided to create the common task lookup objects. The objects that represent
no task being found and task not matching memory requirements do not need to be
created again again and again as they're basically read-only, so we avoid
creating many such objects by sharing a single instance.
* I've cleaned up the logging code. We were logging the same result in multiple
places.
* Some methods and variables have been renamed to better denote what they
represent.
* The core scheduling algorithm, that of obtaining a task from a queue, is all
in one place, within _getTaskFromQueue_. This improves readability and makes it
clear what the code is doing. It will also help with metrics collection, as
there is now one place where the scheduling decisions are taken.
* The thread for redistributing capacity is started only if there is more than
queue.
None of the scheduling logic has changed.
> Cleanup the Capacity Scheduler code
> -----------------------------------
>
> Key: HADOOP-4980
> URL: https://issues.apache.org/jira/browse/HADOOP-4980
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/capacity-sched
> Reporter: Vivek Ratan
> Attachments: 4980.1.patch
>
>
> Given the number of changes that have been made by different folks to the
> Capacity Scheduler code, the code needs to be cleaned up. Some comments and
> variable names are misleading, and the core logic is not in a central place,
> making it harder to understand.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
