[
https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836407#comment-13836407
]
Hudson commented on TAJO-317:
-----------------------------
SUCCESS: Integrated in Tajo-trunk-postcommit #581 (See
[https://builds.apache.org/job/Tajo-trunk-postcommit/581/])
TAJO-317: Improve TajoResourceManager to support more elaborate resource
management. (Keuntae Park via jihoon) (jihoonson:
https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=528c914f9a133bef79df07017cfa424c9fab4412)
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
* CHANGES.txt
* tajo-core/tajo-core-backend/src/main/proto/TajoMasterProtocol.proto
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/util/ApplicationIdUtils.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/ResourceAllocator.java
*
tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TajoTestingCluster.java
* tajo-core/tajo-core-backend/src/main/resources/webapps/admin/cluster.jsp
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/WorkerResource.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/YarnResourceAllocator.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoMasterService.java
* tajo-core/tajo-core-backend/src/main/resources/webapps/admin/index.jsp
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorker.java
*
tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestTajoResourceManager.java
* tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
*
tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/rm/YarnTajoResourceManager.java
> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>
> Key: TAJO-317
> URL: https://issues.apache.org/jira/browse/TAJO-317
> Project: Tajo
> Issue Type: Improvement
> Components: resource manager
> Reporter: Hyunsik Choi
> Assignee: Keuntae Park
> Fix For: 0.8-incubating
>
> Attachments: TAJO-317.patch, TAJO-317_2.patch, TAJO-317_3.patch,
> TAJO-317_4.patch, TAJO-317_5.patch
>
>
> h3. Status of the current Tajo Resource Manager (RM)
> * Tajo RM manages CPU, DISK resource incompletely, and it only provides
> resource management through memory allocations.
> * In addition, Tajo RM considers the memory resource as the fixed number of
> slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU
> and memory consuming job. For example, scan and hash partition or INSERT
> OVERWRITE may be belong to I/O intensive job. In general, Aggregation can be
> belong to CPU-memory consuming job. The current RM is not fit to support
> selectively I/O intensive job or CPU-memory consuming job because it provides
> only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less
> than required resource is not allocated in response to a resource request. It
> is not good to fully utilize the cluster resources. In order to mitigate this
> problem, we need to add resilience to allocation mechanism. For example,
> min-max request would be useful for it.
> h3. Proposal
> * Tajo RM should provides resource management for disk and cpu-memory.
> ** Tajo RM should provide allocation request call with min, max memory
> request, and min, max disk request.
> *** min-max request will be useful to fully utilize remain cluster resources.
> * Each resource request should have a priority. The priority can be disk or
> memory.
> ** If the priority is disk
> *** disk allocation will be limited depending on the remain disk resource
> *** memory allocation will be not limited regardless of the remain memory
> resource, and just reduce the remain memory resource.
> ** If the priority is memory
> *** memory allocation will be limited depending on the remain memory
> resource
> *** disk allocation will be not limited regardless of the remain disk
> resource, and just reduce the remain disk resource.
> * disk resource in each worker is represented as a float value.
> ** The initial disk resource will be the number of disks which participate
> in HDFS data directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
