[
https://issues.apache.org/jira/browse/TAJO-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630912#comment-14630912
]
ASF GitHub Bot commented on TAJO-1397:
--------------------------------------
Github user jinossy commented on a diff in the pull request:
https://github.com/apache/tajo/pull/608#discussion_r34868269
--- Diff:
tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java
---
@@ -251,63 +250,70 @@ public void handle(TaskSchedulerEvent event) {
}
}
- @Override
- public void handleTaskRequestEvent(TaskRequestEvent event) {
-
- taskRequests.handle(event);
- int hosts = scheduledRequests.leafTaskHostMapping.size();
+ private Set<Integer> getWorkerIds(Collection<String> hosts){
+ Set<Integer> workerIds = Sets.newHashSet();
+ if(hosts.isEmpty()) return workerIds;
- // if available cluster resource are large then tasks, the scheduler
thread are working immediately.
- if(remainingScheduledObjectNum() > 0 &&
- (remainingScheduledObjectNum() <= hosts || hosts <=
taskRequests.size())){
- synchronized (schedulingThread){
- schedulingThread.notifyAll();
+ for (WorkerConnectionInfo worker :
stage.getContext().getWorkerMap().values()) {
+ if(hosts.contains(worker.getHost())){
+ workerIds.add(worker.getId());
}
}
+ return workerIds;
}
- @Override
- public int remainingScheduledObjectNum() {
- return scheduledObjectNum;
- }
- private class TaskRequests implements EventHandler<TaskRequestEvent> {
- private final LinkedBlockingQueue<TaskRequestEvent> taskRequestQueue =
- new LinkedBlockingQueue<TaskRequestEvent>();
+ protected LinkedList<TaskRequestEvent> createTaskRequest() throws
Exception {
+ LinkedList<TaskRequestEvent> taskRequestEvents = new
LinkedList<TaskRequestEvent>();
- @Override
- public void handle(TaskRequestEvent event) {
- if(LOG.isDebugEnabled()){
- LOG.debug("TaskRequest: " + event.getContainerId() + "," +
event.getExecutionBlockId());
- }
+ int requestContainerNum = Math.max(remainingScheduledObjectNum(), 1);
--- End diff --
Nice finding! It is my mistake. I will remove the Math.max
> Resource allocation should be fine grained.
> -------------------------------------------
>
> Key: TAJO-1397
> URL: https://issues.apache.org/jira/browse/TAJO-1397
> Project: Tajo
> Issue Type: Improvement
> Components: QueryMaster, resource manager, Worker
> Reporter: Hyunsik Choi
> Assignee: Jinho Kim
> Fix For: 0.11.0
>
> Attachments: ResoruceSequence.jpg, TAJO-1397.patch,
> TAJO-1397_2.patch, old_resource_circuit.png, resource_circuit.png
>
>
> See the comment:
> https://issues.apache.org/jira/browse/TAJO-540?focusedCommentId=14359478&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14359478
> From the discussion in TAJO-540
> {quote}
> In general, query (or job) scheduler aims at the maximum resource
> utilization. For multi-tenancy, we also need to consider the fairness for
> multiple users (or queries). BTW, the maximum resource utilization and
> fairness are usually conflict to each other in many cases. To mitigate this
> problem, many scheduler seems to use preemption approach.
> In this point, our resource and scheduler system has the following problems:
> * A query exclusively uses allocated resources at the first time until the
> query is completed or failed.
> * There is no mechanism to deallocate resources during query processing.
> * Preempt is also not allowed.
> To achieve the multi tenancy, we should change our resource circulation.
> Especially, resource allocation must be fine grained instead of per query.
> So, I'll create a jira issue to change the resource circulation. We have to
> do this issue firstly in my opinion. If we achieve this, implementing
> multi-tenant scheduler would be much easier than now. It would be a good
> starting point of this issue.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
