[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276902#comment-17276902 ] zhuqi commented on YARN-10532: -- [~wangda] [~gandras] Attached a new patch for review. Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10532: - Attachment: YARN-10532.009.patch > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276811#comment-17276811
]
zhuqi edited comment on YARN-10532 at 2/2/21, 5:40 AM:
---
Thanks a lot for [~gandras] patient review, it make sense to me.
# I used to think, parent should also check last submit time passed by leaf
queue, it seems this is not needed.
# But i think we should check all applications in leaf queue, not only
"getNumActiveApplications() == 0".
# The garbage collector in AutoDeletionForExpiredQueuePolicy will make this
more clear, i agree. But i think the logic should not in the for cycle:
{code:java}
Set newMarks = new HashSet<>();
for (Map.Entry queueEntry :
scheduler.getCapacitySchedulerQueueManager().getQueues().entrySet()) {
String queuePath = queueEntry.getKey();
CSQueue queue = queueEntry.getValue();
if (queue instanceof AbstractCSQueue &&
((AbstractCSQueue) queue).isEligibleForAutoDeletion()) {
if (markedForDeletion.contains(queuePath)) {
sentForDeletion.add(queuePath);
markedForDeletion.remove(queuePath);
} else {
newMarks.add(queuePath);
}
}
}
markedForDeletion.clear();
markedForDeletion.addAll(newMarks);
{code}
I will update a new patch later today, your suggestions are very valid, i am
glad to work with you.:)
was (Author: zhuqi):
Thanks a lot for [~gandras] patient review, it make sense to me.
# I used to think, parent should also check last submit time passed by leaf
queue, it seems this is not needed.
# But i think we should check all applications in leaf queue, not only
"getNumActiveApplications() == 0".
# The garbage collector in AutoDeletionForExpiredQueuePolicy will make this
more clear, i agree.
I will update a new patch later today, your suggestions are very valid, i am
glad to work with you.:)
> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is
> not being used
>
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch,
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch,
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch
>
>
> It's better if we can delete auto-created queues when they are not in use for
> a period of time (like 5 mins). It will be helpful when we have a large
> number of auto-created queues (e.g. from 500 users), but only a small subset
> of queues are actively used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276811#comment-17276811 ] zhuqi commented on YARN-10532: -- Thanks a lot for [~gandras] patient review, it make sense to me. # I used to think, parent should also check last submit time passed by leaf queue, it seems this is not needed. # But i think we should check all applications in leaf queue, not only "getNumActiveApplications() == 0". # The garbage collector in AutoDeletionForExpiredQueuePolicy will make this more clear, i agree. I will update a new patch later today, your suggestions are very valid, i am glad to work with you.:) > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 3:08 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
I update a new patch to fix this.
I change the code to :
{code:java}
// When the partition not meet the priority it will return
// AllocationState.PARTITION_SKIPPED, this should also skip.
if (allocationState == AllocationState.PRIORITY_SKIPPED
|| allocationState == AllocationState.PARTITION_SKIPPED) {
continue;
}
{code}
Also , i think the test case is not right, we should add a right test, or this
may not need a new test at my side.
Any other thoughts?
was (Author: zhuqi):
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
I change the code to :
{code:java}
// When the partition not meet the priority it will return
// AllocationState.PARTITION_SKIPPED, this should also skip.
if (allocationState == AllocationState.PRIORITY_SKIPPED
|| allocationState == AllocationState.PARTITION_SKIPPED) {
continue;
}
{code}
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu
[jira] [Updated] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10589:
-
Attachment: YARN-10589-002.patch
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch, YARN-10589-002.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 3:00 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
I change the code to :
{code:java}
// When the partition not meet the priority it will return
// AllocationState.PARTITION_SKIPPED, this should also skip.
if (allocationState == AllocationState.PRIORITY_SKIPPED
|| allocationState == AllocationState.PARTITION_SKIPPED) {
continue;
}
{code}
Any other thoughts?
was (Author: zhuqi):
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
I change the code to :
{code:java}
// When the partition not meet the priority it will return
// AllocationState.PARTITION_SKIPPED, this should also skip.
if (allocationState == AllocationState.PRIORITY_SKIPPED
|| allocationState == AllocationState.PARTITION_SKIPPED) {
continue;
}
{code}
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Maj
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:59 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
I change the code to :
{code:java}
// When the partition not meet the priority it will return
// AllocationState.PARTITION_SKIPPED, this should also skip.
if (allocationState == AllocationState.PRIORITY_SKIPPED
|| allocationState == AllocationState.PARTITION_SKIPPED) {
continue;
}
{code}
Any other thoughts?
was (Author: zhuqi):
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(part
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:48 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
It make sense to me now.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
Any other thoughts?
was (Author: zhuqi):
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:46 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If we change to return ContainerAllocation.PARTITION_SKIPPED in patch, we
should also skip this priority.
Any other thoughts?
was (Author: zhuqi):
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:42 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and the patch.
I have reviewed the patch, it break the PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
Any other thoughts?
was (Author: zhuqi):
Thanks for [~ztang] [~tanu.ajmera] review and patch.
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If this partition not meet this priority, why we can think this partition not
meet all priorities of the applications.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:37 PM:
---
Thanks for [~ztang] [~tanu.ajmera] review and patch.
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If this partition not meet this priority, why we can think this partition not
meet all priorities of the applications.
Any other thoughts?
was (Author: zhuqi):
[~ztang] [~tanu.ajmera]
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If this partition not meet this priority, why we can think this partition not
meet all priorities of the applications.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
>
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:36 PM:
---
[~ztang] [~tanu.ajmera]
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
// will skip priority not meet
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If this partition not meet this priority, why we can think this partition not
meet all priorities of the applications.
Any other thoughts?
was (Author: zhuqi):
[~ztang] [~tanu.ajmera]
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If this partition not meet this priority, why we can think this partition not
meet all priorities of the applications.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi edited comment on YARN-10589 at 2/1/21, 2:35 PM:
---
[~ztang] [~tanu.ajmera]
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
If this partition not meet this priority, why we can think this partition not
meet all priorities of the applications.
Any other thoughts?
was (Author: zhuqi):
[~ztang] [~tanu.ajmera]
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
--
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276369#comment-17276369
]
zhuqi commented on YARN-10589:
--
[~ztang] [~tanu.ajmera]
I have reviewed the patch, i don't think the logic is needed, and it break the
PRIORITY_SKIPPED logic.
In Schedule in priority order:
{code:java}
// Schedule in priority order
for (SchedulerRequestKey schedulerKey : application.getSchedulerKeys()) {
ContainerAllocation result = allocate(clusterResource, candidates,
schedulingMode, resourceLimits, schedulerKey, null);
AllocationState allocationState = result.getAllocationState();
if (allocationState == AllocationState.PRIORITY_SKIPPED) {
continue;
}
return getCSAssignmentFromAllocateResult(clusterResource, result,
null, node);
}
if (reservedContainer == null) {
result = preCheckForNodeCandidateSet(clusterResource, node,
schedulingMode, resourceLimits, schedulerKey);
if (null != result) {
continue;
}
}
{code}
I think the original logic is right, in preCheckForNodeCandidateSet:
{code:java}
// Is the nodePartition of pending request matches the node's partition
// If not match, jump to next priority.
Optional dcOpt = activitiesManager == null ?
Optional.empty() :
activitiesManager.getOptionalDiagnosticsCollector();
if (!appInfo.precheckNode(schedulerKey, node, schedulingMode, dcOpt)) {
ActivitiesLogger.APP.recordSkippedAppActivityWithoutAllocation(
activitiesManager, node, application, schedulerKey,
ActivityDiagnosticConstant.
NODE_DO_NOT_MATCH_PARTITION_OR_PLACEMENT_CONSTRAINTS
+ ActivitiesManager.getDiagnostics(dcOpt),
ActivityLevel.NODE);
return ContainerAllocation.PRIORITY_SKIPPED;
}
{code}
We have skipped the priority, and the next priority may need this node
partition. We just skipped the priority of application.
Any other thoughts?
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10589-001.patch
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275832#comment-17275832 ] zhuqi commented on YARN-10532: -- [~wangda] [~gandras] I have updated a new patch to make the code more clear. I have created a new Policy "(maybe we can make it runnable by default so we don't have to create another config) as [~wangda] suggested". The policy just simply monitor queue last used time and delete queues when needed. We can enable this, by adding AutoDeletionForExpiredQueuePolicy to the conf : "scheduler.monitor.policies". I also handled deletion of ParentQueues which without child queues. And i removed the reinitialize related logic, i think we don't need it when default enabled auto deletion. If you any other thoughts. Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10532: - Attachment: YARN-10532.008.patch > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10178:
-
Attachment: YARN-10178.004.patch
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch,
> YARN-10178.003.patch, YARN-10178.004.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
> if (app.accept(cluster, request, updatePending)
> && app.apply(cluster, request, updatePending)) { // apply this
> resource
> ...
> }
> }
> }
[jira] [Commented] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274240#comment-17274240 ] zhuqi commented on YARN-10352: -- Thanks [~ztang] for review. > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352-010.patch, YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10590: - Description: "Because as we discussed in YARN-10504 , the initialization of auto created queues from template was changed (see comment and comment)." 1. As the comment we discussed, we found the effective core is different(the gap), because the update effective will override the absolute auto created leaf queue. 2. But actually, the new logic in YARN-10504 override is right, the difference is caused by test case , don't consider the caculation loss of multi resource type, the cap/absolute are all caculated by one type, (memory) in default resourcecaculator, (dominant type) in dominant resourcecaculator. As we known in the comment, the absolute auto created leaf queue will merge the effective resource by cap/absolute caculated result, this caused the gap. 2. In other case(not absolute case) in the auto created leaf queue, the merge will not cause the gap, in update effective resource override will also use the one type caculated result. 3. So this jira just make the test right, the caculation result is already right. was: Because we have introduced YARN-10504. In auto created leaf queue with absolute mode, the caculation of effective min resource will different from other mode. We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > > "Because as we discussed in YARN-10504 , the initialization of auto created > queues from template was changed (see comment and comment)." > 1. As the comment we discussed, we found the effective core is different(the > gap), because the update effective will override the absolute auto created > leaf queue. > 2. But actually, the new logic in YARN-10504 override is right, the > difference is caused by test case , don't consider the caculation loss of > multi resource type, the cap/absolute are all caculated by one type, (memory) > in default resourcecaculator, (dominant type) in dominant resourcecaculator. > As we known in the comment, the absolute auto created leaf queue will merge > the effective resource by cap/absolute caculated result, this caused the gap. > 2. In other case(not absolute case) in the auto created leaf queue, the merge > will not cause the gap, in update effective resource override will also use > the one type caculated result. > 3. So this jira just make the test right, the caculation result is already > right. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274178#comment-17274178 ] zhuqi edited comment on YARN-10590 at 1/29/21, 7:09 AM: Thanks [~bteke] for review. "Because as we discussed in YARN-10504 , the initialization of auto created queues from template was changed (see comment and comment)." 1. As the comment we discussed, we found the effective core is different(the gap), because the update effective will override the absolute auto created leaf queue. 2. But actually, the new logic in YARN-10504 override is right, the difference is caused by test case , don't consider the caculation loss of multi resource type, the cap/absolute are all caculated by one type, (memory) in default resourcecaculator, (dominant type) in dominant resourcecaculator. As we known in the comment, the absolute auto created leaf queue will merge the effective resource by cap/absolute caculated result, this caused the gap. 2. In other case(not absolute case) in the auto created leaf queue, the merge will not cause the gap, in update effective resource override will also use the one type caculated result. 3. So this jira just make the test right, the caculation result is already right. In future we should make multi type caculation of cap/absolute, consistent with the absolute multi resource type. In Jira : YARN-10592 I introduced the vector of cap/absolute, it will make this consistent . Also in the future, we will handle other cases about multi resource type cases, it is a more huge thing to do, may be we need a new Umbrella to forward this. cc [~wangda] if you any thoughts about this and YARN-10592. Thanks. was (Author: zhuqi): Thanks [~bteke] for review. "Because as we discussed in YARN-10504 , the initialization of auto created queues from template was changed (see comment and comment)." 1. As the comment we discussed, we found the effective core is different(the gap), because the update effective will override the absolute auto created leaf queue. 2. But actually, the override is right, the difference is caused by test case , don't consider the caculation loss of multi resource type, the cap/absolute are all caculated by one type, (memory) in default resourcecaculator, (dominant type) in dominant resourcecaculator. As we known in the comment, the absolute auto created leaf queue will merge the effective resource by cap/absolute caculated result, this caused the gap. 2. In other case(not absolute case) in the auto created leaf queue, the merge will not cause the gap, in update effective resource override will also use the one type caculated result. 3. So this jira just make the test right, the caculation result is already right. In future we should make multi type caculation of cap/absolute, consistent with the absolute multi resource type. In Jira : YARN-10592 I introduced the vector of cap/absolute, it will make this consistent . Also in the future, we will handle other cases about multi resource type cases, it is a more huge thing to do, may be we need a new Umbrella to forward this. cc [~wangda] if you any thoughts about this and YARN-10592. Thanks. > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274178#comment-17274178 ] zhuqi commented on YARN-10590: -- Thanks [~bteke] for review. "Because as we discussed in YARN-10504 , the initialization of auto created queues from template was changed (see comment and comment)." 1. As the comment we discussed, we found the effective core is different(the gap), because the update effective will override the absolute auto created leaf queue. 2. But actually, the override is right, the difference is caused by test case , don't consider the caculation loss of multi resource type, the cap/absolute are all caculated by one type, (memory) in default resourcecaculator, (dominant type) in dominant resourcecaculator. As we known in the comment, the absolute auto created leaf queue will merge the effective resource by cap/absolute caculated result, this caused the gap. 2. In other case(not absolute case) in the auto created leaf queue, the merge will not cause the gap, in update effective resource override will also use the one type caculated result. 3. So this jira just make the test right, the caculation result is already right. In future we should make multi type caculation of cap/absolute, consistent with the absolute multi resource type. In Jira : YARN-10592 I introduced the vector of cap/absolute, it will make this consistent . Also in the future, we will handle other cases about multi resource type cases, it is a more huge thing to do, may be we need a new Umbrella to forward this. cc [~wangda] if you any thoughts about this and YARN-10592. Thanks. > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274163#comment-17274163
]
zhuqi commented on YARN-10178:
--
[~wangda] [~bteke]
I have updated a patch to sort PriorityQueueResourcesForSorting, and add
reference to queue.
I also add tests to prevent side effect/regression.
After the performance test, i found the there seems no performance cost:
1. And i am surprise that the queue size is about 1000, the new sort fast than
old.
2. When the queue size is huge big : 1, the old fast than the old, but the
gap is always less than 10s.
If you any thoughts about this?
Thanks a lot.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch,
> YARN-10178.003.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getA
[jira] [Updated] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10178:
-
Attachment: YARN-10178.003.patch
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch,
> YARN-10178.003.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
> if (app.accept(cluster, request, updatePending)
> && app.apply(cluster, request, updatePending)) { // apply this
> resource
> ...
> }
> }
> }
> return isSuccess;
[jira] [Commented] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17274122#comment-17274122
]
zhuqi commented on YARN-10178:
--
Thanks a lot for [~wangda] [~bteke] review.
It makes sense to me, [~wangda] you are right.
Compare will be called multiple times during one sorting call, i am not aware
this, this is the key to change my code.
I will try to add tests to prevent side effect/regression, after change the
logic.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be
[jira] [Comment Edited] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270581#comment-17270581
]
zhuqi edited comment on YARN-10178 at 1/28/21, 8:52 AM:
cc [~wangda] [~bteke] [~BilwaST] [~tuyu] [~gandras]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
And if we need to add some performance test cases to confirm the performance
loss about the snapshot.
Thanks.
was (Author: zhuqi):
cc [~wangda] [~bteke] [~BilwaST] [~tuyu]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
And if we need to add some performance test cases to confirm the performance
loss about the snapshot.
Thanks.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource clu
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17273287#comment-17273287 ] zhuqi commented on YARN-10532: -- [~gandras] Thanks for your review and valid suggestions. It makes sense: 1. I will remove the old auto created leaf deletion related logic. 2. Actually, the event triggered by an edit policy thread, which already a background thread that periodically checks , i will make it more readable or just add a new thread. 3. I will handle deletion of ParentQueues. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271183#comment-17271183 ] zhuqi edited comment on YARN-10592 at 1/26/21, 9:20 AM: cc [~wangda] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. And i think the multi resource type based cap/abs/weight , and related change is huge, if we should move this related to a new issue? was (Author: zhuqi): cc [~wangda] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch, YARN-10592.002.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Attachment: YARN-10592.002.patch > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch, YARN-10592.002.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[
https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269888#comment-17269888
]
zhuqi edited comment on YARN-10590 at 1/26/21, 2:22 AM:
cc [~wangda] [~sunilg] [~bteke]
When i deep into the loss of the caculation about
{code:java}
// TODO: Wangda, I think this is a wrong test, it doesn't consider rounding
// loss of multiplication, the right value should be <10240, 2>, but the
// test expects <10240, 1>
{code}
I think the test logic should change to :
{code:java}
// When in absolute mode.
// We can only assertEquals in memory size, because the cap/absolute
// is caculated by memory, we can only make sure this.
if (leafQueue.getCapacityConfigType() ==
AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) {
assertEquals(effMinCapacity.getMemorySize(),
leafQueue.getEffectiveCapacity(label).getMemorySize());
assertTrue(effMinCapacity.getVirtualCores()
<= leafQueue.getEffectiveCapacity(label).getVirtualCores());
} else {
assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label));
}
{code}
Because in absolute mode with auto created leaf queue, we get the cap/absolute
by :
{code:java}
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
...
}
}
{code}
In default resource caculator, we get the cap/absolute only by memory:
{code:java}
public float ratio(Resource a, Resource b) {
return divideSafelyAsFloat(a.getMemorySize(), b.getMemorySize());
}
{code}
This cause the loss of multiply by cap/absolute, with the actual effective min
resource.
The actual effective min resource will be updated in :
{code:java}
// Update effective resource (min/max) to each child queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
}{code}
We will get the actual effective min resource, different with the get the
cap/absolute only by memory.
So in absolute auto created leaf queue , the effective min resource only can
meet the memory caculated by cap/absolute with default resource caculator.
The logic is wright, but the test we should fix.
Thoughts?
Thanks.
was (Author: zhuqi):
cc [~leftnoteasy] [~sunilg] [~bteke]
When i deep into the loss of the caculation about
{code:java}
// TODO: Wangda, I think this is a wrong test, it doesn't consider rounding
// loss of multiplication, the right value should be <10240, 2>, but the
// test expects <10240, 1>
{code}
I think the test logic should change to :
{code:java}
// When in absolute mode.
// We can only assertEquals in memory size, because the cap/absolute
// is caculated by memory, we can only make sure this.
if (leafQueue.getCapacityConfigType() ==
AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) {
assertEquals(effMinCapacity.getMemorySize(),
leafQueue.getEffectiveCapacity(label).getMemorySize());
assertTrue(effMinCapacity.getVirtualCores()
<= leafQueue.getEffectiveCapacity(label).getVirtualCores());
} else {
assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label));
}
{code}
Because in absolute mode with auto created leaf queue, we get the cap/absolute
by :
{code:java}
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
...
}
}
{code}
In default resource caculator, we get the cap/absolute only by memory:
{co
[jira] [Comment Edited] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271183#comment-17271183 ] zhuqi edited comment on YARN-10592 at 1/26/21, 2:21 AM: cc [~wangda] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. was (Author: zhuqi): cc [~leftnoteasy] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271405#comment-17271405 ] zhuqi commented on YARN-10585: -- Thank [~shuzirra] for working on this. The latest patch LGTM. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Summary: Support QueueCapacities to use vector based multi resource types, and update absolute related to use first. (was: Support QueueCapacities to use vector, and update absolute related to use first.) > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271186#comment-17271186
]
zhuqi commented on YARN-10531:
--
Thanks [~wangda] for final review.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271183#comment-17271183 ] zhuqi edited comment on YARN-10592 at 1/25/21, 9:16 AM: cc [~leftnoteasy] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. was (Author: zhuqi): cc [~leftnoteasy] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271183#comment-17271183 ] zhuqi commented on YARN-10592: -- cc [~leftnoteasy] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Description: Because we need to support YARN-10524 and related other multi type support. Support QueueCapacities to use vector. Also this patch has to update absolute related to use first, to make sure no loss of multi type resource fixed by YARN-10590. was: Because we need to support YARN-10524 and related other multi type support. Support QueueCapacities to use vector. Also this patch has to > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Description: Because we need to support YARN-10524 and related other multi type support. Support QueueCapacities to use vector. Also this patch has to was: Because we need to support YARN-10524 and related other multi type support. We should support cap/absolute vector > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Description: Because we need to support YARN-10524 and related other multi type support. We should support cap/absolute vector > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we need to support YARN-10524 and related other multi type support. > We should support cap/absolute vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
zhuqi created YARN-10592: Summary: Support QueueCapacities to use vector, and update absolute related to use first. Key: YARN-10592 URL: https://issues.apache.org/jira/browse/YARN-10592 Project: Hadoop YARN Issue Type: Sub-task Reporter: zhuqi Assignee: zhuqi -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270635#comment-17270635
]
zhuqi edited comment on YARN-10589 at 1/23/21, 2:04 PM:
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
The partition with too many nodes has been improved already.
And the fix version should not set, if not fixed, i removed it.
was (Author: zhuqi):
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
The partition with too many nodes has been improved already.
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10589:
-
Fix Version/s: (was: 3.4.0)
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270635#comment-17270635
]
zhuqi edited comment on YARN-10589 at 1/23/21, 11:44 AM:
-
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
The partition with too many nodes has been improved already.
was (Author: zhuqi):
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
The partition with too much nodes has been improved already.
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Fix For: 3.4.0
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270635#comment-17270635
]
zhuqi edited comment on YARN-10589 at 1/23/21, 11:43 AM:
-
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
The partition with too much nodes has been improved already.
was (Author: zhuqi):
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Fix For: 3.4.0
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10589) Improve logic of multi-node allocation
[
https://issues.apache.org/jira/browse/YARN-10589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270635#comment-17270635
]
zhuqi commented on YARN-10589:
--
[~tanu.ajmera]
The logic has been improved from multi (node->partition) to one (partition).
> Improve logic of multi-node allocation
> --
>
> Key: YARN-10589
> URL: https://issues.apache.org/jira/browse/YARN-10589
> Project: Hadoop YARN
> Issue Type: Task
>Affects Versions: 3.3.0
>Reporter: Tanu Ajmera
>Assignee: Tanu Ajmera
>Priority: Major
> Fix For: 3.4.0
>
>
> {code:java}
> for (String partititon : partitions) {
> if (current++ > start) {
> break;
> }
> CandidateNodeSet candidates =
> cs.getCandidateNodeSet(partititon);
> if (candidates == null) {
> continue;
> }
> cs.allocateContainersToNode(candidates, false);
> }{code}
> In above logic, if we have thousands of node in one partition, we will still
> repeatedly access all nodes of the partition thousands of times. There is no
> break point where if the partition is not same for the first node, it should
> stop checking other nodes in that partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10178:
-
Attachment: YARN-10178.002.patch
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
> if (app.accept(cluster, request, updatePending)
> && app.apply(cluster, request, updatePending)) { // apply this
> resource
> ...
> }
> }
> }
> return isSuccess;
> }
> }
> {code}
> {code
[jira] [Updated] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10178:
-
Target Version/s: 3.4.0
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
> if (app.accept(cluster, request, updatePending)
> && app.apply(cluster, request, updatePending)) { // apply this
> resource
> ...
> }
> }
> }
> return isSuccess;
> }
> }
> {code}
> {code:java}
> public boolean apply(R
[jira] [Comment Edited] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270581#comment-17270581
]
zhuqi edited comment on YARN-10178 at 1/23/21, 7:28 AM:
cc [~wangda] [~bteke] [~BilwaST] [~tuyu]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
And if we need to add some performance test cases to confirm the performance
loss about the snapshot.
Thanks.
was (Author: zhuqi):
cc [~wangda] [~bteke] [~BilwaST]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
And if we need to add some performance test cases to confirm the performance
loss about the snapshot.
Thanks.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> bool
[jira] [Comment Edited] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270581#comment-17270581
]
zhuqi edited comment on YARN-10178 at 1/23/21, 7:12 AM:
cc [~wangda] [~bteke] [~BilwaST]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
And if we need to add some performance test cases to confirm the performance
loss about the snapshot.
Thanks.
was (Author: zhuqi):
cc [~wangda] [~bteke] [~BilwaST]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
Thanks.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (Reso
[jira] [Commented] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270581#comment-17270581
]
zhuqi commented on YARN-10178:
--
cc [~wangda] [~bteke] [~BilwaST]
Since the bug and related YARN-8737 have raised for more than 2 months, i have
faced the same issue recently. i have assigned to myself, and attached a patch
to fix it.
If you can help review it?
Thanks.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAtte
[jira] [Assigned] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi reassigned YARN-10178:
Assignee: zhuqi (was: Benjamin Teke)
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
> if (app.accept(cluster, request, updatePending)
> && app.apply(cluster, request, updatePending)) { // apply this
> resource
> ...
> }
> }
> }
> return isSuccess;
> }
> }
> {code}
> {code:java}
> public boolean apply(Resource cluster, Resource
[jira] [Commented] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270518#comment-17270518 ] zhuqi commented on YARN-10352: -- Thanks [~bibinchundatt] for review. > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352-010.patch, YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10555) missing access check before getAppAttempts
[
https://issues.apache.org/jira/browse/YARN-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269915#comment-17269915
]
zhuqi commented on YARN-10555:
--
Thanks for [~xiaoheipangzi] for this.
The patch LGTM.
> missing access check before getAppAttempts
> ---
>
> Key: YARN-10555
> URL: https://issues.apache.org/jira/browse/YARN-10555
> Project: Hadoop YARN
> Issue Type: Bug
> Components: webapp
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Labels: pull-request-available, security
> Attachments: YARN-10555_1.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> It seems that we miss a security check before getAppAttempts, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1127]
> thus we can get the some sensitive information, like logs link.
> {code:java}
> application_1609318368700_0002 belong to user2
> user1@hadoop11$ curl --negotiate -u :
> http://hadoop11:8088/ws/v1/cluster/apps/application_1609318368700_0002/appattempts/|jq
> {
> "appAttempts": {
> "appAttempt": [
> {
> "id": 1,
> "startTime": 1609318411566,
> "containerId": "container_1609318368700_0002_01_01",
> "nodeHttpAddress": "hadoop12:8044",
> "nodeId": "hadoop12:36831",
> "logsLink":
> "http://hadoop12:8044/node/containerlogs/container_1609318368700_0002_01_01/user2";,
> "blacklistedNodes": "",
> "nodesBlacklistedBySystem": ""
> }
> ]
> }
> }
> {code}
> Other apis, like getApps and getApp, has access check like "hasAccess(app,
> hsr)", they would hide the logs link if the appid do not belong to query
> user, see
> [https://github.com/apache/hadoop/blob/513f1995adc9b73f9c7f4c7beb89725b51b313ac/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java#L1098]
> We need add hasAccess(app, hsr) for getAppAttempts.
>
> Besides, at
> [https://github.com/apache/hadoop/blob/580a6a75a3e3d3b7918edeffd6e93fc211166884/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMAppBlock.java#L145]
> it seems that we have a access check in its caller, so now i pass "true" to
> AppAttemptInfo in the patch.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10590: - Attachment: YARN-10590.002.patch > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[
https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269888#comment-17269888
]
zhuqi commented on YARN-10590:
--
cc [~leftnoteasy] [~sunilg] [~bteke]
When i deep into the loss of the caculation about
{code:java}
// TODO: Wangda, I think this is a wrong test, it doesn't consider rounding
// loss of multiplication, the right value should be <10240, 2>, but the
// test expects <10240, 1>
{code}
I think the test logic should change to :
{code:java}
// When in absolute mode.
// We can only assertEquals in memory size, because the cap/absolute
// is caculated by memory, we can only make sure this.
if (leafQueue.getCapacityConfigType() ==
AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) {
assertEquals(effMinCapacity.getMemorySize(),
leafQueue.getEffectiveCapacity(label).getMemorySize());
assertTrue(effMinCapacity.getVirtualCores()
<= leafQueue.getEffectiveCapacity(label).getVirtualCores());
} else {
assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label));
}
{code}
Because in absolute mode with auto created leaf queue, we get the cap/absolute
by :
{code:java}
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
...
}
}
{code}
In default resource caculator, we get the cap/absolute only by memory:
{code:java}
public float ratio(Resource a, Resource b) {
return divideSafelyAsFloat(a.getMemorySize(), b.getMemorySize());
}
{code}
This cause the loss of multiply by cap/absolute, with the actual effective min
resource.
The actual effective min resource will be updated in :
{code:java}
// Update effective resource (min/max) to each child queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
}{code}
We will get the actual effective min resource, different with the get the
cap/absolute only by memory.
So in absolute auto created leaf queue , the effective min resource only can
meet the memory caculated by cap/absolute with default resource caculator.
The logic is wright, but the test we should fix.
Thoughts?
Thanks.
> Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute
> caculation loss.
> -
>
> Key: YARN-10590
> URL: https://issues.apache.org/jira/browse/YARN-10590
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10590.001.patch
>
>
> Because we have introduced YARN-10504.
> In auto created leaf queue with absolute mode, the caculation of effective
> min resource will different from other mode.
> We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10590: - Description: Because we have introduced YARN-10504. In auto created leaf queue with absolute mode, the caculation of effective min resource will different from other mode. We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. was: Because we have introduced YARN-10504. In auto created leaf queue with absolute mode, the caculation of effective min resource will different from other mode. We should fix the related test in > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10590: - Description: Because we have introduced YARN-10504. In auto created leaf queue with absolute mode, the caculation of effective min resource will different from other mode. We should fix the related test in > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10590: - Summary: Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss. (was: Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation.) > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation.
zhuqi created YARN-10590: Summary: Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation. Key: YARN-10590 URL: https://issues.apache.org/jira/browse/YARN-10590 Project: Hadoop YARN Issue Type: Sub-task Reporter: zhuqi Assignee: zhuqi -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269009#comment-17269009
]
zhuqi commented on YARN-10531:
--
[~wangda]
I have fixed above, in latest patch.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269009#comment-17269009
]
zhuqi edited comment on YARN-10531 at 1/21/21, 3:29 AM:
Thanks [~wangda] for review.
I have fixed above, in latest patch.
was (Author: zhuqi):
[~wangda]
I have fixed above, in latest patch.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10531:
-
Attachment: YARN-10531.007.patch
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10531:
-
Attachment: YARN-10531.006.patch
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268983#comment-17268983 ] zhuqi commented on YARN-10587: -- [~wangda] [~sunilg] Fix related TestAbsoluteResourceConfiguration in latest patch. > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch, YARN-10587.002.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: YARN-10587.002.patch > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch, YARN-10587.002.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: (was: YARN-10587.001.patch) > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: (was: YARN-10587.001.patch) > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Attachment: YARN-10587.001.patch > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10587.001.patch > > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[
https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268670#comment-17268670
]
zhuqi commented on YARN-10587:
--
cc [~leftnoteasy] [~bteke]
When i deep into the testAutoCreateLeafQueueCreation failed about :
{code:java}
// TODO: Wangda: I think this test case is not correct, Sunil could help look
// into details.
{code}
I find the root reason is , when absolute mode enabled in auto created leaf
queue, we finally will update cap related for the UI in :
{code:java}
private void deriveCapacityFromAbsoluteConfigurations(String label,
Resource clusterResource, ResourceCalculator rc) {
/*
* In case when queues are configured with absolute resources, it is better
* to update capacity/max-capacity etc w.r.t absolute resource as well. In
* case of computation, these values wont be used any more. However for
* metrics and UI, its better these values are pre-computed here itself.
*/
// 1. Update capacity as a float based on parent's minResource
float f = rc.divide(clusterResource,
queueResourceQuotas.getEffectiveMinResource(label),
parent.getQueueResourceQuotas().getEffectiveMinResource(label));
queueCapacities.setCapacity(label, Float.isInfinite(f) ? 0 : f);
// 2. Update max-capacity as a float based on parent's maxResource
f = rc.divide(clusterResource,
queueResourceQuotas.getEffectiveMaxResource(label),
parent.getQueueResourceQuotas().getEffectiveMaxResource(label));
queueCapacities.setMaximumCapacity(label, Float.isInfinite(f) ? 0 : f);
// 3. Update absolute capacity as a float based on parent's minResource and
// cluster resource.
queueCapacities.setAbsoluteCapacity(label,
queueCapacities.getCapacity(label) * parent.getQueueCapacities()
.getAbsoluteCapacity(label));
// 4. Update absolute max-capacity as a float based on parent's maxResource
// and cluster resource.
queueCapacities.setAbsoluteMaximumCapacity(label,
queueCapacities.getMaximumCapacity(label) * parent.getQueueCapacities()
.getAbsoluteMaximumCapacity(label));
// Re-visit max applications for a queue based on absolute capacity if
// needed.
if (this instanceof LeafQueue) {
LeafQueue leafQueue = (LeafQueue) this;
CapacitySchedulerConfiguration conf = csContext.getConfiguration();
int maxApplications = conf.getMaximumApplicationsPerQueue(queuePath);
if (maxApplications < 0) {
int maxGlobalPerQueueApps = conf.getGlobalMaximumApplicationsPerQueue();
if (maxGlobalPerQueueApps > 0) {
maxApplications = (int) (maxGlobalPerQueueApps * queueCapacities
.getAbsoluteCapacity(label));
} else{
maxApplications =
(int) (conf.getMaximumSystemApplications() * queueCapacities
.getAbsoluteCapacity(label));
}
}
leafQueue.setMaxApplications(maxApplications);
int maxApplicationsPerUser = Math.min(maxApplications,
(int) (maxApplications * (leafQueue.getUsersManager().getUserLimit()
/ 100.0f) * leafQueue.getUsersManager().getUserLimitFactor()));
leafQueue.setMaxApplicationsPerUser(maxApplicationsPerUser);
LOG.info("LeafQueue:" + leafQueue.getQueuePath() + ", maxApplications="
+ maxApplications + ", maxApplicationsPerUser="
+ maxApplicationsPerUser + ", Abs Cap:" + queueCapacities
.getAbsoluteCapacity(label) + ", Cap: " + queueCapacities
.getCapacity(label) + ", MaxCap : " + queueCapacities
.getMaximumCapacity(label));
}
}
{code}
But the queueResourceQuotas.getEffectiveMinResource is not correct when we add
a absolute auto created leaf queue, just exceed the capacity:
{code:java}
@Override
public AutoCreatedLeafQueueConfig getInitialLeafQueueConfiguration(
AbstractAutoCreatedLeafQueue leafQueue)
throws SchedulerDynamicEditException {
...
float availableCapacity = managedParentQueue.getQueueCapacities().
getAbsoluteCapacity(nodeLabel) - parentQueueState.
getAbsoluteActivatedChildQueueCapacity(nodeLabel) + EPSILON;
if (availableCapacity >= leafQueueTemplateCapacities
.getAbsoluteCapacity(nodeLabel)) {
...
} else{
updateToZeroCapacity(capacities, nodeLabel, leafQueue);
}
...
}
{code}
In updateToZeroCapacity we should change to ,For absolute auto created leaf
queue:
{code:java}
private void updateToZeroCapacity(QueueCapacities capacities,
String nodeLabel, LeafQueue leafQueue) {
capacities.setCapacity(nodeLabel, 0.0f);
capacities.setMaximumCapacity(nodeLabel,
leafQueueTemplateCapacities.getMaximumCapacity(nodeLabel));
leafQueue.getQueueResourceQuotas().
setConfiguredMinResource(nodeLabel, Resource.newInstance(0,0));
}
{code}
Then in calculateEffectiveResourcesAndCapacity the ratio will be correct, when
the absolute auto created leaf queue will not add t
[jira] [Updated] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
[ https://issues.apache.org/jira/browse/YARN-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10587: - Description: When introduced YARN-10504. The logic related to auto created leaf queue changed. The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. > - > > Key: YARN-10587 > URL: https://issues.apache.org/jira/browse/YARN-10587 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > When introduced YARN-10504. > The logic related to auto created leaf queue changed. > The test in testAutoCreateLeafQueueCreation failed, we should fix the Error. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10587) Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode.
zhuqi created YARN-10587: Summary: Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. Key: YARN-10587 URL: https://issues.apache.org/jira/browse/YARN-10587 Project: Hadoop YARN Issue Type: Sub-task Reporter: zhuqi Assignee: zhuqi -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10352: - Attachment: YARN-10352-010.patch > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352-010.patch, YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268457#comment-17268457
]
zhuqi commented on YARN-10531:
--
cc [~wangda] [~sunilg]
If you any other advice for this?
Thanks.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268387#comment-17268387 ] zhuqi edited comment on YARN-10352 at 1/20/21, 6:35 AM: [~bibinchundatt] [~sunilg] I have rebased it in YARN-10352.009.patch . With some changes YARN-8557 which will skip not running node also. Thanks. was (Author: zhuqi): [~bibinchundatt] I have rebased it in YARN-10352.009.patch . With some changes YARN-8557 which will skip not running node also. Thanks. > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi resolved YARN-10572. -- Resolution: Duplicate > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268387#comment-17268387 ] zhuqi edited comment on YARN-10352 at 1/20/21, 6:16 AM: [~bibinchundatt] I have rebased it in YARN-10352.009.patch . With some changes YARN-8557 which will skip not running node also. Thanks. was (Author: zhuqi): [~bibinchundatt] I have rebased it with some changes YARN-8557 which will skip not running node also. Thanks. > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268387#comment-17268387 ] zhuqi commented on YARN-10352: -- [~bibinchundatt] I have rebased it with some changes YARN-8557 which will skip not running node also. Thanks. > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10352: - Attachment: YARN-10352.009.patch > Skip schedule on not heartbeated nodes in Multi Node Placement > -- > > Key: YARN-10352 > URL: https://issues.apache.org/jira/browse/YARN-10352 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0, 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: capacityscheduler, multi-node-placement > Attachments: YARN-10352-001.patch, YARN-10352-002.patch, > YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, > YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, > YARN-10352.009.patch > > > When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM > Active Nodes will be still having those stopped nodes until NM Liveliness > Monitor Expires after configured timeout > (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, > Multi Node Placement assigns the containers on those nodes. They need to > exclude the nodes which has not heartbeated for configured heartbeat interval > (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to > Asynchronous Capacity Scheduler Threads. > (CapacityScheduler#shouldSkipNodeSchedule) > *Repro:* > 1. Enable Multi Node Placement > (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery > Enabled (yarn.node.recovery.enabled) > 2. Have only one NM running say worker0 > 3. Stop worker0 and start any other NM say worker1 > 4. Submit a sleep job. The containers will timeout as assigned to stopped NM > worker0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268383#comment-17268383 ] zhuqi commented on YARN-10572: -- [~sunilg] [~bibinchundatt] Ok i will close this, rebase this in YARN-10352, and add YARN-8557 some small difference which will skip not running node also. Thanks for your reply. > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10524) Support multi resource type based weight mode in CS.
[ https://issues.apache.org/jira/browse/YARN-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10524: - Description: In fair scheduler support multi resource based absolute and pecentage of the cluster. For minResources, maxResources, maxContainerAllocation and maxChildResources properties one can give the parameter in one of the following formats: * _Old format_: “X mb, Y vcores”, “X% cpu, Y% memory”, “X%”. When not a single percentage is provided, configuring both memory and cpu is mandatory, while other resource types are ignored, and set to zero. * _New format (recommended)_: “vcores=X, memory-mb=Y” or “vcores=X%, memory-mb=Y%”. As seen, in this format one can give either a percentage or an integer resource value without units. In the latter case the units will be inferred from the default units configured for that resource. This format is required when resources other than memory and CPU are specified. Any unspecified resource will be set to 0 in case of minResources, or to the maximum for that resource in case of maxResources, maxContainerAllocation and maxChildResources. We should to discuss how to do in CapacityScheduler ? > Support multi resource type based weight mode in CS. > > > Key: YARN-10524 > URL: https://issues.apache.org/jira/browse/YARN-10524 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > In fair scheduler support multi resource based absolute and pecentage of the > cluster. > For minResources, maxResources, maxContainerAllocation and maxChildResources > properties one can give the parameter in one of the following formats: > * _Old format_: “X mb, Y vcores”, “X% cpu, Y% memory”, “X%”. When not a > single percentage is provided, configuring both memory and cpu is mandatory, > while other resource types are ignored, and set to zero. > * _New format (recommended)_: “vcores=X, memory-mb=Y” or “vcores=X%, > memory-mb=Y%”. As seen, in this format one can give either a percentage or an > integer resource value without units. In the latter case the units will be > inferred from the default units configured for that resource. This format is > required when resources other than memory and CPU are specified. Any > unspecified resource will be set to 0 in case of minResources, or to the > maximum for that resource in case of maxResources, maxContainerAllocation and > maxChildResources. > We should to discuss how to do in CapacityScheduler ? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/19/21, 12:29 PM: - [~wangda] [~bibinchundatt] [~BilwaST] [~gandras] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~wangda] [~bibinchundatt] [~bilwa_st] [~gandras] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/19/21, 12:29 PM: - [~wangda] [~bibinchundatt] [~bilwa_st] [~gandras] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~wangda] [~bibinchundatt] [~gandras] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10531:
-
Attachment: YARN-10531.005.patch
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267771#comment-17267771
]
zhuqi commented on YARN-10531:
--
[~BilwaST]
Thanks a lot for your review, fix checkstyle in latest patch.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi updated YARN-10531:
-
Attachment: YARN-10531.004.patch
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267736#comment-17267736 ] zhuqi edited comment on YARN-10532 at 1/19/21, 8:19 AM: [~wangda] [~gandras] The test is not related, now it can be reviewed. Another question is : It realized auto deletion for old auto created also, if we just want to support new auto created queue. Thanks. was (Author: zhuqi): [~wangda] [~gandras] Now it can be reviewed. Another question is : It realized auto deletion for old auto created also, if we just want to support new auto created queue. Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267736#comment-17267736 ] zhuqi commented on YARN-10532: -- [~wangda] [~gandras] Now it can be reviewed. Another question is : It realized auto deletion for old auto created also, if we just want to support new auto created queue. Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10532: - Attachment: YARN-10532.007.patch > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267620#comment-17267620 ] zhuqi commented on YARN-10532: -- Fix the test case in patch 006. When policy edit is not finished, may get the not deleted queue. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10532: - Attachment: YARN-10532.006.patch > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10578) Fix Auto Queue Creation parent handling
[ https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267601#comment-17267601 ] zhuqi commented on YARN-10578: -- [~gandras] If we need created without submit app, we should need write lock. LGTM +1 But the lastSubmittedTimestamp we should use later in auto deletion. > Fix Auto Queue Creation parent handling > --- > > Key: YARN-10578 > URL: https://issues.apache.org/jira/browse/YARN-10578 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10578.001.patch > > > YARN-10506 introduced the new auto queue creation logic, however a parent == > null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels > queue to be created. We need to revert it back to the normal logic, also, we > should wrap the auto queue handling with a lock. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10532: - Attachment: YARN-10532.005.patch > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267111#comment-17267111
]
zhuqi edited comment on YARN-10532 at 1/18/21, 1:27 PM:
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
"
This will not happen:
Scenario A confirmed by :
Double check before deletion, pass the latest last submitted time, and get
before remove again and compare them. All will in the queue write lock.
{code:java}
// Double check for the lastSubmitTime has been expired.
// In case if now, there is a new submitted app.
if (queue instanceof LeafQueue &&
((LeafQueue) queue).isDynamicQueue()) {
LeafQueue underDeleted = (LeafQueue)queue;
if (underDeleted.getLastSubmittedTimestamp() != lastSubmittedTime) {
throw new SchedulerDynamicEditException("This should not happen, " +
"trying to remove queue= " + childQueuePath
+ ", however the queue has new submitted apps.");
}
} else {
throw new SchedulerDynamicEditException(
"This should not happen, can't remove queue= " + childQueuePath
+ " is not a leafQueue or not a dynamic queue.");
}
// Now we can do remove and update
this.childQueues.remove(queue);
this.scheduler.getCapacitySchedulerQueueManager()
.removeQueue(queue.getQueuePath());
{code}
Signal will also update this in the write lock:
{code:java}
@Override
public void submitApplication(ApplicationId applicationId, String userName,
String queue) throws AccessControlException {
// Careful! Locking order is important!
validateSubmitApplication(applicationId, userName, queue);
// Signal to queue submit time in dynamic queue
if (this.isDynamicQueue()) {
signalToSubmitToQueue();
}
// Inform the parent queue
try {
getParent().submitApplication(applicationId, userName, queue);
} catch (AccessControlException ace) {
LOG.info("Failed to submit application to parent-queue: " +
getParent().getQueuePath(), ace);
throw ace;
}
}
// "Tab" the queue, so this queue won't be removed because of idle timeout.
public void signalToSubmitToQueue() {
writeLock.lock();
try {
this.lastSubmittedTimestamp = System.currentTimeMillis();
} finally {
writeLock.unlock();
}
}
{code}
Scenario B confirmed by :
in addApplication
and addApplicationOnRecovery.
{code:java}
//- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
//- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
//- At T2 (T2 > T1), an app submitted to scheduler.
//
//Scheduler should immediately recreate the queue, in another word,
// deleting an dynamic queue should NEVER fail a submitted application.
// This will not happen, because :
// The writelock in addApplication
// and in addApplicationOnRecovery.
// Will make sure the create and submit atomic.
// Also the capacity scheduler writelock will be held in remove logic.
private void addApplication(ApplicationId applicationId, String queueName,
String user, Priority priority,
ApplicationPlacementContext placementContext) {
writeLock.lock();
...
}
// The remove will hold writelock
private CSQueue removeDynamicChildQueue(String childQueuePath, boolean isLeaf,
long lastSubmittedTime)
throws SchedulerDynamicEditException {
writeLock.lock();
...
}{code}
Above is for policy auto deletion.
When reinitializeQueues, already in capacity scheduler write lock. It will be
safe.
was (Author: zhuqi):
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word,
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267111#comment-17267111
]
zhuqi edited comment on YARN-10532 at 1/18/21, 1:27 PM:
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
"
This will not happen:
Scenario A confirmed by :
Double check before deletion, pass the latest last submitted time, and get
before remove again and compare them. All will in the queue write lock.
{code:java}
// Double check for the lastSubmitTime has been expired.
// In case if now, there is a new submitted app.
if (queue instanceof LeafQueue &&
((LeafQueue) queue).isDynamicQueue()) {
LeafQueue underDeleted = (LeafQueue)queue;
if (underDeleted.getLastSubmittedTimestamp() != lastSubmittedTime) {
throw new SchedulerDynamicEditException("This should not happen, " +
"trying to remove queue= " + childQueuePath
+ ", however the queue has new submitted apps.");
}
} else {
throw new SchedulerDynamicEditException(
"This should not happen, can't remove queue= " + childQueuePath
+ " is not a leafQueue or not a dynamic queue.");
}
// Now we can do remove and update
this.childQueues.remove(queue);
this.scheduler.getCapacitySchedulerQueueManager()
.removeQueue(queue.getQueuePath());
{code}
Signal will also update this in the write lock:
{code:java}
@Override
public void submitApplication(ApplicationId applicationId, String userName,
String queue) throws AccessControlException {
// Careful! Locking order is important!
validateSubmitApplication(applicationId, userName, queue);
// Signal to queue submit time in dynamic queue
if (this.isDynamicQueue()) {
signalToSubmitToQueue();
}
// Inform the parent queue
try {
getParent().submitApplication(applicationId, userName, queue);
} catch (AccessControlException ace) {
LOG.info("Failed to submit application to parent-queue: " +
getParent().getQueuePath(), ace);
throw ace;
}
}
// "Tab" the queue, so this queue won't be removed because of idle timeout.
public void signalToSubmitToQueue() {
writeLock.lock();
try {
this.lastSubmittedTimestamp = System.currentTimeMillis();
} finally {
writeLock.unlock();
}
}
{code}
Scenario B confirmed by :
in addApplication
and addApplicationOnRecovery.
{code:java}
//- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
//- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
//- At T2 (T2 > T1), an app submitted to scheduler.
//
//Scheduler should immediately recreate the queue, in another word,
// deleting an dynamic queue should NEVER fail a submitted application.
// This will not happen, because :
// The writelock in addApplication
// and in addApplicationOnRecovery.
// Will make sure the create and submit atomic.
// Also the capacity scheduler writelock will be held in remove logic.
private void addApplication(ApplicationId applicationId, String queueName,
String user, Priority priority,
ApplicationPlacementContext placementContext) {
writeLock.lock();
...
}
// The remove will hold writelock
private CSQueue removeDynamicChildQueue(String childQueuePath, boolean isLeaf,
long lastSubmittedTime)
throws SchedulerDynamicEditException {
writeLock.lock();
...
}{code}
When reinitializeQueues, already in capacity scheduler write lock.
was (Author: zhuqi):
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a sub
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267111#comment-17267111
]
zhuqi edited comment on YARN-10532 at 1/18/21, 1:10 PM:
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
"
This will not happen:
Scenario A confirmed by :
Double check before deletion, pass the latest last submitted time, and get
before remove again and compare them. All will in the queue write lock.
{code:java}
// Double check for the lastSubmitTime has been expired.
// In case if now, there is a new submitted app.
if (queue instanceof LeafQueue &&
((LeafQueue) queue).isDynamicQueue()) {
LeafQueue underDeleted = (LeafQueue)queue;
if (underDeleted.getLastSubmittedTimestamp() != lastSubmittedTime) {
throw new SchedulerDynamicEditException("This should not happen, " +
"trying to remove queue= " + childQueuePath
+ ", however the queue has new submitted apps.");
}
} else {
throw new SchedulerDynamicEditException(
"This should not happen, can't remove queue= " + childQueuePath
+ " is not a leafQueue or not a dynamic queue.");
}
// Now we can do remove and update
this.childQueues.remove(queue);
this.scheduler.getCapacitySchedulerQueueManager()
.removeQueue(queue.getQueuePath());
{code}
Signal will also update this in the write lock:
{code:java}
@Override
public void submitApplication(ApplicationId applicationId, String userName,
String queue) throws AccessControlException {
// Careful! Locking order is important!
validateSubmitApplication(applicationId, userName, queue);
// Signal to queue submit time in dynamic queue
if (this.isDynamicQueue()) {
signalToSubmitToQueue();
}
// Inform the parent queue
try {
getParent().submitApplication(applicationId, userName, queue);
} catch (AccessControlException ace) {
LOG.info("Failed to submit application to parent-queue: " +
getParent().getQueuePath(), ace);
throw ace;
}
}
// "Tab" the queue, so this queue won't be removed because of idle timeout.
public void signalToSubmitToQueue() {
writeLock.lock();
try {
this.lastSubmittedTimestamp = System.currentTimeMillis();
} finally {
writeLock.unlock();
}
}
{code}
Scenario B confirmed by :
in addApplication
and addApplicationOnRecovery.
{code:java}
//- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
//- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
//- At T2 (T2 > T1), an app submitted to scheduler.
//
//Scheduler should immediately recreate the queue, in another word,
// deleting an dynamic queue should NEVER fail a submitted application.
// This will not happen, because :
// The writelock in addApplication
// and in addApplicationOnRecovery.
// Will make sure the create and submit atomic.
// Also the capacity scheduler writelock will be held in remove logic.
private void addApplication(ApplicationId applicationId, String queueName,
String user, Priority priority,
ApplicationPlacementContext placementContext) {
writeLock.lock();
...
}
// The remove will hold writelock
private CSQueue removeDynamicChildQueue(String childQueuePath, boolean isLeaf,
long lastSubmittedTime)
throws SchedulerDynamicEditException {
writeLock.lock();
...
}{code}
was (Author: zhuqi):
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
"
This will not happen:
Scenario A con
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/18/21, 9:34 AM: [~wangda] [~bibinchundatt] [~gandras] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~wangda] [~bibinchundatt] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10532: - Attachment: YARN-10532.004.patch > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267111#comment-17267111
]
zhuqi commented on YARN-10532:
--
The latest patch, double check the
"An additional requirement we should keep in mind:
Scenario A:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- Before the signal arrives to scheduler, an app submitted to scheduler (T1).
T1 > T0
- When at T2 (T2 > T1), the signal arrived at scheduler, scheduler should avoid
removing the queue A because now it is used.{code}
Scenario B:
{code:java}
- At time T0, policy signals scheduler to delete queue A (an auto created
queue).
- At T1 (T1 > T0), scheduler got the signal and deleted the queue.
- At T2 (T2 > T1), an app submitted to scheduler.
Scheduler should immediately recreate the queue, in another word, deleting an
dynamic queue should NEVER fail a submitted application.{code}
"
This will not happen:
Scenario A confirmed by :
Double check before deletion, pass the latest last submitted time, and get
before remove again and compare them. All will in the queue write lock.
{code:java}
// Double check for the lastSubmitTime has been expired.
// In case if now, there is a new submitted app.
if (queue instanceof LeafQueue &&
((LeafQueue) queue).isDynamicQueue()) {
LeafQueue underDeleted = (LeafQueue)queue;
if (underDeleted.getLastSubmittedTimestamp() != lastSubmittedTime) {
throw new SchedulerDynamicEditException("This should not happen, " +
"trying to remove queue= " + childQueuePath
+ ", however the queue has new submitted apps.");
}
} else {
throw new SchedulerDynamicEditException(
"This should not happen, can't remove queue= " + childQueuePath
+ " is not a leafQueue or not a dynamic queue.");
}
// Now we can do remove and update
this.childQueues.remove(queue);
this.scheduler.getCapacitySchedulerQueueManager()
.removeQueue(queue.getQueuePath());
{code}
Signal will also update this in the write lock:
{code:java}
@Override
public void submitApplication(ApplicationId applicationId, String userName,
String queue) throws AccessControlException {
// Careful! Locking order is important!
validateSubmitApplication(applicationId, userName, queue);
// Signal to queue submit time in dynamic queue
if (this.isDynamicQueue()) {
signalToSubmitToQueue();
}
// Inform the parent queue
try {
getParent().submitApplication(applicationId, userName, queue);
} catch (AccessControlException ace) {
LOG.info("Failed to submit application to parent-queue: " +
getParent().getQueuePath(), ace);
throw ace;
}
}
// "Tab" the queue, so this queue won't be removed because of idle timeout.
public void signalToSubmitToQueue() {
writeLock.lock();
try {
this.lastSubmittedTimestamp = System.currentTimeMillis();
} finally {
writeLock.unlock();
}
}
{code}
Scenario B confirmed by :
in addApplication
and addApplicationOnRecovery.
{code:java}
// If the queue has been deleted for expired.
// - At time T0, policy signals scheduler to delete queue A (an auto created
queue).
// - At T1 (T1 > T0), scheduler got the signal and deleted the queue.
// - At T2 (T2 > T1), an app submitted to scheduler.
//
// Scheduler should immediately recreate the queue, in another word,
// deleting an dynamic queue should NEVER fail a submitted application.
// This case queue may be null later
// So add queue write lock here
try {
((AbstractCSQueue) queue).writeLock.lock();
}...{code}
> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is
> not being used
>
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch,
> YARN-10532.003.patch
>
>
> It's better if we can delete auto-created queues when they are not in use for
> a period of time (like 5 mins). It will be helpful when we have a large
> number of auto-created queues (e.g. from 500 users), but only a small subset
> of queues are actively used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10572) Merge YARN-8557 and YARN-10352, and rebase based YARN-10380.
[ https://issues.apache.org/jira/browse/YARN-10572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264155#comment-17264155 ] zhuqi edited comment on YARN-10572 at 1/18/21, 6:30 AM: [~wangda] [~bibinchundatt] [~prabhujoseph] [~tangzhankun] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? was (Author: zhuqi): [~wangda] [~bibinchundatt] [~prabhujoseph] I have updated a patch to rebase YARN-10352 and merge the difference in YARN-8557. Also refactor some method. If you could review and merge it? > Merge YARN-8557 and YARN-10352, and rebase based YARN-10380. > > > Key: YARN-10572 > URL: https://issues.apache.org/jira/browse/YARN-10572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10572.001.patch > > > The work is : > 1. Because of YARN-10380, We should rebase YARN-10352 > 2. Also merge YARN-8557 for not running case skip. > 3. Refactor some method in YARN-10380 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266529#comment-17266529 ] zhuqi edited comment on YARN-10532 at 1/18/21, 5:47 AM: [~wangda] [~gandras] I add a draft poc patch for 1, 2, 3 above. The expired time is just used for old auto created leaf queue, i reuse the expired logic to trigger old auto created expired for deletion. And the new auto created leaf queue , i have used the submit time. I will fix and deep into some other cases for more details. Thanks. was (Author: zhuqi): [~wangda] [~gandras] I add a draft poc patch for 1, 2, 3 above. I will fix and deep into some other cases for more details. Thanks. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
