[jira] [Comment Edited] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[
https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269888#comment-17269888
]
zhuqi edited comment on YARN-10590 at 1/26/21, 2:22 AM:
cc [~wangda] [~sunilg] [~bteke]
When i deep into the loss of the caculation about
{code:java}
// TODO: Wangda, I think this is a wrong test, it doesn't consider rounding
// loss of multiplication, the right value should be <10240, 2>, but the
// test expects <10240, 1>
{code}
I think the test logic should change to :
{code:java}
// When in absolute mode.
// We can only assertEquals in memory size, because the cap/absolute
// is caculated by memory, we can only make sure this.
if (leafQueue.getCapacityConfigType() ==
AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) {
assertEquals(effMinCapacity.getMemorySize(),
leafQueue.getEffectiveCapacity(label).getMemorySize());
assertTrue(effMinCapacity.getVirtualCores()
<= leafQueue.getEffectiveCapacity(label).getVirtualCores());
} else {
assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label));
}
{code}
Because in absolute mode with auto created leaf queue, we get the cap/absolute
by :
{code:java}
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
...
}
}
{code}
In default resource caculator, we get the cap/absolute only by memory:
{code:java}
public float ratio(Resource a, Resource b) {
return divideSafelyAsFloat(a.getMemorySize(), b.getMemorySize());
}
{code}
This cause the loss of multiply by cap/absolute, with the actual effective min
resource.
The actual effective min resource will be updated in :
{code:java}
// Update effective resource (min/max) to each child queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
}{code}
We will get the actual effective min resource, different with the get the
cap/absolute only by memory.
So in absolute auto created leaf queue , the effective min resource only can
meet the memory caculated by cap/absolute with default resource caculator.
The logic is wright, but the test we should fix.
Thoughts?
Thanks.
was (Author: zhuqi):
cc [~leftnoteasy] [~sunilg] [~bteke]
When i deep into the loss of the caculation about
{code:java}
// TODO: Wangda, I think this is a wrong test, it doesn't consider rounding
// loss of multiplication, the right value should be <10240, 2>, but the
// test expects <10240, 1>
{code}
I think the test logic should change to :
{code:java}
// When in absolute mode.
// We can only assertEquals in memory size, because the cap/absolute
// is caculated by memory, we can only make sure this.
if (leafQueue.getCapacityConfigType() ==
AbstractCSQueue.CapacityConfigType.ABSOLUTE_RESOURCE) {
assertEquals(effMinCapacity.getMemorySize(),
leafQueue.getEffectiveCapacity(label).getMemorySize());
assertTrue(effMinCapacity.getVirtualCores()
<= leafQueue.getEffectiveCapacity(label).getVirtualCores());
} else {
assertEquals(effMinCapacity, leafQueue.getEffectiveCapacity(label));
}
{code}
Because in absolute mode with auto created leaf queue, we get the cap/absolute
by :
{code:java}
private void updateQueueCapacities(QueueCapacities queueCapacities) {
for (String label : queueCapacities.getExistingNodeLabels()) {
queueCapacities.setCapacity(label,
this.csContext.getResourceCalculator().divide(
this.csContext.getClusterResource(),
this.csContext.getConfiguration().getMinimumResourceRequirement(
label,
this.csContext.getConfiguration()
.getAutoCreatedQueueTemplateConfPrefix(getQueuePath()),
resourceTypes),
getQueueResourceQuotas().getConfiguredMinResource(label)));
queueCapacities.setAbsoluteCapacity(
label, queueCapacities.getCapacity(label)
* getQueueCapacities().getAbsoluteCapacity(label));
...
}
}
{code}
In default resource caculator, we get the cap/absolute only by memory:
{code:java}
public
[jira] [Comment Edited] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271183#comment-17271183 ] zhuqi edited comment on YARN-10592 at 1/26/21, 2:21 AM: cc [~wangda] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. was (Author: zhuqi): cc [~leftnoteasy] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10164) Allow NM to start even when custom resource type not defined
[
https://issues.apache.org/jira/browse/YARN-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-10164.
---
Resolution: Won't Do
> Allow NM to start even when custom resource type not defined
>
>
> Key: YARN-10164
> URL: https://issues.apache.org/jira/browse/YARN-10164
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
>
> In the [custom resource
> documentation|https://hadoop.apache.org/docs/r3.2.1/hadoop-yarn/hadoop-yarn-site/ResourceModel.html],
> it tells you to add the number of custom resources to a property called
> {{yarn.nodemanager.resource-type.}} in a file called
> {{node-resources.xml}}.
> For GPU resources, this would look something like
> {code:xml}
>
> yarn.nodemanager.resource-type.gpu
> 16
>
> {code}
> A corresponding config property must also be in {{resource-types.xml}} called
> yarn.resource-types:
> {code:xml}
>
> yarn.resource-types
> gpu
> Custom resources to be used for scheduling.
>
> {code}
> If the yarn.nodemanager.resource-type.gpu property exists without the
> corresponding yarn.resource-types property, the nodemanager fails to start.
> I would like the option to automatically create the node-resources.xml on all
> new nodes regardless of whether or not the cluster supports GPU resources so
> that if I deploy a GPU node into an existing cluster that does not (yet)
> support GPU resources, the nodemanager will at least start. Even though it
> doesn't support the GPU resource, the other supported resources will still be
> available to be used by the apps in the cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271563#comment-17271563
]
Hadoop QA commented on YARN-10596:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
8s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 28s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/542/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271419#comment-17271419
]
Benjamin Teke commented on YARN-10596:
--
[~gandras] thanks for the patch. I have a suggestion for making the condition a
bit more readable (for me at least). Consider the following part:
{code:java}
boolean isEligibleForAutoQueueCreation = false;
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isEligibleForAutoQueueCreation = ((ParentQueue)
oldQueue).isDynamicQueue();
}
// if a queue is eligible for auto queue creation v2
// it must be a ParentQueue
if (!isEligibleForAutoQueueCreation) {
isEligibleForAutoQueueCreation =
conf.isAutoQueueCreationV2Enabled(fullQueueName);
}
{code}
Basically the condition is: if the queue is a dynamic parent, or it has the the
AutoQueueCreationV2 enabled then it can be an empty parent. As the old
autoQueueEnabled flag is parsed and stored as a boolean I think it would be
nicer if you got the conf.isAutoQueueCreationV2Enabled there and kept the
isDynamicParent intact (as isDynamicParent induces the eligibility for auto
creation, however it is not the same). You could skip the (bit) misleading if
(!isEligibleForAutoQueueCreation) part, and instead of the if
(childQueueNames.size() == 0 && !isEligibleForAutoQueueCreation) condition you
could say that the queue should not have 0 childQueues, should not be a dynamic
parent and should not be eligible for auto creation based on the config.
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271405#comment-17271405 ] zhuqi commented on YARN-10585: -- Thank [~shuzirra] for working on this. The latest patch LGTM. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271371#comment-17271371 ] Benjamin Teke edited comment on YARN-10585 at 1/25/21, 3:34 PM: Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s comments the patch looks good to me, so generally +1 (non-binding) from my side. Another patch might be needed because the 002 generates license warnings and has a checkstyle issue. was (Author: bteke): Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s comments the patch looks good to me, so +1 (non-binding) from my side. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271371#comment-17271371 ] Benjamin Teke commented on YARN-10585: -- Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s comments the patch looks good to me, so +1 (non-binding) from my side. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Summary: Support QueueCapacities to use vector based multi resource types, and update absolute related to use first. (was: Support QueueCapacities to use vector, and update absolute related to use first.) > Support QueueCapacities to use vector based multi resource types, and update > absolute related to use first. > --- > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271355#comment-17271355 ] Peter Bacsko edited comment on YARN-10596 at 1/25/21, 3:04 PM: --- Ok, I've just read this sentence: _Since it is not an edge case, we also need to support the creation of a ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, even when no child queue is defined under the parent._ My understanding was that by default, all parents are allowed of having dynamic queues created. If this is not true, that means: 1. If someone wants all parents to have dynamic queues (which is standard for FS users who are migrating), then this property must be set for all parents. 2. This affects fs2cs tool, because it does not set "auto-queue-creation-v2" at all. I pointed this out in my comment https://issues.apache.org/jira/browse/YARN-10506?focusedCommentId=17264903=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17264903. See paragraph which begins with "We also have to think about the following scenario" and conclusion point #2. was (Author: pbacsko): Ok, I've just read this sentence: _Since it is not an edge case, we also need to support the creation of a ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, even when no child queue is defined under the parent._ My understanding was that by default, all parents are allowed of having dynamic queues created. If this is not true, that means: 1. If someone wants all parents to have dynamic queues (which is standard for FS users who are migrating), then this property must be set for all parents. 2. This affects fs2cs tool, because it does not set "auto-queue-creation-v2" at all. I pointed this out in my comment https://issues.apache.org/jira/browse/YARN-10506?focusedCommentId=17264903=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17264903. See paragraph which begins with "We also have to think about the following scenario" and conclusion point #2. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10596.001.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271355#comment-17271355 ] Peter Bacsko commented on YARN-10596: - Ok, I've just read this sentence: _Since it is not an edge case, we also need to support the creation of a ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, even when no child queue is defined under the parent._ My understanding was that by default, all parents are allowed of having dynamic queues created. If this is not true, that means: 1. If someone wants all parents to have dynamic queues (which is standard for FS users who are migrating), then this property must be set for all parents. 2. This affects fs2cs tool, because it does not set "auto-queue-creation-v2" at all. I pointed this out in my comment https://issues.apache.org/jira/browse/YARN-10506?focusedCommentId=17264903=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17264903. See paragraph which begins with "We also have to think about the following scenario" and conclusion point #2. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10596.001.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271342#comment-17271342
]
Peter Bacsko edited comment on YARN-10596 at 1/25/21, 2:53 PM:
---
[~gandras] I have a question.
{noformat}
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
isEligibleForAutoQueueCreation = ((ParentQueue) oldQueue).isDynamicQueue();
}
{noformat}
Without explicitly saying if a queue is a parent or not, how do you distinguish
between a parent with no leafs and an ordinary leaf queue? How do you detect
that a queue with no children is intended to be a parent?
was (Author: pbacsko):
[~gandras] I have a question.
{noformat}
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
isEligibleForAutoQueueCreation = ((ParentQueue) oldQueue).isDynamicQueue();
}
{noformat}
Without explicitly saying if a queue is a parent or not, how do you distinguish
between a parent with no leafs and an ordinary leaf queu? How do you detect
that a queue with no children is intended to be a parent?
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271342#comment-17271342
]
Peter Bacsko edited comment on YARN-10596 at 1/25/21, 2:38 PM:
---
[~gandras] I have a question.
{noformat}
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
isEligibleForAutoQueueCreation = ((ParentQueue) oldQueue).isDynamicQueue();
}
{noformat}
Without explicitly saying if a queue is a parent or not, how do you distinguish
between a parent with no leafs and an ordinary leaf queu? How do you detect
that a queue with no children is intended to be a parent?
was (Author: pbacsko):
[~gandras] I have a question.
{noformat}
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
isEligibleForAutoQueueCreation = ((ParentQueue) oldQueue).isDynamicQueue();
}
{noformat}
Without explicitly saying if a queue is a parent or not, how do you distinguish
between a parent with no leafs and an ordinary leaf queu? How do you detect
that a queue with no children is intended to be a parent?
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271342#comment-17271342
]
Peter Bacsko commented on YARN-10596:
-
[~gandras] I have a question.
{noformat}
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
isEligibleForAutoQueueCreation = ((ParentQueue) oldQueue).isDynamicQueue();
}
{noformat}
Without explicitly saying if a queue is a parent or not, how do you distinguish
between a parent with no leafs and an ordinary leaf queu? How do you detect
that a queue with no children is intended to be a parent?
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10425) Replace the legacy placement engine in CS with the new one
[
https://issues.apache.org/jira/browse/YARN-10425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271331#comment-17271331
]
Gergely Pollak commented on YARN-10425:
---
[~ahussein]Thank you for the feedback, I've created the followup Jira
YARN-10597 . You are right, this shouldn't instantiated by the
{{CSMappingPlacementRule}} class, because it will break the cache.
The oversight was when I checked it, I thought the legacy code worked this way,
and that's why I kept it here. However now I've double checked, and I realized,
I must had checked an older branch last time, since it was fixed about a year
ago. So thank you for pointing this out. We'll need to update a few tests, but
I'll take a look at them.
> Replace the legacy placement engine in CS with the new one
> --
>
> Key: YARN-10425
> URL: https://issues.apache.org/jira/browse/YARN-10425
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10425.001.patch, YARN-10425.002.patch,
> YARN-10425.003.patch, YARN-10425.004.patch, YARN-10425.005.patch,
> YARN-10425.006.patch, YARN-10425.007.patch
>
>
> Remove the UserGroupMapping and ApplicationName mapping classes, and use the
> new CSMappingPlacementRule instead. Also cleanup the orphan classes which are
> used by these classes only.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10597) CSMappingPlacementRule should not create new instance of Groups
[ https://issues.apache.org/jira/browse/YARN-10597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak reassigned YARN-10597: - Assignee: Gergely Pollak > CSMappingPlacementRule should not create new instance of Groups > --- > > Key: YARN-10597 > URL: https://issues.apache.org/jira/browse/YARN-10597 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > > As [~ahussein] pointed out in YARN-10425, no new Groups instance should be > created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10597) CSMappingPlacementRule should not create new instance of Groups
Gergely Pollak created YARN-10597: - Summary: CSMappingPlacementRule should not create new instance of Groups Key: YARN-10597 URL: https://issues.apache.org/jira/browse/YARN-10597 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak As [~ahussein] pointed out in YARN-10425, no new Groups instance should be created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
Andras Gyori created YARN-10596: --- Summary: Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled Key: YARN-10596 URL: https://issues.apache.org/jira/browse/YARN-10596 Project: Hadoop YARN Issue Type: Sub-task Reporter: Andras Gyori Assignee: Andras Gyori The old auto queue creation/managed queue logic allowed the definition of childless parents to be created statically, if the auto-create-child-queue flag was turned on the parent (thus making it a ManagedParentQueue). Since it is not an edge case, we also need to support the creation of a ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10595) Support config the AM-RM poll time in AMRMClient
Wanqiang Ji created YARN-10595: -- Summary: Support config the AM-RM poll time in AMRMClient Key: YARN-10595 URL: https://issues.apache.org/jira/browse/YARN-10595 Project: Hadoop YARN Issue Type: Improvement Reporter: Wanqiang Ji Assignee: Wanqiang Ji AMRMClient provides the unregisterApplicationMaster method but the client can't config the poll time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10594) Split the debug log when execute privileged operation
Wanqiang Ji created YARN-10594: -- Summary: Split the debug log when execute privileged operation Key: YARN-10594 URL: https://issues.apache.org/jira/browse/YARN-10594 Project: Hadoop YARN Issue Type: Improvement Reporter: Wanqiang Ji Assignee: Wanqiang Ji Before execute *exec.execute();* statement should print the command log rather than after. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271305#comment-17271305
]
Hadoop QA commented on YARN-10559:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 4
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
15s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 39s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
55s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/541/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 520 unchanged - 0 fixed = 521 total (was 520)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 51s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (YARN-10593) Fix incorrect string comparison in GpuDiscoverer
[
https://issues.apache.org/jira/browse/YARN-10593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271298#comment-17271298
]
Peter Bacsko commented on YARN-10593:
-
Unfortunately, Zoltan Siegl's comment has not been addressed:
https://issues.apache.org/jira/browse/YARN-9217?focusedCommentId=16763445=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16763445
> Fix incorrect string comparison in GpuDiscoverer
>
>
> Key: YARN-10593
> URL: https://issues.apache.org/jira/browse/YARN-10593
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> The following comparison in {{GpuDiscoverer}} is invalid:
> {noformat}
> binaryPath = configuredBinaryFile;
> // If path exists but file name is incorrect don't execute the file
> String fileName = binaryPath.getName();
> if (DEFAULT_BINARY_NAME.equals(fileName)) { <--- inverse condition
> needed
> String msg = String.format("Please check the configuration value of"
> +" %s. It should point to an %s binary.",
> YarnConfiguration.NM_GPU_PATH_TO_EXEC,
> DEFAULT_BINARY_NAME);
> throwIfNecessary(new YarnException(msg), config);
> LOG.warn(msg);
> }{noformat}
> Obviously it should be other way around - we should log a warning or throw an
> exception if the file names *differ*, not when they're equal.
> Consider adding a unit test for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271297#comment-17271297
]
Gergely Pollak edited comment on YARN-10585 at 1/25/21, 1:01 PM:
-
[~gandras] thank you for the review and your comments, let me address your
concerns.
* Nit: in splitRule#line:213 the classic for loop could be replaced by the
enhanced loop/stream, which is nicer to readI
It is VERY subjective what is nice / easy to read, but in this case it's quite
hard to contradict this statement. Literally ANY developer who ever wrote a
loop, will be able to read a simple for loop, while to understand streams they
need some deeper java knowledge, so I think it points out very well, why is the
for loop easier (this nicer) to read.
But let's get technical a bit, because the overuse of the java streams is a pet
peeve of mine. Java streams with lambdas will create an unnecessary (well for
streams they ARE necessary) anonymous class in the background, and wrap the
actual loop core with multiple method calls, which inherently put extra strain
on the stack, gc and on the performance, for a very subjective little gain.
{code:java}
java.lang.ArithmeticException: / by zero
at fiddle.main(fiddle.java:7)
java.lang.ArithmeticException: / by zero
at fiddle.lambda$main$0(fiddle.java:14)
at
java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:581)
at fiddle.main(fiddle.java:14){code}
We can see the extra method calls and the created class as well.
This has an effect on performance as well:
{code:java}
int sum = 0;
for (int i = 0; i < 10; i++) {
sum += i;
}
{code}
Runs on my computer about 330ms
{code:java}
IntStream.range(0, 10).sum();{code}
This has a bit of a range between 380-420 ms, but even in best case it is still
a 20% performance degradation.
So all in all, I prefer to avoid streams, when they are not strictly needed,
because they are not to replace regular loops but to add some functional
programming support to java. They have their place in the language, for example
when one wants to use parallel streams they can really improve the performance,
with much less hassle about thread handling and concurrency.
* Nit: in creatUserMappingRule, you replace the match argument with its json
equivalent (*). Overwriting arguments are generally not a good idea, because it
could produce hard-to-find errors, also made the debugging somewhat harder.
As a general rule it's true, however I'm converting the argument in place,
before using it, it is not a repurpose of the variable, but I'm making sure,
the function receives the input it will properly process, in this case I don't
really find it problematic, since the goal here is to "hide" the incorrect
value from the rest of the method, hence the overwrite.
* I am not sure about this, but is the following format accepted in the legacy
format:
Yeah, it is a complex question. It is NOT supported by the legacy engine, but
it works in the new engine with legacy mode... however I've took a look at it,
and actually the converter DOES support this conversion, it converts it to a
customRule. I don't want to add any validation to the input, besides it's
syntax, because the tool will do a best effort conversion for unsupported
cases. It should be able to convert EVERY use case supported by the legacy
engine, and all cases supported by the new engine in legacy format, however we
don't encourage to use the new features in the legacy format. So if someone
still uses them we might be able to convert those rules as well (probably they
will become custom rules) so I don't want to reject those just because they are
not officially supported.
* userGroupMappingRules and applicationNameMappingRules might be initialised
as empty sets. This would eliminate the null checks in convert loops, but the
empty cases could still be checked in the beginning.
Since I expose the collection setter method then I'd have to move the null
check there, so we don't really save null checks, but at least we don't create
unnecessary objects. However I like the idea to have a centralised null check,
so I'll see the other feedbacks, and if I'll make a new patch set I'll consider
this suggestion.
was (Author: shuzirra):
[~gandras] thank you for the review and your comments, let me address your
concerns.
* Nit: in splitRule#line:213 the classic for loop could be replaced by the
enhanced loop/stream, which is nicer to readI
It is VERY subjective what is nice / easy to read, but in this case it's quite
hard to contradict this statement. Literally ANY developer who ever wrote a
loop, will be able to read a simple for loop, while to understand streams they
need some deeper java knowledge, so I think it
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271297#comment-17271297
]
Gergely Pollak commented on YARN-10585:
---
[~gandras] thank you for the review and your comments, let me address your
concerns.
* Nit: in splitRule#line:213 the classic for loop could be replaced by the
enhanced loop/stream, which is nicer to readI
It is VERY subjective what is nice / easy to read, but in this case it's quite
hard to contradict this statement. Literally ANY developer who ever wrote a
loop, will be able to read a simple for loop, while to understand streams they
need some deeper java knowledge, so I think it points out very well, why is the
for loop easier (this nicer) to read.
But let's get technical a bit, because the overuse of the java streams is a pet
peeve of mine. Java streams with lambdas will create an unnecessary (well for
streams they ARE necessary) anonymous class in the background, and wrap the
actual loop core with multiple method calls, which inherently put extra strain
on the stack, gc and on the performance, for a very subjective little gain.
{code:java}
java.lang.ArithmeticException: / by zero
at fiddle.main(fiddle.java:7)
java.lang.ArithmeticException: / by zero
at fiddle.lambda$main$0(fiddle.java:14)
at
java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110)
at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:581)
at fiddle.main(fiddle.java:14){code}
We can see the extra method calls and the created class as well.
This has an effect on performance as well:
{code:java}
int sum = 0;
for (int i = 0; i < 10; i++) {
sum += i;
}
{code}
Runs on my computer about 330ms
{code:java}
IntStream.range(0, 10).sum();{code}
This has a bit of a range between 380-420 ms, but even in best case it is still
a 20% performance degradation.
So all in all, I prefer to avoid streams, when they are not strictly needed,
because they are not to replace regular loops but to add some functional
programming support to java. They have their place in the language, for example
when one wants to use parallel streams they can really improve the performance,
with much less hassle about thread handling and concurrency.
* Nit: in creatUserMappingRule, you replace the match argument with its json
equivalent (*). Overwriting arguments are generally not a good idea, because it
could produce hard-to-find errors, also made the debugging somewhat harder.
As a general rule it's true, however I'm converting the argument in place,
before using it, it is not a repurpose of the variable, but I'm making sure,
the function receives the input it will properly process, in this case I don't
really find it problematic, since the goal here is to "hide" the incorrect
value from the rest of the method, hence the overwrite.
* I am not sure about this, but is the following format accepted in the legacy
format:
Yeah, it is a complex question. It is NOT supported by the legacy engine, but
it works in the new engine with legacy mode... however I've took a look at it,
and actually the converter DOES support this conversion, it converts it to a
customRule. I don't want to add any validation to the input, besides it's
syntax, because the tool will do a best effort conversion for unsupported
cases. It should be able to convert EVERY use case supported by the legacy
engine, and all cases supported by the new engine in legacy format, however we
don't encourage to use the new features in the legacy format. So if someone
still uses them we might be able to convert those rules as well (probably they
will become custom rules) so I don't want to reject those just because they are
not officially supported.
* userGroupMappingRules and applicationNameMappingRules might be initialised
as empty sets. This would eliminate the null checks in convert loops, but the
empty cases could still be checked in the beginning.
Since I expose the collection setter method then I'd have to move the null
check there, so we don't really save null checks, but at least we don't create
unnecessary objects. However I like the idea to have a centralised null check,
so I'll see the other feedbacks, and if I'll make a new patch set I'll consider
this suggestion.
> Create a class which can convert from legacy mapping rule format to the new
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch
>
>
> To make transition easier
[jira] [Created] (YARN-10593) Fix incorrect string comparison in GpuDiscoverer
Peter Bacsko created YARN-10593:
---
Summary: Fix incorrect string comparison in GpuDiscoverer
Key: YARN-10593
URL: https://issues.apache.org/jira/browse/YARN-10593
Project: Hadoop YARN
Issue Type: Bug
Components: resourcemanager
Reporter: Peter Bacsko
Assignee: Peter Bacsko
The following comparison in {{GpuDiscoverer}} is invalid:
{noformat}
binaryPath = configuredBinaryFile;
// If path exists but file name is incorrect don't execute the file
String fileName = binaryPath.getName();
if (DEFAULT_BINARY_NAME.equals(fileName)) { <--- inverse condition needed
String msg = String.format("Please check the configuration value of"
+" %s. It should point to an %s binary.",
YarnConfiguration.NM_GPU_PATH_TO_EXEC,
DEFAULT_BINARY_NAME);
throwIfNecessary(new YarnException(msg), config);
LOG.warn(msg);
}{noformat}
Obviously it should be other way around - we should log a warning or throw an
exception if the file names *differ*, not when they're equal.
Consider adding a unit test for this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[
https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271289#comment-17271289
]
Hadoop QA commented on YARN-10592:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
9s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
45s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 1s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
5s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for
patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 23s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/539/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt{color}
| {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 19 new +
164 unchanged - 0 fixed = 183 total (was 164) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271281#comment-17271281
]
Hadoop QA commented on YARN-10564:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
17s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 30s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
12s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 35s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
34s{color} |
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271253#comment-17271253
]
Andras Gyori commented on YARN-10585:
-
Thank you [~shuzirra] for the patch! I have analysed the patch and here is my
findings:
* Nit: in splitRule#line:213 the classic for loop could be replaced by the
enhanced loop/stream, which is nicer to read
* Nit: in creatUserMappingRule, you replace the match argument with its json
equivalent (*). Overwriting arguments are generally not a good idea, because it
could produce hard-to-find errors, also made the debugging somewhat harder.
* I am not sure about this, but is the following format accepted in the legacy
format:
{noformat}
u:%user:root.%user.QUEUE{noformat}
Namely, if the %user matcher is not the last item of the rule. This would
complicate things, and I think the current code could not handle this. If it is
not supported, can you make an additional error checking on this case?
* userGroupMappingRules and applicationNameMappingRules might be initialised
as empty sets. This would eliminate the null checks in convert loops, but the
empty cases could still be checked in the beginning.
Since you have written a lot of unit test cases, I think the logic itself is
correct.
> Create a class which can convert from legacy mapping rule format to the new
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch
>
>
> To make transition easier we need to create tooling to support the migration
> effort. The first step is to create a class which can migrate from legacy to
> the new JSON format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10576) Update Capacity Scheduler documentation about JSON-based placement mapping
[ https://issues.apache.org/jira/browse/YARN-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10576: Summary: Update Capacity Scheduler documentation about JSON-based placement mapping (was: Update Capacity Scheduler about JSON-based placement mapping) > Update Capacity Scheduler documentation about JSON-based placement mapping > -- > > Key: YARN-10576 > URL: https://issues.apache.org/jira/browse/YARN-10576 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > > The weight mode and AQC also affects how the new placement engine in CS works > and the documentation has to reflect that. > Certain statements in the documentation are no longer valid, for example: > * create flag: "Only applies to managed queue parents" - there is no > ManagedParentQueue in weight mode. > * "The nested rules primaryGroupUser and secondaryGroupUser expects the > parent queues to exist, ie. they cannot be created automatically". This only > applies to the legacy absolute/percentage mode. > Find all statements that mentions possible limitations and fix them if > necessary. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10531) Be able to disable user limit factor for CapacityScheduler Leaf Queue
[
https://issues.apache.org/jira/browse/YARN-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271186#comment-17271186
]
zhuqi commented on YARN-10531:
--
Thanks [~wangda] for final review.
> Be able to disable user limit factor for CapacityScheduler Leaf Queue
> -
>
> Key: YARN-10531
> URL: https://issues.apache.org/jira/browse/YARN-10531
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10531.001.patch, YARN-10531.002.patch,
> YARN-10531.003.patch, YARN-10531.004.patch, YARN-10531.005.patch,
> YARN-10531.006.patch, YARN-10531.007.patch
>
>
> User limit factor is used to define max cap of how much resource can be
> consumed by single user.
> Under Auto Queue Creation context, it doesn't make much sense to set user
> limit factor, because initially every queue will set weight to 1.0, we want
> user can consume more resource if possible. It is hard to pre-determine how
> to set up user limit factor. So it makes more sense to add a new value (like
> -1) to indicate we will disable user limit factor
> Logic need to be changed is below:
> (Inside LeafQueue.java)
> {code}
> Resource maxUserLimit = Resources.none();
> if (schedulingMode == SchedulingMode.RESPECT_PARTITION_EXCLUSIVITY) {
> maxUserLimit = Resources.multiplyAndRoundDown(queueCapacity,
> getUserLimitFactor());
> } else if (schedulingMode == SchedulingMode.IGNORE_PARTITION_EXCLUSIVITY)
> {
> maxUserLimit = partitionResource;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271183#comment-17271183 ] zhuqi edited comment on YARN-10592 at 1/25/21, 9:16 AM: cc [~leftnoteasy] [~sunilg] [~epayne] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. was (Author: zhuqi): cc [~leftnoteasy] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271183#comment-17271183 ] zhuqi commented on YARN-10592: -- cc [~leftnoteasy] [~bteke] [~gandras] Any thoughts about this, we use vector update in QueueCapacities for future need in YARN-10524. And the patch has checked the absolute related, which can fix YARN-10590. > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10592.001.patch > > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Description: Because we need to support YARN-10524 and related other multi type support. Support QueueCapacities to use vector. Also this patch has to update absolute related to use first, to make sure no loss of multi type resource fixed by YARN-10590. was: Because we need to support YARN-10524 and related other multi type support. Support QueueCapacities to use vector. Also this patch has to > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to update absolute related to use first, to make sure no > loss of multi type resource fixed by YARN-10590. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Description: Because we need to support YARN-10524 and related other multi type support. Support QueueCapacities to use vector. Also this patch has to was: Because we need to support YARN-10524 and related other multi type support. We should support cap/absolute vector > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we need to support YARN-10524 and related other multi type support. > Support QueueCapacities to use vector. > Also this patch has to -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
[ https://issues.apache.org/jira/browse/YARN-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10592: - Description: Because we need to support YARN-10524 and related other multi type support. We should support cap/absolute vector > Support QueueCapacities to use vector, and update absolute related to use > first. > > > Key: YARN-10592 > URL: https://issues.apache.org/jira/browse/YARN-10592 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > > Because we need to support YARN-10524 and related other multi type support. > We should support cap/absolute vector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10592) Support QueueCapacities to use vector, and update absolute related to use first.
zhuqi created YARN-10592: Summary: Support QueueCapacities to use vector, and update absolute related to use first. Key: YARN-10592 URL: https://issues.apache.org/jira/browse/YARN-10592 Project: Hadoop YARN Issue Type: Sub-task Reporter: zhuqi Assignee: zhuqi -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
