[jira] [Updated] (YARN-10621) GPU management using OpenCL instead of vendor-specific solutions
[
https://issues.apache.org/jira/browse/YARN-10621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sotiris Niarchos updated YARN-10621:
Description:
As part of the [E2Data research project|https://e2data.eu/], we at the
[Institute of Communication and Computer Systems
(ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
University of Athens, Greece, have been working on a modified version of Hadoop
Yarn where the GPU devices that are available in the underlying cluster are
discovered via a Java wrapper of the OpenCL framework API (namely
[JOCL|https://github.com/gpu/JOCL]), instead of vendor-specific binaries.
In other words, we have shifted towards *a more uniform and high-level handling
of GPUs as "OpenCL-enabled" devices*. This way, we manage to *decouple GPU
discovery/management from vendor-specific technicalities*; every GPU, no matter
the vendor, is the same for E2Data YARN (more specifically, for the
{{NodeManager}} component), provided that the OpenCL runtime and drivers for
the GPU(s) of interest are installed on the respective node(s) of the cluster.
This way, we *managed to use GPUs other than NVIDIA* (which are the only ones
officially supported via the {{nvidia-smi}} binary) with minimal additional
effort, after our initial changes.
Ultimately, our goal is to *unify every processing unit* that YARN can possible
utilize (CPU cores, GPUs, FPGAs) *behind a common, simple, high-level
interface; that of the OpenCL-enabled device*.
The only drawback of our approach is that vendor-specific info regarding the
GPUs is lost (e.g. temperature). We believe, however, that the lost information
is not necessary for YARN; everything that Hadoop needs in order to discover
and handle GPU devices is provided by OpenCL.
This is just a proposition/a prompt for discussion for the time being. This
modified version is a work in progress. We consider community feedback
regarding the core concept (and the fact that it may constitute a paradigm
shift for YARN) crucial before attaching any patch file and diving into more
(technical) details.
was:
As part of the [E2Data research project|https://e2data.eu/], we at the
[Institute of Communication and Computer Systems
(ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
University of Athens, Greece, have been working on a modified version of Hadoop
Yarn where the GPU devices that are available in the underlying cluster are
discovered via a Java wrapper of the OpenCL framework API (namely
[JOCL|https://github.com/gpu/JOCL]), instead of vendor-specific binaries.
In other words, we have shifted towards *a more uniform and high-level handling
of GPUs as "OpenCL-enabled" devices*. This way, we manage to *decouple GPU
discovery/management from vendor-specific technicalities*; every GPU, no matter
the vendor, is the same for E2Data YARN (more specifically, by the
{{NodeManager}} component), provided that the OpenCL runtime and drivers for
the GPU(s) of interest are installed on the respective node(s) of the cluster.
This way, we *managed to use GPUs other than NVIDIA* (which are the only ones
officially supported via the {{nvidia-smi}} binary) with minimal additional
effort, after our initial changes.
Ultimately, our goal is to *unify every processing unit* that YARN can possible
utilize (CPU cores, GPUs, FPGAs) *behind a common, simple, high-level
interface; that of the OpenCL-enabled device*.
The only drawback of our approach is that vendor-specific info regarding the
GPUs is lost (e.g. temperature). We believe, however, that the lost information
is not necessary for YARN; everything that Hadoop needs in order to discover
and handle GPU devices is provided by OpenCL.
This is just a proposition/a prompt for discussion for the time being. This
modified version is a work in progress. We consider community feedback
regarding the core concept (and the fact that it may constitute a paradigm
shift for YARN) crucial before attaching any patch file and diving into more
(technical) details.
> GPU management using OpenCL instead of vendor-specific solutions
>
>
> Key: YARN-10621
> URL: https://issues.apache.org/jira/browse/YARN-10621
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Sotiris Niarchos
>Priority: Minor
>
> As part of the [E2Data research project|https://e2data.eu/], we at the
> [Institute of Communication and Computer Systems
> (ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
> University of Athens, Greece, have been working on a modified version of
> Hadoop Yarn where the GPU devices that are available in the underlying
> cluster are discovered via a Java wrapper of the OpenCL framewo
[jira] [Commented] (YARN-10588) Percentage of queue and cluster is zero in WebUI
[
https://issues.apache.org/jira/browse/YARN-10588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282097#comment-17282097
]
Eric Payne commented on YARN-10588:
---
[~Jim_Brennan], it does seem like DominantResourceCalculator.isInvalidDivisor()
should match the logic of DominantResourceCalculator.divide(). However, I
wouldn't do that as part of this JIRA. I think just taking out the
isInvalidDivisor check is fine in this JIRA.
[~BilwaST], I am curious why the change was necessary in FiCaSchedulerApp.java?
I'm nervous about making any change to FiCaSchedulerApp.java, even one for
getResourceUsageReport.
> Percentage of queue and cluster is zero in WebUI
> -
>
> Key: YARN-10588
> URL: https://issues.apache.org/jira/browse/YARN-10588
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10588.001.patch, YARN-10588.002.patch,
> YARN-10588.003.patch
>
>
> Steps to reproduce:
> Configure below property in resource-types.xml
> {code:java}
>
> yarn.resource-types
> yarn.io/gpu
> {code}
> Submit a job
> In UI you can see % Of Queue and % Of Cluster is zero for the submitted
> application
>
> This is because in SchedulerApplicationAttempt has below check for
> calculating queueUsagePerc and clusterUsagePerc
> {code:java}
> if (!calc.isInvalidDivisor(cluster)) {
> float queueCapacityPerc = queue.getQueueInfo(false, false)
> .getCapacity();
> queueUsagePerc = calc.divide(cluster, usedResourceClone,
> Resources.multiply(cluster, queueCapacityPerc)) * 100;
> if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
> queueUsagePerc = 0.0f;
> }
> clusterUsagePerc =
> calc.divide(cluster, usedResourceClone, cluster) * 100;
> }
> {code}
> calc.isInvalidDivisor(cluster) always returns true as gpu resource is 0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10621) GPU management using OpenCL instead of vendor-specific solutions
[
https://issues.apache.org/jira/browse/YARN-10621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sotiris Niarchos updated YARN-10621:
Description:
As part of the [E2Data research project|https://e2data.eu/], we at the
[Institute of Communication and Computer Systems
(ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
University of Athens, Greece, have been working on a modified version of Hadoop
Yarn where the GPU devices that are available in the underlying cluster are
discovered via a Java wrapper of the OpenCL framework API (namely
[JOCL|https://github.com/gpu/JOCL]), instead of vendor-specific binaries.
In other words, we have shifted towards *a more uniform and high-level handling
of GPUs as "OpenCL-enabled" devices*. This way, we manage to *decouple GPU
discovery/management from vendor-specific technicalities*; every GPU, no matter
the vendor, is the same for E2Data YARN (more specifically, by the
{{NodeManager}} component), provided that the OpenCL runtime and drivers for
the GPU(s) of interest are installed on the respective node(s) of the cluster.
This way, we *managed to use GPUs other than NVIDIA* (which are the only ones
officially supported via the {{nvidia-smi}} binary) with minimal additional
effort, after our initial changes.
Ultimately, our goal is to *unify every processing unit* that YARN can possible
utilize (CPU cores, GPUs, FPGAs) *behind a common, simple, high-level
interface; that of the OpenCL-enabled device*.
The only drawback of our approach is that vendor-specific info regarding the
GPUs is lost (e.g. temperature). We believe, however, that the lost information
is not necessary for YARN; everything that Hadoop needs in order to discover
and handle GPU devices is provided by OpenCL.
This is just a proposition/a prompt for discussion for the time being. This
modified version is a work in progress. We consider community feedback
regarding the core concept (and the fact that it may constitute a paradigm
shift for YARN) crucial before attaching any patch file and diving into more
(technical) details.
was:
As part of the [E2Data research project|https://e2data.eu/], we at the
[Institute of Communication and Computer Systems
(ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
University of Athens, Greece, Athens have been working on a modified version of
Hadoop Yarn where the GPU devices that are available in the underlying cluster
are discovered via a Java wrapper of the OpenCL framework API (namely
[JOCL|https://github.com/gpu/JOCL]), instead of vendor-specific binaries.
In other words, we have shifted towards *a more uniform and high-level handling
of GPUs as "OpenCL-enabled" devices*. This way, we manage to *decouple GPU
discovery/management from vendor-specific technicalities*; every GPU, no matter
the vendor, is the same for E2Data YARN (more specifically, by the
{{NodeManager}} component), provided that the OpenCL runtime and drivers for
the GPU(s) of interest are installed on the respective node(s) of the cluster.
This way, we *managed to use GPUs other than NVIDIA* (which are the only ones
officially supported via the {{nvidia-smi}} binary) with minimal additional
effort, after our initial changes.
Ultimately, our goal is to *unify every processing unit* that YARN can possible
utilize (CPU cores, GPUs, FPGAs) *behind a common, simple, high-level
interface; that of the OpenCL-enabled device*.
The only drawback of our approach is that vendor-specific info regarding the
GPUs is lost (e.g. temperature). We believe, however, that the lost information
is not necessary for YARN; everything that Hadoop needs in order to discover
and handle GPU devices is provided by OpenCL.
This is just a proposition/a prompt for discussion for the time being. This
modified version is a work in progress. We consider community feedback
regarding the core concept (and the fact that it may constitute a paradigm
shift for YARN) crucial before attaching any patch file and diving into more
(technical) details.
> GPU management using OpenCL instead of vendor-specific solutions
>
>
> Key: YARN-10621
> URL: https://issues.apache.org/jira/browse/YARN-10621
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, yarn
>Reporter: Sotiris Niarchos
>Priority: Minor
>
> As part of the [E2Data research project|https://e2data.eu/], we at the
> [Institute of Communication and Computer Systems
> (ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
> University of Athens, Greece, have been working on a modified version of
> Hadoop Yarn where the GPU devices that are available in the underlying
> cluster are discovered via a Java wrapper of the OpenCL f
[jira] [Created] (YARN-10621) GPU management using OpenCL instead of vendor-specific solutions
Sotiris Niarchos created YARN-10621:
---
Summary: GPU management using OpenCL instead of vendor-specific
solutions
Key: YARN-10621
URL: https://issues.apache.org/jira/browse/YARN-10621
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager, yarn
Reporter: Sotiris Niarchos
As part of the [E2Data research project|https://e2data.eu/], we at the
[Institute of Communication and Computer Systems
(ICCS)|https://www.iccs.gr/en/?noredirect=en_US] of the National Technical
University of Athens, Greece, Athens have been working on a modified version of
Hadoop Yarn where the GPU devices that are available in the underlying cluster
are discovered via a Java wrapper of the OpenCL framework API (namely
[JOCL|https://github.com/gpu/JOCL]), instead of vendor-specific binaries.
In other words, we have shifted towards *a more uniform and high-level handling
of GPUs as "OpenCL-enabled" devices*. This way, we manage to *decouple GPU
discovery/management from vendor-specific technicalities*; every GPU, no matter
the vendor, is the same for E2Data YARN (more specifically, by the
{{NodeManager}} component), provided that the OpenCL runtime and drivers for
the GPU(s) of interest are installed on the respective node(s) of the cluster.
This way, we *managed to use GPUs other than NVIDIA* (which are the only ones
officially supported via the {{nvidia-smi}} binary) with minimal additional
effort, after our initial changes.
Ultimately, our goal is to *unify every processing unit* that YARN can possible
utilize (CPU cores, GPUs, FPGAs) *behind a common, simple, high-level
interface; that of the OpenCL-enabled device*.
The only drawback of our approach is that vendor-specific info regarding the
GPUs is lost (e.g. temperature). We believe, however, that the lost information
is not necessary for YARN; everything that Hadoop needs in order to discover
and handle GPU devices is provided by OpenCL.
This is just a proposition/a prompt for discussion for the time being. This
modified version is a work in progress. We consider community feedback
regarding the core concept (and the fact that it may constitute a paradigm
shift for YARN) crucial before attaching any patch file and diving into more
(technical) details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282020#comment-17282020
]
Hadoop QA commented on YARN-10620:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 50s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {c
[jira] [Commented] (YARN-10619) CS Mapping Rule %specified rule catches default submissions
[ https://issues.apache.org/jira/browse/YARN-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281900#comment-17281900 ] Szilard Nemeth commented on YARN-10619: --- Thanks [~shuzirra] for working on this. Patch LGTM, committed to trunk. > CS Mapping Rule %specified rule catches default submissions > --- > > Key: YARN-10619 > URL: https://issues.apache.org/jira/browse/YARN-10619 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10619.001.patch > > > If we have a mapping rule which places the application to the %specified > queue, then application submissions without specified queues (default) will > get placed to default. > The expected behaviour would be to fail the specified placement when no queue > was specified, and move on or reject based on the fallback action of the > rule. > Also it is impossible to differentiate between explicitly specified 'default' > and when the user does not specify any actual queue, so these will be handled > the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10619) CS Mapping Rule %specified rule catches default submissions
[ https://issues.apache.org/jira/browse/YARN-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10619: -- Fix Version/s: 3.4.0 > CS Mapping Rule %specified rule catches default submissions > --- > > Key: YARN-10619 > URL: https://issues.apache.org/jira/browse/YARN-10619 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10619.001.patch > > > If we have a mapping rule which places the application to the %specified > queue, then application submissions without specified queues (default) will > get placed to default. > The expected behaviour would be to fail the specified placement when no queue > was specified, and move on or reject based on the fallback action of the > rule. > Also it is impossible to differentiate between explicitly specified 'default' > and when the user does not specify any actual queue, so these will be handled > the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10620:
Attachment: YARN-10620-001.patch
> fs2cs: parentQueue for certain placement rules are not set during conversion
>
>
> Key: YARN-10620
> URL: https://issues.apache.org/jira/browse/YARN-10620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10620-001.patch
>
>
> There are some placement rules in FS which are currently not handled properly
> by fs2cs:
> {noformat}
>
>
>
>
> {noformat}
> The first rule means that if the user queue doesn't exist, it should be
> created as {{root.}}.
> The second means the same thing, except refers to the primary group instead
> of the submitting user: {{root.}}.
> The problem is that in order for the create="true" setting to take effect, we
> must set the parent queue in the generated JSON:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "create" : true
> } ]
> }
> {noformat}
> This is missing right now and it need to be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10619) CS Mapping Rule %specified rule catches default submissions
[
https://issues.apache.org/jira/browse/YARN-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281894#comment-17281894
]
Hadoop QA commented on YARN-10619:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
5s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 18s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
6s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config;
considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 40s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {c
[jira] [Updated] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10620:
Description:
There are some placement rules in FS which are currently not handled properly
by fs2cs:
{noformat}
{noformat}
The first rule means that if the user queue doesn't exist, it should be created
as {{root.}}.
The second means the same thing, except refers to the primary group instead of
the submitting user: {{root.}}.
The problem is that in order for the create="true" setting to take effect, we
must set the parent queue in the generated JSON:
{noformat}
{
"rules" : [ {
"type" : "user",
"matches" : "*",
"policy" : "user",
"fallbackResult" : "skip",
"create" : true
}, {
"type" : "user",
"matches" : "*",
"policy" : "primaryGroup",
"fallbackResult" : "skip",
"create" : true
} ]
}
{noformat}
This is missing right now and it need to be fixed.
> fs2cs: parentQueue for certain placement rules are not set during conversion
>
>
> Key: YARN-10620
> URL: https://issues.apache.org/jira/browse/YARN-10620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> There are some placement rules in FS which are currently not handled properly
> by fs2cs:
> {noformat}
>
>
>
>
> {noformat}
> The first rule means that if the user queue doesn't exist, it should be
> created as {{root.}}.
> The second means the same thing, except refers to the primary group instead
> of the submitting user: {{root.}}.
> The problem is that in order for the create="true" setting to take effect, we
> must set the parent queue in the generated JSON:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "create" : true
> } ]
> }
> {noformat}
> This is missing right now and it need to be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
Peter Bacsko created YARN-10620: --- Summary: fs2cs: parentQueue for certain placement rules are not set during conversion Key: YARN-10620 URL: https://issues.apache.org/jira/browse/YARN-10620 Project: Hadoop YARN Issue Type: Sub-task Reporter: Peter Bacsko Assignee: Peter Bacsko -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10619) CS Mapping Rule %specified rule catches default submissions
Gergely Pollak created YARN-10619: - Summary: CS Mapping Rule %specified rule catches default submissions Key: YARN-10619 URL: https://issues.apache.org/jira/browse/YARN-10619 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gergely Pollak Assignee: Gergely Pollak If we have a mapping rule which places the application to the %specified queue, then application submissions without specified queues (default) will get placed to default. The expected behaviour would be to fail the specified placement when no queue was specified, and move on or reject based on the fallback action of the rule. Also it is impossible to differentiate between explicitly specified 'default' and when the user does not specify any actual queue, so these will be handled the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
