[jira] [Resolved] (YARN-7054) Yarn Service Phase 2
[ https://issues.apache.org/jira/browse/YARN-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He resolved YARN-7054. --- Resolution: Fixed Assignee: Jian He Fix Version/s: yarn-native-services Committed to yarn-native-services branch, thanks Billie ! > Yarn Service Phase 2 > > > Key: YARN-7054 > URL: https://issues.apache.org/jira/browse/YARN-7054 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jian He >Assignee: Jian He > Fix For: yarn-native-services > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7127) Merge yarn-native-service branch into trunk
[ https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223870#comment-16223870 ] Jian He commented on YARN-7127: --- [~aw], we have addressed all these three, if you got time, please help check. I plan to restart the vote to trunk, once the jenkins comes green. bq. 1) Obviously the RegistryDNS 100% cpu issue. [I'm truly surprised that no one else had noticed its awful performance characteristics. it is fixed. bq. 2) Banish the separate API server, now that YARN-6626 has been committed. It's confusing and greatly increases the operating costs (and worse, potential security exposure) for little-to-no real benefit vs just using the REST API from the RM. So just remove it from the docs and the yarn command. it is removed. bq. 3) Integrate the yarn service commands into yarn application as mentioned by Eric Yang. it is merged into yarn application command. > Merge yarn-native-service branch into trunk > --- > > Key: YARN-7127 > URL: https://issues.apache.org/jira/browse/YARN-7127 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7127.01.patch, YARN-7127.02.patch, > YARN-7127.03.patch, YARN-7127.04.patch, YARN-7127.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-7054) Yarn Service Phase 2
[ https://issues.apache.org/jira/browse/YARN-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reopened YARN-7054: --- > Yarn Service Phase 2 > > > Key: YARN-7054 > URL: https://issues.apache.org/jira/browse/YARN-7054 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jian He >Assignee: Jian He > Fix For: yarn-native-services > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-7054) Yarn Service Phase 2
[ https://issues.apache.org/jira/browse/YARN-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7054: -- Comment: was deleted (was: Committed to yarn-native-services branch, thanks Billie !) > Yarn Service Phase 2 > > > Key: YARN-7054 > URL: https://issues.apache.org/jira/browse/YARN-7054 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jian He >Assignee: Jian He > Fix For: yarn-native-services > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7127) Merge yarn-native-service branch into trunk
[ https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7127: -- Attachment: YARN-7127.05.patch Uploaded a latest combined patch of yarn-native-services branch > Merge yarn-native-service branch into trunk > --- > > Key: YARN-7127 > URL: https://issues.apache.org/jira/browse/YARN-7127 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7127.01.patch, YARN-7127.02.patch, > YARN-7127.03.patch, YARN-7127.04.patch, YARN-7127.05.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7224) Support GPU isolation for docker container
[ https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223861#comment-16223861 ] Hudson commented on YARN-7224: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13157 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13157/]) YARN-7224. Support GPU isolation for docker container. Contributed by (sunilg: rev 9114d7a5a0159bbe70e9c289dbe1fc5db9101db5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/conf/container-executor.cfg * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuResourcePlugin.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/GpuResourceHandlerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/LinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/JavaSandboxLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/NvidiaDockerV1CommandPlugin.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerVolumeCommand.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/ResourcePlugin.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDockerCommandPluginFactory.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/DockerCommandPlugin.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java * (add)
[jira] [Commented] (YARN-7380) Check findbugs in timeline service branch-2
[
https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223921#comment-16223921
]
Hadoop QA commented on YARN-7380:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
in branch-2 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7380 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894589/YARN-7380-branch-2.0001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3152b4a05989 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 2654d99 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/18228/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18228/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-YARN-Build/18228/artifact/out/patch-asflicense-problems.txt
|
| modules | C:
[jira] [Comment Edited] (YARN-7391) Consider square root instead of natural log for size-based weight
[ https://issues.apache.org/jira/browse/YARN-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223996#comment-16223996 ] Steven Rand edited comment on YARN-7391 at 10/29/17 1:27 PM: - [~templedf] and [~yufeigu], thanks for commenting. Apologies for not including my use case in the original description. We run multiple long-running Spark applications, each of which uses Spark's dynamic allocation feature, and therefore has a demand which fluctuates over time. At any point, the demand of any given app can be quite low (e.g., only an AM container), or quite high (e.g., hundreds of executors). Historically, we've run each app in its own leaf queue, since the Fair Scheduler has not always supported preemption inside a leaf queue. We've found that since the fair share of a parent queue is split evenly among all of its active leaf queues, the fair share of each app is the same, regardless of its demand. This causes our apps with higher demand to have fair shares that are too low for them to preempt enough resources to even get close to meeting their demand. If fair share were based on demand, then our apps with lower demand would be unaffected, but our apps with higher demand could have high enough weights to preempt a reasonable number of resources away from apps that are over their fair shares. This problem led us to consider running more apps inside the same leaf queue, which is no longer an issue now that the Fair Scheduler supports preemption between apps in the same leaf queue. We'd hoped to use the size-based weight feature to achieve the goal of the more demanding apps having high enough fair shares to preempt sufficient resources away from other apps. However, in experimenting with this feature, the results were somewhat underwhelming. Yes, the more demanding apps now have higher fair shares, but not by enough to significantly impact allocation. Consider, for example, the rather extreme case of 10 apps running in a leaf queue, where 9 of them are requesting 20GB each, and 1 of them is requesting 1024GB. The weight of each of the 9 less demanding apps is about 14.3, and the weight of the highly demanding app is about 20.0. So the highly demanding app winds up with about 13.5% (20/148) of the queue's fair share, despite having a demand that's more than 5x that of the other 9 put together, as opposed to the 10% it would have with size-based weight turned off. I know the example is a bit silly, but I wanted to show that even with huge differences in demand, the current behavior of size-based weight doesn't produce major differences in weights. Does that make sense? Happy to provide more info if helpful. was (Author: steven rand): [~templedf] and [~yufeigu], thanks for commenting. Apologies for not including my use case in the original description. We run multiple long-running Spark applications, each of which uses Spark's dynamic allocation feature, and therefore has a demand which fluctuates over time. At any point, the demand of any given app can be quite low (e.g., only an AM container), or quite high (e.g., hundreds of executors). Historically, we've run each app in its own leaf queue, since the Fair Scheduler has not always supported preemption inside a leaf queue. We've found that since the fair share of a parent queue is split evenly among all of its active leaf queues, the fair share of each app is the same, regardless of its demand. This causes our apps with higher demand to have fair shares that are too low for them to preempt enough resources to even get close to meeting their demand. If fair share were based on demand, then our apps with lower demand would be unaffected, but our apps with higher demand could have high enough weights to preempt a reasonable number of resources away from apps that over their fair shares. This problem led us to consider running more apps inside the same leaf queue, which is no longer an issue now that the Fair Scheduler supports preemption inside a leaf queue. We'd hoped to use the size-based weight feature to achieve the goal of the more demanding apps having high enough fair shares to preempt sufficient resources away from other apps. However, in experimenting with this feature, the results were somewhat underwhelming. Yes, the more demanding apps now have higher fair shares, but not by enough to significantly impact allocation. Consider, for example, the rather extreme case of 10 apps running in a leaf queue, where 9 of them are requesting 20GB each, and 1 of them is requesting 1024GB. The weight of each of the 9 less demanding apps is about 14.3, and the weight of the highly demanding app is about 20.0. So the highly demanding app winds up with about 13.5% (20/148) of the queue's fair share, despite having a demand that's more than 5x that of the other 9 put together, as opposed to the 10% it would
[jira] [Commented] (YARN-7391) Consider square root instead of natural log for size-based weight
[
https://issues.apache.org/jira/browse/YARN-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223996#comment-16223996
]
Steven Rand commented on YARN-7391:
---
[~templedf] and [~yufeigu], thanks for commenting. Apologies for not including
my use case in the original description. We run multiple long-running Spark
applications, each of which uses Spark's dynamic allocation feature, and
therefore has a demand which fluctuates over time. At any point, the demand of
any given app can be quite low (e.g., only an AM container), or quite high
(e.g., hundreds of executors).
Historically, we've run each app in its own leaf queue, since the Fair
Scheduler has not always supported preemption inside a leaf queue. We've found
that since the fair share of a parent queue is split evenly among all of its
active leaf queues, the fair share of each app is the same, regardless of its
demand. This causes our apps with higher demand to have fair shares that are
too low for them to preempt enough resources to even get close to meeting their
demand. If fair share were based on demand, then our apps with lower demand
would be unaffected, but our apps with higher demand could have high enough
weights to preempt a reasonable number of resources away from apps that over
their fair shares.
This problem led us to consider running more apps inside the same leaf queue,
which is no longer an issue now that the Fair Scheduler supports preemption
inside a leaf queue. We'd hoped to use the size-based weight feature to achieve
the goal of the more demanding apps having high enough fair shares to preempt
sufficient resources away from other apps. However, in experimenting with this
feature, the results were somewhat underwhelming. Yes, the more demanding apps
now have higher fair shares, but not by enough to significantly impact
allocation.
Consider, for example, the rather extreme case of 10 apps running in a leaf
queue, where 9 of them are requesting 20GB each, and 1 of them is requesting
1024GB. The weight of each of the 9 less demanding apps is about 14.3, and the
weight of the highly demanding app is about 20.0. So the highly demanding app
winds up with about 13.5% (20/148) of the queue's fair share, despite having a
demand that's more than 5x that of the other 9 put together, as opposed to the
10% it would have with size-based weight turned off. I know the example is a
bit silly, but I wanted to show that even with huge differences in demand, the
current behavior of size-based weight doesn't produce major differences in
weights.
Does that make sense? Happy to provide more info if helpful.
> Consider square root instead of natural log for size-based weight
> -
>
> Key: YARN-7391
> URL: https://issues.apache.org/jira/browse/YARN-7391
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-beta1
>Reporter: Steven Rand
>
> Currently for size-based weight, we compute the weight of an app using this
> code from
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#L377:
> {code}
> if (sizeBasedWeight) {
> // Set weight based on current memory demand
> weight = Math.log1p(app.getDemand().getMemorySize()) / Math.log(2);
> }
> {code}
> Because the natural log function grows slowly, the weights of two apps with
> hugely different memory demands can be quite similar. For example, {{weight}}
> evaluates to 14.3 for an app with a demand of 20 GB, and evaluates to 19.9
> for an app with a demand of 1000 GB. The app with the much larger demand will
> still have a higher weight, but not by a large amount relative to the sum of
> those weights.
> I think it's worth considering a switch to a square root function, which will
> grow more quickly. In the above example, the app with a demand of 20 GB now
> has a weight of 143, while the app with a demand of 1000 GB now has a weight
> of 1012. These weights seem more reasonable relative to each other given the
> difference in demand between the two apps.
> The above example is admittedly a bit extreme, but I believe that a square
> root function would also produce reasonable results in general.
> The code I have in mind would look something like:
> {code}
> if (sizeBasedWeight) {
> // Set weight based on current memory demand
> weight = Math.sqrt(app.getDemand().getMemorySize());
> }
> {code}
> Would people be comfortable with this change?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe,
[jira] [Comment Edited] (YARN-7378) Documentation changes post branch-2 merge
[
https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223924#comment-16223924
]
Varun Saxena edited comment on YARN-7378 at 10/29/17 9:48 AM:
--
[~vrushalic], one more comment.
In the documentation, we are referring to version 3.0.0-alpha1. Atleast for the
jars, this is incorrect.
We can change the file extension from {{md}} to {{md.vm}} and refer to
{{$\{project.version\}}} instead of specific version inside the documentation.
Ideally this change should be made for trunk as well.
was (Author: varun_saxena):
[~vrushalic], one more comment.
In the documentation, we are referring to version 3.0.0-alpha1. Atleast for the
jars, this is incorrect.
We can rename the change the file extension from {{md}} to {{md.vm}} and refer
to {{$\{project.version\}}} instead of specific version inside the
documentation. Ideally this change should be made for trunk as well.
> Documentation changes post branch-2 merge
> -
>
> Key: YARN-7378
> URL: https://issues.apache.org/jira/browse/YARN-7378
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
> Attachments: YARN-7378-branch-2.0001.patch,
> YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch, schema creation
> documentation.png
>
>
> Need to update the documentation for the schema creator command. It should
> include the timeline-service-hbase jar as well as hbase-server jar in
> classpath when the command is to be run. Due to YARN-7190 classpath changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7391) Consider square root instead of natural log for size-based weight
[
https://issues.apache.org/jira/browse/YARN-7391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Rand updated YARN-7391:
--
Attachment: YARN-7391-001.patch
I know this is still under discussion, but attached a patch just to make the
intent/scope of the proposed change totally clear.
> Consider square root instead of natural log for size-based weight
> -
>
> Key: YARN-7391
> URL: https://issues.apache.org/jira/browse/YARN-7391
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-beta1
>Reporter: Steven Rand
> Attachments: YARN-7391-001.patch
>
>
> Currently for size-based weight, we compute the weight of an app using this
> code from
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#L377:
> {code}
> if (sizeBasedWeight) {
> // Set weight based on current memory demand
> weight = Math.log1p(app.getDemand().getMemorySize()) / Math.log(2);
> }
> {code}
> Because the natural log function grows slowly, the weights of two apps with
> hugely different memory demands can be quite similar. For example, {{weight}}
> evaluates to 14.3 for an app with a demand of 20 GB, and evaluates to 19.9
> for an app with a demand of 1000 GB. The app with the much larger demand will
> still have a higher weight, but not by a large amount relative to the sum of
> those weights.
> I think it's worth considering a switch to a square root function, which will
> grow more quickly. In the above example, the app with a demand of 20 GB now
> has a weight of 143, while the app with a demand of 1000 GB now has a weight
> of 1012. These weights seem more reasonable relative to each other given the
> difference in demand between the two apps.
> The above example is admittedly a bit extreme, but I believe that a square
> root function would also produce reasonable results in general.
> The code I have in mind would look something like:
> {code}
> if (sizeBasedWeight) {
> // Set weight based on current memory demand
> weight = Math.sqrt(app.getDemand().getMemorySize());
> }
> {code}
> Would people be comfortable with this change?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7380) Check findbugs in timeline service branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223930#comment-16223930 ] Varun Saxena commented on YARN-7380: Thanks [~vrushalic] for the fix. Changes LGTM. Will commit it shortly. ASF License warning is unrelated. cc [~subru], for your notice. > Check findbugs in timeline service branch-2 > --- > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7380) Fix findbugs warning in timeline service on branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7380: --- Summary: Fix findbugs warning in timeline service on branch-2 (was: Check findbugs in timeline service branch-2) > Fix findbugs warning in timeline service on branch-2 > > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7380) Fix findbugs warning in timeline service on branch-2
[ https://issues.apache.org/jira/browse/YARN-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7380: --- Fix Version/s: 2.9.0 > Fix findbugs warning in timeline service on branch-2 > > > Key: YARN-7380 > URL: https://issues.apache.org/jira/browse/YARN-7380 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0 > > Attachments: YARN-7380-branch-2.0001.patch > > > Some findbugs warnings have been noticed in the branch-2 nightly builds. I > haven't investigated them yet but filing to confirm/fix if possible. > I recollect some known findbugs issues with the webservices function calls > with number of parameters which we could not fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7378) Documentation changes post branch-2 merge
[
https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223924#comment-16223924
]
Varun Saxena commented on YARN-7378:
[~vrushalic], one more comment.
In the documentation, we are referring to version 3.0.0-alpha1. Atleast for the
jars, this is incorrect.
We can rename the change the file extension from {{md}} to {{md.vm}} and refer
to {{$\{project.version\}}} instead of specific version inside the
documentation. Ideally this change should be made for trunk as well.
> Documentation changes post branch-2 merge
> -
>
> Key: YARN-7378
> URL: https://issues.apache.org/jira/browse/YARN-7378
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
> Attachments: YARN-7378-branch-2.0001.patch,
> YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch, schema creation
> documentation.png
>
>
> Need to update the documentation for the schema creator command. It should
> include the timeline-service-hbase jar as well as hbase-server jar in
> classpath when the command is to be run. Due to YARN-7190 classpath changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7378) Documentation changes post branch-2 merge
[
https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224404#comment-16224404
]
Hadoop QA commented on YARN-7378:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
45s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7378 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12894680/YARN-7378-branch-2.0004.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 8ad599d34de9 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 243b70e |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| asflicense |
https://builds.apache.org/job/PreCommit-YARN-Build/18234/artifact/out/patch-asflicense-problems.txt
|
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18234/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Documentation changes post branch-2 merge
> -
>
> Key: YARN-7378
> URL: https://issues.apache.org/jira/browse/YARN-7378
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Vrushali C
> Attachments: YARN-7378-branch-2.0001.patch,
> YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch,
> YARN-7378-branch-2.0004.patch, schema creation documentation.png
>
>
> Need to update the documentation for the schema creator command. It should
> include the timeline-service-hbase jar as well as hbase-server jar in
> classpath when the command is to be run. Due to YARN-7190 classpath changes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[
https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224363#comment-16224363
]
Hadoop QA commented on YARN-7330:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 17 new + 56 unchanged - 1 fixed = 73 total (was 57) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 22s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
3s{color} | {color:green} hadoop-yarn-ui in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
30s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
| |
[jira] [Updated] (YARN-7127) Merge yarn-native-service branch into trunk
[ https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7127: -- Attachment: YARN-7127.06.patch Fixed the javac warning, the rest are not related > Merge yarn-native-service branch into trunk > --- > > Key: YARN-7127 > URL: https://issues.apache.org/jira/browse/YARN-7127 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7127.01.patch, YARN-7127.02.patch, > YARN-7127.03.patch, YARN-7127.04.patch, YARN-7127.05.patch, YARN-7127.06.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224331#comment-16224331 ] Sunil G commented on YARN-7159: --- [~maniraj...@gmail.com] Some of test case failures are related. Pls help to check the same > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7378) Documentation changes post branch-2 merge
[ https://issues.apache.org/jira/browse/YARN-7378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7378: - Attachment: YARN-7378-branch-2.0004.patch Thanks [~varun_saxena] , uploading v004 for branch-2. We can mostly likely use this same patch for trunk once the classpath changes for trunk are done. > Documentation changes post branch-2 merge > - > > Key: YARN-7378 > URL: https://issues.apache.org/jira/browse/YARN-7378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Varun Saxena >Assignee: Vrushali C > Attachments: YARN-7378-branch-2.0001.patch, > YARN-7378-branch-2.0002.patch, YARN-7378-branch-2.0003.patch, > YARN-7378-branch-2.0004.patch, schema creation documentation.png > > > Need to update the documentation for the schema creator command. It should > include the timeline-service-hbase jar as well as hbase-server jar in > classpath when the command is to be run. Due to YARN-7190 classpath changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224093#comment-16224093 ] Daniel Templeton commented on YARN-6927: +1 > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Attachments: YARN-6927.000.patch, YARN-6927.001.patch, > YARN-6927.002.patch, YARN-6927.003.patch, YARN-6927.004.patch, > YARN-6927.005.patch, YARN-6927.006.patch, YARN-6927.007.patch, > YARN-6927.008.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7159: --- Attachment: YARN-7159.006.patch > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7159: --- Attachment: (was: YARN-7159.006.patch) > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[
https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224160#comment-16224160
]
Hadoop QA commented on YARN-7159:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 4 new + 111 unchanged - 0 fixed = 115 total (was 111) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 35s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.util.resource.TestResourceUtils |
| | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| | hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
| | hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher |
| | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.TestSignalContainer |
| | hadoop.yarn.server.resourcemanager.TestDecommissioningNodesWatcher
[jira] [Updated] (YARN-7374) Improve performance of DRF comparisons for resource types in fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-7374:
---
Attachment: YARN-7374.005.patch
Whoops. Just noticed that I needed annotations on the exposed constants in
{{Resource}}. Wanna gimme another quick +1, [~yufeigu]
> Improve performance of DRF comparisons for resource types in fair scheduler
> ---
>
> Key: YARN-7374
> URL: https://issues.apache.org/jira/browse/YARN-7374
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7374.001.patch, YARN-7374.002.patch,
> YARN-7374.003.patch, YARN-7374.004.patch, YARN-7374.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224098#comment-16224098 ] Daniel Templeton commented on YARN-6927: Bad news, [~grepas]. It needs a rebase. :( > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Attachments: YARN-6927.000.patch, YARN-6927.001.patch, > YARN-6927.002.patch, YARN-6927.003.patch, YARN-6927.004.patch, > YARN-6927.005.patch, YARN-6927.006.patch, YARN-6927.007.patch, > YARN-6927.008.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-7159: --- Attachment: YARN-7159.006.patch > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7374) Improve performance of DRF comparisons for resource types in fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224153#comment-16224153
]
Hadoop QA commented on YARN-7374:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 6 unchanged - 1 fixed = 7 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
27s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 55m
35s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.DominantResourceFairnessPolicy$DominantResourceFairnessComparator2
implements Comparator but not Serializable At
DominantResourceFairnessPolicy.java:Serializable At
DominantResourceFairnessPolicy.java:[lines 367-508] |
| |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.policies.DominantResourceFairnessPolicy$DominantResourceFairnessComparatorN
implements Comparator but not Serializable At
DominantResourceFairnessPolicy.java:Serializable At
[jira] [Commented] (YARN-6747) TestFSAppStarvation.testPreemptionEnable fails intermittently
[
https://issues.apache.org/jira/browse/YARN-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224255#comment-16224255
]
Hudson commented on YARN-6747:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13158 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13158/])
YARN-6747. TestFSAppStarvation.testPreemptionEnable fails (yufei: rev
d4811c8cfae02f42a7aae1f775e87b6726caa3c9)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSAppStarvation.java
> TestFSAppStarvation.testPreemptionEnable fails intermittently
> -
>
> Key: YARN-6747
> URL: https://issues.apache.org/jira/browse/YARN-6747
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sunil G
>Assignee: Miklos Szegedi
> Fix For: 2.9.0, 3.1.0, 3.0
>
> Attachments: YARN-6747.000.patch, YARN-6747.001.patch,
> YARN-6747.002.patch
>
>
> *Error Message*
> Apps re-added even before starvation delay passed expected:<4> but was:<3>
> *Stacktrace*
> java.lang.AssertionError: Apps re-added even before starvation delay passed
> expected:<4> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:117)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7159) Normalize unit of resource objects in RM and avoid to do unit conversion in critical path
[ https://issues.apache.org/jira/browse/YARN-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224102#comment-16224102 ] Manikandan R commented on YARN-7159: [~sunilg] Thanks for your comments. Incorporated all comments. Attached new patch. > Normalize unit of resource objects in RM and avoid to do unit conversion in > critical path > - > > Key: YARN-7159 > URL: https://issues.apache.org/jira/browse/YARN-7159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Wangda Tan >Assignee: Manikandan R >Priority: Critical > Attachments: YARN-7159.001.patch, YARN-7159.002.patch, > YARN-7159.003.patch, YARN-7159.004.patch, YARN-7159.005.patch, > YARN-7159.006.patch > > > Currently resource conversion could happen in critical code path when > different unit is specified by client. This could impact performance and > throughput of RM a lot. We should do unit normalization when resource passed > to RM and avoid expensive unit conversion every time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7374) Improve performance of DRF comparisons for resource types in fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224243#comment-16224243
]
Yufei Gu commented on YARN-7374:
+1 for the patch 005. The annotation makes sense to me. Java doc of class
{{InterfaceAudience}} doesn't mention we could put a annotation on variables
though. Would you mind file a jira for that?
> Improve performance of DRF comparisons for resource types in fair scheduler
> ---
>
> Key: YARN-7374
> URL: https://issues.apache.org/jira/browse/YARN-7374
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7374.001.patch, YARN-7374.002.patch,
> YARN-7374.003.patch, YARN-7374.004.patch, YARN-7374.005.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6747) TestFSAppStarvation.testPreemptionEnable fails intermittently
[
https://issues.apache.org/jira/browse/YARN-6747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224249#comment-16224249
]
Yufei Gu commented on YARN-6747:
Committed to trunk, branch-3.0 and branch-2. Thanks for the patch,
[~miklos.szeg...@cloudera.com]. Thanks for the review, [~templedf].
> TestFSAppStarvation.testPreemptionEnable fails intermittently
> -
>
> Key: YARN-6747
> URL: https://issues.apache.org/jira/browse/YARN-6747
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sunil G
>Assignee: Miklos Szegedi
> Fix For: 2.9.0, 3.1.0, 3.0
>
> Attachments: YARN-6747.000.patch, YARN-6747.001.patch,
> YARN-6747.002.patch
>
>
> *Error Message*
> Apps re-added even before starvation delay passed expected:<4> but was:<3>
> *Stacktrace*
> java.lang.AssertionError: Apps re-added even before starvation delay passed
> expected:<4> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation.testPreemptionEnabled(TestFSAppStarvation.java:117)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7127) Merge yarn-native-service branch into trunk
[
https://issues.apache.org/jira/browse/YARN-7127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224299#comment-16224299
]
Hadoop QA commented on YARN-7127:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 30 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-yarn-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 24s{color}
| {color:red} root generated 2 new + 1248 unchanged - 0 fixed = 1250 total (was
1248) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 30s{color} | {color:orange} root: The patch generated 899 new + 503
unchanged - 77 fixed = 1402 total (was 580) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
25s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 64 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
3s{color} | {color:red} The patch 12 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
18s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services .
hadoop-yarn-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
[jira] [Commented] (YARN-7224) Support GPU isolation for docker container
[
https://issues.apache.org/jira/browse/YARN-7224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224272#comment-16224272
]
Wangda Tan commented on YARN-7224:
--
Thanks [~sunilg] for review and commit!
> Support GPU isolation for docker container
> --
>
> Key: YARN-7224
> URL: https://issues.apache.org/jira/browse/YARN-7224
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.1.0
>
> Attachments: YARN-7224.001.patch, YARN-7224.002-wip.patch,
> YARN-7224.003.patch, YARN-7224.004.patch, YARN-7224.005.patch,
> YARN-7224.006.patch, YARN-7224.007.patch, YARN-7224.008.patch,
> YARN-7224.009.patch
>
>
> This patch is to address issues when docker container is being used:
> 1. GPU driver and nvidia libraries: If GPU drivers and NV libraries are
> pre-packaged inside docker image, it could conflict to driver and
> nvidia-libraries installed on Host OS. An alternative solution is to detect
> Host OS's installed drivers and devices, mount it when launch docker
> container. Please refer to \[1\] for more details.
> 2. Image detection:
> From \[2\], the challenge is:
> bq. Mounting user-level driver libraries and device files clobbers the
> environment of the container, it should be done only when the container is
> running a GPU application. The challenge here is to determine if a given
> image will be using the GPU or not. We should also prevent launching
> containers based on a Docker image that is incompatible with the host NVIDIA
> driver version, you can find more details on this wiki page.
> 3. GPU isolation.
> *Proposed solution*:
> a. Use nvidia-docker-plugin \[3\] to address issue #1, this is the same
> solution used by K8S \[4\]. issue #2 could be addressed in a separate JIRA.
> We won't ship nvidia-docker-plugin with out releases and we require cluster
> admin to preinstall nvidia-docker-plugin to use GPU+docker support on YARN.
> "nvidia-docker" is a wrapper of docker binary which can address #3 as well,
> however "nvidia-docker" doesn't provide same semantics of docker, and it
> needs to setup additional environments such as PATH/LD_LIBRARY_PATH to use
> it. To avoid introducing additional issues, we plan to use
> nvidia-docker-plugin + docker binary approach.
> b. To address GPU driver and nvidia libraries, we uses nvidia-docker-plugin
> \[3\] to create a volume which includes GPU-related libraries and mount it
> when docker container being launched. Changes include:
> - Instead of using {{volume-driver}}, this patch added {{docker volume
> create}} command to c-e and NM Java side. The reason is {{volume-driver}} can
> only use single volume driver for each launched docker container.
> - Updated {{c-e}} and Java side, if a mounted volume is a named volume in
> docker, skip checking file existence. (Named-volume still need to be added to
> permitted list of container-executor.cfg).
> c. To address isolation issue:
> We found that, cgroup + docker doesn't work under newer docker version which
> uses {{runc}} as default runtime. Setting {{--cgroup-parent}} to a cgroup
> which include any {{devices.deny}} causes docker container cannot be launched.
> Instead this patch passes allowed GPU devices via {{--device}} to docker
> launch command.
> References:
> \[1\] https://github.com/NVIDIA/nvidia-docker/wiki/NVIDIA-driver
> \[2\] https://github.com/NVIDIA/nvidia-docker/wiki/Image-inspection
> \[3\] https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker-plugin
> \[4\] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7416) Use "docker volume inspect" to make sure that volumes for GPU drivers/libs are properly mounted.
Wangda Tan created YARN-7416: Summary: Use "docker volume inspect" to make sure that volumes for GPU drivers/libs are properly mounted. Key: YARN-7416 URL: https://issues.apache.org/jira/browse/YARN-7416 Project: Hadoop YARN Issue Type: Sub-task Reporter: Wangda Tan Assignee: Wangda Tan -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7374) Improve performance of DRF comparisons for resource types in fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224310#comment-16224310 ] Hudson commented on YARN-7374: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13159 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13159/]) YARN-7374. Improve performance of DRF comparisons for resource types in (templedf: rev 9711b78998ca3a1f7734058a78c7baddd130ce0f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/TestDominantResourceFairnessPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceInformation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Resource.java > Improve performance of DRF comparisons for resource types in fair scheduler > --- > > Key: YARN-7374 > URL: https://issues.apache.org/jira/browse/YARN-7374 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Fix For: 3.1.0 > > Attachments: YARN-7374.001.patch, YARN-7374.002.patch, > YARN-7374.003.patch, YARN-7374.004.patch, YARN-7374.005.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
