[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[ https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775825#comment-16775825 ] Szilard Nemeth commented on YARN-9213: -- Sure, I will do that! > RM Web UI v1 does not show custom resource allocations for containers page > -- > > Key: YARN-9213 > URL: https://issues.apache.org/jira/browse/YARN-9213 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen > Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch, > YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch, > YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch, > YARN-9213.branch3.2.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[ https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775824#comment-16775824 ] Sunil Govindan commented on YARN-9213: -- Could you try to avoid the use of CustomResourceTypesConfigurationProvider? > RM Web UI v1 does not show custom resource allocations for containers page > -- > > Key: YARN-9213 > URL: https://issues.apache.org/jira/browse/YARN-9213 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen > Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch, > YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch, > YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch, > YARN-9213.branch3.2.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775794#comment-16775794
]
Prabhu Joseph commented on YARN-9317:
-
Thanks [~bibinchundatt]!
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch,
> YARN-9317-branch-3.1.001.patch, YARN-9317-branch-3.2.001.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9329) Doing updatePriority is blocking when using FairScheduler
[
https://issues.apache.org/jira/browse/YARN-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775786#comment-16775786
]
Hadoop QA commented on YARN-9329:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 37s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9329 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959865/YARN-9329.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6887bc696f8e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 05bce33 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23504/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23504/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Updated] (YARN-9329) Doing updatePriority is blocking when using FairScheduler
[
https://issues.apache.org/jira/browse/YARN-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiandan Yang updated YARN-9329:
Attachment: YARN-9329.001.patch
> Doing updatePriority is blocking when using FairScheduler
> -
>
> Key: YARN-9329
> URL: https://issues.apache.org/jira/browse/YARN-9329
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Attachments: YARN-9329.001.patch
>
>
> updating priority is blocked when using FairScheduler, the command is as
> follows:
> {code:java}
> bin/yarn application -appId application_1550834280474_0003 -updatePriority
> 500
> {code}
> The RM rpc handler thread is blocked in the future.get(), although
> FairScheduler does not support priority currently, it is not friendly to
> user when executing updatePriority.
> The trace of handler is:
> {code:java}
> "IPC Server handler 44 on 18040" #258 daemon prio=5 os_prio=0
> tid=0x7f8e65701000 nid=0x1b9f1 waiting on condition [0x7f8e29123000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000671f13da0> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.getChecked(RMAppManager.java:225)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.updateApplicationPriority(RMAppManager.java:734)
> - locked <0x00067872ed98> (a
> org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.updateApplicationPriority(ClientRMService.java:1658)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.updateApplicationPriority(ApplicationClientProtocolPBServiceImpl.java:622)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:659)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:881)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:827)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2699)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9329) Doing updatePriority is blocking when using FairScheduler
Jiandan Yang created YARN-9329:
---
Summary: Doing updatePriority is blocking when using FairScheduler
Key: YARN-9329
URL: https://issues.apache.org/jira/browse/YARN-9329
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jiandan Yang
Assignee: Jiandan Yang
updating priority is blocked when using FairScheduler, the command is as
follows:
{code:java}
bin/yarn application -appId application_1550834280474_0003 -updatePriority 500
{code}
The RM rpc handler thread is blocked in the future.get(), although
FairScheduler does not support priority currently, it is not friendly to user
when executing updatePriority.
The trace of handler is:
{code:java}
"IPC Server handler 44 on 18040" #258 daemon prio=5 os_prio=0
tid=0x7f8e65701000 nid=0x1b9f1 waiting on condition [0x7f8e29123000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000671f13da0> (a
com.google.common.util.concurrent.AbstractFuture$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.getChecked(RMAppManager.java:225)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.updateApplicationPriority(RMAppManager.java:734)
- locked <0x00067872ed98> (a
org.apache.hadoop.yarn.api.records.impl.pb.ApplicationIdPBImpl)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.updateApplicationPriority(ClientRMService.java:1658)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.updateApplicationPriority(ApplicationClientProtocolPBServiceImpl.java:622)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:659)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:881)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:827)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2699)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9244) Document docker registry deployment with direct S3 driver
[
https://issues.apache.org/jira/browse/YARN-9244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775721#comment-16775721
]
Hudson commented on YARN-9244:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16034 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16034/])
YARN-9244. Document docker registry deployment with direct S3 driver.
(eyang: rev 05bce33d6e8b46c416a37b8d12fd84eb53474d6d)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md
> Document docker registry deployment with direct S3 driver
> -
>
> Key: YARN-9244
> URL: https://issues.apache.org/jira/browse/YARN-9244
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9244.1.patch, YARN-9244.2.patch
>
>
> On Amazon, it would be possible to deploy docker registry in YARN using
> Docker's own S3 driver. The same micro-benchmark can also be applied to show
> the docker registry performance difference. The sample YARN service
> configuration looks like this:
> {code}
> {
> "name": "docker-registry",
> "version": "1.0",
> "kerberos_principal" : {
> "principal_name" : "hbase/_h...@example.com",
> "keytab" : "file:///etc/security/keytabs/hbase.service.keytab"
> },
> "components" :
> [
> {
> "name": "registry",
> "number_of_containers": 1,
> "artifact": {
> "id": "registry:latest",
> "type": "DOCKER"
> },
> "resource": {
> "cpus": 1,
> "memory": "256"
> },
> "run_privileged_container": true,
> "configuration": {
> "env": {
> "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
>
> "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/path/to/config.yml:/etc/docker/registry/config.yml"
> },
> "properties": {
> "docker.network": "host"
> }
> }
> }
> ]
> }
> {code}
> Where /path/to/config.yml maps to a Docker S3 storage option config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9139) Simplify initializer code of GpuDiscoverer
[
https://issues.apache.org/jira/browse/YARN-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775674#comment-16775674
]
Hadoop QA commented on YARN-9139:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 221 unchanged - 1 fixed = 222 total (was 222) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
32s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9139 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959848/YARN-9139.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 73e7c5713dd0 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7057980 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Updated] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-999: - Description: In current design and implementation, when we decrease resource on node to less than resource consumption of current running tasks, tasks can still be running until the end. But just no new task get assigned on this node (because AvailableResource < 0) until some tasks are finished and AvailableResource > 0 again. This is good for most cases but in case of long running task, it could be too slow for resource setting to actually work so preemption could be used here. (was: In current design and implementation, when we decrease resource on node to less than resource consumption of current running tasks, tasks can still be running until the end. But just no new task get assigned on this node (because AvailableResource < 0) until some tasks are finished and AvailableResource > 0 again. This is good for most cases but in case of long running task, it could be too slow for resource setting to actually work so preemption could be hired here.) > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be used here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9138) Test error handling of nvidia-smi binary execution of GpuDiscoverer
[
https://issues.apache.org/jira/browse/YARN-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775642#comment-16775642
]
Hadoop QA commented on YARN-9138:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 5 unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
6s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959847/YARN-9138.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2b641bd93877 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7057980 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23502/testReport/ |
| Max. process+thread count | 443 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23502/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (YARN-9244) Document docker registry deployment with direct S3 driver
[
https://issues.apache.org/jira/browse/YARN-9244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775635#comment-16775635
]
Eric Yang commented on YARN-9244:
-
[~suma.shivaprasad] Thank you for patch 2.
+1 looks good to me.
> Document docker registry deployment with direct S3 driver
> -
>
> Key: YARN-9244
> URL: https://issues.apache.org/jira/browse/YARN-9244
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-9244.1.patch, YARN-9244.2.patch
>
>
> On Amazon, it would be possible to deploy docker registry in YARN using
> Docker's own S3 driver. The same micro-benchmark can also be applied to show
> the docker registry performance difference. The sample YARN service
> configuration looks like this:
> {code}
> {
> "name": "docker-registry",
> "version": "1.0",
> "kerberos_principal" : {
> "principal_name" : "hbase/_h...@example.com",
> "keytab" : "file:///etc/security/keytabs/hbase.service.keytab"
> },
> "components" :
> [
> {
> "name": "registry",
> "number_of_containers": 1,
> "artifact": {
> "id": "registry:latest",
> "type": "DOCKER"
> },
> "resource": {
> "cpus": 1,
> "memory": "256"
> },
> "run_privileged_container": true,
> "configuration": {
> "env": {
> "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
>
> "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/path/to/config.yml:/etc/docker/registry/config.yml"
> },
> "properties": {
> "docker.network": "host"
> }
> }
> }
> ]
> }
> {code}
> Where /path/to/config.yml maps to a Docker S3 storage option config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775629#comment-16775629
]
Hadoop QA commented on YARN-9322:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
40s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 18s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9322 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959834/YARN-9322.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f1695597482d 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed13cf8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[ https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775614#comment-16775614 ] Szilard Nemeth commented on YARN-9121: -- [~sunilg]: Findbugs complains about GpuResourcePlugin.gpuResourceHandler is accessed inconsistently in terms of synchronization. As I haven't touched this field at all, could we ignore this error for this patch? Thanks! > Users of GpuDiscoverer.getInstance() are not possible to test as instance is > a static field > --- > > Key: YARN-9121 > URL: https://issues.apache.org/jira/browse/YARN-9121 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9121.001.patch, YARN-9121.002.patch > > > The clients of GpuDiscoverer are very hard to test as they call > GpuDiscoverer.getInstance() internally. > For example, writing tests for > org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo > is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not > interchangeable as GpuDiscoverer is not mockable since we cannot inject it in > tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9139) Simplify initializer code of GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775603#comment-16775603 ] Szilard Nemeth commented on YARN-9139: -- I had to upload a new patch as patch004 was not applied to trunk without conflicts. > Simplify initializer code of GpuDiscoverer > -- > > Key: YARN-9139 > URL: https://issues.apache.org/jira/browse/YARN-9139 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9139.001.patch, YARN-9139.002.patch, > YARN-9139.003.patch, YARN-9139.004.patch, YARN-9139.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9139) Simplify initializer code of GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9139: - Attachment: YARN-9139.005.patch > Simplify initializer code of GpuDiscoverer > -- > > Key: YARN-9139 > URL: https://issues.apache.org/jira/browse/YARN-9139 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9139.001.patch, YARN-9139.002.patch, > YARN-9139.003.patch, YARN-9139.004.patch, YARN-9139.005.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9138) Test error handling of nvidia-smi binary execution of GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775585#comment-16775585 ] Szilard Nemeth commented on YARN-9138: -- Hi [~adam.antal]! Thanks for your comments, they are very detailed and valuable. 1. Good point, extracted most of the repetitive stuff into methods. 2. As GpuDiscoverer finds out where nvidia-smi lives based on the path provided in the config, I wanted to keep the behaviour in tests as close as possible to the production code. As the script is invoked by a call to Shell.execCommand(), we can count this as a hard-dependency of this class and it's kinda hard to mock this and if I done that, it would change GpuDiscoverer in a more fundamental way. To be precise, the bash script I "generate" in the test is not creating any new files, just echoing the contents of a very basic XML. I would like to keep this as it is. The only change I made with my new patch regarding this is the extraction of common things into methods. 3. Logging is not a common thing in tests as far my experience tells. I'm not saying that it's good or bad, at least that's what I have been seeing. Anyways, I added some logging instead of the comments in testGetGpuDeviceInformationFaultyNvidiaSmiScriptConsecutiveRun. If you have ideas on how to have better logs in this test class, feel free to report a new jira under YARN-9304. About the less concerning things: 1. It was a great idea to extract the parent directory name to a constant so I did that! 2. I gues "RunLinuxGpuResourceDiscoverPluginConfigTest" is set by either the user running the JVM (with a system property) or by some jenkins job. Probably [~sunilg] can tell you more on that as I didn't modify the code and he was the committer of this back in end of 2017. 3. Separation of testLinuxGpuResourceDiscoverPluginConfig: I agree, but I would create a follow-up jira for that. The purpose of my change was not to refactor but rather extend the test coverage. 4. I didn't get your comment about the separation of "getNumberOfUsableGpusFromConfig". Please check my latest patch! > Test error handling of nvidia-smi binary execution of GpuDiscoverer > --- > > Key: YARN-9138 > URL: https://issues.apache.org/jira/browse/YARN-9138 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9138.001.patch, YARN-9138.002.patch, > YARN-9138.003.patch > > > The code that executes nvidia-smi (doing GPU device auto-discovery) don't > have much test coverage. > This patch adds tests to this part of the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9138) Test error handling of nvidia-smi binary execution of GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9138: - Attachment: YARN-9138.004.patch > Test error handling of nvidia-smi binary execution of GpuDiscoverer > --- > > Key: YARN-9138 > URL: https://issues.apache.org/jira/browse/YARN-9138 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9138.001.patch, YARN-9138.002.patch, > YARN-9138.003.patch, YARN-9138.004.patch > > > The code that executes nvidia-smi (doing GPU device auto-discovery) don't > have much test coverage. > This patch adds tests to this part of the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[
https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775575#comment-16775575
]
Hadoop QA commented on YARN-999:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 29s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 368 unchanged - 15 fixed = 371 total (was 383) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 1s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-999 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959829/YARN-999.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 519ee3f5e158 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed13cf8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[
https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775547#comment-16775547
]
Hadoop QA commented on YARN-9121:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 15 unchanged - 2 fixed = 15 total (was 17) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
17s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
46s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| | Inconsistent synchronization of
org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin.gpuResourceHandler;
locked 75% of time Unsynchronized access at GpuResourcePlugin.java:75% of
time Unsynchronized access at GpuResourcePlugin.java:[line 87] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9121 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959832/YARN-9121.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e71531c0bf59 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk /
[jira] [Commented] (YARN-9324) TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11
[
https://issues.apache.org/jira/browse/YARN-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775546#comment-16775546
]
Hadoop QA commented on YARN-9324:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 49s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9324 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959826/YARN-9324-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fd39f4da2214 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed13cf8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23497/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23497/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-9287) Consecutive String Builder Append Should Reuse
[
https://issues.apache.org/jira/browse/YARN-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775541#comment-16775541
]
Hadoop QA commented on YARN-9287:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m
0s{color} | {color:red} The patch appears to contain 1 files with names longer
than 240 {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
19s{color} | {color:red} hadoop-submarine-yarnservice-runtime in trunk failed.
{color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
59s{color} | {color:green} root: The patch generated 0 new + 932 unchanged - 15
fixed = 932 total (was 947) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red} hadoop-submarine-yarnservice-runtime in the patch
failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
56s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775517#comment-16775517
]
Hadoop QA commented on YARN-9317:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
16s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} branch-3.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} branch-3.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
56s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
47s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:080e9d0 |
| JIRA Issue | YARN-9317 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959819/YARN-9317-branch-3.1.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fd7373a5eb0b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.1 / 7db50ff |
| maven | version: Apache Maven 3.3.9 |
| Default
[jira] [Commented] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775490#comment-16775490
]
Hadoop QA commented on YARN-9317:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
11s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
33s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 40s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocationAsync
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | YARN-9317 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959816/YARN-9317-branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux cfe8e989549e 4.4.0-139-generic #165-Ubuntu SMP
[jira] [Comment Edited] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[
https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775469#comment-16775469
]
Szilard Nemeth edited comment on YARN-9121 at 2/22/19 6:22 PM:
---
[~sunilg]: Sure, {{new GpuDiscoverer()}} only happens once in the production
code and that single instance is used everywhere. Also added patch002 as
patch001 was not applyable to trunk without conflicts.
was (Author: snemeth):
[~sunilg]: Sure, {{new GpuDiscoverer()}} only happens once in the production
code and that single instance is used everywhere.
> Users of GpuDiscoverer.getInstance() are not possible to test as instance is
> a static field
> ---
>
> Key: YARN-9121
> URL: https://issues.apache.org/jira/browse/YARN-9121
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9121.001.patch, YARN-9121.002.patch
>
>
> The clients of GpuDiscoverer are very hard to test as they call
> GpuDiscoverer.getInstance() internally.
> For example, writing tests for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo
> is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not
> interchangeable as GpuDiscoverer is not mockable since we cannot inject it in
> tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9322: - Attachment: YARN-9322.003.patch > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775478#comment-16775478 ] Szilard Nemeth commented on YARN-9322: -- Added patch003 to fix checkstyle issues. > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch, YARN-9322.003.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[
https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775469#comment-16775469
]
Szilard Nemeth commented on YARN-9121:
--
[~sunilg]: Sure, {{new GpuDiscoverer()}} only happens once in the production
code and that single instance is used everywhere.
> Users of GpuDiscoverer.getInstance() are not possible to test as instance is
> a static field
> ---
>
> Key: YARN-9121
> URL: https://issues.apache.org/jira/browse/YARN-9121
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-9121.001.patch
>
>
> The clients of GpuDiscoverer are very hard to test as they call
> GpuDiscoverer.getInstance() internally.
> For example, writing tests for
> org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo
> is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not
> interchangeable as GpuDiscoverer is not mockable since we cannot inject it in
> tests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[ https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9121: - Attachment: YARN-9121.002.patch > Users of GpuDiscoverer.getInstance() are not possible to test as instance is > a static field > --- > > Key: YARN-9121 > URL: https://issues.apache.org/jira/browse/YARN-9121 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9121.001.patch, YARN-9121.002.patch > > > The clients of GpuDiscoverer are very hard to test as they call > GpuDiscoverer.getInstance() internally. > For example, writing tests for > org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo > is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not > interchangeable as GpuDiscoverer is not mockable since we cannot inject it in > tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9244) Document docker registry deployment with direct S3 driver
[
https://issues.apache.org/jira/browse/YARN-9244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775456#comment-16775456
]
Hadoop QA commented on YARN-9244:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
25m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9244 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959827/YARN-9244.2.patch |
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 1dcd3e7986ef 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ed13cf8 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 410 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23498/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Document docker registry deployment with direct S3 driver
> -
>
> Key: YARN-9244
> URL: https://issues.apache.org/jira/browse/YARN-9244
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-9244.1.patch, YARN-9244.2.patch
>
>
> On Amazon, it would be possible to deploy docker registry in YARN using
> Docker's own S3 driver. The same micro-benchmark can also be applied to show
> the docker registry performance difference. The sample YARN service
> configuration looks like this:
> {code}
> {
> "name": "docker-registry",
> "version": "1.0",
> "kerberos_principal" : {
> "principal_name" : "hbase/_h...@example.com",
> "keytab" : "file:///etc/security/keytabs/hbase.service.keytab"
> },
> "components" :
> [
> {
> "name": "registry",
> "number_of_containers": 1,
> "artifact": {
> "id": "registry:latest",
> "type": "DOCKER"
> },
> "resource": {
> "cpus": 1,
> "memory": "256"
> },
> "run_privileged_container": true,
> "configuration": {
> "env": {
> "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
>
> "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/path/to/config.yml:/etc/docker/registry/config.yml"
> },
> "properties": {
> "docker.network": "host"
> }
> }
> }
> ]
> }
> {code}
> Where /path/to/config.yml maps to a Docker S3 storage option config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
[jira] [Commented] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[ https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775451#comment-16775451 ] Sunil Govindan commented on YARN-9121: -- On a second thought, will multiple instance of GpuDiscoverer be created any time in this call flow? > Users of GpuDiscoverer.getInstance() are not possible to test as instance is > a static field > --- > > Key: YARN-9121 > URL: https://issues.apache.org/jira/browse/YARN-9121 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9121.001.patch > > > The clients of GpuDiscoverer are very hard to test as they call > GpuDiscoverer.getInstance() internally. > For example, writing tests for > org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo > is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not > interchangeable as GpuDiscoverer is not mockable since we cannot inject it in > tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[
https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775433#comment-16775433
]
Íñigo Goiri commented on YARN-999:
--
I forgot to add the new tests which are used in {{TestCapacityScheduler}} now.
Let's see how unhappy Yetus is now.
> In case of long running tasks, reduce node resource should balloon out
> resource quickly by calling preemption API and suspending running task.
> ---
>
> Key: YARN-999
> URL: https://issues.apache.org/jira/browse/YARN-999
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful, nodemanager, scheduler
>Reporter: Junping Du
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-291.000.patch, YARN-999.001.patch,
> YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch
>
>
> In current design and implementation, when we decrease resource on node to
> less than resource consumption of current running tasks, tasks can still be
> running until the end. But just no new task get assigned on this node
> (because AvailableResource < 0) until some tasks are finished and
> AvailableResource > 0 again. This is good for most cases but in case of long
> running task, it could be too slow for resource setting to actually work so
> preemption could be hired here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-999) In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task.
[ https://issues.apache.org/jira/browse/YARN-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-999: - Attachment: YARN-999.004.patch > In case of long running tasks, reduce node resource should balloon out > resource quickly by calling preemption API and suspending running task. > --- > > Key: YARN-999 > URL: https://issues.apache.org/jira/browse/YARN-999 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful, nodemanager, scheduler >Reporter: Junping Du >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-291.000.patch, YARN-999.001.patch, > YARN-999.002.patch, YARN-999.003.patch, YARN-999.004.patch > > > In current design and implementation, when we decrease resource on node to > less than resource consumption of current running tasks, tasks can still be > running until the end. But just no new task get assigned on this node > (because AvailableResource < 0) until some tasks are finished and > AvailableResource > 0 again. This is good for most cases but in case of long > running task, it could be too slow for resource setting to actually work so > preemption could be hired here. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9244) Document docker registry deployment with direct S3 driver
[
https://issues.apache.org/jira/browse/YARN-9244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suma Shivaprasad updated YARN-9244:
---
Attachment: YARN-9244.2.patch
> Document docker registry deployment with direct S3 driver
> -
>
> Key: YARN-9244
> URL: https://issues.apache.org/jira/browse/YARN-9244
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-9244.1.patch, YARN-9244.2.patch
>
>
> On Amazon, it would be possible to deploy docker registry in YARN using
> Docker's own S3 driver. The same micro-benchmark can also be applied to show
> the docker registry performance difference. The sample YARN service
> configuration looks like this:
> {code}
> {
> "name": "docker-registry",
> "version": "1.0",
> "kerberos_principal" : {
> "principal_name" : "hbase/_h...@example.com",
> "keytab" : "file:///etc/security/keytabs/hbase.service.keytab"
> },
> "components" :
> [
> {
> "name": "registry",
> "number_of_containers": 1,
> "artifact": {
> "id": "registry:latest",
> "type": "DOCKER"
> },
> "resource": {
> "cpus": 1,
> "memory": "256"
> },
> "run_privileged_container": true,
> "configuration": {
> "env": {
> "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
>
> "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/path/to/config.yml:/etc/docker/registry/config.yml"
> },
> "properties": {
> "docker.network": "host"
> }
> }
> }
> ]
> }
> {code}
> Where /path/to/config.yml maps to a Docker S3 storage option config.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9324) TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11
[
https://issues.apache.org/jira/browse/YARN-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9324:
Attachment: YARN-9324-001.patch
> TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11
> -
>
> Key: YARN-9324
> URL: https://issues.apache.org/jira/browse/YARN-9324
> Project: Hadoop YARN
> Issue Type: Test
> Components: capacity scheduler
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9324-001.patch
>
>
> {{TestSchedulingRequestContainerAllocation(Async)}} fails with junit-4.11 in
> branch-3.2 with below exception while using array return type of
> {{@Parameters}} method of the {{Parameterized}} runner. This is supported
> from junit-4.12 and works in branch-3.3. branch-3.3 has junit-4.12 from
> HADOOP-14775
> https://github.com/junit-team/junit4/blob/master/doc/ReleaseNotes4.12.md#parameterized-tests
> {code}
> [ERROR]
> initializationError(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.Exception:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation.placementConstarintHandlers()
> must return an Iterable of arrays.
> at
> org.junit.runners.Parameterized.parametersMethodReturnedWrongType(Parameterized.java:343)
> at org.junit.runners.Parameterized.allParameters(Parameterized.java:296)
> at org.junit.runners.Parameterized.(Parameterized.java:282)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:29)
> at
> org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:21)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9324) TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11
[
https://issues.apache.org/jira/browse/YARN-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9324:
Attachment: (was: YARN-9324-branch-3.2.001.patch)
> TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11
> -
>
> Key: YARN-9324
> URL: https://issues.apache.org/jira/browse/YARN-9324
> Project: Hadoop YARN
> Issue Type: Test
> Components: capacity scheduler
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9324-001.patch
>
>
> {{TestSchedulingRequestContainerAllocation(Async)}} fails with junit-4.11 in
> branch-3.2 with below exception while using array return type of
> {{@Parameters}} method of the {{Parameterized}} runner. This is supported
> from junit-4.12 and works in branch-3.3. branch-3.3 has junit-4.12 from
> HADOOP-14775
> https://github.com/junit-team/junit4/blob/master/doc/ReleaseNotes4.12.md#parameterized-tests
> {code}
> [ERROR]
> initializationError(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation)
> Time elapsed: 0.008 s <<< ERROR!
> java.lang.Exception:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation.placementConstarintHandlers()
> must return an Iterable of arrays.
> at
> org.junit.runners.Parameterized.parametersMethodReturnedWrongType(Parameterized.java:343)
> at org.junit.runners.Parameterized.allParameters(Parameterized.java:296)
> at org.junit.runners.Parameterized.(Parameterized.java:282)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:29)
> at
> org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:21)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:26)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:362)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[ https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775349#comment-16775349 ] Szilard Nemeth commented on YARN-9213: -- I can also copy the class from YARN-9051 and add it to the 3.2 patch of this jira. I think it's also okay, just the 2 branches will diverge more, but if you are fine by that, I can do that. > RM Web UI v1 does not show custom resource allocations for containers page > -- > > Key: YARN-9213 > URL: https://issues.apache.org/jira/browse/YARN-9213 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen > Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch, > YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch, > YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch, > YARN-9213.branch3.2.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8783) Improve the documentation for the docker.trusted.registries configuration
[
https://issues.apache.org/jira/browse/YARN-8783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775351#comment-16775351
]
Eric Badger commented on YARN-8783:
---
Thanks for the update. +1 on patch 002
> Improve the documentation for the docker.trusted.registries configuration
> -
>
> Key: YARN-8783
> URL: https://issues.apache.org/jira/browse/YARN-8783
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Simon Prewo
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker, container-executor, docker
> Attachments: YARN-8783.001.patch, YARN-8783.002.patch
>
>
> I am deploying the default yarn distributed shell example:
> {code:java}
> yarn jar hadoop-yarn-applications-distributedshell.jar -shell_env
> YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos -shell_command "sleep 90" -jar
> hadoop-yarn-applications-distributedshell.jar -num_containers 1{code}
> Having a *single trusted registry configured like this works*:
> {code:java}
> docker.trusted.registries=centos{code}
> But having *a list of trusted registries configured fails* ("Shell error
> output: image: centos is not trusted."):
> {code:java}
> docker.trusted.registries=centos,ubuntu{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[ https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775353#comment-16775353 ] Adam Antal commented on YARN-9121: -- Thanks, +1 (non-binding). > Users of GpuDiscoverer.getInstance() are not possible to test as instance is > a static field > --- > > Key: YARN-9121 > URL: https://issues.apache.org/jira/browse/YARN-9121 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9121.001.patch > > > The clients of GpuDiscoverer are very hard to test as they call > GpuDiscoverer.getInstance() internally. > For example, writing tests for > org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo > is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not > interchangeable as GpuDiscoverer is not mockable since we cannot inject it in > tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9317:
Attachment: YARN-9317-branch-3.1.001.patch
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch,
> YARN-9317-branch-3.1.001.patch, YARN-9317-branch-3.2.001.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[ https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775324#comment-16775324 ] Sunil Govindan commented on YARN-9213: -- [~snemeth] YARN-9051 is a feature change. And i am not very sure abt the impact on 3.2 or 3.1 with this. Meantime could you please check any other alternative if we cant back port YARN-9051. [~leftnoteasy] [~tangzhankun], is YARN-9051 good to backport ? I need to ensure that this patch is fine for these 2 branches. Thanks. > RM Web UI v1 does not show custom resource allocations for containers page > -- > > Key: YARN-9213 > URL: https://issues.apache.org/jira/browse/YARN-9213 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen > Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch, > YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch, > YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch, > YARN-9213.branch3.2.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9317:
Attachment: YARN-9317-branch-3.2.001.patch
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch,
> YARN-9317-branch-3.2.001.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[
https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775312#comment-16775312
]
Szilard Nemeth edited comment on YARN-9213 at 2/22/19 4:03 PM:
---
Hi [~sunilg]!
It's indeed wrong.
The problem is that my fix on trunk uses
CustomResourceTypesConfigurationProvider that is not part of branch-3.2:
{code:java}
??-( szilardnemeth@snemeth-MBP[17:01:00] >1< @trunk )--(
~/development/apache/hadoop )--
└-$ git log origin/branch-3.2 --oneline | grep YARN-9051
??-( szilardnemeth@snemeth-MBP[17:01:05] >1< @trunk )--(
~/development/apache/hadoop )--
└-$ git log origin/trunk --oneline | grep YARN-9051
881230da21c YARN-9051. Integrate multiple
CustomResourceTypesConfigurationProvider implementations into one. (Contributed
by Szilard Nemeth)
{code}
Would you suggest to open a new jira for cherry-picking YARN-9051 to
branch-3.2? Maybe a more lightweight alternative would be to just copy this
class from trunk and include it within the patch for branch-3.2 here.
I would like to use the cleaner approach so I'll let you decide.
Thanks!
was (Author: snemeth):
Hi [~sunilg]!
It's indeed wrong.
The problem is that it my fix on trunk uses
CustomResourceTypesConfigurationProvider that is not part of branch-3.2:
{code:java}
??-( szilardnemeth@snemeth-MBP[17:01:00] >1< @trunk )--(
~/development/apache/hadoop )--
└-$ git log origin/branch-3.2 --oneline | grep YARN-9051
??-( szilardnemeth@snemeth-MBP[17:01:05] >1< @trunk )--(
~/development/apache/hadoop )--
└-$ git log origin/trunk --oneline | grep YARN-9051
881230da21c YARN-9051. Integrate multiple
CustomResourceTypesConfigurationProvider implementations into one. (Contributed
by Szilard Nemeth)
{code}
Would you suggest to open a new jira for cherry-picking YARN-9051 to
branch-3.2? Maybe a more lightweight alternative would be to just copy this
class from trunk and include it within the patch for branch-3.2 here.
I would like to use the cleaner approach so I'll let you decide.
Thanks!
> RM Web UI v1 does not show custom resource allocations for containers page
> --
>
> Key: YARN-9213
> URL: https://issues.apache.org/jira/browse/YARN-9213
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen
> Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch,
> YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch,
> YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch,
> YARN-9213.branch3.2.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[
https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775312#comment-16775312
]
Szilard Nemeth commented on YARN-9213:
--
Hi [~sunilg]!
It's indeed wrong.
The problem is that it my fix on trunk uses
CustomResourceTypesConfigurationProvider that is not part of branch-3.2:
{code:java}
??-( szilardnemeth@snemeth-MBP[17:01:00] >1< @trunk )--(
~/development/apache/hadoop )--
└-$ git log origin/branch-3.2 --oneline | grep YARN-9051
??-( szilardnemeth@snemeth-MBP[17:01:05] >1< @trunk )--(
~/development/apache/hadoop )--
└-$ git log origin/trunk --oneline | grep YARN-9051
881230da21c YARN-9051. Integrate multiple
CustomResourceTypesConfigurationProvider implementations into one. (Contributed
by Szilard Nemeth)
{code}
Would you suggest to open a new jira for cherry-picking YARN-9051 to
branch-3.2? Maybe a more lightweight alternative would be to just copy this
class from trunk and include it within the patch for branch-3.2 here.
I would like to use the cleaner approach so I'll let you decide.
Thanks!
> RM Web UI v1 does not show custom resource allocations for containers page
> --
>
> Key: YARN-9213
> URL: https://issues.apache.org/jira/browse/YARN-9213
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen
> Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch,
> YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch,
> YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch,
> YARN-9213.branch3.2.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9287) Consecutive String Builder Append Should Reuse
[ https://issues.apache.org/jira/browse/YARN-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775310#comment-16775310 ] Ayush Saxena commented on YARN-9287: Thanx [~giovanni.fumarola] for the review. Have uploaded the rebased version. Pls Review :) > Consecutive String Builder Append Should Reuse > -- > > Key: YARN-9287 > URL: https://issues.apache.org/jira/browse/YARN-9287 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: YARN-9287-01.patch, YARN-9287-02.patch, > YARN-9287-03.patch, YARN-9287-04.patch, YARN-9287-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775300#comment-16775300
]
Hadoop QA commented on YARN-9322:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 79 unchanged - 0 fixed = 82 total (was 79) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 23s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9322 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959761/YARN-9322.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5588b4f8c89e 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 632d5e8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/23491/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23491/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
[jira] [Commented] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775298#comment-16775298
]
Hudson commented on YARN-9317:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16030 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16030/])
YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled
(bibinchundatt: rev ed13cf84068ad50a78e66211132c1f08238fb65b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/AMLauncher.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/DefaultAMSProcessor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[ https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775288#comment-16775288 ] Sunil Govindan commented on YARN-9213: -- Hi [~snemeth] branch-3.2 patch seems not good. Could you please help to check again. Thanks > RM Web UI v1 does not show custom resource allocations for containers page > -- > > Key: YARN-9213 > URL: https://issues.apache.org/jira/browse/YARN-9213 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen > Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch, > YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch, > YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch, > YARN-9213.branch3.2.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[
https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775284#comment-16775284
]
Hadoop QA commented on YARN-9213:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
29s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
28s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
24s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m
8s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 27s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | YARN-9213 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959068/YARN-9213.branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux aab917a8bded 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / d75aa33 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/23494/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/23494/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| javac |
[jira] [Updated] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-9317:
---
Fix Version/s: 3.3.0
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775282#comment-16775282
]
Bibin A Chundatt commented on YARN-9317:
[~Prabhu Joseph]
Could you upload patch for 3.1 and 3.2 too
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775283#comment-16775283
]
Prabhu Joseph commented on YARN-9317:
-
[~bibinchundatt] Yes Sure, will upload shortly.
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9317) Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
[
https://issues.apache.org/jira/browse/YARN-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-9317:
---
Summary: Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
(was: DefaultAMSProcessor#allocate timelineServiceV2Enabled check is costly)
> Avoid repeated YarnConfiguration#timelineServiceV2Enabled check
> ---
>
> Key: YARN-9317
> URL: https://issues.apache.org/jira/browse/YARN-9317
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9317-001.patch, YARN-9317-002.patch
>
>
> {code}
> if (YarnConfiguration.timelineServiceV2Enabled(
> getRmContext().getYarnConfiguration()))
> {code}
> DefaultAMSProcessor#init check is required only once and assign to boolean
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
[
https://issues.apache.org/jira/browse/YARN-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775271#comment-16775271
]
Hadoop QA commented on YARN-9327:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
7s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9327 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959792/YARN-9327.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8395c4bdcded 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9c88695 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23493/testReport/ |
| Max. process+thread count | 312 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/23493/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Commented] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[ https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775256#comment-16775256 ] Szilard Nemeth commented on YARN-9121: -- Hi [~adam.antal]! Fair enough. Updated the description. Does this answer your question? > Users of GpuDiscoverer.getInstance() are not possible to test as instance is > a static field > --- > > Key: YARN-9121 > URL: https://issues.apache.org/jira/browse/YARN-9121 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9121.001.patch > > > The clients of GpuDiscoverer are very hard to test as they call > GpuDiscoverer.getInstance() internally. > For example, writing tests for > org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo > is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not > interchangeable as GpuDiscoverer is not mockable since we cannot inject it in > tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9121) Users of GpuDiscoverer.getInstance() are not possible to test as instance is a static field
[ https://issues.apache.org/jira/browse/YARN-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9121: - Description: The clients of GpuDiscoverer are very hard to test as they call GpuDiscoverer.getInstance() internally. For example, writing tests for org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not interchangeable as GpuDiscoverer is not mockable since we cannot inject it in tests. > Users of GpuDiscoverer.getInstance() are not possible to test as instance is > a static field > --- > > Key: YARN-9121 > URL: https://issues.apache.org/jira/browse/YARN-9121 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9121.001.patch > > > The clients of GpuDiscoverer are very hard to test as they call > GpuDiscoverer.getInstance() internally. > For example, writing tests for > org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuResourcePlugin#getNMResourceInfo > is quite hard as the GpuDeviceInformation returned by GpuDiscoverer is not > interchangeable as GpuDiscoverer is not mockable since we cannot inject it in > tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9118) Handle exceptions with parsing user defined GPU devices in GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775243#comment-16775243 ] Hudson commented on YARN-9118: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16029 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16029/]) YARN-9118. Handle exceptions with parsing user defined GPU devices in (sunilg: rev 95fbbfed75dd309b5d56032ece64996165572287) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDeviceSpecificationException.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/TestGpuDiscoverer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/GpuResourceHandlerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuNodeResourceUpdateHandler.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/resourceplugin/gpu/GpuDiscoverer.java > Handle exceptions with parsing user defined GPU devices in GpuDiscoverer > > > Key: YARN-9118 > URL: https://issues.apache.org/jira/browse/YARN-9118 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9118.001.patch, YARN-9118.002.patch, > YARN-9118.003.patch, YARN-9118.004.patch, YARN-9118.005.patch, > YARN-9118.006.patch, YARN-9118.007.patch, YARN-9118.008.patch, > YARN-9118.009.patch > > > getGpusUsableByYarn has the following issues: > - Duplicate GPU device definitions are not denied: This seems to be the > biggest issue as it could increase the number of devices on the node if the > device ID is defined 2 or more times. > - An empty-string is accepted, it works like the user would not want to use > auto-discovery and haven't defined any GPU devices: This will result in an > empty device list, but the empty-string check is never explicitly there in > the code, so this behavior just coincidental. > - Number validation does not happen on GPU device IDs (separated by commas) > Many testcases are added as the coverage was already very low. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9118) Handle exceptions with parsing user defined GPU devices in GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775235#comment-16775235 ] Szilard Nemeth commented on YARN-9118: -- Thanks [~sunilg] > Handle exceptions with parsing user defined GPU devices in GpuDiscoverer > > > Key: YARN-9118 > URL: https://issues.apache.org/jira/browse/YARN-9118 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: YARN-9118.001.patch, YARN-9118.002.patch, > YARN-9118.003.patch, YARN-9118.004.patch, YARN-9118.005.patch, > YARN-9118.006.patch, YARN-9118.007.patch, YARN-9118.008.patch, > YARN-9118.009.patch > > > getGpusUsableByYarn has the following issues: > - Duplicate GPU device definitions are not denied: This seems to be the > biggest issue as it could increase the number of devices on the node if the > device ID is defined 2 or more times. > - An empty-string is accepted, it works like the user would not want to use > auto-discovery and haven't defined any GPU devices: This will result in an > empty device list, but the empty-string check is never explicitly there in > the code, so this behavior just coincidental. > - Number validation does not happen on GPU device IDs (separated by commas) > Many testcases are added as the coverage was already very low. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9213) RM Web UI v1 does not show custom resource allocations for containers page
[ https://issues.apache.org/jira/browse/YARN-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775233#comment-16775233 ] Sunil Govindan commented on YARN-9213: -- lemme kick jenkins again. some issues. > RM Web UI v1 does not show custom resource allocations for containers page > -- > > Key: YARN-9213 > URL: https://issues.apache.org/jira/browse/YARN-9213 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-08 at 21.16.37-before.png, Screen > Shot 2019-02-09 at 9.55.16-after.png, YARN-9213.001.patch, > YARN-9213.002.patch, YARN-9213.003.patch, YARN-9213.branch-3.1.001.patch, > YARN-9213.branch-3.2.001.patch, YARN-9213.branch3.1.001.patch, > YARN-9213.branch3.2.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8891) Documentation of the pluggable device framework
[ https://issues.apache.org/jira/browse/YARN-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775209#comment-16775209 ] Hudson commented on YARN-8891: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16028 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16028/]) YARN-8891. Documentation of the pluggable device framework. Contributed (sunilg: rev 9636fe4114eed9035cdc80108a026c657cd196d9) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DevelopYourOwnDevicePlugin.md * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/PluggableDeviceFramework.md > Documentation of the pluggable device framework > --- > > Key: YARN-8891 > URL: https://issues.apache.org/jira/browse/YARN-8891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8891-trunk.001.patch, YARN-8891-trunk.002.patch, > YARN-8891-trunk.003.patch, YARN-8891-trunk.004.patch, > YARN-8891-trunk.005.patch, YARN-8891-trunk.006.patch, > YARN-8891-trunk.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9118) Handle exceptions with parsing user defined GPU devices in GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-9118: - Summary: Handle exceptions with parsing user defined GPU devices in GpuDiscoverer (was: Handle issues with parsing user defined GPU devices in GpuDiscoverer) > Handle exceptions with parsing user defined GPU devices in GpuDiscoverer > > > Key: YARN-9118 > URL: https://issues.apache.org/jira/browse/YARN-9118 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9118.001.patch, YARN-9118.002.patch, > YARN-9118.003.patch, YARN-9118.004.patch, YARN-9118.005.patch, > YARN-9118.006.patch, YARN-9118.007.patch, YARN-9118.008.patch, > YARN-9118.009.patch > > > getGpusUsableByYarn has the following issues: > - Duplicate GPU device definitions are not denied: This seems to be the > biggest issue as it could increase the number of devices on the node if the > device ID is defined 2 or more times. > - An empty-string is accepted, it works like the user would not want to use > auto-discovery and haven't defined any GPU devices: This will result in an > empty device list, but the empty-string check is never explicitly there in > the code, so this behavior just coincidental. > - Number validation does not happen on GPU device IDs (separated by commas) > Many testcases are added as the coverage was already very low. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9235) If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown
[
https://issues.apache.org/jira/browse/YARN-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775202#comment-16775202
]
Sunil Govindan commented on YARN-9235:
--
Thanks [~bsteinbach]. Make sense. Lets visit and review (YARN-9118, YARN-9213)
and come back here.
> If linux container executor is not set for a GPU cluster
> GpuResourceHandlerImpl is not initialized and NPE is thrown
>
>
> Key: YARN-9235
> URL: https://issues.apache.org/jira/browse/YARN-9235
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Major
> Attachments: YARN-9235.001.patch
>
>
> If GPU plugin is enabled for the NodeManager, it is possible to run jobs with
> GPU.
> However, if LinuxContainerExecutor is not configured, an NPE is thrown when
> calling
> {code:java}
> GpuResourcePlugin.getNMResourceInfo{code}
> Also, there are no warns in the log if GPU is misconfigured like this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9118) Handle issues with parsing user defined GPU devices in GpuDiscoverer
[ https://issues.apache.org/jira/browse/YARN-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775204#comment-16775204 ] Sunil Govindan commented on YARN-9118: -- I am happy with latest patch. Lets get this in later evening. +1 > Handle issues with parsing user defined GPU devices in GpuDiscoverer > > > Key: YARN-9118 > URL: https://issues.apache.org/jira/browse/YARN-9118 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-9118.001.patch, YARN-9118.002.patch, > YARN-9118.003.patch, YARN-9118.004.patch, YARN-9118.005.patch, > YARN-9118.006.patch, YARN-9118.007.patch, YARN-9118.008.patch, > YARN-9118.009.patch > > > getGpusUsableByYarn has the following issues: > - Duplicate GPU device definitions are not denied: This seems to be the > biggest issue as it could increase the number of devices on the node if the > device ID is defined 2 or more times. > - An empty-string is accepted, it works like the user would not want to use > auto-discovery and haven't defined any GPU devices: This will result in an > empty device list, but the empty-string check is never explicitly there in > the code, so this behavior just coincidental. > - Number validation does not happen on GPU device IDs (separated by commas) > Many testcases are added as the coverage was already very low. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8891) Documentation of the pluggable device framework
[ https://issues.apache.org/jira/browse/YARN-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8891: - Component/s: documentation > Documentation of the pluggable device framework > --- > > Key: YARN-8891 > URL: https://issues.apache.org/jira/browse/YARN-8891 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-8891-trunk.001.patch, YARN-8891-trunk.002.patch, > YARN-8891-trunk.003.patch, YARN-8891-trunk.004.patch, > YARN-8891-trunk.005.patch, YARN-8891-trunk.006.patch, > YARN-8891-trunk.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775182#comment-16775182
]
Hudson commented on YARN-9238:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16027 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16027/])
YARN-9238. Avoid allocating opportunistic containers to (wwei: rev
9c88695bcda0ffe4c7f49d643c649dfa1dce9bde)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestOpportunisticContainerAllocatorAMService.java
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
[
https://issues.apache.org/jira/browse/YARN-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-9327:
---
Attachment: YARN-9327.001.patch
> ProtoUtils#convertToProtoFormat block Application Master Service and many more
> --
>
> Key: YARN-9327
> URL: https://issues.apache.org/jira/browse/YARN-9327
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: YARN-9327.001.patch
>
>
> {code}
> public static synchronized ResourceProto convertToProtoFormat(Resource r) {
> return ResourcePBImpl.getProto(r);
> }
> {code}
> {noformat}
> "IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
> tid=0x7f181de72800 nid=0x222 waiting for monitor entry
> [0x7ef153dad000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
> - waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
> - locked <0x7f0fed96f500> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
> at
> com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
> at
> com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
> - eliminated <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684){noformat}
> Seems synchronization is not required here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775159#comment-16775159
]
Weiwei Yang commented on YARN-9238:
---
Pushed to trunk, cherry picked to branch-3.2 and branch-3.1. Thanks for the
contribution [~xiaoheipangzi].
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-9238:
--
Fix Version/s: 3.1.3
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775139#comment-16775139
]
lujie edited comment on YARN-9238 at 2/22/19 1:32 PM:
--
Hi：[~cheersyang]
One more thing. Could please review the patch that fix YARN-9248? That bug also
happens to opportunistic container.
was (Author: xiaoheipangzi):
Hi：[~cheersyang]
One more thing. Could please review the patch that fix YARN-9248? This bug also
happens to opportunistic container.
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775147#comment-16775147
]
Weiwei Yang commented on YARN-9238:
---
Sure [~xiaoheipangzi], I'll take a look. Thank you.
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775139#comment-16775139
]
lujie commented on YARN-9238:
-
Hi：[~cheersyang]
One more thing. Could please review the patch that fix YARN-9248? This bug also
happens to opportunistic container.
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9287) Consecutive String Builder Append Should Reuse
[ https://issues.apache.org/jira/browse/YARN-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated YARN-9287: --- Attachment: YARN-9287-05.patch > Consecutive String Builder Append Should Reuse > -- > > Key: YARN-9287 > URL: https://issues.apache.org/jira/browse/YARN-9287 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: YARN-9287-01.patch, YARN-9287-02.patch, > YARN-9287-03.patch, YARN-9287-04.patch, YARN-9287-05.patch > > > Consecutive calls to StringBuffer/StringBuilder .append should be chained, > reusing the target object. This can improve the performance by producing a > smaller bytecode, reducing overhead and improving inlining. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9328) ParentQueue#apply move log outside writelock
[
https://issues.apache.org/jira/browse/YARN-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph reassigned YARN-9328:
---
Assignee: Prabhu Joseph
> ParentQueue#apply move log outside writelock
>
>
> Key: YARN-9328
> URL: https://issues.apache.org/jira/browse/YARN-9328
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Prabhu Joseph
>Priority: Major
>
> {code}
> LOG.info("assignedContainer" + " queue=" + getQueueName()
> + " usedCapacity=" + getUsedCapacity() + "
> absoluteUsedCapacity="
> + getAbsoluteUsedCapacity() + " used=" + queueUsage.getUsed()
> + " cluster=" + cluster);
> {code}
> Logging can be done after log.. Logging could reduce performance ..
> {code}
> "Thread-16" #40 daemon prio=5 os_prio=0 tid=0x7f181f9bb800 nid=0x125
> waiting for monitor entry [0x7ef163bab000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.log4j.Category.callAppenders(Category.java:204)
> - locked <0x7ef2d803e2b8> (a org.apache.log4j.spi.RootLogger)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:176)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.apply(ParentQueue.java:1336)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.apply(LeafQueue.java:1371)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:665)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2946)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run(CapacityScheduler.java:644)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9328) ParentQueue#apply move log outside writelock
[
https://issues.apache.org/jira/browse/YARN-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-9328:
---
Description:
{code}
LOG.info("assignedContainer" + " queue=" + getQueueName()
+ " usedCapacity=" + getUsedCapacity() + " absoluteUsedCapacity="
+ getAbsoluteUsedCapacity() + " used=" + queueUsage.getUsed()
+ " cluster=" + cluster);
{code}
Logging can be done after log.. Logging could reduce performance ..
{code}
"Thread-16" #40 daemon prio=5 os_prio=0 tid=0x7f181f9bb800 nid=0x125
waiting for monitor entry [0x7ef163bab000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:204)
- locked <0x7ef2d803e2b8> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at
org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:176)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.apply(ParentQueue.java:1336)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.apply(LeafQueue.java:1371)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:665)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2946)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run(CapacityScheduler.java:644)
{code}
was:
{code}
LOG.info("assignedContainer" + " queue=" + getQueueName()
+ " usedCapacity=" + getUsedCapacity() + " absoluteUsedCapacity="
+ getAbsoluteUsedCapacity() + " used=" + queueUsage.getUsed()
+ " cluster=" + cluster);
{code}
Logging can be done after log.. Logging could reduce performance ..
> ParentQueue#apply move log outside writelock
>
>
> Key: YARN-9328
> URL: https://issues.apache.org/jira/browse/YARN-9328
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Major
>
> {code}
> LOG.info("assignedContainer" + " queue=" + getQueueName()
> + " usedCapacity=" + getUsedCapacity() + "
> absoluteUsedCapacity="
> + getAbsoluteUsedCapacity() + " used=" + queueUsage.getUsed()
> + " cluster=" + cluster);
> {code}
> Logging can be done after log.. Logging could reduce performance ..
> {code}
> "Thread-16" #40 daemon prio=5 os_prio=0 tid=0x7f181f9bb800 nid=0x125
> waiting for monitor entry [0x7ef163bab000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.log4j.Category.callAppenders(Category.java:204)
> - locked <0x7ef2d803e2b8> (a org.apache.log4j.spi.RootLogger)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:176)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.apply(ParentQueue.java:1336)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.apply(LeafQueue.java:1371)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:665)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2946)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$ResourceCommitterService.run(CapacityScheduler.java:644)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9328) ParentQueue#apply move log outside writelock
Bibin A Chundatt created YARN-9328:
--
Summary: ParentQueue#apply move log outside writelock
Key: YARN-9328
URL: https://issues.apache.org/jira/browse/YARN-9328
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
{code}
LOG.info("assignedContainer" + " queue=" + getQueueName()
+ " usedCapacity=" + getUsedCapacity() + " absoluteUsedCapacity="
+ getAbsoluteUsedCapacity() + " used=" + queueUsage.getUsed()
+ " cluster=" + cluster);
{code}
Logging can be done after log.. Logging could reduce performance ..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775086#comment-16775086 ] Szilard Nemeth commented on YARN-9322: -- Thanks [~adam.antal] and [~shuzirra] for your review comments! I can agree with your concerns regarding the method visibility changes so I created a new patch to exclude those. Please check the patch again! Thanks! > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
Bibin A Chundatt created YARN-9327:
--
Summary: ProtoUtils#convertToProtoFormat block Application Master
Service and many more
Key: YARN-9327
URL: https://issues.apache.org/jira/browse/YARN-9327
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
{code}
public static synchronized ResourceProto convertToProtoFormat(Resource r) {
return ResourcePBImpl.getProto(r);
}
{code}
{noformat}
"IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
tid=0x7f181de72800 nid=0x222 waiting for monitor entry [0x7ef153dad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
- waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
- locked <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
- locked <0x7f0fed96f500> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
at
com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
at
com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
at
org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
- locked <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
- eliminated <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
- locked <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684){noformat}
synchronization is not required here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
[
https://issues.apache.org/jira/browse/YARN-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775103#comment-16775103
]
Bibin A Chundatt commented on YARN-9327:
cc : [~leftnoteasy]
> ProtoUtils#convertToProtoFormat block Application Master Service and many more
> --
>
> Key: YARN-9327
> URL: https://issues.apache.org/jira/browse/YARN-9327
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
>
> {code}
> public static synchronized ResourceProto convertToProtoFormat(Resource r) {
> return ResourcePBImpl.getProto(r);
> }
> {code}
> {noformat}
> "IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
> tid=0x7f181de72800 nid=0x222 waiting for monitor entry
> [0x7ef153dad000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
> - waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
> - locked <0x7f0fed96f500> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
> at
> com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
> at
> com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
> - eliminated <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684){noformat}
> synchronization is not required here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
[
https://issues.apache.org/jira/browse/YARN-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775103#comment-16775103
]
Bibin A Chundatt edited comment on YARN-9327 at 2/22/19 12:56 PM:
--
cc : [~leftnoteasy],[~sunil.gov...@gmail.com]
was (Author: bibinchundatt):
cc : [~leftnoteasy]
> ProtoUtils#convertToProtoFormat block Application Master Service and many more
> --
>
> Key: YARN-9327
> URL: https://issues.apache.org/jira/browse/YARN-9327
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
>
> {code}
> public static synchronized ResourceProto convertToProtoFormat(Resource r) {
> return ResourcePBImpl.getProto(r);
> }
> {code}
> {noformat}
> "IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
> tid=0x7f181de72800 nid=0x222 waiting for monitor entry
> [0x7ef153dad000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
> - waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
> - locked <0x7f0fed96f500> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
> at
> com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
> at
> com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
> - eliminated <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684){noformat}
> Seems synchronization is not required here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For
[jira] [Updated] (YARN-9238) Avoid allocating opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-9238:
--
Summary: Avoid allocating opportunistic containers to
previous/removed/non-exist application attempt (was: Avoid to allocate
opportunistic containers to previous/removed/non-exist application attempt)
> Avoid allocating opportunistic containers to previous/removed/non-exist
> application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9238) Avoid to allocate opportunistic containers to previous/removed/non-exist application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-9238:
--
Summary: Avoid to allocate opportunistic containers to
previous/removed/non-exist application attempt (was: Allocate on previous or
removed or non existent application attempt)
> Avoid to allocate opportunistic containers to previous/removed/non-exist
> application attempt
>
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
[
https://issues.apache.org/jira/browse/YARN-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-9327:
---
Description:
{code}
public static synchronized ResourceProto convertToProtoFormat(Resource r) {
return ResourcePBImpl.getProto(r);
}
{code}
{noformat}
"IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
tid=0x7f181de72800 nid=0x222 waiting for monitor entry [0x7ef153dad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
- waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
- locked <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
- locked <0x7f0fed96f500> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
at
com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
at
com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
at
org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
- locked <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
- eliminated <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
- locked <0x7f0fed968a30> (a
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684){noformat}
Seems synchronization is not required here.
was:
{code}
public static synchronized ResourceProto convertToProtoFormat(Resource r) {
return ResourcePBImpl.getProto(r);
}
{code}
{noformat}
"IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
tid=0x7f181de72800 nid=0x222 waiting for monitor entry [0x7ef153dad000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
- waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
at
org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
at
[jira] [Assigned] (YARN-9327) ProtoUtils#convertToProtoFormat block Application Master Service and many more
[
https://issues.apache.org/jira/browse/YARN-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt reassigned YARN-9327:
--
Assignee: Bibin A Chundatt
> ProtoUtils#convertToProtoFormat block Application Master Service and many more
> --
>
> Key: YARN-9327
> URL: https://issues.apache.org/jira/browse/YARN-9327
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
>
> {code}
> public static synchronized ResourceProto convertToProtoFormat(Resource r) {
> return ResourcePBImpl.getProto(r);
> }
> {code}
> {noformat}
> "IPC Server handler 41 on 23764" #324 daemon prio=5 os_prio=0
> tid=0x7f181de72800 nid=0x222 waiting for monitor entry
> [0x7ef153dad000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils.convertToProtoFormat(ProtoUtils.java:404)
> - waiting to lock <0x7ef2d8bcf6d8> (a java.lang.Class for
> org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.convertToProtoFormat(NodeReportPBImpl.java:315)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToBuilder(NodeReportPBImpl.java:262)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.mergeLocalToProto(NodeReportPBImpl.java:289)
> at
> org.apache.hadoop.yarn.api.records.impl.pb.NodeReportPBImpl.getProto(NodeReportPBImpl.java:228)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.convertToProtoFormat(AllocateResponsePBImpl.java:844)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.access$500(AllocateResponsePBImpl.java:72)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:810)
> - locked <0x7f0fed96f500> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl$7$1.next(AllocateResponsePBImpl.java:799)
> at
> com.google.protobuf.AbstractMessageLite$Builder.checkForNullValues(AbstractMessageLite.java:336)
> at
> com.google.protobuf.AbstractMessageLite$Builder.addAll(AbstractMessageLite.java:323)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$AllocateResponseProto$Builder.addAllUpdatedNodes(YarnServiceProtos.java:13810)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToBuilder(AllocateResponsePBImpl.java:158)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.mergeLocalToProto(AllocateResponsePBImpl.java:198)
> - eliminated <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl.getProto(AllocateResponsePBImpl.java:103)
> - locked <0x7f0fed968a30> (a
> org.apache.hadoop.yarn.api.protocolrecords.impl.pb.AllocateResponsePBImpl)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:61)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:824)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2684){noformat}
> synchronization is not required here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9322: - Attachment: YARN-9322.002.patch > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch, YARN-9322.002.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1169) Refactor AuxServicesEvent into a AuxServicesAppEvent and AuxServicesContainerEvent
[ https://issues.apache.org/jira/browse/YARN-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775083#comment-16775083 ] Joakim Croona commented on YARN-1169: - Hi! We are a group of 5 students that are working on refactoring as part of an assignment. We are wondering if it is ok for us to attempt this issue? > Refactor AuxServicesEvent into a AuxServicesAppEvent and > AuxServicesContainerEvent > -- > > Key: YARN-1169 > URL: https://issues.apache.org/jira/browse/YARN-1169 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.1.1-beta >Reporter: Alejandro Abdelnur >Priority: Major > > As discussed in YARN-910, we should refactor AuxServicesEvent into a > AuxServicesAppEvent and AuxServicesContainerEvent to avoid having properties > with NULL values. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9316) TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently
[
https://issues.apache.org/jira/browse/YARN-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775066#comment-16775066
]
Hadoop QA commented on YARN-9316:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 28s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9316 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12959741/YARN-9316-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 96c0d1ac5020 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 632d5e8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/23490/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/23490/testReport/ |
| Max. process+thread count | 917 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-8891) Documentation of the pluggable device framework
[ https://issues.apache.org/jira/browse/YARN-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775067#comment-16775067 ] Zhankun Tang commented on YARN-8891: [~sunilg] , the Jenkins result is ok too. Could you help to merge it? Thanks > Documentation of the pluggable device framework > --- > > Key: YARN-8891 > URL: https://issues.apache.org/jira/browse/YARN-8891 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8891-trunk.001.patch, YARN-8891-trunk.002.patch, > YARN-8891-trunk.003.patch, YARN-8891-trunk.004.patch, > YARN-8891-trunk.005.patch, YARN-8891-trunk.006.patch, > YARN-8891-trunk.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9326) Fair Scheduler configuration defaults are not documented in case of min and maxResources
Adam Antal created YARN-9326:
Summary: Fair Scheduler configuration defaults are not documented
in case of min and maxResources
Key: YARN-9326
URL: https://issues.apache.org/jira/browse/YARN-9326
Project: Hadoop YARN
Issue Type: Improvement
Components: docs, documentation, fairscheduler, yarn
Affects Versions: 3.2.0
Reporter: Adam Antal
Assignee: Adam Antal
The FairScheduler's configuration has the following defaults (from the code:
javadoc):
{noformat}
In new style resources, any resource that is not specified will be set to
missing or 0%, as appropriate. Also, in the new style resources, units are not
allowed. Units are assumed from the resource manager's settings for the
resources when the value isn't a percentage. The missing parameter is only used
in the case of new style resources without percentages. With new style
resources with percentages, any missing resources will be assumed to be 100%
because percentages are only used with maximum resource limits.
{noformat}
This is not documented in the hadoop yarn site FairScheduler.html. It is quite
intuitive, but still need to be documented though.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9322) Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774727#comment-16774727 ] Gergely Pollak edited comment on YARN-9322 at 2/22/19 11:21 AM: I'm always concerned when making visibility changes, because that means we may make breaking change in an internal API (obviously we don't break anything within hadoop project, but we might in other projects using our classes). So as a rule of thumb I think we should only change visibility when there is a good reason to. Also I see no harm in public getter methods, hiding certain setters might be a good practice, but I would recommend to hide only the newly added setters. was (Author: shuzirra): I'm always concerned when doing visibility changes, because that means we may make breaking change in an internal API (obviously we don't break anything within hadoop project, but we might in other projects using our classes). So as a rule of thumb I think we should only change visibility when there is a good reason to. Also I see no harm in public getter methods, hiding certain setters might be a good practice, but I would recommend to hide only the newly added setters. > Store metrics for custom resource types into FSQueueMetrics and query them in > FairSchedulerQueueInfo > > > Key: YARN-9322 > URL: https://issues.apache.org/jira/browse/YARN-9322 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2019-02-21 at 12.06.46.png, > YARN-9322.001.patch > > > YARN-8842 implemented storing and exposing of metrics of custom resources. > FSQueueMetrics should have a similar implementation. > All metrics stored in this class should have their custom resource > counterpart. > In a consequence of metrics were not stored for custom resource type, > FairSchedulerQueueInfo haven't contained those values therefore the UI v1 > could not show them, obviously. > See that gpu is missing from the value of "AM Max Resources" on the attached > screenshot. > Additionally, the callees of the following methods (in class > FairSchedulerQueueInfo) should consider to query values for custom resource > types too: > getMaxAMShareMB > getMaxAMShareVCores > getAMResourceUsageMB > getAMResourceUsageVCores -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9316) TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently
[
https://issues.apache.org/jira/browse/YARN-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774946#comment-16774946
]
Prabhu Joseph commented on YARN-9316:
-
[~cheersyang] {{TestPlacementConstraintsUtil#testInterAppConstraintsByAppID}}
fails intermittently. The reason is there are two application with same id and
the timestamp also will
be same when the test case runs faster and so the application3 fails when
registering in PlacementConstraintManagerService with application already
registered.
{code}
ts = System.currentTimeMillis();
ApplicationId application2 = BuilderUtils.newApplicationId(ts, 124);
.
ts = System.currentTimeMillis();
ApplicationId application3 = BuilderUtils.newApplicationId(ts, 124);
{code}
{code}
2019-02-22 14:19:06,348 WARN [main]
constraint.MemoryPlacementConstraintManager
(MemoryPlacementConstraintManager.java:registerApplication(89)) - Application
application_1550825346301_0124 has already been registered.
{code}
Have fixed with unique applicationId. Can you review the same when you get
time. Thanks.
> TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails
> intermittently
>
>
> Key: YARN-9316
> URL: https://issues.apache.org/jira/browse/YARN-9316
> Project: Hadoop YARN
> Issue Type: Test
> Components: capacity scheduler
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9316-001.patch
>
>
> TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails
> intermittently - observed in YARN-8132
> {code}
> [ERROR]
> testInterAppConstraintsByAppID(org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementConstraintsUtil)
> Time elapsed: 0.339 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementConstraintsUtil.testInterAppConstraintsByAppID(TestPlacementConstraintsUtil.java:965)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Updated] (YARN-9316) TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently
[
https://issues.apache.org/jira/browse/YARN-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-9316:
Attachment: YARN-9316-001.patch
> TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails
> intermittently
>
>
> Key: YARN-9316
> URL: https://issues.apache.org/jira/browse/YARN-9316
> Project: Hadoop YARN
> Issue Type: Test
> Components: capacity scheduler
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9316-001.patch
>
>
> TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails
> intermittently - observed in YARN-8132
> {code}
> [ERROR]
> testInterAppConstraintsByAppID(org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementConstraintsUtil)
> Time elapsed: 0.339 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementConstraintsUtil.testInterAppConstraintsByAppID(TestPlacementConstraintsUtil.java:965)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9238) Allocate on previous or removed or non existent application attempt
[
https://issues.apache.org/jira/browse/YARN-9238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774931#comment-16774931
]
Weiwei Yang commented on YARN-9238:
---
Sorry this one got slipped away, I just found the patch is good. +1.
I will commit this shortly. Thanks.
> Allocate on previous or removed or non existent application attempt
> ---
>
> Key: YARN-9238
> URL: https://issues.apache.org/jira/browse/YARN-9238
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Critical
> Attachments: YARN-9238_1.patch, YARN-9238_2.patch, YARN-9238_3.patch,
> hadoop-test-resourcemanager-hadoop11.log
>
>
> See
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.OpportunisticAMSProcessor.allocate
> {code:java}
> // Allocate OPPORTUNISTIC containers.
> 171. SchedulerApplicationAttempt appAttempt =
> 172.((AbstractYarnScheduler)rmContext.getScheduler())
> 173. .getApplicationAttempt(appAttemptId);
> 174.
> 175. OpportunisticContainerContext oppCtx =
> 176. appAttempt.getOpportunisticContainerContext();
> 177. oppCtx.updateNodeList(getLeastLoadedNodes());
> {code}
> MRAppmaster crashes before before allocate#171, ResourceManager will start
> the new appAttempt and do
> {code:java}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplication.setCurrentAppAttempt(T
> currentAttempt){
> this.currentAttempt = currentAttempt;
> }{code}
> hence the allocate#171 will get the new appAttmept and its field
> OpportunisticContainerContext hasn't been initialized.
> so oopCtx ==null at and null pointer happens at line 177
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService$OpportunisticAMSProcessor.allocate(OpportunisticContainerAllocatorAMService.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:424)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:878)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2830) {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
