[jira] [Commented] (YARN-2402) NM restart: Container recovery for Windows
[ https://issues.apache.org/jira/browse/YARN-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126783#comment-16126783 ] Yuqi Wang commented on YARN-2402: - [~jlowe] When the NM is down, if the container process exit with exit code 666, with this patch, 666 will be recorded in the ExitCodeFile. And after NM recovered, NM will detect the container process exited and then read the ExitCodeFile to get the exit code. > NM restart: Container recovery for Windows > -- > > Key: YARN-2402 > URL: https://issues.apache.org/jira/browse/YARN-2402 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Yuqi Wang > Attachments: YARN-2402-v1.patch, YARN-2402-v2.patch > > > We should add container recovery for NM restart on Windows. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[
https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126776#comment-16126776
]
Hadoop QA commented on YARN-6885:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 32 new + 15 unchanged - 1 fixed = 47 total (was 16)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 18s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6885 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881861/YARN-6885.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0f705cef7c75 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d7be1d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16898/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16898/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16898/testReport/ |
| modules | C:
[jira] [Commented] (YARN-5978) ContainerScheduler and ContainerManager changes to support ExecType update
[ https://issues.apache.org/jira/browse/YARN-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126760#comment-16126760 ] Hudson commented on YARN-5978: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12186 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12186/]) YARN-5978. ContainerScheduler and ContainerManager changes to support (arun suresh: rev 4d7be1d8575e9254c59d41460960708e3718503a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerSchedulerEventType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerResync.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/UpdateContainerSchedulerEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/RMContainerTokenSecretManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestNMClient.java > ContainerScheduler and ContainerManager changes to support ExecType update > -- > > Key: YARN-5978 > URL: https://issues.apache.org/jira/browse/YARN-5978 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: kartheek muthyala > Attachments: YARN-5978.001.patch, YARN-5978.002.patch, > YARN-5978.003.patch > > > ContainerScheduler should support updateContainer API for > - Container Resource update > - ExecType update that can change an opportunistic to guaranteed and > vice-versa > Adding a new ContainerState event, UpdateContainerStateEvent to support > UPDATE_CONTAINER call from RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5978) ContainerScheduler and ContainerManager changes to support ExecType update
[ https://issues.apache.org/jira/browse/YARN-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5978: -- Summary: ContainerScheduler and ContainerManager changes to support ExecType update (was: ContainerScheduler and Container state machine changes to support ExecType update) > ContainerScheduler and ContainerManager changes to support ExecType update > -- > > Key: YARN-5978 > URL: https://issues.apache.org/jira/browse/YARN-5978 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: kartheek muthyala > Attachments: YARN-5978.001.patch, YARN-5978.002.patch, > YARN-5978.003.patch > > > ContainerScheduler should support updateContainer API for > - Container Resource update > - ExecType update that can change an opportunistic to guaranteed and > vice-versa > Adding a new ContainerState event, UpdateContainerStateEvent to support > UPDATE_CONTAINER call from RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126745#comment-16126745
]
Hadoop QA commented on YARN-6900:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
38s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
48s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6900 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881856/YARN-6900-010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux f8615b831e5f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55
[jira] [Updated] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[
https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu-Tang Lin updated YARN-6885:
--
Attachment: YARN-6885.007.patch
> AllocationFileLoaderService.loadQueue() should use a switch statement in the
> main tag parsing loop instead of the if/else-if/...
>
>
> Key: YARN-6885
> URL: https://issues.apache.org/jira/browse/YARN-6885
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yu-Tang Lin
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6885.005.patch, YARN-6885.006.patch,
> YARN-6885.007.patch
>
>
> {code} if ("minResources".equals(field.getTagName())) {
> String text = ((Text)field.getFirstChild()).getData().trim();
> Resource val =
> FairSchedulerConfiguration.parseResourceConfigValue(text);
> minQueueResources.put(queueName, val);
> } else if ("maxResources".equals(field.getTagName())) {
> ...{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6882) AllocationFileLoaderService.reloadAllocations() should use the diamond operator
[
https://issues.apache.org/jira/browse/YARN-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126740#comment-16126740
]
Larry Lo commented on YARN-6882:
Is there anything I can do to fix it ?
> AllocationFileLoaderService.reloadAllocations() should use the diamond
> operator
> ---
>
> Key: YARN-6882
> URL: https://issues.apache.org/jira/browse/YARN-6882
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Larry Lo
>Priority: Trivial
> Labels: newbie
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6882.001.patch
>
>
> Here:{code}for (FSQueueType queueType : FSQueueType.values()) {
> configuredQueues.put(queueType, new HashSet());
> }{code} and here:{code}List queueElements = new
> ArrayList();{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7007) NPE in RM while using YarnClientImpl.getApplications() ot get all applications
[
https://issues.apache.org/jira/browse/YARN-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lingfeng Su updated YARN-7007:
--
Description:
{code:java}
java.lang.NullPointerException: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:239)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2198)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2196)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254)
at sun.reflect.GeneratedMethodAccessor731.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy161.getApplications(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479)
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:456)
{code}
When I use YarnClient.getApplications() to get all applications of RM,
Occasionally, it throw a NPE problem.
{code:java}
RMAppAttempt currentAttempt = rmContext.getRMApps()
.get(attemptId.getApplicationId()).getCurrentAppAttempt();
{code}
if the application id is not in ConcurrentMap
getRMApps(), it may throw NPE problem.
Can [-YARN-6102-|https://issues.apache.org/jira/browse/YARN-6102] solve this
problem ?
was:
{code:java}
java.lang.NullPointerException: java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics.getAggregateAppResourceUsage(RMAppAttemptMetrics.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.getApplicationResourceUsageReport(RMAppAttemptImpl.java:857)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.createAndGetApplicationReport(RMAppImpl.java:629)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.verifyAndCreateAppReport(ClientRMService.java:972)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:898)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:734)
at
[jira] [Commented] (YARN-6361) FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big queues
[
https://issues.apache.org/jira/browse/YARN-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126727#comment-16126727
]
YunFan Zhou commented on YARN-6361:
---
[~yufeigu] [~sunilg] Could you please help me review my idea and code? Thank
you.
> FairScheduler: FSLeafQueue.fetchAppsWithDemand CPU usage is high with big
> queues
>
>
> Key: YARN-6361
> URL: https://issues.apache.org/jira/browse/YARN-6361
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: YunFan Zhou
> Attachments: dispatcherthread.png, threads.png, YARN-6361.001.patch
>
>
> FSLeafQueue.fetchAppsWithDemand sorts the applications by the current policy.
> Most of the time is spent in FairShareComparator.compare. We could improve
> this by doing the calculations outside the sort loop {{(O\(n\))}} and we
> sorted by a fixed number inside instead {{O(n*log\(n\))}}. This could be an
> performance issue when there are huge number of applications in a single
> queue. The attachments shows the performance impact when there are 10k
> applications in one queue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-6900:
--
Attachment: YARN-6900-010.patch
Fixing XML.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-010.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126703#comment-16126703
]
Hadoop QA commented on YARN-6900:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 2s{color} |
{color:red} The patch has 1 ill-formed XML file(s). {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 55s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 28s{color}
| {color:red} hadoop-yarn-server-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| | hadoop.yarn.client.TestClientRMProxy |
| | hadoop.yarn.conf.TestYarnConfiguration |
| | hadoop.yarn.security.TestYARNTokenIdentifier |
| | hadoop.yarn.event.TestAsyncDispatcher |
| | hadoop.yarn.logaggregation.TestAggregatedLogsBlock |
| | hadoop.yarn.client.api.impl.TestTimelineClient |
| |
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[
https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126692#comment-16126692
]
Allen Wittenauer commented on YARN-7011:
For completeness, with a fresh build:
{code}
$ sbin/yarn-daemon.sh --config /tmp/hadoop --debug start classpath 2>&1 | grep
/tmp/hadoop
DEBUG: HADOOP_CONF_DIR=/tmp/hadoop
DEBUG: Profiles: importing /tmp/hadoop/shellprofile.d/example.sh
/tmp/hadoop:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/common/lib/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/common/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/hdfs:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/hdfs/lib/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/hdfs/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/mapreduce/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/yarn:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/yarn/lib/*:/Users/aw/H/hadoop-3.0.0-beta1-SNAPSHOT/share/hadoop/yarn/*
{code}
> yarn-daemon.sh is not respecting --config option
>
>
> Key: YARN-7011
> URL: https://issues.apache.org/jira/browse/YARN-7011
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Priority: Blocker
> Fix For: 3.0.0-beta1
>
>
> Steps to reproduce:
> 1. Copy the conf to a temporary location /tmp/Conf
> 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM
> address
> 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf
> 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml
> is not taken in while restarting RM
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7011) yarn-daemon.sh is not respecting --config option
[
https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126689#comment-16126689
]
Allen Wittenauer edited comment on YARN-7011 at 8/15/17 1:02 AM:
-
What does yarn-daemon.sh --debug say? You can also run {{yarn-daemon.sh
--debug start classpath}}. The directory passed via --config should be the
first entry.
was (Author: aw):
What does yarn-daemon.sh --debug say?
> yarn-daemon.sh is not respecting --config option
>
>
> Key: YARN-7011
> URL: https://issues.apache.org/jira/browse/YARN-7011
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Sumana Sathish
>Priority: Blocker
> Fix For: 3.0.0-beta1
>
>
> Steps to reproduce:
> 1. Copy the conf to a temporary location /tmp/Conf
> 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM
> address
> 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf
> 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml
> is not taken in while restarting RM
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7011) yarn-daemon.sh is not respecting --config option
[ https://issues.apache.org/jira/browse/YARN-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126689#comment-16126689 ] Allen Wittenauer commented on YARN-7011: What does yarn-daemon.sh --debug say? > yarn-daemon.sh is not respecting --config option > > > Key: YARN-7011 > URL: https://issues.apache.org/jira/browse/YARN-7011 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Sumana Sathish >Priority: Blocker > Fix For: 3.0.0-beta1 > > > Steps to reproduce: > 1. Copy the conf to a temporary location /tmp/Conf > 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM > address > 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf > 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml > is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-6900:
--
Attachment: YARN-6900-009.patch
Adding missing key in {{yarn-default.xml}}.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-YARN-2915-000.patch, YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126661#comment-16126661
]
Hadoop QA commented on YARN-6900:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6900 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881841/YARN-6900-008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 08265883f661 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Commented] (YARN-6789) new api to get all supported resources from RM
[ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126632#comment-16126632 ] Wangda Tan commented on YARN-6789: -- bq. I this we need to look for NULL as per me. I could handle this condition in this ticket itself if everyone agrees. Thank You. Agree, +[~vvasudev]/[~templedf], could you share your thoughts here? > new api to get all supported resources from RM > -- > > Key: YARN-6789 > URL: https://issues.apache.org/jira/browse/YARN-6789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-6789-YARN-3926.001.patch > > > It will be better to provide an api to get all supported resource types from > RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6594) [API] Introduce SchedulingRequest object
[
https://issues.apache.org/jira/browse/YARN-6594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126623#comment-16126623
]
Konstantinos Karanasos commented on YARN-6594:
--
bq. I just checked the latest patch, beyond the stable api below, patch looks
good. For rest of the debates, I suggest to move it to future tasks and not
block the feature development, what do you guys think?
Hi [~leftnoteasy], I agree. I will fix the issues with the patch within the
next days, so that we can move forward.
> [API] Introduce SchedulingRequest object
>
>
> Key: YARN-6594
> URL: https://issues.apache.org/jira/browse/YARN-6594
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6594.001.patch
>
>
> This JIRA introduces a new SchedulingRequest object.
> It will be part of the {{AllocateRequest}} and will be used to define sizing
> (e.g., number of allocations, size of allocations) and placement constraints
> for allocations.
> Applications can use either this new object (when rich placement constraints
> are required) or the existing {{ResourceRequest}} object.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6851) Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat
[ https://issues.apache.org/jira/browse/YARN-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126619#comment-16126619 ] Wangda Tan commented on YARN-6851: -- +1, thanks [~ywskycn]! > Capacity Scheduler: document configs for controlling # containers allowed to > be allocated per node heartbeat > > > Key: YARN-6851 > URL: https://issues.apache.org/jira/browse/YARN-6851 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: YARN-6851.001.patch > > > YARN-4161 introduces new configs for controlling how many containers allowed > to be allocated in each node heartbeat. And we also have offswitchCount > config before. Would be better to document these configurations in CS section. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7011) yarn-daemon.sh is not respecting --config option
Sumana Sathish created YARN-7011: Summary: yarn-daemon.sh is not respecting --config option Key: YARN-7011 URL: https://issues.apache.org/jira/browse/YARN-7011 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Sumana Sathish Priority: Blocker Fix For: 3.0.0-beta1 Steps to reproduce: 1. Copy the conf to a temporary location /tmp/Conf 2. Modify anything in yarn-site.xml under /tmp/Conf/. Ex: Give invalid RM address 3. Restart the resourcemanager using yarn-daemon.sh using --config /tmp/Conf 4. --config is not respected as the changes made in /tmp/Conf/yarn-site.xml is not taken in while restarting RM -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-6900:
--
Attachment: YARN-6900-008.patch
Fixing unit tests.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5434) Add -client|server argument for graceful decom
[
https://issues.apache.org/jira/browse/YARN-5434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126567#comment-16126567
]
Robert Kanter commented on YARN-5434:
-
[~sunilg], could it be that 127.0.0.1:25003 wasn't doing anything when you ran
the command? If the node isn't busy, it will decommission right away and
appear to be non-graceful.
> Add -client|server argument for graceful decom
> --
>
> Key: YARN-5434
> URL: https://issues.apache.org/jira/browse/YARN-5434
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful
>Affects Versions: 2.8.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: YARN-5343.001.patch, YARN-5343.002.patch,
> YARN-5343.003.patch
>
>
> We should add {{-client|server}} argument to allow the user to specify if
> they want to use the client-side graceful decom tracking, or the server-side
> tracking (YARN-4676).
> Even though the server-side tracking won't go into 2.8, we should add the
> arguments to 2.8 for compatibility between 2.8 and 2.9, when it's added. In
> 2.8, using {{-server}} would just throw an Exception.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[
https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126554#comment-16126554
]
Yufei Gu commented on YARN-6995:
Thanks for the new patch. The message looks really good. Only one thing:
{{String.format(MESSAGE, resource, type)}} should be {{String.format(MESSAGE,
type, resource)}}.
> Improve use of ResourceNotFoundException in resource types code
> ---
>
> Key: YARN-6995
> URL: https://issues.apache.org/jira/browse/YARN-6995
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Minor
> Attachments: YARN-6995.YARN-3926.001.patch,
> YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch,
> YARN-6995.YARN-3926.004.patch
>
>
> Now that all the YarnExceptions have been replaced with
> ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as
> useful as possible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[
https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126507#comment-16126507
]
Hadoop QA commented on YARN-6995:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
26s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
12s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
32s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
50s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api |
| | Format string should use %n rather than n in new
org.apache.hadoop.yarn.exceptions.ResourceNotFoundException(Resource, String)
At ResourceNotFoundException.java:rather than n in new
org.apache.hadoop.yarn.exceptions.ResourceNotFoundException(Resource, String)
At ResourceNotFoundException.java:[line 45] |
| | Format string should use %n rather than n in new
org.apache.hadoop.yarn.exceptions.ResourceNotFoundException(Resource, String,
Throwable) At ResourceNotFoundException.java:rather than n in new
org.apache.hadoop.yarn.exceptions.ResourceNotFoundException(Resource, String,
Throwable) At ResourceNotFoundException.java:[line 50] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881820/YARN-6995.YARN-3926.004.patch
|
| Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126468#comment-16126468
]
Hadoop QA commented on YARN-6610:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
19s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 0 new + 4569 unchanged - 5 fixed = 4569 total (was 4574) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
34s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6610 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881821/YARN-6610.YARN-3926.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fb6886c189b3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / 8f80907 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16893/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16893/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components:
[jira] [Commented] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126465#comment-16126465 ] Vrushali C commented on YARN-6820: -- Ah, yes, so it's YARN-5355_branch2 I believe. cc [~varun_saxena] is that correct? There were some issues with the previous branches, let me confirm and get back. > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch, > YARN-6820-YARN-5355_branch_2.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126463#comment-16126463 ] Jason Lowe commented on YARN-6820: -- Oh sorry, I missed that branch. I noticed there's both a YARN-5355-branch-2 _and_ YARN-5355_branch2, but I don't see a YARN-5355_branch_2. Both branches have commits from August but are not the same, so it's not clear to me which one is "correct." Seems like one of them should be deleted to remove any confusion. > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch, > YARN-6820-YARN-5355_branch_2.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126433#comment-16126433
]
Daniel Templeton commented on YARN-6610:
[~yufeigu], yeah, you're right about unit tests. There is currently the
{{TestResourceCalculator}} class, but it only tests {{compare()}} when the
cluster resource is 0, and it does nothing to test more than 2 resources.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-6820: - Attachment: YARN-6820-YARN-5355_branch_2.patch Attaching patch for YARN_5355_branch_2 Hi [~jlowe] We usually also commit all patches onto this branch-2 branch that we have. If it's not too much trouble, could you also commit this patch to the YARN_5355_branch_2 branch? There are some minor changes in the test case to ensure the compilation works due to the javac version difference between YARN-5355 and YARN-5355_branch_2. > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch, > YARN-6820-YARN-5355_branch_2.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6610:
---
Attachment: YARN-6610.YARN-3926.005.patch
I added a handler for the exception.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch,
> YARN-6610.YARN-3926.005.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126408#comment-16126408 ] Daniel Templeton commented on YARN-6995: CC [~sunilg] > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch, > YARN-6995.YARN-3926.004.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6995: --- Attachment: YARN-6995.YARN-3926.004.patch I reworked the message a bit, and simplified the exception class a little. > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch, > YARN-6995.YARN-3926.004.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2402) NM restart: Container recovery for Windows
[ https://issues.apache.org/jira/browse/YARN-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126396#comment-16126396 ] Jason Lowe commented on YARN-2402: -- bq. Container recovery for Windows has been fully verified on Windows. Excellent news! Curious, how does the container recovery on Windows reconstruct the exit code for containers that completed while the NM was down? We should really dup this JIRA to whatever JIRA added that functionality, but I didn't see any code that handled this on Windows. The prototype patch attached to this JIRA was doing something along those lines, and I didn't see how Windows was properly recovering exit codes for completed containers without something like it. bq. there is also not unit test for getting exit code from the exitCodeFile for Unix or getting pid from the pidFile for Windows, seems it is trivial to test this simple script. Feel free to file a JIRA for adding those tests, and I can help review it. > NM restart: Container recovery for Windows > -- > > Key: YARN-2402 > URL: https://issues.apache.org/jira/browse/YARN-2402 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Yuqi Wang > Attachments: YARN-2402-v1.patch, YARN-2402-v2.patch > > > We should add container recovery for NM restart on Windows. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126394#comment-16126394 ] Jian He commented on YARN-6903: --- I'll fix the rest of the jenkins issues in a followup patch, as this patch is big enough. > Yarn-native-service framework core rewrite > -- > > Key: YARN-6903 > URL: https://issues.apache.org/jira/browse/YARN-6903 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6903.yarn-native-services.01.patch, > YARN-6903.yarn-native-services.02.patch, > YARN-6903.yarn-native-services.03.patch, > YARN-6903.yarn-native-services.04.patch, > YARN-6903.yarn-native-services.05.patch, > YARN-6903.yarn-native-services.06.patch, > YARN-6903.yarn-native-services.07.patch, > YARN-6903.yarn-native-services.08.patch > > > There are some new features like rich placement scheduling, container auto > restart, container upgrade in YARN core that can be taken advantage by the > native-service framework. Besides, there are quite a lot legacy code which > are no longer required. > So we decide to rewrite the core part to have a leaner codebase and make use > of various advanced features in YARN. > And the new code design will be in align with what we have designed for the > service API YARN-4793 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7010) Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps)
Giovanni Matteo Fumarola created YARN-7010: -- Summary: Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps) Key: YARN-7010 URL: https://issues.apache.org/jira/browse/YARN-7010 Project: Hadoop YARN Issue Type: Sub-task Reporter: Giovanni Matteo Fumarola Assignee: Giovanni Matteo Fumarola -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126391#comment-16126391 ] Billie Rinaldi commented on YARN-6903: -- I have done some basic testing of this patch and I think it's in pretty good shape. The precommit build issues seem to be unrelated or in code that is going to be removed. I think we should go ahead and commit this and perform additional cleanup in follow on tickets. > Yarn-native-service framework core rewrite > -- > > Key: YARN-6903 > URL: https://issues.apache.org/jira/browse/YARN-6903 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6903.yarn-native-services.01.patch, > YARN-6903.yarn-native-services.02.patch, > YARN-6903.yarn-native-services.03.patch, > YARN-6903.yarn-native-services.04.patch, > YARN-6903.yarn-native-services.05.patch, > YARN-6903.yarn-native-services.06.patch, > YARN-6903.yarn-native-services.07.patch, > YARN-6903.yarn-native-services.08.patch > > > There are some new features like rich placement scheduling, container auto > restart, container upgrade in YARN core that can be taken advantage by the > native-service framework. Besides, there are quite a lot legacy code which > are no longer required. > So we decide to rewrite the core part to have a leaner codebase and make use > of various advanced features in YARN. > And the new code design will be in align with what we have designed for the > service API YARN-4793 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6917) Queue path is recomputed from scratch on every allocation
[ https://issues.apache.org/jira/browse/YARN-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126382#comment-16126382 ] Hudson commented on YARN-6917: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12183 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12183/]) YARN-6917. Queue path is recomputed from scratch on every allocation. (jlowe: rev 5558792894169425bff054364a1ab4c48b347fb9) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java > Queue path is recomputed from scratch on every allocation > - > > Key: YARN-6917 > URL: https://issues.apache.org/jira/browse/YARN-6917 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.8.1 >Reporter: Jason Lowe >Assignee: Eric Payne >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1, 2.8.2 > > Attachments: YARN-6917.001.patch, YARN-6917.002.patch > > > As part of the discussion in YARN-6901 I noticed that we are recomputing a > queue's path for every allocation. Currently getting the queue's path > involves calling getQueuePath on the parent then building onto that string > with the basename of the queue. In turn the parent's getQueuePath method > does the same, so we end up spending time recomputing a string that will > never change until a reconfiguration. > Ideally the queue path should be computed once during queue initialization > rather than on-demand. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6923) Metrics for Federation Router
[ https://issues.apache.org/jira/browse/YARN-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126388#comment-16126388 ] Giovanni Matteo Fumarola commented on YARN-6923: Hey [~asuresh] can you take a look of this? That checkstyle warning is active for other Metrics classes as well. > Metrics for Federation Router > - > > Key: YARN-6923 > URL: https://issues.apache.org/jira/browse/YARN-6923 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6923.v1.patch, YARN-6923.v2.patch > > > This JIRA proposes addition of metrics for Federation StateStore -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7006) [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol
[
https://issues.apache.org/jira/browse/YARN-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126358#comment-16126358
]
Jian He commented on YARN-7006:
---
looks good to me, one question, why is this change required?
{code}
-event.getUser(), event.getContainer().getContainerId(),
+event.getContainer().getUser(),
{code}
> [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol
>
>
> Key: YARN-7006
> URL: https://issues.apache.org/jira/browse/YARN-7006
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-7006-YARN-5355.01.patch,
> YARN-7006-YARN-5355.02.patch
>
>
> Communication between Collector and NM is via RPC.
> We would do kerberos based authentication for communication between these 2
> components, as of now. Added SecurityInfo implementation for it.
> We can think of adding token based access once collector starts outside of NM.
> Also creation of timeline client within NMTimelinePublisher would be done
> using NM login UGI.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6917) Queue path is recomputed from scratch on every allocation
[ https://issues.apache.org/jira/browse/YARN-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126354#comment-16126354 ] Jason Lowe commented on YARN-6917: -- The unit test failures are unrelated. +1 lgtm. Committing this. > Queue path is recomputed from scratch on every allocation > - > > Key: YARN-6917 > URL: https://issues.apache.org/jira/browse/YARN-6917 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 2.8.1 >Reporter: Jason Lowe >Assignee: Eric Payne >Priority: Minor > Attachments: YARN-6917.001.patch, YARN-6917.002.patch > > > As part of the discussion in YARN-6901 I noticed that we are recomputing a > queue's path for every allocation. Currently getting the queue's path > involves calling getQueuePath on the parent then building onto that string > with the basename of the queue. In turn the parent's getQueuePath method > does the same, so we end up spending time recomputing a string that will > never change until a reconfiguration. > Ideally the queue path should be computed once during queue initialization > rather than on-demand. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6987) Log app attempt during InvalidStateTransition
[
https://issues.apache.org/jira/browse/YARN-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126337#comment-16126337
]
Hudson commented on YARN-6987:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12182 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12182/])
YARN-6987. Log app attempt during InvalidStateTransition. Contributed by
(jlowe: rev 3325ef653d6f364a82dd32485d9ef6d987380ce3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
> Log app attempt during InvalidStateTransition
> -
>
> Key: YARN-6987
> URL: https://issues.apache.org/jira/browse/YARN-6987
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: YARN-6987.001.patch, YARN-6987.002.patch
>
>
> Found this InvalidStateTransition logged in the resource manager log file
> with no way to determine exactly which app attempt this was associated with.
> {noformat}
> 2017-08-04 17:22:29,895 [AsyncDispatcher event handler] ERROR
> attempt.RMAppAttemptImpl: Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> UNREGISTERED at LAUNCHED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:802)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:803)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:784)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6923) Metrics for Federation Router
[
https://issues.apache.org/jira/browse/YARN-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126328#comment-16126328
]
Hadoop QA commented on YARN-6923:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 9s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-yarn-server-router in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6923 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881811/YARN-6923.v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6e5024834e2e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 3325ef6 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16892/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16892/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16892/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Metrics for Federation Router
> -
>
> Key: YARN-6923
> URL: https://issues.apache.org/jira/browse/YARN-6923
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient by caching resource usage
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126324#comment-16126324 ] Yufei Gu commented on YARN-4090: Yes, at least I hope so, but the patch is not ready. > Make Collections.sort() more efficient by caching resource usage > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Xianyin Xin >Assignee: zhangshilong > Attachments: sampling1.jpg, sampling2.jpg, YARN-4090.001.patch, > YARN-4090.002.patch, YARN-4090.003.patch, YARN-4090.004.patch, > YARN-4090.005.patch, YARN-4090.006.patch, YARN-4090-preview.patch, > YARN-4090-TestResult.pdf > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4090) Make Collections.sort() more efficient by caching resource usage
[ https://issues.apache.org/jira/browse/YARN-4090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126309#comment-16126309 ] Ashwin Shankar commented on YARN-4090: -- [~zsl2007] [~yufeigu] is this patch going to be committed? looks like a valuable optimization > Make Collections.sort() more efficient by caching resource usage > > > Key: YARN-4090 > URL: https://issues.apache.org/jira/browse/YARN-4090 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Xianyin Xin >Assignee: zhangshilong > Attachments: sampling1.jpg, sampling2.jpg, YARN-4090.001.patch, > YARN-4090.002.patch, YARN-4090.003.patch, YARN-4090.004.patch, > YARN-4090.005.patch, YARN-4090.006.patch, YARN-4090-preview.patch, > YARN-4090-TestResult.pdf > > > Collections.sort() consumes too much time in a scheduling round. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[
https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126308#comment-16126308
]
Hadoop QA commented on YARN-6995:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-3926 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
16s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
17s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
24s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} YARN-3926 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881804/YARN-6995.YARN-3926.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6721ecbc4662 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / 8f80907 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16891/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16891/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve use of
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126306#comment-16126306 ] Yufei Gu commented on YARN-6995: Looks good to me generally. Pending on Jenkins. One typo: "avaliability" > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6881) LOG is unused in AllocationConfiguration
[ https://issues.apache.org/jira/browse/YARN-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126302#comment-16126302 ] Hudson commented on YARN-6881: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12181 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12181/]) YARN-6881. LOG is unused in AllocationConfiguration (Contributed by (templedf: rev 6b09c327057947049ef7984afbb5ed225f15fc2d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java > LOG is unused in AllocationConfiguration > > > Key: YARN-6881 > URL: https://issues.apache.org/jira/browse/YARN-6881 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: weiyuan > Labels: newbie > Fix For: 3.0.0-beta1 > > Attachments: YARN-6881.001.patch, YARN-6881.002.patch > > > The variable can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[
https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126298#comment-16126298
]
Daniel Templeton commented on YARN-6885:
I think it's also OK to drop the blocks when the only contents are an _if_.
> AllocationFileLoaderService.loadQueue() should use a switch statement in the
> main tag parsing loop instead of the if/else-if/...
>
>
> Key: YARN-6885
> URL: https://issues.apache.org/jira/browse/YARN-6885
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yu-Tang Lin
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6885.005.patch, YARN-6885.006.patch
>
>
> {code} if ("minResources".equals(field.getTagName())) {
> String text = ((Text)field.getFirstChild()).getData().trim();
> Resource val =
> FairSchedulerConfiguration.parseResourceConfigValue(text);
> minQueueResources.put(queueName, val);
> } else if ("maxResources".equals(field.getTagName())) {
> ...{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[
https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126297#comment-16126297
]
Daniel Templeton commented on YARN-6885:
Can we pull the breaks into the blocks so that the indentation isn't quite so
wacky?
> AllocationFileLoaderService.loadQueue() should use a switch statement in the
> main tag parsing loop instead of the if/else-if/...
>
>
> Key: YARN-6885
> URL: https://issues.apache.org/jira/browse/YARN-6885
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: Yu-Tang Lin
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0-alpha4
>
> Attachments: YARN-6885.005.patch, YARN-6885.006.patch
>
>
> {code} if ("minResources".equals(field.getTagName())) {
> String text = ((Text)field.getFirstChild()).getData().trim();
> Resource val =
> FairSchedulerConfiguration.parseResourceConfigValue(text);
> minQueueResources.put(queueName, val);
> } else if ("maxResources".equals(field.getTagName())) {
> ...{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6923) Metrics for Federation Router
[ https://issues.apache.org/jira/browse/YARN-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-6923: --- Attachment: YARN-6923.v2.patch > Metrics for Federation Router > - > > Key: YARN-6923 > URL: https://issues.apache.org/jira/browse/YARN-6923 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6923.v1.patch, YARN-6923.v2.patch > > > This JIRA proposes addition of metrics for Federation StateStore -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6890) If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.
[ https://issues.apache.org/jira/browse/YARN-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-6890: -- Fix Version/s: 2.8.2 3.0.0-beta1 > If UI is not secured, we allow user to kill other users' job even yarn > cluster is secured. > -- > > Key: YARN-6890 > URL: https://issues.apache.org/jira/browse/YARN-6890 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Junping Du >Priority: Critical > Fix For: 3.0.0-beta1, 2.8.2 > > Attachments: YARN-6890.patch, YARN-6890-v2.patch, YARN-6890-v3.patch > > > Configuring SPNEGO for web browser could be a head ache, so many production > cluster choose to configure a unsecured UI access even for a secured cluster. > In this setup, users (login as some random guy) could watch other users job > which is expected. However, the kill button (added in YARN-3249 which enabled > by default) shouldn't work in this situation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6987) Log app attempt during InvalidStateTransition
[
https://issues.apache.org/jira/browse/YARN-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126283#comment-16126283
]
Jason Lowe commented on YARN-6987:
--
+1 lgtm. Committing this.
> Log app attempt during InvalidStateTransition
> -
>
> Key: YARN-6987
> URL: https://issues.apache.org/jira/browse/YARN-6987
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: YARN-6987.001.patch, YARN-6987.002.patch
>
>
> Found this InvalidStateTransition logged in the resource manager log file
> with no way to determine exactly which app attempt this was associated with.
> {noformat}
> 2017-08-04 17:22:29,895 [AsyncDispatcher event handler] ERROR
> attempt.RMAppAttemptImpl: Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> UNREGISTERED at LAUNCHED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:802)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:803)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:784)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[
https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126268#comment-16126268
]
Hudson commented on YARN-6905:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12180 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12180/])
YARN-6905 Multiple HBaseTimelineStorage test failures due to missing (vrushali:
rev 608a06cca5d68b3155bd70a94bf29ae0942b9ca0)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageApps.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestHBaseTimelineStorageEntities.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/TestCustomApplicationIdConversion.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/AppIdKeyConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/common/HBaseTimelineStorageUtils.java
> Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
> ---
>
> Key: YARN-6905
> URL: https://issues.apache.org/jira/browse/YARN-6905
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 3.0.0-beta1
> Environment: Ubuntu 14.04
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Haibo Chen
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6905.00.patch, YARN-6905.01.patch
>
>
> There are multiple test failing in Hadoop YARN Timeline Service HBase tests
> project with the following error :
> {code}
> java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat
> at
> org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104)
> {code}
> Below are the failing tests :
> {code}
> TestHBaseTimelineStorageApps.testWriteApplicationToHBase
> TestHBaseTimelineStorageApps.testEvents
> TestHBaseTimelineStorageEntities.testEventsEscapeTs
> TestHBaseTimelineStorageEntities.testWriteEntityToHBase
> TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6871) Add additional deSelects params in getAppReport
[
https://issues.apache.org/jira/browse/YARN-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126261#comment-16126261
]
Giovanni Matteo Fumarola commented on YARN-6871:
+1 LGTM. [~sunilg] Can you review it?
> Add additional deSelects params in getAppReport
> ---
>
> Key: YARN-6871
> URL: https://issues.apache.org/jira/browse/YARN-6871
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager, router
>Reporter: Giovanni Matteo Fumarola
>Assignee: Tanuj Nayak
> Attachments: YARN-6871.002.patch, YARN-6871.003.patch,
> YARN-6871.004.patch, YARN-6871.proto.patch
>
>
> This jira tracks the effort to add additional deSelect params to the
> GetAppReport to make it lighter and faster.
> With the current one we are facing a scalability issues.
> E.g. with ~500 applications running the AppReport can reach up to 300MB in
> size due to the {{ResourceRequest}} in the {{AppInfo}}.
> Yarn RM will return the new result faster and it will use less compute cycles
> to create the report and it will improve the YARN RM and Client's
> performances.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126237#comment-16126237
]
Subru Krishnan commented on YARN-6900:
--
[~elgoiri], to answer your question, I feel we should update the default store
to ZK. The test cases are fairly straightforward to fix:
* All the policy based ones use {{FederationPoliciesTestUtil::initFacade}}, so
simply set the store impl to in-memory there.
* We should add a before method in {{TestFederationStateStoreFacadeRetry}} and
move the config creation, max retry and in-memory store setting there.
* Similarly we can set the in-memory store in the
{{TestFederationStateStoreFacade}} constructor.
Can you take a look at why {{TestZookeeperFederationStateStore}} is failing.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6917) Queue path is recomputed from scratch on every allocation
[
https://issues.apache.org/jira/browse/YARN-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126230#comment-16126230
]
Hadoop QA commented on YARN-6917:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 47m 14s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6917 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881798/YARN-6917.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 78bc6ecdcd51 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ce797a1 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/16890/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16890/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16890/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (YARN-6996) Change javax.cache library implementation from JSR107 to Apache Geronimo
[
https://issues.apache.org/jira/browse/YARN-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126231#comment-16126231
]
Hudson commented on YARN-6996:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12179 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12179/])
YARN-6996. Change javax.cache library implementation from JSR107 to (subru: rev
18f3603bce37e0e07c9075811b1179afc2c227eb)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
* (edit) hadoop-project/pom.xml
> Change javax.cache library implementation from JSR107 to Apache Geronimo
>
>
> Key: YARN-6996
> URL: https://issues.apache.org/jira/browse/YARN-6996
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Blocker
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6996.001.patch
>
>
> With YARN Federation, we added YARN-3672, which adds the following to
> {noformat}
> javax.cache
> cache-api
> {noformat}
> This third-party library has some murky license history, as documented in
> this [really long comment
> thread|https://github.com/jsr107/jsr107spec/issues/333]. The summary of the
> thread is that "the library is officially APL (take our word for it), but
> there hasn't been a subsequent release with the license file change".
> LEGAL-325 has been filed to discuss the validity of this license for Apache.
> Before we get to final Hadoop 3 release, I'm wondering if anyone else has
> concerns about using this library. Just from looking at the various javax
> Maven artifacts in our pom.xml files, I see a lot of other javax.* library
> entries (although we may not ship the .jars if they're part of the Java
> runtime).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126228#comment-16126228
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
45s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 59s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 25s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| |
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.DockerCommand.toString()
concatenates strings using + in a loop At DockerCommand.java:in a loop At
DockerCommand.java:[line 87] |
| | Unread field:field be static? At
[jira] [Updated] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[
https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vrushali C updated YARN-6905:
-
Fix Version/s: 3.0.0-beta1
> Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
> ---
>
> Key: YARN-6905
> URL: https://issues.apache.org/jira/browse/YARN-6905
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 3.0.0-beta1
> Environment: Ubuntu 14.04
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Haibo Chen
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6905.00.patch, YARN-6905.01.patch
>
>
> There are multiple test failing in Hadoop YARN Timeline Service HBase tests
> project with the following error :
> {code}
> java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat
> at
> org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104)
> {code}
> Below are the failing tests :
> {code}
> TestHBaseTimelineStorageApps.testWriteApplicationToHBase
> TestHBaseTimelineStorageApps.testEvents
> TestHBaseTimelineStorageEntities.testEventsEscapeTs
> TestHBaseTimelineStorageEntities.testWriteEntityToHBase
> TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6995: --- Attachment: YARN-6995.YARN-3926.003.patch Rebased > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch, YARN-6995.YARN-3926.003.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[
https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126214#comment-16126214
]
Vrushali C commented on YARN-6905:
--
Thanks [~haibo.chen] , committed to trunk as part of
https://github.com/apache/hadoop/commit/608a06cca5d68b3155bd70a94bf29ae0942b9ca0
> Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
> ---
>
> Key: YARN-6905
> URL: https://issues.apache.org/jira/browse/YARN-6905
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 3.0.0-beta1
> Environment: Ubuntu 14.04
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Haibo Chen
> Attachments: YARN-6905.00.patch, YARN-6905.01.patch
>
>
> There are multiple test failing in Hadoop YARN Timeline Service HBase tests
> project with the following error :
> {code}
> java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat
> at
> org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104)
> {code}
> Below are the failing tests :
> {code}
> TestHBaseTimelineStorageApps.testWriteApplicationToHBase
> TestHBaseTimelineStorageApps.testEvents
> TestHBaseTimelineStorageEntities.testEventsEscapeTs
> TestHBaseTimelineStorageEntities.testWriteEntityToHBase
> TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126194#comment-16126194 ] Robert Kanter commented on YARN-5536: - To put a few more details on the work required here from the comments in YARN-5464 (the JIRA doesn't say much), as I understand it: # Remove the XML format as-is # Make the input format pluggable, similar to HDFS-9005 # Add a JSON format that is compatible with HDFS-9005 so users can use the same file for both Is that correct [~mingma]? This is currently a blocker for Hadoop 3 beta 1. [~mingma] do you think you'll be able to take care of this in time? The [Hadoop 3 release wiki page|https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+Release] says that we only have about a month left. If not, we have a couple faster options we can do in the meantime to make this not a blocker: # If we plan to keep the XML format when doing this, it won't be backwards incompatible anymore. We can still rewrite the code to be in the pluggable implementation, but we'd have to keep the same XML format itself. # We could remove the XML format now (so we'd only have the original format), and do the rest of this work later. Users won't be able to specify individual different timeouts per host in the meantime though. I think if we do one of those, then we can take care of this issue later if we don't have time to take care of it now for beta 1. > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Priority: Blocker > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6987) Log app attempt during InvalidStateTransition
[
https://issues.apache.org/jira/browse/YARN-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126189#comment-16126189
]
Jonathan Eagles commented on YARN-6987:
---
Test failure and timeout are unrelated to this logging jira. [~jlowe], can you
have a look at patch 002?
> Log app attempt during InvalidStateTransition
> -
>
> Key: YARN-6987
> URL: https://issues.apache.org/jira/browse/YARN-6987
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: YARN-6987.001.patch, YARN-6987.002.patch
>
>
> Found this InvalidStateTransition logged in the resource manager log file
> with no way to determine exactly which app attempt this was associated with.
> {noformat}
> 2017-08-04 17:22:29,895 [AsyncDispatcher event handler] ERROR
> attempt.RMAppAttemptImpl: Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> UNREGISTERED at LAUNCHED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:802)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:803)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:784)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126183#comment-16126183
]
Hudson commented on YARN-6959:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12178 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12178/])
YARN-6959. RM may allocate wrong AM Container for new attempt. (jianhe: rev
e2f6299f6f580d7a03f2377d19ac85f55fd4e73b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959.004.patch, YARN-6959.005.patch,
> YARN-6959-branch-2.7.001.patch, YARN-6959.yarn_nm.log.zip,
> YARN-6959.yarn_rm.log.zip
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7009) TestNMClient.testNMClientNoCleanupOnStop is flaky by design
Miklos Szegedi created YARN-7009:
Summary: TestNMClient.testNMClientNoCleanupOnStop is flaky by
design
Key: YARN-7009
URL: https://issues.apache.org/jira/browse/YARN-7009
Project: Hadoop YARN
Issue Type: Bug
Reporter: Miklos Szegedi
Assignee: Miklos Szegedi
The sleeps to wait for a transition to reinit and than back to running is not
long enough, it can miss the reinit event.
{code}
java.lang.AssertionError: Exception is not expected:
org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform RE_INIT on
[container_1502735389852_0001_01_01]. Current state is [REINITIALIZING,
isReInitializing=true].
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.preReInitializeOrLocalizeCheck(ContainerManagerImpl.java:1772)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1697)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1668)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.reInitializeContainer(ContainerManagementProtocolPBServiceImpl.java:214)
at
org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:237)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at
org.apache.hadoop.yarn.client.api.impl.TestNMClient.testReInitializeContainer(TestNMClient.java:567)
at
org.apache.hadoop.yarn.client.api.impl.TestNMClient.testContainerManagement(TestNMClient.java:405)
at
org.apache.hadoop.yarn.client.api.impl.TestNMClient.testNMClientNoCleanupOnStop(TestNMClient.java:214)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Cannot perform
RE_INIT on [container_1502735389852_0001_01_01]. Current state is
[REINITIALIZING, isReInitializing=true].
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.preReInitializeOrLocalizeCheck(ContainerManagerImpl.java:1772)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1697)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.reInitializeContainer(ContainerManagerImpl.java:1668)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.reInitializeContainer(ContainerManagementProtocolPBServiceImpl.java:214)
at
org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:237)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
[jira] [Commented] (YARN-5978) ContainerScheduler and Container state machine changes to support ExecType update
[ https://issues.apache.org/jira/browse/YARN-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126178#comment-16126178 ] Arun Suresh commented on YARN-5978: --- Thanks for the updated patch [~kartheek] +1, will commit this EOD if there are no other objections. > ContainerScheduler and Container state machine changes to support ExecType > update > - > > Key: YARN-5978 > URL: https://issues.apache.org/jira/browse/YARN-5978 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: kartheek muthyala > Attachments: YARN-5978.001.patch, YARN-5978.002.patch, > YARN-5978.003.patch > > > ContainerScheduler should support updateContainer API for > - Container Resource update > - ExecType update that can change an opportunistic to guaranteed and > vice-versa > Adding a new ContainerState event, UpdateContainerStateEvent to support > UPDATE_CONTAINER call from RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA
[ https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126174#comment-16126174 ] Robert Kanter commented on YARN-5464: - Sounds good. Thanks! > Server-Side NM Graceful Decommissioning with RM HA > -- > > Key: YARN-5464 > URL: https://issues.apache.org/jira/browse/YARN-5464 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Robert Kanter >Priority: Blocker > Attachments: YARN-5464.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126169#comment-16126169
]
Daniel Templeton commented on YARN-6610:
Thanks for the review. I don't see how to avoid sorting the shares in
{{DRC#compare()}}. We have to compare the two resources' shares in order of
dominance, which requires that we order them by dominance. Instead of sorting
we could hunt for the dominant share by going through the arrays, which would
be faster than sorting only if the dominant shares aren't equal. Worst case is
that all shares are equal, which makes that approach slower than sorting. It
seems to me that it's too early to optimize this part.
#2 is an interesting point. It shouldn't ever happen that resources disagree
on how many resource types there are. If it happens, it's a critical issue
that should take the RM down. We could put the whole thing in a _try/catch_ to
have the RM go down with a friendlier message than an
{{ArrayIndexOutOfBoundsException}}, but there is a small cost to adding the
_try_ block, mostly that we limit some kinds of compiler optimizations. I
think it's probably worth it, though.
For #3, we can't do the comparison until we've sorted the shares arrays. Even
if we were to try being clever about merging the sort into the calculation,
we'd still need another pass to do the comparison.
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6995) Improve use of ResourceNotFoundException in resource types code
[ https://issues.apache.org/jira/browse/YARN-6995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126160#comment-16126160 ] Yufei Gu commented on YARN-6995: Thanks for filing this, [~templedf]. It does't apply to the branch. Can you rebase it? > Improve use of ResourceNotFoundException in resource types code > --- > > Key: YARN-6995 > URL: https://issues.apache.org/jira/browse/YARN-6995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6995.YARN-3926.001.patch, > YARN-6995.YARN-3926.002.patch > > > Now that all the YarnExceptions have been replaced with > ResourceNotFoundExceptions, we should make the ResourceNotFoundExceptions as > useful as possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6994) Remove last uses of Long from resource types code
[ https://issues.apache.org/jira/browse/YARN-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126146#comment-16126146 ] Yufei Gu commented on YARN-6994: Thanks for the patch, [~templedf]. Committed to branch branch-yarn-3926. > Remove last uses of Long from resource types code > - > > Key: YARN-6994 > URL: https://issues.apache.org/jira/browse/YARN-6994 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: YARN-3926 > > Attachments: YARN-6994.YARN-3926.001.patch > > > Most of the uses have been removed over the last few patches. There's only > one left that I see. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA
[ https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126139#comment-16126139 ] Junping Du commented on YARN-5464: -- I tried to find the bandwidth to do in next 2 weeks. Let's give it a try. If not, we can have a quick document patch as you mentioned. Sounds good? > Server-Side NM Graceful Decommissioning with RM HA > -- > > Key: YARN-5464 > URL: https://issues.apache.org/jira/browse/YARN-5464 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Robert Kanter >Priority: Blocker > Attachments: YARN-5464.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7006) [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol
[ https://issues.apache.org/jira/browse/YARN-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7006: --- Description: Communication between Collector and NM is via RPC. We would do kerberos based authentication for communication between these 2 components, as of now. Added SecurityInfo implementation for it. We can think of adding token based access once collector starts outside of NM. Also creation of timeline client within NMTimelinePublisher would be done using NM login UGI. > [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol > > > Key: YARN-7006 > URL: https://issues.apache.org/jira/browse/YARN-7006 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-7006-YARN-5355.01.patch, > YARN-7006-YARN-5355.02.patch > > > Communication between Collector and NM is via RPC. > We would do kerberos based authentication for communication between these 2 > components, as of now. Added SecurityInfo implementation for it. > We can think of adding token based access once collector starts outside of NM. > Also creation of timeline client within NMTimelinePublisher would be done > using NM login UGI. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6994) Remove last uses of Long from resource types code
[ https://issues.apache.org/jira/browse/YARN-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126132#comment-16126132 ] Yufei Gu commented on YARN-6994: +1 > Remove last uses of Long from resource types code > - > > Key: YARN-6994 > URL: https://issues.apache.org/jira/browse/YARN-6994 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: YARN-6994.YARN-3926.001.patch > > > Most of the uses have been removed over the last few patches. There's only > one left that I see. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7006) [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol
[
https://issues.apache.org/jira/browse/YARN-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126126#comment-16126126
]
Hadoop QA commented on YARN-7006:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5355 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
33s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} YARN-5355 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
48s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in YARN-5355 has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} YARN-5355 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
56s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-7006 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881789/YARN-7006-YARN-5355.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8d7fe00d494d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / ee5d80d |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16888/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16888/testReport/ |
| modules | C:
[jira] [Comment Edited] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA
[
https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126119#comment-16126119
]
Robert Kanter edited comment on YARN-5464 at 8/14/17 6:08 PM:
--
This is currently a blocker for Hadoop 3 beta 1. [~djp] do you think you'll be
able to take care of this in time? The [Hadoop 3 release wiki
page|https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+Release]
says that we only have about a month left.
If not, we have a couple faster options we can do in the meantime to make this
not a blocker:
# Document that the {{-server}} version is not recommended because it will not
work correctly during an RM failover
# Disable/Remove the {{-server}} version so you can only do the {{-client}}
version for now
I think if we do one of those, then we can take care of this issue later if we
don't have time to take care of it now for beta 1.
was (Author: rkanter):
This is currently a blocker for Hadoop 3 beta 1. [~djp] do you think you'll be
able to take care of this in time? The [Hadoop 3 release wiki
page|https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+Release]
says that we only have about a month left.
If not, we have a couple faster options we can do in the meantime to make this
not a blocker:
# Document that the {{-server}} version is not recommended because it will not
work correctly during an RM failover
# Disable the {{-server}} version so you can only do the {{-client}} version
for now
I think if we do one of those, then we can take care of this issue later if we
don't have time to take care of it now for beta 1.
> Server-Side NM Graceful Decommissioning with RM HA
> --
>
> Key: YARN-5464
> URL: https://issues.apache.org/jira/browse/YARN-5464
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: YARN-5464.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA
[
https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126119#comment-16126119
]
Robert Kanter commented on YARN-5464:
-
This is currently a blocker for Hadoop 3 beta 1. [~djp] do you think you'll be
able to take care of this in time? The [Hadoop 3 release wiki
page|https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+Release]
says that we only have about a month left.
If not, we have a couple faster options we can do in the meantime to make this
not a blocker:
# Document that the {{-server}} version is not recommended because it will not
work correctly during an RM failover
# Disable the {{-server}} version so you can only do the {{-client}} version
for now
I think if we do one of those, then we can take care of this issue later if we
don't have time to take care of it now for beta 1.
> Server-Side NM Graceful Decommissioning with RM HA
> --
>
> Key: YARN-5464
> URL: https://issues.apache.org/jira/browse/YARN-5464
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful
>Reporter: Robert Kanter
>Priority: Blocker
> Attachments: YARN-5464.wip.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126117#comment-16126117 ] Giovanni Matteo Fumarola commented on YARN-6972: +1 LGTM. [~asuresh] Can you please review it? > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch, YARN-6972.002.patch, > YARN-6972.003.patch, YARN-6972.004.patch, YARN-6972.005.patch, > YARN-6972.006.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6687) Validate that the duration of the periodic reservation is less than the periodicity
[ https://issues.apache.org/jira/browse/YARN-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126111#comment-16126111 ] Subru Krishnan commented on YARN-6687: -- Thanks [~curino], I cherry-picked it back to branch-2 cleanly. > Validate that the duration of the periodic reservation is less than the > periodicity > --- > > Key: YARN-6687 > URL: https://issues.apache.org/jira/browse/YARN-6687 > Project: Hadoop YARN > Issue Type: Sub-task > Components: reservation system >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6687-v1.patch > > > Duration of a reservation (endTime - startTime) is specified independently of > the recurrence. This can lead to requests where the duration exceeds the > periodicity, for e.g: an hourly reservation that runs for 2 hours. This will > result in multiple concurrently active instances for the same reservation > which is hard both to manage as well as grok. This jira proposes to reject > such requests by clearly informing the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6687) Validate that the duration of the periodic reservation is less than the periodicity
[ https://issues.apache.org/jira/browse/YARN-6687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-6687: - Fix Version/s: 2.9.0 > Validate that the duration of the periodic reservation is less than the > periodicity > --- > > Key: YARN-6687 > URL: https://issues.apache.org/jira/browse/YARN-6687 > Project: Hadoop YARN > Issue Type: Sub-task > Components: reservation system >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6687-v1.patch > > > Duration of a reservation (endTime - startTime) is specified independently of > the recurrence. This can lead to requests where the duration exceeds the > periodicity, for e.g: an hourly reservation that runs for 2 hours. This will > result in multiple concurrently active instances for the same reservation > which is hard both to manage as well as grok. This jira proposes to reject > such requests by clearly informing the user. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126094#comment-16126094
]
Manikandan R commented on YARN-6933:
[~sunilg] Thanks for the review.
1. I was under the impression only memory-mb and vcores are mandatory
resources. Will do check even for "memory". Need to define constant in
{{ResourceInformation}}. Shall I define "M" as default for "memory"?
2. Yes, will need to fix.
3. vcores default unit is empty string (""). It is intentional pass to produce
the error case.
4. Yes, we will need to do changes in junits accordingly.
> ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources()
> are duplicating work
> ---
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6959) RM may allocate wrong AM Container for new attempt
[
https://issues.apache.org/jira/browse/YARN-6959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126093#comment-16126093
]
Jian He commented on YARN-6959:
---
[~yqwang], yep, I've committed to trunk and branch-2,
branch-2.8 doesn't apply, could you provide a patch for branch-2.8 ?
> RM may allocate wrong AM Container for new attempt
> --
>
> Key: YARN-6959
> URL: https://issues.apache.org/jira/browse/YARN-6959
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, fairscheduler, scheduler
>Affects Versions: 2.7.1
>Reporter: Yuqi Wang
>Assignee: Yuqi Wang
> Labels: patch
> Fix For: 2.7.1, 3.0.0-alpha4
>
> Attachments: YARN-6959.001.patch, YARN-6959.002.patch,
> YARN-6959.003.patch, YARN-6959.004.patch, YARN-6959.005.patch,
> YARN-6959-branch-2.7.001.patch, YARN-6959.yarn_nm.log.zip,
> YARN-6959.yarn_rm.log.zip
>
>
> *Issue Summary:*
> Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests. These mis-recorded ResourceRequests may confuse AM
> Container Request and Allocation for current attempt.
> *Issue Pipeline:*
> {code:java}
> // Executing precondition check for the incoming attempt id.
> ApplicationMasterService.allocate() ->
> scheduler.allocate(attemptId, ask, ...) ->
> // Previous precondition check for the attempt id may be outdated here,
> // i.e. the currentAttempt may not be the corresponding attempt of the
> attemptId.
> // Such as the attempt id is corresponding to the previous attempt.
> currentAttempt = scheduler.getApplicationAttempt(attemptId) ->
> // Previous attempt ResourceRequest may be recorded into current attempt
> ResourceRequests
> currentAttempt.updateResourceRequests(ask) ->
> // RM may allocate wrong AM Container for the current attempt, because its
> ResourceRequests
> // may come from previous attempt which can be any ResourceRequests previous
> AM asked
> // and there is not matching logic for the original AM Container
> ResourceRequest and
> // the returned amContainerAllocation below.
> AMContainerAllocatedTransition.transition(...) ->
> amContainerAllocation = scheduler.allocate(currentAttemptId, ...)
> {code}
> *Patch Correctness:*
> Because after this Patch, RM will definitely record ResourceRequests from
> different attempt into different objects of
> SchedulerApplicationAttempt.AppSchedulingInfo.
> So, even if RM still record ResourceRequests from old attempt at any time,
> these ResourceRequests will be recorded in old AppSchedulingInfo object which
> will not impact current attempt's resource requests and allocation.
> *Concerns:*
> The getApplicationAttempt function in AbstractYarnScheduler is so confusing,
> we should better rename it to getCurrentApplicationAttempt. And reconsider
> whether there are any other bugs related to getApplicationAttempt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6908) ResourceProfilesManagerImpl is missing @Overrides on methods
[
https://issues.apache.org/jira/browse/YARN-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126091#comment-16126091
]
Daniel Templeton commented on YARN-6908:
Thanks for the patch, [~sunilg]. The checkstyle warnings are legit. They're
also the source of the javadoc errors. While you're fixing those, let's also
fix {{@param profile Name of Resource profile}} to be {{@param profile name of
resource profile}} and add a more useful message for the {{@throws}} clauses,
maybe something like, "when an issue is encountered loading the configuration".
> ResourceProfilesManagerImpl is missing @Overrides on methods
>
>
> Key: YARN-6908
> URL: https://issues.apache.org/jira/browse/YARN-6908
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6908-YARN-3926.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[
https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126090#comment-16126090
]
Yufei Gu commented on YARN-6610:
Thanks [~templedf]'s patch and [~sunilg]'s review. The patch looks much better.
Some comments:
# Is it guaranteed that {{lhs}} and {{rhs}} get the same length? If not, we
need to handle the {{ArrayIndexOutOfBoundsException}} nicely. It seems OK to
ignore the exception.
# Still think unit tests for {{DominantResourceCalculator#compare()}} are worth
to do. I've searched around callers and callers of callers. Didn't find good
enough tests for this. Please let me know if they are there.
# Not sure how we can avoid this and merge two loops in [~sunilg]'s comment #3.
We could alway do further optimization, but it is premature without performance
tests and detailed performance analysis.
{code}
137 Arrays.sort(lhsShares);
138 Arrays.sort(rhsShares);
{code}
> DominantResourceCalculator.getResourceAsValue() dominant param is no longer
> appropriate
> ---
>
> Key: YARN-6610
> URL: https://issues.apache.org/jira/browse/YARN-6610
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch,
> YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch
>
>
> The {{dominant}} param assumes there are only two resources, i.e. true means
> to compare the dominant, and false means to compare the subordinate. Now
> that there are _n_ resources, this parameter no longer makes sense.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[
https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126076#comment-16126076
]
Haibo Chen commented on YARN-6905:
--
Thanks [~vrushalic] for your review!
> Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
> ---
>
> Key: YARN-6905
> URL: https://issues.apache.org/jira/browse/YARN-6905
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: 3.0.0-beta1
> Environment: Ubuntu 14.04
> x86, ppc64le
> $ java -version
> openjdk version "1.8.0_111"
> OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)
>Reporter: Sonia Garudi
>Assignee: Haibo Chen
> Attachments: YARN-6905.00.patch, YARN-6905.01.patch
>
>
> There are multiple test failing in Hadoop YARN Timeline Service HBase tests
> project with the following error :
> {code}
> java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat
> at
> org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104)
> {code}
> Below are the failing tests :
> {code}
> TestHBaseTimelineStorageApps.testWriteApplicationToHBase
> TestHBaseTimelineStorageApps.testEvents
> TestHBaseTimelineStorageEntities.testEventsEscapeTs
> TestHBaseTimelineStorageEntities.testWriteEntityToHBase
> TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3050) [Data Serving] Implement new flow-based ATS queries in the new ATS design
[ https://issues.apache.org/jira/browse/YARN-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-3050: - Fix Version/s: YARN-5355-branch-2 YARN-5355 > [Data Serving] Implement new flow-based ATS queries in the new ATS design > - > > Key: YARN-3050 > URL: https://issues.apache.org/jira/browse/YARN-3050 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vrushali C > Labels: YARN-5355 > Fix For: YARN-5355, YARN-5355-branch-2 > > Attachments: Flow based queries.docx > > > Implement new flow-based ATS queries in the new ATS design. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3050) [Data Serving] Implement new flow-based ATS queries in the new ATS design
[ https://issues.apache.org/jira/browse/YARN-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C resolved YARN-3050. -- Resolution: Implemented I would like to resolve this as "Implemented" since ATSv2 now has a bunch of flow based query APIs. It does not yet have the user level or queue level aggregation APIs but we can track those in separate jiras. > [Data Serving] Implement new flow-based ATS queries in the new ATS design > - > > Key: YARN-3050 > URL: https://issues.apache.org/jira/browse/YARN-3050 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Vrushali C > Labels: YARN-5355 > Attachments: Flow based queries.docx > > > Implement new flow-based ATS queries in the new ATS design. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6881) LOG is unused in AllocationConfiguration
[ https://issues.apache.org/jira/browse/YARN-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126069#comment-16126069 ] Daniel Templeton commented on YARN-6881: LGTM +1 > LOG is unused in AllocationConfiguration > > > Key: YARN-6881 > URL: https://issues.apache.org/jira/browse/YARN-6881 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: weiyuan > Labels: newbie > Attachments: YARN-6881.001.patch, YARN-6881.002.patch > > > The variable can be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6780) ResourceWeights.toString() cleanup
[
https://issues.apache.org/jira/browse/YARN-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-6780:
---
Description: The {{toString()}} method should have {{@Override}} and should
use a {{StringBuilder}} instead of a {{StringBuffer}}. (was: The
{{toString()}} method should have {{@Overrides}} and should use a
{{StringBuilder}} instead of a {{StringBuffer}}.)
> ResourceWeights.toString() cleanup
> --
>
> Key: YARN-6780
> URL: https://issues.apache.org/jira/browse/YARN-6780
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: weiyuan
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6780.001.patch
>
>
> The {{toString()}} method should have {{@Override}} and should use a
> {{StringBuilder}} instead of a {{StringBuffer}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6780) ResourceWeights.toString() cleanup
[
https://issues.apache.org/jira/browse/YARN-6780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126067#comment-16126067
]
Daniel Templeton commented on YARN-6780:
The patch doesn't compile, because the annotation is actually {{@Override}}.
My bad.
> ResourceWeights.toString() cleanup
> --
>
> Key: YARN-6780
> URL: https://issues.apache.org/jira/browse/YARN-6780
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler
>Affects Versions: 2.8.1, 3.0.0-alpha4
>Reporter: Daniel Templeton
>Assignee: weiyuan
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6780.001.patch
>
>
> The {{toString()}} method should have {{@Overrides}} and should use a
> {{StringBuilder}} instead of a {{StringBuffer}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6996) Change javax.cache library implementation from JSR107 to Apache Geronimo
[
https://issues.apache.org/jira/browse/YARN-6996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vijay updated YARN-6996:
Priority: Blocker (was: Major)
> Change javax.cache library implementation from JSR107 to Apache Geronimo
>
>
> Key: YARN-6996
> URL: https://issues.apache.org/jira/browse/YARN-6996
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
>Priority: Blocker
> Attachments: YARN-6996.001.patch
>
>
> With YARN Federation, we added YARN-3672, which adds the following to
> {noformat}
> javax.cache
> cache-api
> {noformat}
> This third-party library has some murky license history, as documented in
> this [really long comment
> thread|https://github.com/jsr107/jsr107spec/issues/333]. The summary of the
> thread is that "the library is officially APL (take our word for it), but
> there hasn't been a subsequent release with the license file change".
> LEGAL-325 has been filed to discuss the validity of this license for Apache.
> Before we get to final Hadoop 3 release, I'm wondering if anyone else has
> concerns about using this library. Just from looking at the various javax
> Maven artifacts in our pom.xml files, I see a lot of other javax.* library
> entries (although we may not ship the .jars if they're part of the Java
> runtime).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6880) FSQueue.reservedResource can be final
[ https://issues.apache.org/jira/browse/YARN-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-6880: -- Assignee: HondaWei > FSQueue.reservedResource can be final > - > > Key: YARN-6880 > URL: https://issues.apache.org/jira/browse/YARN-6880 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: HondaWei >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6917) Queue path is recomputed from scratch on every allocation
[
https://issues.apache.org/jira/browse/YARN-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-6917:
-
Attachment: YARN-6917.002.patch
Thanks [~jlowe]. Uploading patch YARN-6917.002.patch that makes {{String
queuePath}} private.
> Queue path is recomputed from scratch on every allocation
> -
>
> Key: YARN-6917
> URL: https://issues.apache.org/jira/browse/YARN-6917
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Affects Versions: 2.8.1
>Reporter: Jason Lowe
>Assignee: Eric Payne
>Priority: Minor
> Attachments: YARN-6917.001.patch, YARN-6917.002.patch
>
>
> As part of the discussion in YARN-6901 I noticed that we are recomputing a
> queue's path for every allocation. Currently getting the queue's path
> involves calling getQueuePath on the parent then building onto that string
> with the basename of the queue. In turn the parent's getQueuePath method
> does the same, so we end up spending time recomputing a string that will
> never change until a reconfiguration.
> Ideally the queue path should be computed once during queue initialization
> rather than on-demand.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6851) Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat
[ https://issues.apache.org/jira/browse/YARN-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126015#comment-16126015 ] Wei Yan commented on YARN-6851: --- [~sunilg], [~wangda], any comments? > Capacity Scheduler: document configs for controlling # containers allowed to > be allocated per node heartbeat > > > Key: YARN-6851 > URL: https://issues.apache.org/jira/browse/YARN-6851 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: YARN-6851.001.patch > > > YARN-4161 introduces new configs for controlling how many containers allowed > to be allocated in each node heartbeat. And we also have offswitchCount > config before. Would be better to document these configurations in CS section. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6783) DominantResourceCalculator.resourceNames should be a constant
[ https://issues.apache.org/jira/browse/YARN-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-6783. --- Resolution: Duplicate This is no longer needed post YARN-6892 > DominantResourceCalculator.resourceNames should be a constant > - > > Key: YARN-6783 > URL: https://issues.apache.org/jira/browse/YARN-6783 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Priority: Trivial > Labels: newbie > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6984) DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
[
https://issues.apache.org/jira/browse/YARN-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126005#comment-16126005
]
Sunil G commented on YARN-6984:
---
Thanks [~templedf] I will help to share a patch here.
> DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
> -
>
> Key: YARN-6984
> URL: https://issues.apache.org/jira/browse/YARN-6984
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>
> The method currently tests only memory and CPU. It looks to me like it
> should test all resources, i.e. it should do what {{isInvalidDivisor()}} does
> and should, in fact, replace that method. [~sunilg], since you wrote the
> method originally, can you comment on what its intended semantics are?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6984) DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
[
https://issues.apache.org/jira/browse/YARN-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G reassigned YARN-6984:
-
Assignee: Sunil G
> DominantResourceCalculator.isAnyMajorResourceZero() should test all resources
> -
>
> Key: YARN-6984
> URL: https://issues.apache.org/jira/browse/YARN-6984
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Sunil G
>
> The method currently tests only memory and CPU. It looks to me like it
> should test all resources, i.e. it should do what {{isInvalidDivisor()}} does
> and should, in fact, replace that method. [~sunilg], since you wrote the
> method originally, can you comment on what its intended semantics are?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
[
https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125991#comment-16125991
]
Hadoop QA commented on YARN-5764:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-5764 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-5764 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862266/YARN-5764-v2.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16889/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> NUMA awareness support for launching containers
> ---
>
> Key: YARN-5764
> URL: https://issues.apache.org/jira/browse/YARN-5764
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, yarn
>Reporter: Olasoji
>Assignee: Devaraj K
> Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance
> Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing
> costly remote memory accesses on non SMP systems. Yarn containers, on launch,
> will be pinned to a specific NUMA node and all subsequent memory allocations
> will be served by the same node, reducing remote memory accesses. The current
> default behavior is to spread memory across all NUMA nodes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5764) NUMA awareness support for launching containers
[
https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125987#comment-16125987
]
Miklos Szegedi commented on YARN-5764:
--
Thank you, [~devaraj.k] for the patch. I have a few quick comments.
First of all a quick question. Is the list of numa nodes, dynamic, can they be
enabled/disabled runtime?
{code}
143 // Remove from NUMA Resources Manager if enabled
144 if (YarnConfiguration.numaAwarenessEnabled(getConfig())) {
145containerManager.getNumaNodesScheduler()
146.releaseNumaResources(containerId);
147 }
{code}
It is probably a safer practice to release the resources, if
containerManager.getNumaNodesScheduler() is not null
{code}
NUMA_NODEIDS_REGEX
{code}
I saw this pattern at the other regexes as well. Should not you use \\s+
instead of \\s* in some places?
{code}
NUMA_NODE_MEMORY_REGEX
{code}
([0-9]*) should be ([0-9]+)
{code}
([0-9\\s]*)
{code}
Should not this be ([0-9\\s]+)?
Is there any reason to throw YarnRuntimeException instead of YarnException in
serviceInit?
{code}
105 long memory = parseMemory(outputLines, nodeIdRange);
{code}
Note: Technically this is not a node Id range anymore, if it is not a space.
Also, could it contain commas?
{code}
115long mem = conf.getLong(
116"yarn.nodemanager.numa-awareness." + nodeId + ".memory", 0);
117int cpus = conf
118.getInt("yarn.nodemanager.numa-awareness." + nodeId +
".cpus", 0);
{code}
It is probably a good idea to default here to some meaningful value like 1024
for memory and 1 for cpu.
{code}
132String[] args = new String[] {"numactl", "--hardware"};
{code}
It might be safer to default this to /usr/bin/numactl and make it configurable.
{code}
131 String executeNGetCmdOutput() throws YarnRuntimeException {
{code}
YarnRuntimeException is a runtime exception, it does not need to be declared.
However, it is better to use YarnException, I think.
{code}
172 } else if (KB.equals(units)) {
173memory = memory / 1024;
174 }
{code}
If memory equals 100 KB, this will become 100/1024=0 as long. Is this expected.
We might need a check that to throw, if memory is 0. Also there is no check for
negative values.
{code}
52 public synchronized boolean isResourcesAvailable(Resource resource) {
53LOG.info(
54"Memory available:" + (totalMemory - usedMemory) + ", CPUs
available:"
55+ (totalCpus - usedCpus) + ", requested:" + resource);
{code}
This is running in a loop, so it might be better to LOG.debug
> NUMA awareness support for launching containers
> ---
>
> Key: YARN-5764
> URL: https://issues.apache.org/jira/browse/YARN-5764
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager, yarn
>Reporter: Olasoji
>Assignee: Devaraj K
> Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance
> Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing
> costly remote memory accesses on non SMP systems. Yarn containers, on launch,
> will be pinned to a specific NUMA node and all subsequent memory allocations
> will be served by the same node, reducing remote memory accesses. The current
> default behavior is to spread memory across all NUMA nodes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7006) [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol
[ https://issues.apache.org/jira/browse/YARN-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7006: --- Attachment: YARN-7006-YARN-5355.02.patch Fixed findbugs issue. > [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol > > > Key: YARN-7006 > URL: https://issues.apache.org/jira/browse/YARN-7006 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-7006-YARN-5355.01.patch, > YARN-7006-YARN-5355.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[
https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125926#comment-16125926
]
Varun Vasudev commented on YARN-6988:
-
[~ebadger] - I'm increasing the EXECUTOR_PATH_MAX to 128K as part of YARN-6623.
Can you please take a look?
> container-executor fails for docker when command length > 4096 B
>
>
> Key: YARN-6988
> URL: https://issues.apache.org/jira/browse/YARN-6988
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
>
> {{run_docker}} and {{launch_docker_container_as_user}} allocate their command
> arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in
> configuration.h. Because of this, the full docker command can only be 4096
> characters. If it is longer, it will be truncated and the command will fail
> with a parsing error. Because of the bind-mounting of volumes, the arguments
> to the docker command can quickly get large. For example, I passed the 4096
> limit with an 11 disk node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
