[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412984#comment-15412984
]
Bibin A Chundatt commented on YARN-4833:
[~Naganarasimha]/[~sunilg]
Another issue in ClientRMService is for AccessControlException in below code is
of type {{import java.security.AccessControlException}}
{code}
{if (!checkAccess(callerUGI, application.getUser(),
ApplicationAccessType.MODIFY_APP, application)) {
RMAuditLogger.logFailure(callerUGI.getShortUserName(),
AuditConstants.UPDATE_APP_PRIORITY,
"User doesn't have permissions to "
+ ApplicationAccessType.MODIFY_APP.toString(), "ClientRMService",
AuditConstants.UNAUTHORIZED_USER, applicationId);
throw RPCUtil.getRemoteException(new AccessControlException("User "
+ callerUGI.getShortUserName() + " cannot perform operation "
+ ApplicationAccessType.MODIFY_APP.name() + " on " + applicationId));
}}
{code}
Will update in {{YARN-4833.0001.patch}} . In {{ClientRMService}} all other
location we are handling {{AccessControlException}} after wrapping in
{{RemoteException}}.
No need to handle in {{RetryPolicies#shouldRetry}} and behaviour will be same
as earlier
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
>
[jira] [Commented] (YARN-5483) Optimize RMAppAttempt#pullJustFinishedContainers
[ https://issues.apache.org/jira/browse/YARN-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412962#comment-15412962 ] sandflee commented on YARN-5483: test failure seems not related to the patch, the test may fail with or without the patch. modified the test timeout from 5s to 50s, the test could always succ > Optimize RMAppAttempt#pullJustFinishedContainers > > > Key: YARN-5483 > URL: https://issues.apache.org/jira/browse/YARN-5483 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5483.01.patch, YARN-5483.02.patch, jprofiler-cpu.png > > > about 1000 app running on cluster, jprofiler found pullJustFinishedContainers > cost too much cpu. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412898#comment-15412898
]
Bibin A Chundatt commented on YARN-4833:
[~Jobo]
Thank you for looking into the patch .. It is {{YarnException}}
{code}
public class RPCUtil {
/**
* Returns an instance of {@link YarnException}
*/
public static YarnException getRemoteException(Throwable t) {
return new YarnException(t);
}
/**
{code}
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Commented] (YARN-5474) Typo mistake in AMRMClient#getRegisteredTimeineClient API
[
https://issues.apache.org/jira/browse/YARN-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412874#comment-15412874
]
Sangjin Lee commented on YARN-5474:
---
{{AMRMClientAsync.java}} also has a method with the same typo.
> Typo mistake in AMRMClient#getRegisteredTimeineClient API
> -
>
> Key: YARN-5474
> URL: https://issues.apache.org/jira/browse/YARN-5474
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Naganarasimha G R
>Priority: Trivial
> Labels: newbie
> Attachments: YARN-5474.v1.001.patch
>
>
> Just found that typo mistake in the API, It can be fixed since ATS is not
> release in any version.
> {code}
> /**
>* Get registered timeline client.
>* @return the registered timeline client
>*/
> public TimelineClient getRegisteredTimeineClient() {
> return this.timelineClient;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412869#comment-15412869 ] Jian He commented on YARN-5382: --- ah, I missed it.. thanks for catching this ! > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382-branch-2.7.12.patch, > YARN-5382.06.patch, YARN-5382.07.patch, YARN-5382.08.patch, > YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch, > YARN-5382.12.patch, YARN-5382.13.patch, YARN-5382.14.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[ https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5457: -- Attachment: YARN-5457.004.patch Renamed the class based on [~kkaranasos]'s suggestion and fixed the javadoc typo [~subru] noticed. > Refactor DistributedScheduling framework to pull out common functionality > - > > Key: YARN-5457 > URL: https://issues.apache.org/jira/browse/YARN-5457 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5457.001.patch, YARN-5457.002.patch, > YARN-5457.003.patch, YARN-5457.004.patch > > > Opening this JIRA to track the some refactoring missed in YARN-5113: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412859#comment-15412859
]
Varun Saxena edited comment on YARN-4833 at 8/9/16 3:21 AM:
True. Actually we can handle it the 1st way (amongst the 2 approaches) by
adding it to some exception map and then handling it while retrying on client
side. That is we adopt approach 1.
It was just a followup comment to Naga's comment.
Your patch is also fine.
was (Author: varun_saxena):
Hmm. Actually we can handle it the 1st way (amongst the 2 approaches) by adding
it to some exception map and then handling it while retrying on client side.
That is we adopt approach 1.
It was just a followup comment to Naga's comment.
Your patch is also fine.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
>
[jira] [Comment Edited] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412859#comment-15412859
]
Varun Saxena edited comment on YARN-4833 at 8/9/16 3:19 AM:
Hmm. Actually we can handle it the 1st way (amongst the 2 approaches) by adding
it to some exception map and then handling it while retrying on client side.
That is we adopt approach 1.
It was just a followup comment to Naga's comment.
Your patch is also fine.
was (Author: varun_saxena):
Hmm. Actually we can handle it the 1st way (amongst the 2 approaches) by adding
it to some exception map and then handling it while retrying on client side.
It was just a followup comment to Naga's comment.
Your patch is also fine.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412859#comment-15412859
]
Varun Saxena commented on YARN-4833:
Hmm. Actually we can handle it the 1st way (amongst the 2 approaches) by adding
it to some exception map and then handling it while retrying on client side.
It was just a followup comment to Naga's comment.
Your patch is also fine.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412853#comment-15412853
]
Bob.zhao commented on YARN-4833:
Hi, Bibin, thanks for you patch!
Here one thing want to know, why not directly throw YarnException directly but
warp with RemoteException?
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at
[jira] [Commented] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[
https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412841#comment-15412841
]
Konstantinos Karanasos commented on YARN-5457:
--
Looks good to me too, thanks [~asuresh].
(Very minor suggestion: it might look better to rename
{{OpportunisticContainersAllocatingAMService}} to
{{OpportunisticContainersAllocatorAMService}} or simply
{{OpportunisticContainersAMService}}).
> Refactor DistributedScheduling framework to pull out common functionality
> -
>
> Key: YARN-5457
> URL: https://issues.apache.org/jira/browse/YARN-5457
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5457.001.patch, YARN-5457.002.patch,
> YARN-5457.003.patch
>
>
> Opening this JIRA to track the some refactoring missed in YARN-5113:
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2098) App priority support in Fair Scheduler
[ https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412839#comment-15412839 ] stefanlee commented on YARN-2098: - [~ywskycn] how about the progress of this jira? > App priority support in Fair Scheduler > -- > > Key: YARN-2098 > URL: https://issues.apache.org/jira/browse/YARN-2098 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.5.0 >Reporter: Ashwin Shankar >Assignee: Wei Yan > Attachments: YARN-2098.patch, YARN-2098.patch > > > This jira is created for supporting app priorities in fair scheduler. > AppSchedulable hard codes priority of apps to 1,we should > change this to get priority from ApplicationSubmissionContext. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[ https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412832#comment-15412832 ] Arun Suresh commented on YARN-5457: --- Actually... Ill wait for a day before pushing this in.. > Refactor DistributedScheduling framework to pull out common functionality > - > > Key: YARN-5457 > URL: https://issues.apache.org/jira/browse/YARN-5457 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5457.001.patch, YARN-5457.002.patch, > YARN-5457.003.patch > > > Opening this JIRA to track the some refactoring missed in YARN-5113: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5407) In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore
[
https://issues.apache.org/jira/browse/YARN-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412813#comment-15412813
]
Subru Krishnan edited comment on YARN-5407 at 8/9/16 2:06 AM:
--
Thanks [~ellenfkh] for the patch.
I realized while reviewing this patch that {{SubClusterPolicyConfiguration}} is
missing *queue* field. Consequently we can remove it from
{{SetSubClusterPolicyConfigurationRequest}}. Can you kindly make the changes?
The patch itself LGTM. I have a minor suggestion for
{{FederationStateStoreBaseTest}} - we should have helper methods to add
applicationSubCluster mapping and policy similar to
_queryApplicationHomeSC/queryPolicy_ as they are used by all other tests and
now we have repeating code.
[~jianhe], can you take a quick look?
was (Author: subru):
Thanks [~ellenfkh] for the patch.
I realized while reviewing this patch that {{SubClusterPolicyConfiguration}} is
missing *queue* field. Consequently we can remove it from
{{SetSubClusterPolicyConfigurationRequest}}. Can you kindly make the changes?
It LGTM. I have a minor suggestion for {{FederationStateStoreBaseTest}} - we
should have helper methods to add applicationSubCluster mapping and policy
similar to _queryApplicationHomeSC/queryPolicy_ as they are used by all other
tests and now we have repeating code.
[~jianhe], can you take a quick look?
> In-memory based implementation of the FederationApplicationStateStore,
> FederationPolicyStateStore
> -
>
> Key: YARN-5407
> URL: https://issues.apache.org/jira/browse/YARN-5407
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5407-YARN-2915.v0.patch,
> YARN-5407-YARN-2915.v1.patch
>
>
> YARN-5307 defines the FederationApplicationStateStore API. YARN-3664 defines
> the FederationPolicyStateStore API. This JIRA tracks an in-memory based
> implementation which is useful for both single-box testing and for future
> unit tests that depend on the state store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5407) In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore
[
https://issues.apache.org/jira/browse/YARN-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412813#comment-15412813
]
Subru Krishnan edited comment on YARN-5407 at 8/9/16 2:06 AM:
--
Thanks [~ellenfkh] for the patch.
I realized while reviewing this patch that {{SubClusterPolicyConfiguration}} is
missing *queue* field. Consequently we can remove it from
{{SetSubClusterPolicyConfigurationRequest}}. Can you kindly make the changes?
It LGTM. I have a minor suggestion for {{FederationStateStoreBaseTest}} - we
should have helper methods to add applicationSubCluster mapping and policy
similar to _queryApplicationHomeSC/queryPolicy_ as they are used by all other
tests and now we have repeating code.
[~jianhe], can you take a quick look?
was (Author: subru):
Thanks [~ellenfkh] for the patch.
It LGTM. I have a minor suggestion for {{FederationStateStoreBaseTest}} - we
should have helper methods to add applicationSubCluster mapping and policy
similar to _queryApplicationHomeSC/queryPolicy_ as they are used by all other
tests and now we have repeating code.
[~jianhe], can you take a quick look before I commit tomorrow?
> In-memory based implementation of the FederationApplicationStateStore,
> FederationPolicyStateStore
> -
>
> Key: YARN-5407
> URL: https://issues.apache.org/jira/browse/YARN-5407
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5407-YARN-2915.v0.patch,
> YARN-5407-YARN-2915.v1.patch
>
>
> YARN-5307 defines the FederationApplicationStateStore API. YARN-3664 defines
> the FederationPolicyStateStore API. This JIRA tracks an in-memory based
> implementation which is useful for both single-box testing and for future
> unit tests that depend on the state store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
[
https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412817#comment-15412817
]
He Tianyi commented on YARN-5479:
-
Thanks for comments. [~rchiang]. [~jlowe].
bq. I'd be careful with having multiple implementations or multiple APIs for
doing the same thing with Resource. Resource is used a lot of places in the
Hadoop codebase and this could add confusion, even with accurate Javadocs.
Yes, multiple implementations would be confusing. I tried to replace
{{ResourcePBImpl}} directly with the implementation I mentioned and looks like
no other issue is raised. Maybe we could still stick to single version of
implementation by making it faster.
bq. The nodeUpdate() changes will conflict with YARN-5047 unless you plan on
doing the same changes for CapacityScheduler and FifoScheduler.
Most changes can be done in {{attemptScheduling}}, which is dedicated to
FairScheduler. So perhaps we can keep it that way.
bq. Minimally I think we should approach this as two (or more) separate JIRAs
since there are two vastly different approaches to improving performance here.
Agreed. Will fill separate JIRAs to address each aspect.
bq. I don't think we should start loosening the guarantees of the scheduler for
performance reasons until we've exhausted the other ways we can improve
performance
Certainly. However, the approach would be quite simple for implementing. While
doing so does not seemly cause any problem in production (fairness is slightly
damaged locally, but within acceptable range. and there is no effect globally.
though not carefully investigated yet).
So if one must figure out how to balance between resource utilization and
fairness (since resource costs), providing such option (e.g. through
configuration) may be viable.
Shall we make this issue an umbrella? There are still many approaches to
deliver better performance in FairScheduler.
> FairScheduler: Scheduling performance improvement
> -
>
> Key: YARN-5479
> URL: https://issues.apache.org/jira/browse/YARN-5479
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for
> scheduling. As number of nodes grows, more events need to be processed in
> time in FairScheduler. Also, increased number of applications & queues slows
> down processing of each single event.
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds.
> This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of
> some nodes cannot get processed before next heartbeat. This brings
> inefficiency handling burst requests (i.e. newly submitted MapReduce
> application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single
> cluster is scaled to several K of nodes (even with {{assignmultiple}}
> enabled).
> This issue proposes to perform several optimization towards performance in
> FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped
> (or should this be called 'delayed sorting'?). we can either start another
> dedicated thread to do the sorting & updating, or actually perform sorting
> after current result have been used several times (say sort once in every 100
> calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at
> least 2 objects ({{ResourceImpl}} and its proto builder) is created each time
> (using 'immutable' apis). the overhead can be eliminated with a
> light-weighted implementation of Resource, which do not instantiate a builder
> until necessary, because most instances are used as intermediate result in
> scheduler instead of being exchanged via IPC. Also, {{createResource}} is
> using reflection, which can be replaced by a plain {{new}} (for scheduler
> usage only). furthermore, perhaps we could 'intern' resource to avoid
> allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to
> {{update}}, making root queue metrics eventual consistent (which may
> satisfies most of the needs). or introduce counters to {{getResourceUsage}}
> and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[ https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412815#comment-15412815 ] Arun Suresh commented on YARN-5457: --- Thanks for the review [~subru] and the +1. Committing this shortly (Will fix the typos as I check in) > Refactor DistributedScheduling framework to pull out common functionality > - > > Key: YARN-5457 > URL: https://issues.apache.org/jira/browse/YARN-5457 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5457.001.patch, YARN-5457.002.patch, > YARN-5457.003.patch > > > Opening this JIRA to track the some refactoring missed in YARN-5113: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5407) In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore
[
https://issues.apache.org/jira/browse/YARN-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412813#comment-15412813
]
Subru Krishnan commented on YARN-5407:
--
Thanks [~ellenfkh] for the patch.
It LGTM. I have a minor suggestion for {{FederationStateStoreBaseTest}} - we
should have helper methods to add applicationSubCluster mapping and policy
similar to _queryApplicationHomeSC/queryPolicy_ as they are used by all other
tests and now we have repeating code.
[~jianhe], can you take a quick look before I commit tomorrow?
> In-memory based implementation of the FederationApplicationStateStore,
> FederationPolicyStateStore
> -
>
> Key: YARN-5407
> URL: https://issues.apache.org/jira/browse/YARN-5407
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5407-YARN-2915.v0.patch,
> YARN-5407-YARN-2915.v1.patch
>
>
> YARN-5307 defines the FederationApplicationStateStore API. YARN-3664 defines
> the FederationPolicyStateStore API. This JIRA tracks an in-memory based
> implementation which is useful for both single-box testing and for future
> unit tests that depend on the state store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4910) Fix incomplete log info in ResourceLocalizationService
[
https://issues.apache.org/jira/browse/YARN-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412803#comment-15412803
]
Jun Gong commented on YARN-4910:
Thanks [~varun_saxena] for the review and commit!
> Fix incomplete log info in ResourceLocalizationService
> --
>
> Key: YARN-4910
> URL: https://issues.apache.org/jira/browse/YARN-4910
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jun Gong
>Assignee: Jun Gong
>Priority: Trivial
> Fix For: 2.9.0
>
> Attachments: YARN-4910.01.patch
>
>
> When debugging, find a lot of incomplete log info from
> ResourceLocalizationService, it is a little confusing.
> {quote}
> 2016-03-30 22:47:29,703 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Writing credentials to the nmPrivate file
> /data6/yarnenv/local/nmPrivate/container_1456839788316_4159_01_04_37.tokens.
> Credentials list:
> {quote}
> The content of credentials list will only be printed for DEBUG log level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5483) Optimize RMAppAttempt#pullJustFinishedContainers
[
https://issues.apache.org/jira/browse/YARN-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412711#comment-15412711
]
Hadoop QA commented on YARN-5483:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 112 unchanged - 1 fixed = 112 total (was 113)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 10s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822692/YARN-5483.02.patch |
| JIRA Issue | YARN-5483 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f167dec99c26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0705489 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12682/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12682/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12682/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[
https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412709#comment-15412709
]
Hadoop QA commented on YARN-5382:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 214 unchanged - 1 fixed = 215 total (was 215)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 36s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 14s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 9s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822694/YARN-5382.14.patch |
| JIRA Issue | YARN-5382 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 168893af76d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0705489 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12683/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/12683/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12683/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12683/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org
[jira] [Commented] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412694#comment-15412694 ] Brook Zhou commented on YARN-5472: -- Yes, this is something I have verified on our NM's running on machine versions up to server 2012 r2 - the max path limitation without using long-path prefixing is 259 characters. > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > Attachments: YARN-5472-v0.patch > > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" on Windows platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-5382: - Attachment: YARN-5382.14.patch Thanks [~jlowe], yes updating the patch for trunk. > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382-branch-2.7.12.patch, > YARN-5382.06.patch, YARN-5382.07.patch, YARN-5382.08.patch, > YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch, > YARN-5382.12.patch, YARN-5382.13.patch, YARN-5382.14.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[
https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412643#comment-15412643
]
Hadoop QA commented on YARN-5457:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 49s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 29s
{color} | {color:red} root: The patch generated 3 new + 364 unchanged - 14
fixed = 367 total (was 378) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 32s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 34s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 42s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 38s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 114m 43s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 232m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.logaggregation.TestAggregatedLogFormat |
| | hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | hadoop.yarn.client.api.impl.TestYarnClient |
[jira] [Commented] (YARN-5483) Optimize RMAppAttempt#pullJustFinishedContainers
[ https://issues.apache.org/jira/browse/YARN-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412639#comment-15412639 ] sandflee commented on YARN-5483: thanks [~templedf], update the patch to add diamond operator to finishedContainersSentToAM and justFinishedContainers related code. > Optimize RMAppAttempt#pullJustFinishedContainers > > > Key: YARN-5483 > URL: https://issues.apache.org/jira/browse/YARN-5483 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5483.01.patch, YARN-5483.02.patch, jprofiler-cpu.png > > > about 1000 app running on cluster, jprofiler found pullJustFinishedContainers > cost too much cpu. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5483) Optimize RMAppAttempt#pullJustFinishedContainers
[ https://issues.apache.org/jira/browse/YARN-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandflee updated YARN-5483: --- Attachment: YARN-5483.02.patch > Optimize RMAppAttempt#pullJustFinishedContainers > > > Key: YARN-5483 > URL: https://issues.apache.org/jira/browse/YARN-5483 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5483.01.patch, YARN-5483.02.patch, jprofiler-cpu.png > > > about 1000 app running on cluster, jprofiler found pullJustFinishedContainers > cost too much cpu. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5407) In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore
[
https://issues.apache.org/jira/browse/YARN-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412633#comment-15412633
]
Hadoop QA commented on YARN-5407:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 33s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822683/YARN-5407-YARN-2915.v1.patch
|
| JIRA Issue | YARN-5407 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4f0a542d82d9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / dbaebf8 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12681/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12681/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12681/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> In-memory based implementation of the FederationApplicationStateStore,
> FederationPolicyStateStore
>
[jira] [Commented] (YARN-5382) RM does not audit log kill request for active applications
[ https://issues.apache.org/jira/browse/YARN-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412593#comment-15412593 ] Jason Lowe commented on YARN-5382: -- The auditLogKillEvent method is now being called from the AppRejectedTransition instead of the KillAttemptTransition which I assume is unintentional. I think we need the audit log event in the KillAttemptTransition to make sure apps that are killed before they start running still get logged. > RM does not audit log kill request for active applications > -- > > Key: YARN-5382 > URL: https://issues.apache.org/jira/browse/YARN-5382 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.7.2 >Reporter: Jason Lowe >Assignee: Vrushali C > Attachments: YARN-5382-branch-2.7.01.patch, > YARN-5382-branch-2.7.02.patch, YARN-5382-branch-2.7.03.patch, > YARN-5382-branch-2.7.04.patch, YARN-5382-branch-2.7.05.patch, > YARN-5382-branch-2.7.09.patch, YARN-5382-branch-2.7.10.patch, > YARN-5382-branch-2.7.11.patch, YARN-5382-branch-2.7.12.patch, > YARN-5382.06.patch, YARN-5382.07.patch, YARN-5382.08.patch, > YARN-5382.09.patch, YARN-5382.10.patch, YARN-5382.11.patch, > YARN-5382.12.patch, YARN-5382.13.patch > > > ClientRMService will audit a kill request but only if it either fails to > issue the kill or if the kill is sent to an already finished application. It > does not create a log entry when the application is active which is arguably > the most important case to audit. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5407) In-memory based implementation of the FederationApplicationStateStore, FederationPolicyStateStore
[ https://issues.apache.org/jira/browse/YARN-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ellen Hui updated YARN-5407: Attachment: YARN-5407-YARN-2915.v1.patch > In-memory based implementation of the FederationApplicationStateStore, > FederationPolicyStateStore > - > > Key: YARN-5407 > URL: https://issues.apache.org/jira/browse/YARN-5407 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5407-YARN-2915.v0.patch, > YARN-5407-YARN-2915.v1.patch > > > YARN-5307 defines the FederationApplicationStateStore API. YARN-3664 defines > the FederationPolicyStateStore API. This JIRA tracks an in-memory based > implementation which is useful for both single-box testing and for future > unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5478) [YARN-4902] Define Java API for generalized & unified scheduling-strategies.
[ https://issues.apache.org/jira/browse/YARN-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5478: - Attachment: YARN-5478.preliminary-poc.2.patch Attached ver.2 preliminary POC patch, make builder easier to use (still refer to guts.Examples) > [YARN-4902] Define Java API for generalized & unified scheduling-strategies. > > > Key: YARN-5478 > URL: https://issues.apache.org/jira/browse/YARN-5478 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5478.preliminary-poc.1.patch, > YARN-5478.preliminary-poc.2.patch > > > Define Java API for application to specify generic scheduling requirements > described in YARN-4902 design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5137) Make DiskChecker pluggable in NodeManager
[ https://issues.apache.org/jira/browse/YARN-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-5137: --- Attachment: YARN-5137.006.patch Patch 006 is the same to patch 005 except remove the bug fix code in YARN-5444. > Make DiskChecker pluggable in NodeManager > - > > Key: YARN-5137 > URL: https://issues.apache.org/jira/browse/YARN-5137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ray Chiang >Assignee: Yufei Gu > Labels: supportability > Attachments: YARN-5137.001.patch, YARN-5137.002.patch, > YARN-5137.003.patch, YARN-5137.004.patch, YARN-5137.005.patch, > YARN-5137.006.patch > > > It would be nice to have the option for a DiskChecker that has more > sophisticated checking capabilities. In order to do this, we would first > need DiskChecker to be pluggable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5408) Compose Federation membership/application/policy APIs into an uber FederationStateStore API
[
https://issues.apache.org/jira/browse/YARN-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412532#comment-15412532
]
Hadoop QA commented on YARN-5408:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 50s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822664/YARN-5408-YARN-2915.v2.patch
|
| JIRA Issue | YARN-5408 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a7bc41796fc6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / a6a43c0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12679/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12679/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Compose Federation membership/application/policy APIs into an uber
> FederationStateStore API
> ---
>
> Key: YARN-5408
> URL: https://issues.apache.org/jira/browse/YARN-5408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
>
[jira] [Comment Edited] (YARN-5137) Make DiskChecker pluggable in NodeManager
[ https://issues.apache.org/jira/browse/YARN-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412533#comment-15412533 ] Yufei Gu edited comment on YARN-5137 at 8/8/16 9:43 PM: Patch 006 is the same to patch 005 except removing the bug fix code in YARN-5444. was (Author: yufeigu): Patch 006 is the same to patch 005 except remove the bug fix code in YARN-5444. > Make DiskChecker pluggable in NodeManager > - > > Key: YARN-5137 > URL: https://issues.apache.org/jira/browse/YARN-5137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ray Chiang >Assignee: Yufei Gu > Labels: supportability > Attachments: YARN-5137.001.patch, YARN-5137.002.patch, > YARN-5137.003.patch, YARN-5137.004.patch, YARN-5137.005.patch, > YARN-5137.006.patch > > > It would be nice to have the option for a DiskChecker that has more > sophisticated checking capabilities. In order to do this, we would first > need DiskChecker to be pluggable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412530#comment-15412530 ] Hudson commented on YARN-5470: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10237 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10237/]) YARN-5470. Addedum to differentiate exactly matching of log-files with (vinodkv: rev 83a2ffec408fc3ebb89ab13ab4788d9f02fae33b) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.1.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412511#comment-15412511 ] Vinod Kumar Vavilapalli commented on YARN-5470: --- The updated patch looks good to me. The check-style warnings already exist and are not addressable as we are validating the CLI output, without sacrificing a lot of readability - not worth it. The test failures are unrelated - YARN-5442 / YARN-5389 track this. Checking the addendum patch in. > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.1.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412506#comment-15412506
]
Hadoop QA commented on YARN-4833:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 37m 37s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 9s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822650/YARN-4833.0002.patch |
| JIRA Issue | YARN-4833 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 088c1affb815 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 58e1523 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12678/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12678/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A
[jira] [Updated] (YARN-5408) Compose Federation membership/application/policy APIs into an uber FederationStateStore API
[ https://issues.apache.org/jira/browse/YARN-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ellen Hui updated YARN-5408: Attachment: YARN-5408-YARN-2915.v2.patch Address comments from [~subru]. > Compose Federation membership/application/policy APIs into an uber > FederationStateStore API > --- > > Key: YARN-5408 > URL: https://issues.apache.org/jira/browse/YARN-5408 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5408-YARN-2915.v1.patch, > YARN-5408-YARN-2915.v2.patch > > > This is a simple composition of the three APIs defined in YARN-3664, > YARN-5307, YARN-3662. This is for convenience so that we provide a single API > for both implementations and for consumers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4767) Network issues can cause persistent RM UI outage
[ https://issues.apache.org/jira/browse/YARN-4767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412447#comment-15412447 ] Daniel Templeton commented on YARN-4767: Flood ping, [~xgong], [~vinodkv]. It would be great to get some feedback before we talk about committing. > Network issues can cause persistent RM UI outage > > > Key: YARN-4767 > URL: https://issues.apache.org/jira/browse/YARN-4767 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-4767.001.patch, YARN-4767.002.patch, > YARN-4767.003.patch, YARN-4767.004.patch, YARN-4767.005.patch, > YARN-4767.006.patch, YARN-4767.007.patch > > > If a network issue causes an AM web app to resolve the RM proxy's address to > something other than what's listed in the allowed proxies list, the > AmIpFilter will 302 redirect the RM proxy's request back to the RM proxy. > The RM proxy will then consume all available handler threads connecting to > itself over and over, resulting in an outage of the web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[ https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412428#comment-15412428 ] Subru Krishnan commented on YARN-5457: -- Thanks [~asuresh] for the patch. +1 from my side (pending jenkins). Nit: there are a couple of typos in Javadocs and redundant whitespaces in imports. BTW there is a MR test class change which I guess is acceptable to be part of this patch as otherwise there will be compilation issues. > Refactor DistributedScheduling framework to pull out common functionality > - > > Key: YARN-5457 > URL: https://issues.apache.org/jira/browse/YARN-5457 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5457.001.patch, YARN-5457.002.patch, > YARN-5457.003.patch > > > Opening this JIRA to track the some refactoring missed in YARN-5113: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5487) Can't kill nodemanager process when running in the foreground
[ https://issues.apache.org/jira/browse/YARN-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-5487: --- Affects Version/s: 2.7.2 > Can't kill nodemanager process when running in the foreground > - > > Key: YARN-5487 > URL: https://issues.apache.org/jira/browse/YARN-5487 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Allen Wittenauer > Labels: regression > > It looks like there is a misconfigured signal handler somewhere in the mix. > Hitting ctrl-c results in this message: > 16/08/08 20:26:37 ERROR nodemanager.NodeManager: RECEIVED SIGNAL 2: SIGINT > ... which is then summarily ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5487) Can't kill nodemanager process when running in the foreground
[ https://issues.apache.org/jira/browse/YARN-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-5487: --- Labels: regression (was: ) > Can't kill nodemanager process when running in the foreground > - > > Key: YARN-5487 > URL: https://issues.apache.org/jira/browse/YARN-5487 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Allen Wittenauer > Labels: regression > > It looks like there is a misconfigured signal handler somewhere in the mix. > Hitting ctrl-c results in this message: > 16/08/08 20:26:37 ERROR nodemanager.NodeManager: RECEIVED SIGNAL 2: SIGINT > ... which is then summarily ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5487) Can't kill nodemanager process when running in the foreground
[ https://issues.apache.org/jira/browse/YARN-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated YARN-5487: --- Component/s: nodemanager > Can't kill nodemanager process when running in the foreground > - > > Key: YARN-5487 > URL: https://issues.apache.org/jira/browse/YARN-5487 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Allen Wittenauer > Labels: regression > > It looks like there is a misconfigured signal handler somewhere in the mix. > Hitting ctrl-c results in this message: > 16/08/08 20:26:37 ERROR nodemanager.NodeManager: RECEIVED SIGNAL 2: SIGINT > ... which is then summarily ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5487) Can't kill nodemanager process when running in the foreground
Allen Wittenauer created YARN-5487: -- Summary: Can't kill nodemanager process when running in the foreground Key: YARN-5487 URL: https://issues.apache.org/jira/browse/YARN-5487 Project: Hadoop YARN Issue Type: Bug Reporter: Allen Wittenauer It looks like there is a misconfigured signal handler somewhere in the mix. Hitting ctrl-c results in this message: 16/08/08 20:26:37 ERROR nodemanager.NodeManager: RECEIVED SIGNAL 2: SIGINT ... which is then summarily ignored. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4833:
---
Attachment: YARN-4833.0002.patch
Attaching patch handling the exception in {{ClientRMService}} .
Tested the same manually its working fine
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch,
> YARN-4833.0002.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412396#comment-15412396
]
Bibin A Chundatt commented on YARN-4833:
{quote}
AccessControlException in RetryPolicy and should go to fallbackPolicy
{quote}
Want to keep the behaviour similar to earlier.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at
[jira] [Commented] (YARN-5137) Make DiskChecker pluggable in NodeManager
[ https://issues.apache.org/jira/browse/YARN-5137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412392#comment-15412392 ] Ray Chiang commented on YARN-5137: -- Thanks [~vvasudev] for your help on YARN-5444. I look forward to getting this fix unblocked. > Make DiskChecker pluggable in NodeManager > - > > Key: YARN-5137 > URL: https://issues.apache.org/jira/browse/YARN-5137 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Ray Chiang >Assignee: Yufei Gu > Labels: supportability > Attachments: YARN-5137.001.patch, YARN-5137.002.patch, > YARN-5137.003.patch, YARN-5137.004.patch, YARN-5137.005.patch > > > It would be nice to have the option for a DiskChecker that has more > sophisticated checking capabilities. In order to do this, we would first > need DiskChecker to be pluggable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412350#comment-15412350
]
Bibin A Chundatt edited comment on YARN-4833 at 8/8/16 7:56 PM:
[~varun_saxena]
For HA the flow will be different.
{code}
// Handle HA case first
if (isHAEnabled) {
...
return RetryPolicies.failoverOnNetworkException(
RetryPolicies.TRY_ONCE_THEN_FAIL, maxFailoverAttempts,
failoverSleepBaseMs, failoverSleepMaxMs);
}
{code}
was (Author: bibinchundatt):
[~varun_saxena]
For HA the flow will be different.
// Handle HA case first
if (isHAEnabled) {
...
return RetryPolicies.failoverOnNetworkException(
RetryPolicies.TRY_ONCE_THEN_FAIL, maxFailoverAttempts,
failoverSleepBaseMs, failoverSleepMaxMs);
}
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
>
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412350#comment-15412350
]
Bibin A Chundatt commented on YARN-4833:
[~varun_saxena]
For HA the flow will be different.
// Handle HA case first
if (isHAEnabled) {
...
return RetryPolicies.failoverOnNetworkException(
RetryPolicies.TRY_ONCE_THEN_FAIL, maxFailoverAttempts,
failoverSleepBaseMs, failoverSleepMaxMs);
}
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Comment Edited] (YARN-5408) Compose Federation membership/application/policy APIs into an uber FederationStateStore API
[
https://issues.apache.org/jira/browse/YARN-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412301#comment-15412301
]
Subru Krishnan edited comment on YARN-5408 at 8/8/16 7:43 PM:
--
Thanks [~ellenfkh] for the patch. It mostly LGTM, couple of minor comments:
* Can you rename {{FederationStore}} to {{FederationStateStore}}.
* Also rename {{getFederationStoreVersion()}} to {{getCurrentVersion()}} and
add a {{loadVersion()}} to align with *RMStateStore*.
* If possible, can you add little more meat to the Javadocs (class/method) of
{{FederationStateStore}}.
was (Author: subru):
Thanks [~ellenfkh] for the patch. It mostly LGTM, couple of minor comments:
* Can you rename {{FederationStore}} to {{FederationStateStore}}.
* Also rename {{getFederationStoreVersion}} to {{getCurrentVersion}} to align
with *RMStateStore*.
* If possible, can you add little more meat to the Javadocs (class/method) of
{{FederationStateStore}}.
> Compose Federation membership/application/policy APIs into an uber
> FederationStateStore API
> ---
>
> Key: YARN-5408
> URL: https://issues.apache.org/jira/browse/YARN-5408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5408-YARN-2915.v1.patch
>
>
> This is a simple composition of the three APIs defined in YARN-3664,
> YARN-5307, YARN-3662. This is for convenience so that we provide a single API
> for both implementations and for consumers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[ https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5457: -- Attachment: YARN-5457.003.patch Adding package-info.java and fixing javadocs. > Refactor DistributedScheduling framework to pull out common functionality > - > > Key: YARN-5457 > URL: https://issues.apache.org/jira/browse/YARN-5457 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5457.001.patch, YARN-5457.002.patch, > YARN-5457.003.patch > > > Opening this JIRA to track the some refactoring missed in YARN-5113: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes
[
https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412324#comment-15412324
]
Hadoop QA commented on YARN-4765:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
3s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} YARN-5355 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests:
The patch generated 6 new + 3 unchanged - 4 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 49s
{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 19s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822634/YARN-4765-YARN-5355.01.patch
|
| JIRA Issue | YARN-4765 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e135e2facf39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5355 / 5d7ad39 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12676/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12676/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12676/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Split TestHBaseTimelineStorage into multiple test classes
>
[jira] [Updated] (YARN-5478) [YARN-4902] Define Java API for generalized & unified scheduling-strategies.
[ https://issues.apache.org/jira/browse/YARN-5478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-5478: - Attachment: YARN-5478.preliminary-poc.1.patch Attached preliminary patch to show what the APIs look like in my mind. Please feel free to share your thoughts! You can take a look at org.apache.hadoop.yarn.util.guts.Examples as a starting point. + [~vinodkv], [~kasha], [~subru], [~asuresh], [~kkaranasos]. > [YARN-4902] Define Java API for generalized & unified scheduling-strategies. > > > Key: YARN-5478 > URL: https://issues.apache.org/jira/browse/YARN-5478 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-5478.preliminary-poc.1.patch > > > Define Java API for application to specify generic scheduling requirements > described in YARN-4902 design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5408) Compose Federation membership/application/policy APIs into an uber FederationStateStore API
[
https://issues.apache.org/jira/browse/YARN-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412301#comment-15412301
]
Subru Krishnan commented on YARN-5408:
--
Thanks [~ellenfkh] for the patch. It mostly LGTM, couple of minor comments:
* Can you rename {{FederationStore}} to {{FederationStateStore}}.
* Also rename {{getFederationStoreVersion}} to {{getCurrentVersion}} to align
with *RMStateStore*.
* If possible, can you add little more meat to the Javadocs (class/method) of
{{FederationStateStore}}.
> Compose Federation membership/application/policy APIs into an uber
> FederationStateStore API
> ---
>
> Key: YARN-5408
> URL: https://issues.apache.org/jira/browse/YARN-5408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5408-YARN-2915.v1.patch
>
>
> This is a simple composition of the three APIs defined in YARN-3664,
> YARN-5307, YARN-3662. This is for convenience so that we provide a single API
> for both implementations and for consumers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes
[ https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412279#comment-15412279 ] Varun Saxena commented on YARN-4765: Simple fix. Moved data generation to a separate utility class, split tests for entity and application table. > Split TestHBaseTimelineStorage into multiple test classes > - > > Key: YARN-4765 > URL: https://issues.apache.org/jira/browse/YARN-4765 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355 > Attachments: YARN-4765-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5408) Compose Federation membership/application/policy APIs into an uber FederationStateStore API
[
https://issues.apache.org/jira/browse/YARN-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412280#comment-15412280
]
Hadoop QA commented on YARN-5408:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
10s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s {color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822629/YARN-5408-YARN-2915.v1.patch
|
| JIRA Issue | YARN-5408 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6240509d1bbf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / a6a43c0 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12675/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12675/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Compose Federation membership/application/policy APIs into an uber
> FederationStateStore API
> ---
>
> Key: YARN-5408
> URL: https://issues.apache.org/jira/browse/YARN-5408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
>
[jira] [Updated] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes
[ https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4765: --- Attachment: YARN-4765-YARN-5355.01.patch > Split TestHBaseTimelineStorage into multiple test classes > - > > Key: YARN-4765 > URL: https://issues.apache.org/jira/browse/YARN-4765 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355 > Attachments: YARN-4765-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5472) WIN_MAX_PATH logic is off by one
[
https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412277#comment-15412277
]
Hadoop QA commented on YARN-5472:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 59s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822615/YARN-5472-v0.patch |
| JIRA Issue | YARN-5472 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 20e81106a887 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6255859 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12674/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12674/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12674/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12674/console
[jira] [Updated] (YARN-5408) Compose Federation membership/application/policy APIs into an uber FederationStateStore API
[ https://issues.apache.org/jira/browse/YARN-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ellen Hui updated YARN-5408: Attachment: YARN-5408-YARN-2915.v1.patch > Compose Federation membership/application/policy APIs into an uber > FederationStateStore API > --- > > Key: YARN-5408 > URL: https://issues.apache.org/jira/browse/YARN-5408 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > Attachments: YARN-5408-YARN-2915.v1.patch > > > This is a simple composition of the three APIs defined in YARN-3664, > YARN-5307, YARN-3662. This is for convenience so that we provide a single API > for both implementations and for consumers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5486) Update OpportunisticConatinerAllocatioAMService allocate method to handle OPPORTUNISTIC container requests
[ https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5486: -- Assignee: Konstantinos Karanasos (was: Arun Suresh) > Update OpportunisticConatinerAllocatioAMService allocate method to handle > OPPORTUNISTIC container requests > -- > > Key: YARN-5486 > URL: https://issues.apache.org/jira/browse/YARN-5486 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos > > YARN-5457 refactors the Distributed Scheduling framework to move the > container allocator to yarn-server-common. > This JIRA proposes to update the allocate method in the new AM service to use > the OpportunisticContainerAllocator to allocate opportunistic containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412215#comment-15412215
]
Hadoop QA commented on YARN-4833:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 34m 23s
{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 11s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822608/YARN-4833.0001.patch |
| JIRA Issue | YARN-4833 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fa78c9709539 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 37d939a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12672/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12672/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A
[jira] [Created] (YARN-5486) Update OpportunisticConatinerAllocatioAMService allocate method to handle OPPORTUNISTIC container requests
Arun Suresh created YARN-5486: - Summary: Update OpportunisticConatinerAllocatioAMService allocate method to handle OPPORTUNISTIC container requests Key: YARN-5486 URL: https://issues.apache.org/jira/browse/YARN-5486 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun Suresh Assignee: Arun Suresh YARN-5457 refactors the Distributed Scheduling framework to move the container allocator to yarn-server-common. This JIRA proposes to update the allocate method in the new AM service to use the OpportunisticContainerAllocator to allocate opportunistic containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412089#comment-15412089
]
Varun Saxena edited comment on YARN-4833 at 8/8/16 6:15 PM:
We can add AccessControlException with fail retry policy in
exceptiontopolicymap (in RMProxy) as well. I do not think ACL related errors
will change upon retrying.
was (Author: varun_saxena):
We can add AccessControlException in exceptiontopolicymap in RMProxy as well.
I do not think ACL related errors will change upon retrying.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at
[jira] [Commented] (YARN-5479) FairScheduler: Scheduling performance improvement
[
https://issues.apache.org/jira/browse/YARN-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412177#comment-15412177
]
Jason Lowe commented on YARN-5479:
--
Agree the proposals are interesting. I'd love to get the overhead of Resource
reduced, since as you and Ray point out it's used everywhere.
Minimally I think we should approach this as two (or more) separate JIRAs since
there are two vastly different approaches to improving performance here. One
is optimizing the existing algorithm while the other is proposing to change the
requirements to allow more optimization. I don't think we should start
loosening the guarantees of the scheduler for performance reasons until we've
exhausted the other ways we can improve performance. So personally I'd rather
see the Resource-related improvements before the others that change the
guarantees to which users have grown accustomed.
> FairScheduler: Scheduling performance improvement
> -
>
> Key: YARN-5479
> URL: https://issues.apache.org/jira/browse/YARN-5479
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, resourcemanager
>Affects Versions: 2.6.0
>Reporter: He Tianyi
>Assignee: He Tianyi
>
> Currently ResourceManager uses a single thread to handle async events for
> scheduling. As number of nodes grows, more events need to be processed in
> time in FairScheduler. Also, increased number of applications & queues slows
> down processing of each single event.
> There are two cases that slow processing of nodeUpdate events is problematic:
> A. global throughput is lower than number of nodes through heartbeat rounds.
> This keeps resource from being allocated since the inefficiency.
> B. global throughput meets the need, but for some of these rounds, events of
> some nodes cannot get processed before next heartbeat. This brings
> inefficiency handling burst requests (i.e. newly submitted MapReduce
> application cannot get its all task launched soon given enough resource).
> Pretty sure some people will encounter the problem eventually after a single
> cluster is scaled to several K of nodes (even with {{assignmultiple}}
> enabled).
> This issue proposes to perform several optimization towards performance in
> FairScheduler {{nodeUpdate}} method. To be specific:
> A. trading off fairness with efficiency, queue & app sorting can be skipped
> (or should this be called 'delayed sorting'?). we can either start another
> dedicated thread to do the sorting & updating, or actually perform sorting
> after current result have been used several times (say sort once in every 100
> calls.)
> B. performing calculation on {{Resource}} instances is expensive, since at
> least 2 objects ({{ResourceImpl}} and its proto builder) is created each time
> (using 'immutable' apis). the overhead can be eliminated with a
> light-weighted implementation of Resource, which do not instantiate a builder
> until necessary, because most instances are used as intermediate result in
> scheduler instead of being exchanged via IPC. Also, {{createResource}} is
> using reflection, which can be replaced by a plain {{new}} (for scheduler
> usage only). furthermore, perhaps we could 'intern' resource to avoid
> allocation.
> C. other minor changes: such as move {{updateRootMetrics}} call to
> {{update}}, making root queue metrics eventual consistent (which may
> satisfies most of the needs). or introduce counters to {{getResourceUsage}}
> and make changing of resource incrementally instead of recalculate each time.
> With A and B, I was looking at 4 times improvement in a cluster with 2K nodes.
> Suggestions? Opinions?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[
https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412169#comment-15412169
]
Hadoop QA commented on YARN-5470:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The
patch generated 60 new + 39 unchanged - 46 fixed = 99 total (was 85) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 24s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822609/YARN-5470.addendum.1.patch
|
| JIRA Issue | YARN-5470 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3ee0e5717534 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 37d939a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/12673/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12673/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12673/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12673/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12673/console |
| Powered by | Apache Yetus
[jira] [Commented] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412163#comment-15412163 ] Daniel Templeton commented on YARN-5472: It would appear you are correct. Have you tested it to confirm? > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > Attachments: YARN-5472-v0.patch > > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" on Windows platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5472) WIN_MAX_PATH logic is off by one
[ https://issues.apache.org/jira/browse/YARN-5472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brook Zhou updated YARN-5472: - Attachment: YARN-5472-v0.patch > WIN_MAX_PATH logic is off by one > > > Key: YARN-5472 > URL: https://issues.apache.org/jira/browse/YARN-5472 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.6.0 > Environment: Windows >Reporter: Brook Zhou >Assignee: Brook Zhou >Priority: Minor > Attachments: YARN-5472-v0.patch > > > The following check is incorrect in DefaultContainerExecutor: > if (Shell.WINDOWS && sb.getWrapperScriptPath().toString().length() > > WIN_MAX_PATH) > should be >=, as the max path is defined as "D:\some 256-character path > string" on Windows platforms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412089#comment-15412089
]
Varun Saxena edited comment on YARN-4833 at 8/8/16 5:07 PM:
We can add AccessControlException in exceptiontopolicymap in RMProxy as well.
I do not think ACL related errors will change upon retrying.
was (Author: varun_saxena):
We can add AccessControlException in exceptiontopolicymap in RMProxy as well.
I do not see ACL related errors to change upon retrying.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
[jira] [Commented] (YARN-4910) Fix incomplete log info in ResourceLocalizationService
[
https://issues.apache.org/jira/browse/YARN-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412090#comment-15412090
]
Hudson commented on YARN-4910:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #10233 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10233/])
YARN-4910. Fix incomplete log info in ResourceLocalizationService (Jun
(varunsaxena: rev 37d939a453d1052878a2a1aab7588cbac48a824d)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ResourceLocalizationService.java
> Fix incomplete log info in ResourceLocalizationService
> --
>
> Key: YARN-4910
> URL: https://issues.apache.org/jira/browse/YARN-4910
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jun Gong
>Assignee: Jun Gong
>Priority: Trivial
> Fix For: 2.9.0
>
> Attachments: YARN-4910.01.patch
>
>
> When debugging, find a lot of incomplete log info from
> ResourceLocalizationService, it is a little confusing.
> {quote}
> 2016-03-30 22:47:29,703 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Writing credentials to the nmPrivate file
> /data6/yarnenv/local/nmPrivate/container_1456839788316_4159_01_04_37.tokens.
> Credentials list:
> {quote}
> The content of credentials list will only be printed for DEBUG log level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412089#comment-15412089
]
Varun Saxena commented on YARN-4833:
We can add AccessControlException in exceptiontopolicymap in RMProxy as well.
I do not see ACL related errors to change upon retrying.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at
[jira] [Reopened] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reopened YARN-5470: - > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.1.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412081#comment-15412081 ] Xuan Gong edited comment on YARN-5470 at 8/8/16 5:02 PM: - Thanks for the comments. [~vinodkv] Reopen the jira and attached an addendum patch to address the comments. was (Author: xgong): Thanks for the comments. [~vinodkv] Reopen the jiar and attached a addendum patch to address the comments. > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.1.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412081#comment-15412081 ] Xuan Gong commented on YARN-5470: - Thanks for the comments. [~vinodkv] Reopen the jiar and attached a addendum patch to address the comments. > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.1.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5470: Attachment: YARN-5470.addendum.1.patch > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.1.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-4833:
---
Attachment: YARN-4833.0001.patch
Attaching rebased patch.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch, YARN-4833.0001.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
> at
>
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412075#comment-15412075
]
Naganarasimha G R commented on YARN-4833:
-
Hi [~bibinchundatt],
Why not the first approach ??(should handle the AccessControlException in
RetryPolicy and should go to fallbackPolicy)?? which you had initially
mentioned ?,
And if its approach2 as per the patch i would suggest to handle it in
{{ClientRMService}} rather than {{RMAppManager}}
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Updated] (YARN-5470) Differentiate exactly match with regex in yarn log CLI
[ https://issues.apache.org/jira/browse/YARN-5470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-5470: Attachment: YARN-5470.addendum.patch > Differentiate exactly match with regex in yarn log CLI > -- > > Key: YARN-5470 > URL: https://issues.apache.org/jira/browse/YARN-5470 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0 > > Attachments: YARN-5470.1.patch, YARN-5470.2.patch, YARN-5470.3.patch, > YARN-5470.3.patch, YARN-5470.addendum.patch > > > Since YARN-5089, we support regular expression in YARN log CLI "-logFiles" > option. However, we should differentiate exactly match with regex match as > user could put something like "system.out" here which have different > semantics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R resolved YARN-5484. - Resolution: Duplicate Yes guys its duplicate and we were discussing the solution and planned to post the same > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5484 > URL: https://issues.apache.org/jira/browse/YARN-5484 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 only for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4910) Fix incomplete log info in ResourceLocalizationService
[
https://issues.apache.org/jira/browse/YARN-4910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412058#comment-15412058
]
Varun Saxena commented on YARN-4910:
LGTM. Committing it.
> Fix incomplete log info in ResourceLocalizationService
> --
>
> Key: YARN-4910
> URL: https://issues.apache.org/jira/browse/YARN-4910
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jun Gong
>Assignee: Jun Gong
>Priority: Trivial
> Attachments: YARN-4910.01.patch
>
>
> When debugging, find a lot of incomplete log info from
> ResourceLocalizationService, it is a little confusing.
> {quote}
> 2016-03-30 22:47:29,703 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> Writing credentials to the nmPrivate file
> /data6/yarnenv/local/nmPrivate/container_1456839788316_4159_01_04_37.tokens.
> Credentials list:
> {quote}
> The content of credentials list will only be printed for DEBUG log level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5483) Optimize RMAppAttempt#pullJustFinishedContainers
[
https://issues.apache.org/jira/browse/YARN-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412043#comment-15412043
]
Daniel Templeton commented on YARN-5483:
Looks good to me. Looks like the call to {{pullJustFinishedContainers()}}
wasn't clearing the list of just finished containers per the method contract.
Nice catch.
With these lines:
{code}
finishedContainersSentToAM.putIfAbsent(nodeId,
new ArrayList());
{code}
can you please use the diamond operator in the list constructor and merge them
back into one line?
> Optimize RMAppAttempt#pullJustFinishedContainers
>
>
> Key: YARN-5483
> URL: https://issues.apache.org/jira/browse/YARN-5483
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: sandflee
>Assignee: sandflee
> Attachments: YARN-5483.01.patch, jprofiler-cpu.png
>
>
> about 1000 app running on cluster, jprofiler found pullJustFinishedContainers
> cost too much cpu.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412021#comment-15412021
]
Sunil G commented on YARN-4833:
---
[~bibinchundatt] ,could you please rebase the patch.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
> at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
> at
>
[jira] [Commented] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412017#comment-15412017 ] Sunil G commented on YARN-5484: --- Yes it is. I think a patch is already available there. We can push that in. I will help to review the same. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5484 > URL: https://issues.apache.org/jira/browse/YARN-5484 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 only for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412013#comment-15412013 ] Daniel Templeton commented on YARN-5484: Looks like it to me. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5484 > URL: https://issues.apache.org/jira/browse/YARN-5484 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 only for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4833) For Queue AccessControlException client retries multiple times on both RM
[
https://issues.apache.org/jira/browse/YARN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411982#comment-15411982
]
Hadoop QA commented on YARN-4833:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} YARN-4833 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12801651/0001-YARN-4833.patch |
| JIRA Issue | YARN-4833 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12671/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> For Queue AccessControlException client retries multiple times on both RM
> -
>
> Key: YARN-4833
> URL: https://issues.apache.org/jira/browse/YARN-4833
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4833.patch
>
>
> Submit application to queue where ACL is enabled and submitted user is not
> having access. Client retries till failMaxattempt 10 times.
> {noformat}
> 16/03/18 10:01:06 INFO retry.RetryInvocationHandler: Exception while invoking
> submitApplication of class ApplicationClientProtocolPBClientImpl over rm1.
> Trying to fail over immediately.
> org.apache.hadoop.security.AccessControlException: User hdfs does not have
> permission to submit application_1458273884145_0001 to queue default
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:291)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:618)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:252)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:483)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2360)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2356)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2356)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateIOException(RPCUtil.java:80)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:119)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:272)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:257)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
> at com.sun.proxy.$Proxy23.submitApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:261)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:295)
> at
[jira] [Commented] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411977#comment-15411977 ] Bibin A Chundatt commented on YARN-5484: Dupe of YARN-4833? > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5484 > URL: https://issues.apache.org/jira/browse/YARN-5484 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 only for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2398) TestResourceTrackerOnHA crashes
[
https://issues.apache.org/jira/browse/YARN-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411970#comment-15411970
]
Hadoop QA commented on YARN-2398:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 27s {color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestYarnClient |
| | hadoop.yarn.client.TestApplicationClientProtocolOnHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822588/YARN-2398.patch |
| JIRA Issue | YARN-2398 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d3911a1b19c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4d3af47 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/12670/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-YARN-Build/12670/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/12670/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/12670/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestResourceTrackerOnHA crashes
> ---
>
> Key: YARN-2398
> URL:
[jira] [Updated] (YARN-5482) ContainerMetric Lead to memory leaks
[
https://issues.apache.org/jira/browse/YARN-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-5482:
-
Assignee: tangshangwen (was: Jason Lowe)
This is really more of a duplicate of YARN-5341 since this was reported against
2.7. It's fixed in 2.7.4. Backporting YARN-5296 could also fix the issue but
causes other problems, hence why we fixed it specifically for 2.7 in YAN-5341.
> ContainerMetric Lead to memory leaks
>
>
> Key: YARN-5482
> URL: https://issues.apache.org/jira/browse/YARN-5482
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: tangshangwen
> Attachments: oom1.png, oom2.png
>
>
> In our cluster, I often find NodeManager OOM, I dump the heap file and found
> ContainerMetric takes up a lot of memory
> {code}
> export YARN_NODEMANAGER_OPTS="-Xmx2g -Xms2g -Xmn1g -XX:PermSize=128M
> -XX:MaxPermSize=128M -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/data1/yarn-logs/nm_dump.log -Dcom.sun.management.jmxremote
> -Xloggc:/data1/yarn-logs/nm_gc.log -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCApplicationConcurrentTime -XX:+PrintTenuringDistribution
> -XX:ErrorFile=/data1/yarn-logs/nm_err_pid"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5482) ContainerMetric Lead to memory leaks
[
https://issues.apache.org/jira/browse/YARN-5482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned YARN-5482:
Assignee: Jason Lowe (was: tangshangwen)
> ContainerMetric Lead to memory leaks
>
>
> Key: YARN-5482
> URL: https://issues.apache.org/jira/browse/YARN-5482
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: tangshangwen
>Assignee: Jason Lowe
> Attachments: oom1.png, oom2.png
>
>
> In our cluster, I often find NodeManager OOM, I dump the heap file and found
> ContainerMetric takes up a lot of memory
> {code}
> export YARN_NODEMANAGER_OPTS="-Xmx2g -Xms2g -Xmn1g -XX:PermSize=128M
> -XX:MaxPermSize=128M -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/data1/yarn-logs/nm_dump.log -Dcom.sun.management.jmxremote
> -Xloggc:/data1/yarn-logs/nm_gc.log -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime
> -XX:+PrintGCApplicationConcurrentTime -XX:+PrintTenuringDistribution
> -XX:ErrorFile=/data1/yarn-logs/nm_err_pid"
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5483) Optimize RMAppAttempt#pullJustFinishedContainers
[ https://issues.apache.org/jira/browse/YARN-5483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411939#comment-15411939 ] Jason Lowe commented on YARN-5483: -- Thanks for the report and patch, [~sandflee]! +1, patch looks good to me. I will commit this tomorrow if there are no objections. > Optimize RMAppAttempt#pullJustFinishedContainers > > > Key: YARN-5483 > URL: https://issues.apache.org/jira/browse/YARN-5483 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: sandflee >Assignee: sandflee > Attachments: YARN-5483.01.patch, jprofiler-cpu.png > > > about 1000 app running on cluster, jprofiler found pullJustFinishedContainers > cost too much cpu. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2398) TestResourceTrackerOnHA crashes
[ https://issues.apache.org/jira/browse/YARN-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S updated YARN-2398: -- Attachment: YARN-2398.patch Attached with modification. Please review > TestResourceTrackerOnHA crashes > --- > > Key: YARN-2398 > URL: https://issues.apache.org/jira/browse/YARN-2398 > Project: Hadoop YARN > Issue Type: Test >Reporter: Jason Lowe >Assignee: Ajith S > Attachments: YARN-2398.patch > > > TestResourceTrackerOnHA is currently crashing and failing trunk builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5430) Get container's ip and host from NM
[
https://issues.apache.org/jira/browse/YARN-5430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411871#comment-15411871
]
Hadoop QA commented on YARN-5430:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 8
new + 166 unchanged - 2 fixed = 174 total (was 168) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 15s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 4s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.logaggregation.TestAggregatedLogFormat |
| |
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.TestDockerInspectCommand
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12822509/YARN-5430.5.patch |
| JIRA Issue | YARN-5430 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 618ff66f4513 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build
[jira] [Updated] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5484: --- Description: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 only for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5484 > URL: https://issues.apache.org/jira/browse/YARN-5484 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 only for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5484) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[
https://issues.apache.org/jira/browse/YARN-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411815#comment-15411815
]
Bob.zhao commented on YARN-5484:
I think just like other place on yarn where process AccessControlException to
YarnException , here we can warp AccessControlException to YarnException :
{code}
try {
} catch (AccessControlException e) {
throw new YarnException(e);
}
{code}
Any ideas?
> YARN client will still retry for many times on failover even though RM
> server throw AccessControlException
> ---
>
> Key: YARN-5484
> URL: https://issues.apache.org/jira/browse/YARN-5484
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0
>Reporter: Bob.zhao
>
> 1. Enable yarn.acl.enable
> 2. Set up some queue for users on yarn, for example queue1 onle for user1
> 3. Using user1 to submit app to queue1, it can run successfully.
> 4. Change user2 to submit app to queue1, it will not be permit submit to
> queue1.
> So, At RM server side, it will throw
> IOException(ClientRMService.java#submitApplication), which is the parent
> implementation of the
> AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this
> IOException will be threw to client that cause yarn client frequently
> failover for many times.
> We'b better avoid this behavior that if client got permission deny from
> server, it should try once and exit, no need to retry.
> This issue was introduced by YARN-4522.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException)
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411802#comment-15411802 ] Bob.zhao commented on YARN-5485: rt. sorry to make a mistake for that. let track this issue on YARN-5484. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException(IOException) > > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException)
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G resolved YARN-5485. --- Resolution: Duplicate Dup of YARN-5484 ? > YARN client will still retry for many times on failover even though RM > server throw AccessControlException(IOException) > > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2398) TestResourceTrackerOnHA crashes
[
https://issues.apache.org/jira/browse/YARN-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411790#comment-15411790
]
Ajith S commented on YARN-2398:
---
I encountered with same scenario. I guess the bug is because of
{{org.apache.hadoop.yarn.client.ProtocolHATestBase.MiniYARNClusterForHATesting.waittingForFailOver()}}
{code}
private boolean waittingForFailOver() {
int maximumWaittingTime = 50;
int count = 0;
while (!failoverTriggered.get() && count >= maximumWaittingTime) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
// DO NOTHING
}
count++;
}
...
{code}
here it should be {{count <= maximumWaittingTime}}
As otherwise, the while loop will be exit at first check itself, and there is
no actual wait hence it causes racecondition between failover and registerNM
event
> TestResourceTrackerOnHA crashes
> ---
>
> Key: YARN-2398
> URL: https://issues.apache.org/jira/browse/YARN-2398
> Project: Hadoop YARN
> Issue Type: Test
>Reporter: Jason Lowe
>Assignee: Ajith S
>
> TestResourceTrackerOnHA is currently crashing and failing trunk builds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException)
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Summary: YARN client will still retry for many times on failover even though RM server throw AccessControlException(IOException) (was: YARN client will still retry for many times on failover even though RM server throw AccessControlException) > YARN client will still retry for many times on failover even though RM > server throw AccessControlException(IOException) > > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Description: Issue reproduced steps 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > Issue reproduced steps > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-2398) TestResourceTrackerOnHA crashes
[ https://issues.apache.org/jira/browse/YARN-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S reopened YARN-2398: --- > TestResourceTrackerOnHA crashes > --- > > Key: YARN-2398 > URL: https://issues.apache.org/jira/browse/YARN-2398 > Project: Hadoop YARN > Issue Type: Test >Reporter: Jason Lowe >Assignee: Ajith S > > TestResourceTrackerOnHA is currently crashing and failing trunk builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-2398) TestResourceTrackerOnHA crashes
[ https://issues.apache.org/jira/browse/YARN-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S reassigned YARN-2398: - Assignee: Ajith S > TestResourceTrackerOnHA crashes > --- > > Key: YARN-2398 > URL: https://issues.apache.org/jira/browse/YARN-2398 > Project: Hadoop YARN > Issue Type: Test >Reporter: Jason Lowe >Assignee: Ajith S > > TestResourceTrackerOnHA is currently crashing and failing trunk builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5485) YARN client will still retry for many times on failover even though RM server throw AccessControlException
[ https://issues.apache.org/jira/browse/YARN-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob.zhao updated YARN-5485: --- Description: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. was: 1. Enable yarn.acl.enable 2. Set up some queue for users on yarn, for example queue1 onle for user1 3. Using user1 to submit app to queue1, it can run successfully. 4. Change user2 to submit app to queue1, it will not be permit submit to queue1. So, At RM server side, it will throw IOException(ClientRMService.java#submitApplication), which is the parent implementation of the AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this IOException will be threw to client that cause yarn client frequently failover for many times. We'b better avoid this behavior that if client got permission deny from server, it should try once and exit, no need to retry. This issue was introduced by YARN-4522. > YARN client will still retry for many times on failover even though RM > server throw AccessControlException > --- > > Key: YARN-5485 > URL: https://issues.apache.org/jira/browse/YARN-5485 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0 >Reporter: Bob.zhao > > 1. Enable yarn.acl.enable > 2. Set up some queue for users on yarn, for example queue1 onle for user1 > 3. Using user1 to submit app to queue1, it can run successfully. > 4. Change user2 to submit app to queue1, it will not be permit submit to > queue1. > So, At RM server side, it will throw > IOException(ClientRMService.java#submitApplication), which is the parent > implementation of the > AccessControlException(RMAppManager.java#createAndPopulateNewRMApp), this > IOException will be threw to client that cause yarn client frequently > failover for many times. > We'b better avoid this behavior that if client got permission deny from > server, it should try once and exit, no need to retry. > This issue was introduced by YARN-4522. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
