[jira] [Updated] (YARN-6184) [YARN-3368] Introduce loading icon in each page of YARN-UI
[ https://issues.apache.org/jira/browse/YARN-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-6184: --- Attachment: YARN-6184.004.patch > [YARN-3368] Introduce loading icon in each page of YARN-UI > -- > > Key: YARN-6184 > URL: https://issues.apache.org/jira/browse/YARN-6184 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akhil PB >Assignee: Akhil PB > Attachments: YARN-6184.001.patch, YARN-6184.002.patch, > YARN-6184.003.patch, YARN-6184.004.patch > > > Add loading icon in each page in new YARN-UI. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6159) Documentation changes for TimelineV2Client
[
https://issues.apache.org/jira/browse/YARN-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871323#comment-15871323
]
Naganarasimha G R commented on YARN-6159:
-
[~varun_saxena], now that YARN-4675 is in i think we can get this too !
> Documentation changes for TimelineV2Client
> --
>
> Key: YARN-6159
> URL: https://issues.apache.org/jira/browse/YARN-6159
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Varun Saxena
>Assignee: Naganarasimha G R
>Priority: Minor
> Attachments: TimelineServiceV2.html, YARN-6159.v1.001.patch,
> YARN-6159.v1.002.patch
>
>
> Make documentation changes for TimelineV2Client i.e. to reflect changes made
> in client API in YARN-4675.
> Also in TimelineServiceV2.md, under section Publishing application specific
> data, we have the following code snippet. Here,
> {{timelineClient.putEntitiesAsync(entity);}} should be
> {{client.putEntitiesAsync(entity);}} instead.
> {code}
> // Create and start the Timeline client v.2
> TimelineClient client = TimelineClient.createTimelineClient(appId);
> client.init(conf);
> client.start();
> try {
> TimelineEntity myEntity = new TimelineEntity();
> myEntity.setEntityType("MY_APPLICATION");
> myEntity.setEntityId("MyApp1")
> // Compose other entity info
> // Blocking write
> client.putEntities(entity);
> TimelineEntity myEntity2 = new TimelineEntity();
> // Compose other info
> // Non-blocking write
> timelineClient.putEntitiesAsync(entity);
> } catch (IOException e) {
> // Handle the exception
> } catch (RuntimeException e) {
> {code}
> Below can also be changed to client to keep it consistent.
> {code}
> amRMClient.registerTimelineClient(timelineClient)Íž
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871322#comment-15871322 ] Naganarasimha G R commented on YARN-4675: - Thanks for the review and commit [~sjlee0] and reviews from [~varun_saxena] and [~gtCarrera9]. > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 3.0.0-alpha3 > > Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, > YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, > YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, > YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, > YARN-4675-YARN-2928.v1.001.patch, YARN-4675-YARN-5355.v2.011.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871308#comment-15871308
]
Naganarasimha G R commented on YARN-6207:
-
bq. Considering this is unlikely to happen unless we are doing mass movement of
apps on a busy cluster, this may be fine.
Need not be, suppose during RM transition "Move app" is done then user might
not be aware that RM transition was happening in the backend. Other approach
what me and [~bibinchundatt] were thinking of was to send SchedulerEvent for
handling this move, so that it gets queued up in the SchedulerEventDispatcher
and not get immediately invoked. We will further check the pro's and cons of
the same !
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4520) FinishAppEvent is leaked in leveldb if no app's container running on this node
[ https://issues.apache.org/jira/browse/YARN-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871306#comment-15871306 ] Akira Ajisaka commented on YARN-4520: - Thanks [~sandflee] and [~jianhe] for the contribution. Filed YARN-6208 for improving the log. > FinishAppEvent is leaked in leveldb if no app's container running on this node > -- > > Key: YARN-4520 > URL: https://issues.apache.org/jira/browse/YARN-4520 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: sandflee >Assignee: sandflee > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-4520.01.patch, YARN-4520.02.patch > > > once we restart nodemanager we see many logs like : > 2015-12-28 11:59:18,725 WARN > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: > Event EventType: FINISH_APPLICATION sent to absent application > application_1446103803043_9892 > we find that the app containers are never started on NM and is released by AM > after allocated. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6208) Improve the log when FinishAppEvent sent to the NodeManager which didn't run the application
Akira Ajisaka created YARN-6208:
---
Summary: Improve the log when FinishAppEvent sent to the
NodeManager which didn't run the application
Key: YARN-6208
URL: https://issues.apache.org/jira/browse/YARN-6208
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Akira Ajisaka
Priority: Minor
When FinishAppEvent of an application is sent to a NodeManager and there are no
applications of the application ran on the NodeManager, we can see the
following log:
{code}
2015-12-28 11:59:18,725 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: FINISH_APPLICATION sent to absent application
application_1446103803043_9892
{code}
YARN-4520 made the log as follows:
{code}
LOG.warn("couldn't find application " + appID + " while processing"
+ " FINISH_APPS event");
{code}
and I'm thinking it can be improved.
* Make the log WARN from INFO
* Add why the NodeManager couldn't find the application. For example, "because
there were no containers of the application ran on the NodeManager."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5501) Container Pooling in YARN
[ https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Sharma updated YARN-5501: Attachment: Container Pooling in YARN.pdf > Container Pooling in YARN > - > > Key: YARN-5501 > URL: https://issues.apache.org/jira/browse/YARN-5501 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Attachments: Container Pooling in YARN.pdf, Container Pooling - one > pager.pdf > > > This JIRA proposes a method for reducing the container launch latency in > YARN. It introduces a notion of pooling *Unattached Pre-Initialized > Containers*. > Proposal in brief: > * Have a *Pre-Initialized Container Factory* service within the NM to create > these unattached containers. > * The NM would then advertise these containers as special resource types > (this should be possible via YARN-3926). > * When a start container request is received by the node manager for > launching a container requesting this specific type of resource, it will take > one of these unattached pre-initialized containers from the pool, and use it > to service the container request. > * Once the request is complete, the pre-initialized container would be > released and ready to serve another request. > This capability would help reduce container launch latencies and thereby > allow for development of more interactive applications on YARN. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5501) Container Pooling in YARN
[ https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871293#comment-15871293 ] Hitesh Sharma commented on YARN-5501: - Hi [~jlowe], thanks for the feedback. I have captured some of the discussion in the attached document. [~arun suresh], [~vvasudev], please have look and share your thoughts. Look forward to the discussion. > Container Pooling in YARN > - > > Key: YARN-5501 > URL: https://issues.apache.org/jira/browse/YARN-5501 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Attachments: Container Pooling in YARN.pdf, Container Pooling - one > pager.pdf > > > This JIRA proposes a method for reducing the container launch latency in > YARN. It introduces a notion of pooling *Unattached Pre-Initialized > Containers*. > Proposal in brief: > * Have a *Pre-Initialized Container Factory* service within the NM to create > these unattached containers. > * The NM would then advertise these containers as special resource types > (this should be possible via YARN-3926). > * When a start container request is received by the node manager for > launching a container requesting this specific type of resource, it will take > one of these unattached pre-initialized containers from the pool, and use it > to service the container request. > * Once the request is complete, the pre-initialized container would be > released and ready to serve another request. > This capability would help reduce container launch latencies and thereby > allow for development of more interactive applications on YARN. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5501) Container Pooling in YARN
[ https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871293#comment-15871293 ] Hitesh Sharma edited comment on YARN-5501 at 2/17/17 6:50 AM: -- Hi [~jlowe], thanks for the feedback. I have captured some of the discussion in the attached document. [~asuresh]], [~vvasudev], please have look and share your thoughts. Look forward to the discussion. was (Author: hrsharma): Hi [~jlowe], thanks for the feedback. I have captured some of the discussion in the attached document. [~arun suresh], [~vvasudev], please have look and share your thoughts. Look forward to the discussion. > Container Pooling in YARN > - > > Key: YARN-5501 > URL: https://issues.apache.org/jira/browse/YARN-5501 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Attachments: Container Pooling in YARN.pdf, Container Pooling - one > pager.pdf > > > This JIRA proposes a method for reducing the container launch latency in > YARN. It introduces a notion of pooling *Unattached Pre-Initialized > Containers*. > Proposal in brief: > * Have a *Pre-Initialized Container Factory* service within the NM to create > these unattached containers. > * The NM would then advertise these containers as special resource types > (this should be possible via YARN-3926). > * When a start container request is received by the node manager for > launching a container requesting this specific type of resource, it will take > one of these unattached pre-initialized containers from the pool, and use it > to service the container request. > * Once the request is complete, the pre-initialized container would be > released and ready to serve another request. > This capability would help reduce container launch latencies and thereby > allow for development of more interactive applications on YARN. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871257#comment-15871257
]
Varun Saxena commented on YARN-6207:
We have 2 options here.
Either send a more descriptive message and related exception back to the
client. Considering this is unlikely to happen unless we are doing mass
movement of apps on a busy cluster, this may be fine.
Or, wait for a while to let the attempt be added to be added to scheduler and
then throw an exception.
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6207) Move application can fail when attempt add event is delayed
[
https://issues.apache.org/jira/browse/YARN-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-6207:
---
Summary: Move application can fail when attempt add event is delayed
(was: Move application can fail when attempt add is delayed)
> Move application can fail when attempt add event is delayed
>
>
> Key: YARN-6207
> URL: https://issues.apache.org/jira/browse/YARN-6207
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> *Steps to reproduce*
> 1.Submit application and delay attempt add to Scheduler
> (Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
> 2. Call move application to destination queue.
> {noformat}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at org.apache.hadoop.ipc.Client.call(Client.java:1339)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
> at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6207) Move application can fail when attempt add is delayed
Bibin A Chundatt created YARN-6207:
--
Summary: Move application can fail when attempt add is delayed
Key: YARN-6207
URL: https://issues.apache.org/jira/browse/YARN-6207
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
*Steps to reproduce*
1.Submit application and delay attempt add to Scheduler
(Simulate using debug at EventDispatcher for SchedulerEventDispatcher)
2. Call move application to destination queue.
{noformat}
Caused by:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.preValidateMoveApplication(CapacityScheduler.java:2086)
at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.moveApplicationAcrossQueue(RMAppManager.java:669)
at
org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.moveApplicationAcrossQueues(ClientRMService.java:1231)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBServiceImpl.java:388)
at
org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:537)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:522)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:867)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:813)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1892)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2659)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
at org.apache.hadoop.ipc.Client.call(Client.java:1429)
at org.apache.hadoop.ipc.Client.call(Client.java:1339)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:115)
at com.sun.proxy.$Proxy7.moveApplicationAcrossQueues(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.moveApplicationAcrossQueues(ApplicationClientProtocolPBClientImpl.java:398)
... 16 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871131#comment-15871131
]
Yuanbo Liu commented on YARN-1728:
--
Would like to take it over, and see why the path is not decoded. We've met such
kind of defect in hadoop-2.7.3.
> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
>
> For example, going to the job history server page
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId:
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent
> encoding.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-1728) History server doesn't understand percent encoded paths
[
https://issues.apache.org/jira/browse/YARN-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu reassigned YARN-1728:
Assignee: Yuanbo Liu
> History server doesn't understand percent encoded paths
> ---
>
> Key: YARN-1728
> URL: https://issues.apache.org/jira/browse/YARN-1728
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Abraham Elmahrek
>Assignee: Yuanbo Liu
>
> For example, going to the job history server page
> http://localhost:19888/jobhistory/logs/localhost%3A8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> results in the following error:
> {code}
> Cannot get container logs. Invalid nodeId:
> test-cdh5-hue.ent.cloudera.com%3A8041
> {code}
> Where the url decoded version works:
> http://localhost:19888/jobhistory/logs/localhost:8041/container_1391466602060_0011_01_01/job_1391466602060_0011/admin/stderr
> It seems like both should be supported as the former is simply percent
> encoding.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6206) RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object
[
https://issues.apache.org/jira/browse/YARN-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871126#comment-15871126
]
Hadoop QA commented on YARN-6206:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
44s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 61 new + 0 unchanged - 0 fixed = 61 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
16s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
42s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
53s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.CancelDelegationTokenRequestPBImpl.proto;
locked 66% of time Unsynchronized access at
CancelDelegationTokenRequestPBImpl.java:66% of time Unsynchronized access at
CancelDelegationTokenRequestPBImpl.java:[line 114] |
| | Inconsistent synchronization of
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.CancelDelegationTokenRequestPBImpl.viaProto;
locked 57% of time Unsynchronized access at
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871125#comment-15871125 ] Rohith Sharma K S commented on YARN-6027: - bq. we're exposing a lot of immediate values in the parsing process of FlowActivityEntityReader Yeah thats true.. while coding I also felt it is huge immense part doing in flowActivityEntityReader. Given FLOW_UID is changed as I commented previously, use FLOW_UID as fromId then these parsings can be avoided from entity reader. > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6027-YARN-5355.0001.patch, > YARN-6027-YARN-5355.0002.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6027) Support fromid(offset) filter for /flows API
[
https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871122#comment-15871122
]
Rohith Sharma K S commented on YARN-6027:
-
bq. On a related note, how would the client provide the value for fromId? Does
it need to compose the fromId, or can we add a UID for flow activity in the
info and they can just grab and send it as fromId? I thought we were doing
something like that for other pagination? If it is the latter, then can we not
use FLOW_UID for this?
To the flowActivity, fromId can NOT be single value since these are tracked by
daily basis!! FromId which user provides has to be complete row key. I wanted
to used flow entity id itself as fromId because user can provide just pick the
flow entity id and send it as fromId. Note that flow entity id has complete row
key values. But flow entity id is not escaped as flow_UID is done. FLOW_UID
does not have time values in it e.g {{"UID": "yarn-cluster!rohithsharmaks!Sleep
job"}}. May be FLOW_UID itself can be changed to add timestamp and use it as
fromId.
> Support fromid(offset) filter for /flows API
>
>
> Key: YARN-6027
> URL: https://issues.apache.org/jira/browse/YARN-6027
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Labels: yarn-5355-merge-blocker
> Attachments: YARN-6027-YARN-5355.0001.patch,
> YARN-6027-YARN-5355.0002.patch
>
>
> In YARN-5585 , fromId is supported for retrieving entities. We need similar
> filter for flows/flowRun apps and flow run and flow as well.
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates
> found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6176) Test whether FS preemption consider child queues over fair share if the parent is under fair share
[
https://issues.apache.org/jira/browse/YARN-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871101#comment-15871101
]
Hadoop QA commented on YARN-6176:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 49s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
| | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6176 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853185/YARN-6176.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3fbe7ad21ff0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 658702e |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14996/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14996/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14996/console |
| Powered by | Apache Yetus
[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871096#comment-15871096 ] Sangjin Lee commented on YARN-4675: --- Thanks Naga. Committed it to YARN-5355 and YARN-5355-branch-2. > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 3.0.0-alpha3 > > Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, > YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, > YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, > YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, > YARN-4675-YARN-2928.v1.001.patch, YARN-4675-YARN-5355.v2.011.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5829) FS preemption should reserve a node before considering containers on it for preemption
[ https://issues.apache.org/jira/browse/YARN-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871054#comment-15871054 ] Tao Jie commented on YARN-5829: --- [~kasha], it seems similar situation as mentioned in YARN-5636. Should we have a common mechanism that supports "reserving certain resource on a certain node for a certain app for a while" ? > FS preemption should reserve a node before considering containers on it for > preemption > -- > > Key: YARN-5829 > URL: https://issues.apache.org/jira/browse/YARN-5829 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > FS preemption evaluates nodes for preemption, and subsequently preempts > identified containers. If this node is not reserved for a specific > application, any other application could be allocated resources on this node. > Reserving the node for the starved application before preempting containers > would help avoid this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871039#comment-15871039 ] Ajay Jadhav commented on YARN-6188: --- Test failure doesn't look related to patch. > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch, > YARN-6188.003.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[
https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871036#comment-15871036
]
Hadoop QA commented on YARN-6188:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6188 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853174/YARN-6188.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f198e1392a97 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / bdad8b7 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14994/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14994/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14994/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix OOM issue with decommissioningNodesWatcher in the case of clusters
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871019#comment-15871019 ] Weiwei Yang commented on YARN-6177: --- Thanks [~gtCarrera9], [~jlowe] and [~jojochuang] for all the comments and suggestions! > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Fix For: 2.8.0, 2.9.0, 3.0.0-alpha3 > > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch, YARN-6177.06.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6176) Test whether FS preemption consider child queues over fair share if the parent is under fair share
[ https://issues.apache.org/jira/browse/YARN-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6176: --- Component/s: fairscheduler > Test whether FS preemption consider child queues over fair share if the > parent is under fair share > -- > > Key: YARN-6176 > URL: https://issues.apache.org/jira/browse/YARN-6176 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6176.001.patch > > > Port the test case in YARN-6151 to trunk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6176) Test whether FS preemption consider child queues over fair share if the parent is under fair share
[ https://issues.apache.org/jira/browse/YARN-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6176: --- Affects Version/s: 2.9.0 3.0.0-alpha2 > Test whether FS preemption consider child queues over fair share if the > parent is under fair share > -- > > Key: YARN-6176 > URL: https://issues.apache.org/jira/browse/YARN-6176 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6176.001.patch > > > Port the test case in YARN-6151 to trunk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6176) Test whether FS preemption consider child queues over fair share if the parent is under fair share
[ https://issues.apache.org/jira/browse/YARN-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6176: --- Attachment: YARN-6176.001.patch > Test whether FS preemption consider child queues over fair share if the > parent is under fair share > -- > > Key: YARN-6176 > URL: https://issues.apache.org/jira/browse/YARN-6176 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6176.001.patch > > > Port the test case in YARN-6151 to trunk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6206) RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object
[
https://issues.apache.org/jira/browse/YARN-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-6206:
--
Attachment: YARN-6206.1.patch
uploaded a patch to fix the issue.
Added synchronized to the getProto of all the request PB object for the
connection to RM
also, add few log messages
> RequestHedgingRMFailoverProvider may have concurrency issues when accessing
> the same request object
> ---
>
> Key: YARN-6206
> URL: https://issues.apache.org/jira/browse/YARN-6206
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-6206.1.patch
>
>
> Here, since multiple threads are accessing the same request object - 'args',
> and the request protobuf implementation is not synchronized. That caused
> errors like below:
> {code}
>Callable c = new Callable() {
> @Override
> public Object call() throws Exception {
> return method.invoke(pInfo.proxy, args);
> }
> };
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.ArrayList.elementData(ArrayList.java:418)
> at java.util.ArrayList.get(ArrayList.java:431)
> at
> com.google.protobuf.LazyStringArrayList.getByteString(LazyStringArrayList.java:157)
> at
> com.google.protobuf.UnmodifiableLazyStringList.getByteString(UnmodifiableLazyStringList.java:66)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationsRequestProto.getSerializedSize(YarnServiceProtos.java:17070)
> at
> com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.write(ProtobufRpcEngine.java:356)
> at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1059)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1398)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy17.getApplications(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:251)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:282)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
> at com.sun.proxy.$Proxy18.getApplications(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870996#comment-15870996
]
Hadoop QA commented on YARN-6175:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
13s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
48s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 111 unchanged - 3 fixed = 111 total (was 114) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests |
[jira] [Updated] (YARN-6176) Test whether FS preemption consider child queues over fair share if the parent is under fair share
[ https://issues.apache.org/jira/browse/YARN-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6176: --- Summary: Test whether FS preemption consider child queues over fair share if the parent is under fair share (was: Test whether FS preemption consider child queues over fair share if the parent is under fairshare) > Test whether FS preemption consider child queues over fair share if the > parent is under fair share > -- > > Key: YARN-6176 > URL: https://issues.apache.org/jira/browse/YARN-6176 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Yufei Gu >Assignee: Yufei Gu > > Port the test case in YARN-6151 to trunk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6176) Test whether FS preemption consider child queues over fair share if the parent is under fairshare
[ https://issues.apache.org/jira/browse/YARN-6176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6176: --- Summary: Test whether FS preemption consider child queues over fair share if the parent is under fairshare (was: Test whether FS preemption consider child queues over fair share if the parent is under) > Test whether FS preemption consider child queues over fair share if the > parent is under fairshare > - > > Key: YARN-6176 > URL: https://issues.apache.org/jira/browse/YARN-6176 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Yufei Gu >Assignee: Yufei Gu > > Port the test case in YARN-6151 to trunk. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870971#comment-15870971 ] Ajay Jadhav commented on YARN-6188: --- Ok. Thanks. > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch, > YARN-6188.003.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5703) ReservationAgents are not correctly configured
[ https://issues.apache.org/jira/browse/YARN-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870966#comment-15870966 ] Naganarasimha G R commented on YARN-5703: - Thanks [~maniraj...@gmail.com], your latest patch looks good to me (whitespace issue can be any way handed while committing), will wait for couple of days before committing. Hope [~seanpo03] could take a look too. > ReservationAgents are not correctly configured > -- > > Key: YARN-5703 > URL: https://issues.apache.org/jira/browse/YARN-5703 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Sean Po >Assignee: Manikandan R > Attachments: YARN-5703.001.patch, YARN-5703.002.patch, > YARN-5703.003.patch, YARN-5703.004.patch, YARN-5703.005.patch, > YARN-5703.006.patch, YARN-5703.007.patch, YARN-5703.008.patch > > > In AbstractReservationSystem, the method that instantiates a ReservationAgent > does not properly initialize it with the appropriate configuration because it > expects the ReservationAgent to implement Configurable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6206) RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object
[
https://issues.apache.org/jira/browse/YARN-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jian He updated YARN-6206:
--
Description:
Here, since multiple threads are accessing the same request object - 'args',
and the request protobuf implementation is not synchronized. That caused errors
like below:
{code}
Callable c = new Callable() {
@Override
public Object call() throws Exception {
return method.invoke(pInfo.proxy, args);
}
};
{code}
{code}
java.lang.ArrayIndexOutOfBoundsException: 1
at java.util.ArrayList.elementData(ArrayList.java:418)
at java.util.ArrayList.get(ArrayList.java:431)
at
com.google.protobuf.LazyStringArrayList.getByteString(LazyStringArrayList.java:157)
at
com.google.protobuf.UnmodifiableLazyStringList.getByteString(UnmodifiableLazyStringList.java:66)
at
org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationsRequestProto.getSerializedSize(YarnServiceProtos.java:17070)
at
com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.write(ProtobufRpcEngine.java:356)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1059)
at org.apache.hadoop.ipc.Client.call(Client.java:1457)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy17.getApplications(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:282)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy18.getApplications(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> RequestHedgingRMFailoverProvider may have concurrency issues when accessing
> the same request object
> ---
>
> Key: YARN-6206
> URL: https://issues.apache.org/jira/browse/YARN-6206
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
>
> Here, since multiple threads are accessing the same request object - 'args',
> and the request protobuf implementation is not synchronized. That caused
> errors like below:
> {code}
>Callable c = new Callable() {
> @Override
> public Object call() throws Exception {
> return method.invoke(pInfo.proxy, args);
> }
> };
> {code}
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at java.util.ArrayList.elementData(ArrayList.java:418)
> at java.util.ArrayList.get(ArrayList.java:431)
> at
> com.google.protobuf.LazyStringArrayList.getByteString(LazyStringArrayList.java:157)
> at
> com.google.protobuf.UnmodifiableLazyStringList.getByteString(UnmodifiableLazyStringList.java:66)
> at
> org.apache.hadoop.yarn.proto.YarnServiceProtos$GetApplicationsRequestProto.getSerializedSize(YarnServiceProtos.java:17070)
> at
> com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcMessageWithHeader.write(ProtobufRpcEngine.java:356)
> at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1059)
> at org.apache.hadoop.ipc.Client.call(Client.java:1457)
> at org.apache.hadoop.ipc.Client.call(Client.java:1398)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at com.sun.proxy.$Proxy17.getApplications(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:251)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Updated] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-4675: Attachment: YARN-4675-YARN-5355.v2.011.patch [~sjlee0], Attaching the file for YARN-5355 branch, hope you could take a quick look and feedback. > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 3.0.0-alpha3 > > Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, > YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, > YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, > YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, > YARN-4675-YARN-2928.v1.001.patch, YARN-4675-YARN-5355.v2.011.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870957#comment-15870957 ] Daniel Templeton commented on YARN-6188: LGTM +1 I'll commit tomorrow in case there are any last-minute comments. > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch, > YARN-6188.003.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton reassigned YARN-6188: -- Assignee: Ajay Jadhav > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch, > YARN-6188.003.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6206) RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object
Jian He created YARN-6206: - Summary: RequestHedgingRMFailoverProvider may have concurrency issues when accessing the same request object Key: YARN-6206 URL: https://issues.apache.org/jira/browse/YARN-6206 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated YARN-6188: -- Attachment: YARN-6188.003.patch > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch, > YARN-6188.003.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870950#comment-15870950 ] Ajay Jadhav commented on YARN-6188: --- Yes, it makes sense. I'm updating the patch. > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870933#comment-15870933
]
Hadoop QA commented on YARN-6193:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m
24s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6193 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853150/YARN-6193.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4547cd9b6ea8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f432999 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14992/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14992/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch,
>
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870932#comment-15870932 ] Daniel Templeton commented on YARN-6188: Right. We only run the method if we're logging at a debug level, but the method logs at the info level. It should be logging at debug. > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6143) Fix incompatible issue caused by YARN-3583
[
https://issues.apache.org/jira/browse/YARN-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870925#comment-15870925
]
Hadoop QA commented on YARN-6143:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
12s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 51s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 14s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 35s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 5s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 2m
23s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}208m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Private method
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.GetLabelsToNodesResponsePBImpl.convertToProtoFormat(NodeLabel)
is never called At GetLabelsToNodesResponsePBImpl.java:called At
GetLabelsToNodesResponsePBImpl.java:[line 157] |
| Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
| | hadoop.yarn.server.resourcemanager.TestClientRMService |
| | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector |
| Timed out junit tests |
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[
https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870920#comment-15870920
]
Ajay Jadhav commented on YARN-6188:
---
I'm not sure what you want me to fix. Currently, the logs are turned on only
during debug mode.
if (!LOG.isDebugEnabled() || decomNodes.size() == 0) {
return;
}
> Fix OOM issue with decommissioningNodesWatcher in the case of clusters with
> large number of nodes
> -
>
> Key: YARN-6188
> URL: https://issues.apache.org/jira/browse/YARN-6188
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Ajay Jadhav
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-6188.001.patch, YARN-6188.002.patch
>
>
> LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses
> StringBuilder to append status of all
> decommissioning nodes for logging purpose.
> In the case of large number of decommissioning nodes, this leads to OOM
> exception. The fix scopes StringBuilder so that in case of memory pressure,
> GC can kick in and free up the memory.
> This is supposed to fix a bug introduced in
> https://issues.apache.org/jira/browse/YARN-4676
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870913#comment-15870913 ] Daniel Templeton commented on YARN-6125: Thanks, [~andras.piros]! LGTM +1. I'll let this sit until tomorrow in case anyone else wants to pile on with comments. :) > The application attempt's diagnostic message should have a maximum size > --- > > Key: YARN-6125 > URL: https://issues.apache.org/jira/browse/YARN-6125 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.0 >Reporter: Daniel Templeton >Assignee: Andras Piros >Priority: Critical > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6125.000.patch, YARN-6125.001.patch, > YARN-6125.002.patch, YARN-6125.003.patch, YARN-6125.004.patch, > YARN-6125.005.patch, YARN-6125.006.patch, YARN-6125.007.patch, > YARN-6125.008.patch, YARN-6125.009.patch > > > We've found through experience that the diagnostic message can grow > unbounded. I've seen attempts that have diagnostic messages over 1MB. Since > the message is stored in the state store, it's a bad idea to allow the > message to grow unbounded. Instead, there should be a property that sets a > maximum size on the message. > I suspect that some of the ZK state store issues we've seen in the past were > due to the size of the diagnostic messages and not to the size of the > classpath, as is the current prevailing opinion. > An open question is how best to prune the message once it grows too large. > Should we > # truncate the tail, > # truncate the head, > # truncate the middle, > # add another property to make the behavior selectable, or > # none of the above? -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3926) Extend the YARN resource model for easier resource-type management and profiles
[ https://issues.apache.org/jira/browse/YARN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870906#comment-15870906 ] Arun Suresh commented on YARN-3926: --- [~vvasudev], can you maybe update the status on this.. I would really love to see this feature get into trunk.. The outstanding JIRA's IMHO do not seem to be a blocker to get the feature in. Happy to help pushing thru any remaining tasks. > Extend the YARN resource model for easier resource-type management and > profiles > --- > > Key: YARN-3926 > URL: https://issues.apache.org/jira/browse/YARN-3926 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: Proposal for modifying resource model and profiles.pdf > > > Currently, there are efforts to add support for various resource-types such > as disk(YARN-2139), network(YARN-2140), and HDFS bandwidth(YARN-2681). These > efforts all aim to add support for a new resource type and are fairly > involved efforts. In addition, once support is added, it becomes harder for > users to specify the resources they need. All existing jobs have to be > modified, or have to use the minimum allocation. > This ticket is a proposal to extend the YARN resource model to a more > flexible model which makes it easier to support additional resource-types. It > also considers the related aspect of âresource profilesâ which allow users to > easily specify the various resources they need for any given container. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870894#comment-15870894
]
Hadoop QA commented on YARN-6193:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 6s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6193 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853150/YARN-6193.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1ebefde79d93 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5d339c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14991/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14991/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14991/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL:
[jira] [Commented] (YARN-6042) Fairscheduler: Dump scheduler state in log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870884#comment-15870884 ] Yufei Gu commented on YARN-6042: Patch v7 has rebased, but PR doesn't. Update PR as well. > Fairscheduler: Dump scheduler state in log > -- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch, > YARN-6042.006.patch, YARN-6042.007.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6171) ConcurrentModificationException on FSAppAttempt.containersToPreempt
[
https://issues.apache.org/jira/browse/YARN-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870881#comment-15870881
]
Hudson commented on YARN-6171:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11268 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11268/])
YARN-6171. ConcurrentModificationException on (kasha: rev
a77f432449aad67da31bd8bf8644b71def741bde)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
> ConcurrentModificationException on FSAppAttempt.containersToPreempt
> ---
>
> Key: YARN-6171
> URL: https://issues.apache.org/jira/browse/YARN-6171
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6171.000.patch, YARN-6171.001.patch,
> YARN-6171.002.patch, YARN-6171.003.patch, YARN-6171.004.patch
>
>
> I have noticed an exception that closes the Application Master occasionally
> with Fair scheduler.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.util.ConcurrentModificationException):
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
> at java.util.HashMap$KeyIterator.next(HashMap.java:956)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:532)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[
https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870875#comment-15870875
]
Daniel Templeton commented on YARN-6188:
Ah, oops. I was looking at the diff patch, and I didn't notice that there were
lines between the changes that I wasn't seeing.
I just took a closer look at tho code, and, wow, the
{{DecommissioningNodesWatcher}} is chatty! Looking at the
{{logDecommissioningNodesStatus()}} method, we have: {code}if
(!LOG.isDebugEnabled() || decomNodes.size() == 0) {
return;
}
...
LOG.info("Decommissioning Nodes: " + sb.toString());{code}. Clearly, the
log was meant to be at debug level, which makes more sense. Can you also fix
that? Your change is still needed because otherwise turning on debug logging
will OOME the RM.
> Fix OOM issue with decommissioningNodesWatcher in the case of clusters with
> large number of nodes
> -
>
> Key: YARN-6188
> URL: https://issues.apache.org/jira/browse/YARN-6188
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Ajay Jadhav
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: YARN-6188.001.patch, YARN-6188.002.patch
>
>
> LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses
> StringBuilder to append status of all
> decommissioning nodes for logging purpose.
> In the case of large number of decommissioning nodes, this leads to OOM
> exception. The fix scopes StringBuilder so that in case of memory pressure,
> GC can kick in and free up the memory.
> This is supposed to fix a bug introduced in
> https://issues.apache.org/jira/browse/YARN-4676
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Fairscheduler: Dump scheduler state in log
[
https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870874#comment-15870874
]
Hadoop QA commented on YARN-6042:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} YARN-6042 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6042 |
| GITHUB PR | https://github.com/apache/hadoop/pull/193 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14993/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fairscheduler: Dump scheduler state in log
> --
>
> Key: YARN-6042
> URL: https://issues.apache.org/jira/browse/YARN-6042
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6042.001.patch, YARN-6042.002.patch,
> YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch,
> YARN-6042.006.patch, YARN-6042.007.patch
>
>
> To improve the debugging of scheduler issues it would be a big improvement to
> be able to dump the scheduler state into a log on request.
> The Dump the scheduler state at a point in time would allow debugging of a
> scheduler that is not hung (deadlocked) but also not assigning containers.
> Currently we do not have a proper overview of what state the scheduler and
> the queues are in and we have to make assumptions or guess
> The scheduler and queue state needed would include (not exhaustive):
> - instantaneous and steady fair share (app / queue)
> - AM share and resources
> - weight
> - app demand
> - application run state (runnable/non runnable)
> - last time at fair/min share
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6042) Fairscheduler: Dump scheduler state in log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6042: --- Attachment: YARN-6042.007.patch Rebase it in patch v7. > Fairscheduler: Dump scheduler state in log > -- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch, > YARN-6042.006.patch, YARN-6042.007.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6188) Fix OOM issue with decommissioningNodesWatcher in the case of clusters with large number of nodes
[ https://issues.apache.org/jira/browse/YARN-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870852#comment-15870852 ] Ajay Jadhav commented on YARN-6188: --- And yes, it does throw an OOM exception without the patch in case of large clusters. > Fix OOM issue with decommissioningNodesWatcher in the case of clusters with > large number of nodes > - > > Key: YARN-6188 > URL: https://issues.apache.org/jira/browse/YARN-6188 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0 >Reporter: Ajay Jadhav > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-6188.001.patch, YARN-6188.002.patch > > > LogDecommissioningNodesStatus method in DecommissioningNodesWatcher uses > StringBuilder to append status of all > decommissioning nodes for logging purpose. > In the case of large number of decommissioning nodes, this leads to OOM > exception. The fix scopes StringBuilder so that in case of memory pressure, > GC can kick in and free up the memory. > This is supposed to fix a bug introduced in > https://issues.apache.org/jira/browse/YARN-4676 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6059) Update paused container state in the state store
[
https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870850#comment-15870850
]
Hadoop QA commented on YARN-6059:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
49s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 12 new + 364 unchanged - 2 fixed = 376 total (was 366)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
15s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 235 unchanged - 0 fixed = 236 total (was 235) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 19s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6059 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853144/YARN-6059-YARN-5972.008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5f42ceab0c5b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5972 / 3a68608 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14990/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/14990/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
[jira] [Comment Edited] (YARN-6050) AMs can't be scheduled on racks or nodes
[
https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870845#comment-15870845
]
Wangda Tan edited comment on YARN-6050 at 2/16/17 11:12 PM:
Thanks [~rkanter],
Some comments:
1) {{RMAppImpl#createNewAttempt}}, we don't have to initialize BlacklistManager
here, correct? Since we will do this in ScheduledTransition. This can be done
in a separate patch.
2) Changes in getApplicableNodeCountForAM doesn't get #Nodes correct, at least
for following cases:
- If host1-3/rack1 have relaxLocality set to true, and host1-3 belongs to
rack1, this method get usable nodes = 3 + #rack1.
- If rack1 has 29 nodes, rack2 has 35 nodes, if relaxLocality set to true for
both of them, usable nodes = 35 instead of 29+35.
3) Not caused by your patch, but we'd better fix: {{ScheduleTransition}}
happens only once for every attempt, this won't get updated for node
add/remove, it's better to periodically update this. This can be done in a
separate patch.
was (Author: leftnoteasy):
Thanks [~rkanter],
Some comments:
1) {{RMAppImpl#createNewAttempt}}, we don't have to initialize BlacklistManager
here, correct? Since we will do this in ScheduledTransition. This can be done
in a separate patch.
2) Changes in getApplicableNodeCountForAM doesn't get #Nodes correct, at least
for following cases:
- If host1-3/rack1 have relaxLocality set to true, and host1-3 belongs to
rack1, this method get usable nodes = 3 + #rack1.
- If rack1 has 29 nodes, rack2 has 35 nodes, if relaxLocality set to true for
both of them, usable nodes = 35 instead of 29+35.
3) Not caused by your patch, but we'd better fix: {{ScheduleTransition}}
happens only once for every attempt, this won't get updated for node
add/remove, it's better to periodically update this. This can be done in a
separate patch.
> AMs can't be scheduled on racks or nodes
>
>
> Key: YARN-6050
> URL: https://issues.apache.org/jira/browse/YARN-6050
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6050.001.patch, YARN-6050.002.patch,
> YARN-6050.003.patch, YARN-6050.004.patch, YARN-6050.005.patch,
> YARN-6050.006.patch, YARN-6050.007.patch, YARN-6050.008.patch
>
>
> Yarn itself supports rack/node aware scheduling for AMs; however, there
> currently are two problems:
> # To specify hard or soft rack/node requests, you have to specify more than
> one {{ResourceRequest}}. For example, if you want to schedule an AM only on
> "rackA", you have to create two {{ResourceRequest}}, like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS,
> true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more
> than one {{ResourceRequest}} in the {{ApplicationSubmissionContext}}. The
> current behavior is to either build one from {{getResource}} or directly from
> {{getAMContainerResourceRequest}}, depending on if
> {{getAMContainerResourceRequest}} is null or not. We'll need to add a third
> method, say {{getAMContainerResourceRequests}}, which takes a list of
> {{ResourceRequest}} so that clients can specify the multiple resource
> requests.
> # There are some places where things are hardcoded to overwrite what the
> client specifies. These are pretty straightforward to fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6050) AMs can't be scheduled on racks or nodes
[
https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870845#comment-15870845
]
Wangda Tan commented on YARN-6050:
--
Thanks [~rkanter],
Some comments:
1) {{RMAppImpl#createNewAttempt}}, we don't have to initialize BlacklistManager
here, correct? Since we will do this in ScheduledTransition. This can be done
in a separate patch.
2) Changes in getApplicableNodeCountForAM doesn't get #Nodes correct, at least
for following cases:
- If host1-3/rack1 have relaxLocality set to true, and host1-3 belongs to
rack1, this method get usable nodes = 3 + #rack1.
- If rack1 has 29 nodes, rack2 has 35 nodes, if relaxLocality set to true for
both of them, usable nodes = 35 instead of 29+35.
3) Not caused by your patch, but we'd better fix: {{ScheduleTransition}}
happens only once for every attempt, this won't get updated for node
add/remove, it's better to periodically update this. This can be done in a
separate patch.
> AMs can't be scheduled on racks or nodes
>
>
> Key: YARN-6050
> URL: https://issues.apache.org/jira/browse/YARN-6050
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6050.001.patch, YARN-6050.002.patch,
> YARN-6050.003.patch, YARN-6050.004.patch, YARN-6050.005.patch,
> YARN-6050.006.patch, YARN-6050.007.patch, YARN-6050.008.patch
>
>
> Yarn itself supports rack/node aware scheduling for AMs; however, there
> currently are two problems:
> # To specify hard or soft rack/node requests, you have to specify more than
> one {{ResourceRequest}}. For example, if you want to schedule an AM only on
> "rackA", you have to create two {{ResourceRequest}}, like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS,
> true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more
> than one {{ResourceRequest}} in the {{ApplicationSubmissionContext}}. The
> current behavior is to either build one from {{getResource}} or directly from
> {{getAMContainerResourceRequest}}, depending on if
> {{getAMContainerResourceRequest}} is null or not. We'll need to add a third
> method, say {{getAMContainerResourceRequests}}, which takes a list of
> {{ResourceRequest}} so that clients can specify the multiple resource
> requests.
> # There are some places where things are hardcoded to overwrite what the
> client specifies. These are pretty straightforward to fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4212) FairScheduler: Can't create a DRF queue under a FAIR policy queue
[
https://issues.apache.org/jira/browse/YARN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-4212:
---
Fix Version/s: 3.0.0-alpha3
> FairScheduler: Can't create a DRF queue under a FAIR policy queue
> -
>
> Key: YARN-4212
> URL: https://issues.apache.org/jira/browse/YARN-4212
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Yufei Gu
> Labels: fairscheduler
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-4212.002.patch, YARN-4212.003.patch,
> YARN-4212.004.patch, YARN-4212.005.patch, YARN-4212.006.patch,
> YARN-4212.007.patch, YARN-4212.008.patch, YARN-4212.009.patch,
> YARN-4212.010.patch, YARN-4212.011.patch, YARN-4212.1.patch
>
>
> The Fair Scheduler, while performing a {{recomputeShares()}} during an
> {{update()}} call, uses the parent queues policy to distribute shares to its
> children.
> If the parent queues policy is 'fair', it only computes weight for memory and
> sets the vcores fair share of its children to 0.
> Assuming a situation where we have 1 parent queue with policy 'fair' and
> multiple leaf queues with policy 'drf', Any app submitted to the child queues
> with vcore requirement > 1 will always be above fairshare, since during the
> recomputeShare process, the child queues were all assigned 0 for fairshare
> vcores.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870838#comment-15870838
]
Daniel Templeton commented on YARN-6193:
Thanks, [~kasha]. A couple more comments:
* {{SchedulingPolicy.instances}} should be {{INSTANCES}}. Not your fault, but
while you're in there...
* Why don't the want the comparator to be static?
* You've changed the behavior of {{FifoPolicy.checkIfUsageOverFairShare()}}.
Are you sure that's the right thing to do?
* You should explain in this JIRA thread why you're upping the test iterations
from 100 back to 1000.
* On a philosophical note, {{checkIfUsageOverFairShare()}} and
{{checkIfUsageUnderFairShare()}} don't actually have anything to do with the
fair share. They're just resource comparisons. Should the {{fairShare}} arg
maybe be replaced with a call to {{getFairShare()}}?
* I took some time to stare at the code for a bit, and I don't see where this
{code}-Resource currentUsage = getResourceUsage();
-Resource fairshare = getFairShare();
-Resource overFairShareBy = Resources.subtract(currentUsage, fairshare);
-
-return (Resources.fitsIn(container.getAllocatedResource(),
-overFairShareBy));
+Resource usageAfterPreemption = Resources.subtract(
+getResourceUsage(), container.getAllocatedResource());
+return !fsQueue.getPolicy().checkIfUsageUnderFairShare(
+usageAfterPreemption, getFairShare());{code} actually makes any
difference. Seems to be the same operation to me, assuming that the app is
over its fair share.
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch,
> YARN-6193.002.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5798) Set UncaughtExceptionHandler for all FairScheduler threads
[ https://issues.apache.org/jira/browse/YARN-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-5798: --- Fix Version/s: 3.0.0-alpha3 > Set UncaughtExceptionHandler for all FairScheduler threads > -- > > Key: YARN-5798 > URL: https://issues.apache.org/jira/browse/YARN-5798 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-5798.001.patch, YARN-5798.002.patch, > YARN-5798.003.patch, YARN-5798.004.patch > > > YARN-5605 added an FSPreemptionThread. The run() method catches > InterruptedException. If this were to run into a RuntimeException, the > preemption thread would crash. We should probably fail the RM itself (or > transition to standby) when this happens. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6163) FS Preemption is a trickle for severely starved applications
[ https://issues.apache.org/jira/browse/YARN-6163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-6163: --- Fix Version/s: 3.0.0-alpha3 > FS Preemption is a trickle for severely starved applications > > > Key: YARN-6163 > URL: https://issues.apache.org/jira/browse/YARN-6163 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6163.004.patch, YARN-6163.005.patch, > YARN-6163.006.patch, yarn-6163-1.patch, yarn-6163-2.patch > > > With current logic, only one RR is considered per each instance of marking an > application starved. This marking happens only on the update call that runs > every 500ms. Due to this, an application that is severely starved takes > forever to reach fairshare based on preemptions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6042) Fairscheduler: Dump scheduler state in log
[ https://issues.apache.org/jira/browse/YARN-6042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870823#comment-15870823 ] Ray Chiang commented on YARN-6042: -- Looks like YARN-4212 means this patch no longer applies cleanly. > Fairscheduler: Dump scheduler state in log > -- > > Key: YARN-6042 > URL: https://issues.apache.org/jira/browse/YARN-6042 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6042.001.patch, YARN-6042.002.patch, > YARN-6042.003.patch, YARN-6042.004.patch, YARN-6042.005.patch, > YARN-6042.006.patch > > > To improve the debugging of scheduler issues it would be a big improvement to > be able to dump the scheduler state into a log on request. > The Dump the scheduler state at a point in time would allow debugging of a > scheduler that is not hung (deadlocked) but also not assigning containers. > Currently we do not have a proper overview of what state the scheduler and > the queues are in and we have to make assumptions or guess > The scheduler and queue state needed would include (not exhaustive): > - instantaneous and steady fair share (app / queue) > - AM share and resources > - weight > - app demand > - application run state (runnable/non runnable) > - last time at fair/min share -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6171) ConcurrentModificationException on FSAppAttempt.containersToPreempt
[
https://issues.apache.org/jira/browse/YARN-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6171:
---
Summary: ConcurrentModificationException on
FSAppAttempt.containersToPreempt (was: ConcurrentModificationException in
ApplicationMasterService.allocate)
> ConcurrentModificationException on FSAppAttempt.containersToPreempt
> ---
>
> Key: YARN-6171
> URL: https://issues.apache.org/jira/browse/YARN-6171
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6171.000.patch, YARN-6171.001.patch,
> YARN-6171.002.patch, YARN-6171.003.patch, YARN-6171.004.patch
>
>
> I have noticed an exception that closes the Application Master occasionally
> with Fair scheduler.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.util.ConcurrentModificationException):
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
> at java.util.HashMap$KeyIterator.next(HashMap.java:956)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:532)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6193) FairScheduler might not trigger preemption when using DRF
[
https://issues.apache.org/jira/browse/YARN-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6193:
---
Attachment: YARN-6193.002.patch
Most test failures correspond to tests that mock FairScheduler.
TestFairSchedulerPreemption is a timeout issue. Fixed them all in the 002.patch.
> FairScheduler might not trigger preemption when using DRF
> -
>
> Key: YARN-6193
> URL: https://issues.apache.org/jira/browse/YARN-6193
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: YARN-6193.000.patch, YARN-6193.001.patch,
> YARN-6193.002.patch
>
>
> {{FSAppAttempt#canContainerBePreempted}} verifies preempting a container
> doesn't lead to new starvation ({{Resources.fitsIn}}). When using DRF, this
> leads to verifying both resources instead of just the dominant resource.
> Ideally, the check should be based on policy.
> Note that current implementation of
> {{DominantResourceFairnessPolicy#checkIfUsageOverFairShare}} is broken.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6205) Default lifetime for native services app is invalid
Billie Rinaldi created YARN-6205: Summary: Default lifetime for native services app is invalid Key: YARN-6205 URL: https://issues.apache.org/jira/browse/YARN-6205 Project: Hadoop YARN Issue Type: Sub-task Reporter: Billie Rinaldi Assignee: Billie Rinaldi Fix For: yarn-native-services The default lifetime for a native services app started at the command line appears to be 0, which is an invalid lifetime. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Sharma updated YARN-6059: Attachment: YARN-6059-YARN-5972.008.patch > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma > Attachments: YARN-5216-YARN-6059.001.patch, > YARN-6059-YARN-5972.001.patch, YARN-6059-YARN-5972.002.patch, > YARN-6059-YARN-5972.003.patch, YARN-6059-YARN-5972.004.patch, > YARN-6059-YARN-5972.005.patch, YARN-6059-YARN-5972.006.patch, > YARN-6059-YARN-5972.007.patch, YARN-6059-YARN-5972.008.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870774#comment-15870774 ] Hudson commented on YARN-6177: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11267 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11267/]) YARN-6177. Yarn client should exit with an informative error message if (gtcarrera9: rev 5d339c46f5b16b951afd82afd9e907b9aa2ded9a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Fix For: 2.8.0, 2.9.0, 3.0.0-alpha3 > > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch, YARN-6177.06.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6175:
---
Attachment: YARN-6175.branch-2.8.002.patch
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch, YARN-6175.branch-2.8.002.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6175:
---
Attachment: (was: YARN-6175.002.patch)
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch, YARN-6175.branch-2.8.002.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870730#comment-15870730
]
Yufei Gu commented on YARN-6175:
Uploaded patch v2 which is in line with YARN-6163.
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch, YARN-6175.002.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6175:
---
Attachment: YARN-6175.002.patch
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch, YARN-6175.002.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870725#comment-15870725 ] Li Lu commented on YARN-6177: - Committing... > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch, YARN-6177.06.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6059) Update paused container state in the state store
[
https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870700#comment-15870700
]
Hadoop QA commented on YARN-6059:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
28s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 12 new + 363 unchanged - 2 fixed = 375 total (was 365)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
14s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 235 unchanged - 0 fixed = 236 total (was 235) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 18s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6059 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853126/YARN-6059-YARN-5972.007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d097eef376fe 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5972 / 3a68608 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14987/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/14987/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| unit |
[jira] [Commented] (YARN-6143) Fix incompatible issue caused by YARN-3583
[ https://issues.apache.org/jira/browse/YARN-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870663#comment-15870663 ] Wangda Tan commented on YARN-6143: -- And just executed script (https://issues.apache.org/jira/browse/HDFS-11096?focusedCommentId=15816405=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15816405) to compare proto files after patch applied, from what I can see, all incompat changes introduced by YARN-3583 are fixed by this patch. > Fix incompatible issue caused by YARN-3583 > -- > > Key: YARN-6143 > URL: https://issues.apache.org/jira/browse/YARN-6143 > Project: Hadoop YARN > Issue Type: Sub-task > Components: rolling upgrade >Reporter: Wangda Tan >Assignee: Sunil G >Priority: Blocker > Attachments: YARN-6143.0001.patch, YARN-6143.0002.patch, > YARN-6143.0003.patch > > > As mentioned by comment: > https://issues.apache.org/jira/browse/YARN-6142?focusedCommentId=15852009=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15852009. > We need to fix the incompatible issue caused by YARN-3583. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Sharma updated YARN-6059: Attachment: YARN-6059-YARN-5972.007.patch > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma > Attachments: YARN-5216-YARN-6059.001.patch, > YARN-6059-YARN-5972.001.patch, YARN-6059-YARN-5972.002.patch, > YARN-6059-YARN-5972.003.patch, YARN-6059-YARN-5972.004.patch, > YARN-6059-YARN-5972.005.patch, YARN-6059-YARN-5972.006.patch, > YARN-6059-YARN-5972.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Sharma updated YARN-6059: Attachment: YARN-6059-YARN-5972.006.patch > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma > Attachments: YARN-5216-YARN-6059.001.patch, > YARN-6059-YARN-5972.001.patch, YARN-6059-YARN-5972.002.patch, > YARN-6059-YARN-5972.003.patch, YARN-6059-YARN-5972.004.patch, > YARN-6059-YARN-5972.005.patch, YARN-6059-YARN-5972.006.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870628#comment-15870628 ] Hudson commented on YARN-4675: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11266 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11266/]) YARN-4675. Reorganize TimelineClient and TimelineClientImpl into (sjlee: rev 4fa1afdb883dab8786d2fb5c72a195dd2e87d711) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineV2ClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClientV2Impl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/TestTimelineServiceClientIntegration.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineV2Client.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineConnector.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/impl/AMRMClientAsyncImpl.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/TestNMTimelinePublisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AMRMClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/timelineservice/NMTimelinePublisher.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 3.0.0-alpha3 > > Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, > YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, > YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, > YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, > YARN-4675-YARN-2928.v1.001.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870590#comment-15870590 ] Sangjin Lee commented on YARN-4675: --- I went ahead and committed it to trunk. I'll wait for the YARN-5355 branch patch to commit it there as well. Thanks [~Naganarasimha] for your contribution, and [~varun_saxena] and [~gtCarrera9] for your review! > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: YARN-4675.v2.002.patch, YARN-4675.v2.003.patch, > YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, YARN-4675.v2.006.patch, > YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, YARN-4675.v2.009.patch, > YARN-4675.v2.010.patch, YARN-4675.v2.011.patch, > YARN-4675-YARN-2928.v1.001.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6171) ConcurrentModificationException in ApplicationMasterService.allocate
[
https://issues.apache.org/jira/browse/YARN-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870574#comment-15870574
]
Hadoop QA commented on YARN-6171:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 16s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6171 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853112/YARN-6171.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1c087fa41f3d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5690b51 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/14983/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14983/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14983/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ConcurrentModificationException in ApplicationMasterService.allocate
>
[jira] [Commented] (YARN-6059) Update paused container state in the state store
[
https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870556#comment-15870556
]
Hadoop QA commented on YARN-6059:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
11s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} YARN-5972 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 12 new + 366 unchanged - 0 fixed = 378 total (was 366)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 1 new + 235 unchanged - 0 fixed = 236 total (was 235) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6059 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853113/YARN-6059-YARN-5972.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1fbeee927d99 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-5972 / 3a68608 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14984/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/14984/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14984/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870541#comment-15870541 ] Li Lu commented on YARN-6177: - LGTM. Will commit in a few hours if nobody objects. > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch, YARN-6177.06.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6143) Fix incompatible issue caused by YARN-3583
[
https://issues.apache.org/jira/browse/YARN-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870540#comment-15870540
]
Wangda Tan commented on YARN-6143:
--
Thanks [~sunilg],
Only one minor comment: {{initLocalNodeLabels}} of
{{GetClusterNodeLabelsResponsePBImpl}} doesn't have to read from deprecated-,
the reason is we don't expect new client talk to old server, and the new server
always set both. So reading from non-deprecated field should be enough.
Submitted patch to trigger Jenkins.
> Fix incompatible issue caused by YARN-3583
> --
>
> Key: YARN-6143
> URL: https://issues.apache.org/jira/browse/YARN-6143
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: rolling upgrade
>Reporter: Wangda Tan
>Assignee: Sunil G
>Priority: Blocker
> Attachments: YARN-6143.0001.patch, YARN-6143.0002.patch,
> YARN-6143.0003.patch
>
>
> As mentioned by comment:
> https://issues.apache.org/jira/browse/YARN-6142?focusedCommentId=15852009=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15852009.
> We need to fix the incompatible issue caused by YARN-3583.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6185) Apply SLIDER-1199 to yarn native services for blacklisting nodes
[
https://issues.apache.org/jira/browse/YARN-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870533#comment-15870533
]
Hadoop QA commented on YARN-6185:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core:
The patch generated 2 new + 482 unchanged - 0 fixed = 484 total (was 482)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-slider-core in the patch passed. {color}
|
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6185 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853114/YARN-6185-yarn-native-services.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ff6baee89c7c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 92df1cb |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/14985/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-slider_hadoop-yarn-slider-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14985/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core
|
| Console output |
[jira] [Commented] (YARN-5946) Create YarnConfigurationStore interface and InMemoryConfigurationStore class
[
https://issues.apache.org/jira/browse/YARN-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870522#comment-15870522
]
Wangda Tan commented on YARN-5946:
--
[~jhung],
APIs mentioned in your latest make sense to me. I suggest to keep
{{List> getConfirmedConfHistory(long fromId)}} in the
interface but we don't have to implement/use it as of now. The reason is, while
implementing store, we can keep in mind that this is a future requirement so we
can consider it while designing storage format.
> Create YarnConfigurationStore interface and InMemoryConfigurationStore class
>
>
> Key: YARN-5946
> URL: https://issues.apache.org/jira/browse/YARN-5946
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-5946.001.patch, YARN-5946-YARN-5734.002.patch
>
>
> This class provides the interface to persist YARN configurations in a backing
> store.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
[
https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870504#comment-15870504
]
Andras Piros commented on YARN-6125:
The tests {{TestRMRestart}}, {{TestDelegationTokenRenewer}}, and
{{TestFairSchedulerPreemption}} are passing locally and seem unrelated.
> The application attempt's diagnostic message should have a maximum size
> ---
>
> Key: YARN-6125
> URL: https://issues.apache.org/jira/browse/YARN-6125
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: Daniel Templeton
>Assignee: Andras Piros
>Priority: Critical
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6125.000.patch, YARN-6125.001.patch,
> YARN-6125.002.patch, YARN-6125.003.patch, YARN-6125.004.patch,
> YARN-6125.005.patch, YARN-6125.006.patch, YARN-6125.007.patch,
> YARN-6125.008.patch, YARN-6125.009.patch
>
>
> We've found through experience that the diagnostic message can grow
> unbounded. I've seen attempts that have diagnostic messages over 1MB. Since
> the message is stored in the state store, it's a bad idea to allow the
> message to grow unbounded. Instead, there should be a property that sets a
> maximum size on the message.
> I suspect that some of the ZK state store issues we've seen in the past were
> due to the size of the diagnostic messages and not to the size of the
> classpath, as is the current prevailing opinion.
> An open question is how best to prune the message once it grows too large.
> Should we
> # truncate the tail,
> # truncate the head,
> # truncate the middle,
> # add another property to make the behavior selectable, or
> # none of the above?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6175) Negative vcore for resource needed to preempt
[
https://issues.apache.org/jira/browse/YARN-6175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870487#comment-15870487
]
Yufei Gu commented on YARN-6175:
[~kasha], would you like take a look since YARN-6163 is done.
> Negative vcore for resource needed to preempt
> -
>
> Key: YARN-6175
> URL: https://issues.apache.org/jira/browse/YARN-6175
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6175.001.patch
>
>
> Both old preemption code (2.8 and before) and new preemption code could have
> negative vcores while calculating resources needed to preempt.
> For old preemption, you can find following messages in RM logs:
> {code}
> Should preempt
> {code}
> The related code is in method {{resourceDeficit()}}.
> For new preemption code, there are no messages in RM logs, the related code
> is in method {{fairShareStarvation()}}.
> The negative value isn't only a display issue, but also may cause missing
> necessary preemption.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6185) Apply SLIDER-1199 to yarn native services for blacklisting nodes
[ https://issues.apache.org/jira/browse/YARN-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-6185: - Attachment: YARN-6185-yarn-native-services.001.patch > Apply SLIDER-1199 to yarn native services for blacklisting nodes > > > Key: YARN-6185 > URL: https://issues.apache.org/jira/browse/YARN-6185 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Fix For: yarn-native-services > > Attachments: YARN-6185-yarn-native-services.001.patch > > > Enable blacklisting of nodes by Slider AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Sharma updated YARN-6059: Attachment: YARN-6059-YARN-5972.005.patch > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma > Attachments: YARN-5216-YARN-6059.001.patch, > YARN-6059-YARN-5972.001.patch, YARN-6059-YARN-5972.002.patch, > YARN-6059-YARN-5972.003.patch, YARN-6059-YARN-5972.004.patch, > YARN-6059-YARN-5972.005.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6171) ConcurrentModificationException in ApplicationMasterService.allocate
[
https://issues.apache.org/jira/browse/YARN-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-6171:
-
Attachment: YARN-6171.004.patch
Addressing comments
> ConcurrentModificationException in ApplicationMasterService.allocate
>
>
> Key: YARN-6171
> URL: https://issues.apache.org/jira/browse/YARN-6171
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6171.000.patch, YARN-6171.001.patch,
> YARN-6171.002.patch, YARN-6171.003.patch, YARN-6171.004.patch
>
>
> I have noticed an exception that closes the Application Master occasionally
> with Fair scheduler.
> {code}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(java.util.ConcurrentModificationException):
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922)
> at java.util.HashMap$KeyIterator.next(HashMap.java:956)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:1005)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:532)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6204) Set UncaughtExceptionHandler for event handling thread in AsyncDispatcher
[ https://issues.apache.org/jira/browse/YARN-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870433#comment-15870433 ] Yufei Gu commented on YARN-6204: The problem is AsyncDispatcher is in yarn common, but RMCriticalThreadUncaughtExceptionHandler is in yarn server. Seems like we need an interface to make it happen. > Set UncaughtExceptionHandler for event handling thread in AsyncDispatcher > - > > Key: YARN-6204 > URL: https://issues.apache.org/jira/browse/YARN-6204 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > > The event handling thread in AsyncDispatcher is a critical thread in RM. We > should set UncaughtExceptionHandler introduced in YARN-6061 for it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6204) Set UncaughtExceptionHandler for event handling thread in AsyncDispatcher
Yufei Gu created YARN-6204: -- Summary: Set UncaughtExceptionHandler for event handling thread in AsyncDispatcher Key: YARN-6204 URL: https://issues.apache.org/jira/browse/YARN-6204 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.0.0-alpha2, 2.9.0 Reporter: Yufei Gu Assignee: Yufei Gu The event handling thread in AsyncDispatcher is a critical thread in RM. We should set UncaughtExceptionHandler introduced in YARN-6061 for it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6050) AMs can't be scheduled on racks or nodes
[
https://issues.apache.org/jira/browse/YARN-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870378#comment-15870378
]
Wangda Tan commented on YARN-6050:
--
Sorry [~rkanter] for my late responses, I will try to get some time today or
tomorrow to review the patch.
Thanks,
> AMs can't be scheduled on racks or nodes
>
>
> Key: YARN-6050
> URL: https://issues.apache.org/jira/browse/YARN-6050
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-6050.001.patch, YARN-6050.002.patch,
> YARN-6050.003.patch, YARN-6050.004.patch, YARN-6050.005.patch,
> YARN-6050.006.patch, YARN-6050.007.patch, YARN-6050.008.patch
>
>
> Yarn itself supports rack/node aware scheduling for AMs; however, there
> currently are two problems:
> # To specify hard or soft rack/node requests, you have to specify more than
> one {{ResourceRequest}}. For example, if you want to schedule an AM only on
> "rackA", you have to create two {{ResourceRequest}}, like this:
> {code}
> ResourceRequest.newInstance(PRIORITY, ANY, CAPABILITY, NUM_CONTAINERS, false);
> ResourceRequest.newInstance(PRIORITY, "rackA", CAPABILITY, NUM_CONTAINERS,
> true);
> {code}
> The problem is that the Yarn API doesn't actually allow you to specify more
> than one {{ResourceRequest}} in the {{ApplicationSubmissionContext}}. The
> current behavior is to either build one from {{getResource}} or directly from
> {{getAMContainerResourceRequest}}, depending on if
> {{getAMContainerResourceRequest}} is null or not. We'll need to add a third
> method, say {{getAMContainerResourceRequests}}, which takes a list of
> {{ResourceRequest}} so that clients can specify the multiple resource
> requests.
> # There are some places where things are hardcoded to overwrite what the
> client specifies. These are pretty straightforward to fix.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[
https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870306#comment-15870306
]
Hadoop QA commented on YARN-6177:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
51s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6177 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12853093/YARN-6177.06.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 44ad5624d182 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5690b51 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/14982/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/14982/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Yarn client should exit with an informative error message if an incompatible
> Jersey library is used at client
> -
>
> Key: YARN-6177
> URL: https://issues.apache.org/jira/browse/YARN-6177
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0, 3.0.0-alpha2
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: spark2-job-output-after-besteffort.out,
> spark2-job-output-after.out,
[jira] [Commented] (YARN-6191) CapacityScheduler preemption by container priority can be problematic for MapReduce
[ https://issues.apache.org/jira/browse/YARN-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870275#comment-15870275 ] Sunil G commented on YARN-6191: --- Hi [~jlowe] During inter-queue preemption improvement time, there were a bunch of thoughts regarding a plugin-policy to select containers from an app for preemption. Now we do this based on container priority. Few more good params were - % of work completed - time remaining to finish a container - locality of preempted container (Whether this will help the demanding queue's app for better placement) - +type of container+ as discussed here (map/reduce is better to preempt) However all or some of these may not be available always or it may not well suit for a given usecase. An idea of having a *pre-computed preemption cost* per container may be a good idea. And priority of container could attribute to that cost, and other params as well (if configured). > CapacityScheduler preemption by container priority can be problematic for > MapReduce > --- > > Key: YARN-6191 > URL: https://issues.apache.org/jira/browse/YARN-6191 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Jason Lowe > > A MapReduce job with thousands of reducers and just a couple of maps left to > go was running in a preemptable queue. Periodically other queues would get > busy and the RM would preempt some resources from the job, but it _always_ > picked the job's map tasks first because they use the lowest priority > containers. Even though the reducers had a shorter running time, most were > spared but the maps were always shot. Since the map tasks ran for a longer > time than the preemption period, the job was in a perpetual preemption loop. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated YARN-6177: -- Attachment: YARN-6177.06.patch > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch, YARN-6177.06.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6191) CapacityScheduler preemption by container priority can be problematic for MapReduce
[ https://issues.apache.org/jira/browse/YARN-6191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870131#comment-15870131 ] Jason Lowe commented on YARN-6191: -- Thanks, Chris! Having the AM react to the preemption message in the heartbeat will definitely help a lot for common cases, even if it doesn't do any work-conserving logic and just kills the reducers. However there's still an issue because the preemption message is too general. For example, if the message says "going to preempt 60GB of resources" and the AM kills 10 reducers that are 6GB each on 6 different nodes, the RM can still kill the maps because the RM needed 60GB of contiguous resources. Fixing that requires the preemption message to be more expressive/specific so the AM knows that its actions will indeed prevent the preemption of other containers. I still wonder about the logic of preferring lower container priorities regardless of how long they've been running. I'm not sure container priority always translates well to how important a container is to the application, and we might be better served by preferring to minimize total lost work regardless of container priority. > CapacityScheduler preemption by container priority can be problematic for > MapReduce > --- > > Key: YARN-6191 > URL: https://issues.apache.org/jira/browse/YARN-6191 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Reporter: Jason Lowe > > A MapReduce job with thousands of reducers and just a couple of maps left to > go was running in a preemptable queue. Periodically other queues would get > busy and the RM would preempt some resources from the job, but it _always_ > picked the job's map tasks first because they use the lowest priority > containers. Even though the reducers had a shorter running time, most were > spared but the maps were always shot. Since the map tasks ran for a longer > time than the preemption period, the job was in a perpetual preemption loop. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870127#comment-15870127 ] Weiwei Yang commented on YARN-6177: --- Hello [~jlowe] Thanks for the input, it's quite helpful. Option #1 was not to change best-effort default value, I meant to say with the patch provided in this ticket, user will be able to work around this jar conflict problem if they want to set "yarn.timeline-service.client.best-effort" to "true". If both of you think the best-effort property was not suitable to handle an error like this, let me upload a new patch and just give the error message (suggest user to disable timeline service to work around). Thanks you! > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6177) Yarn client should exit with an informative error message if an incompatible Jersey library is used at client
[ https://issues.apache.org/jira/browse/YARN-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870096#comment-15870096 ] Jason Lowe commented on YARN-6177: -- The best effort setting was primarily targeting the scenario where the timeline server is down and not wanting jobs to fail as a result. It wasn't intended to mask jar conflicts like this. Maybe I'm misunderstanding option #1 from above, but we should not be changing the best-effort default value. That's an incompatible and surprising change. When someone sets best-effort to true then they desire the timeline server to work but don't want transient errors to block the job's progress. This case is not a transient error -- the timeline client is never going to work with that jar conflict in place. Therefore I agree with [~gtCarrera9] that we should not mask this error. If it's common then I think we should check for it and provide a useful error message stating the user needs to address the classpath conflict or disable the timeline client, but the error should still be fatal. > Yarn client should exit with an informative error message if an incompatible > Jersey library is used at client > - > > Key: YARN-6177 > URL: https://issues.apache.org/jira/browse/YARN-6177 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.8.0, 3.0.0-alpha2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: spark2-job-output-after-besteffort.out, > spark2-job-output-after.out, spark2-job-output-before.out, > YARN-6177.01.patch, YARN-6177.02.patch, YARN-6177.03.patch, > YARN-6177.04.patch, YARN-6177.05.patch > > > Per discussion in YARN-5271, lets provide an error message to suggest user to > disable timeline service instead of disabling for them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6069) CORS support in timeline v2
[ https://issues.apache.org/jira/browse/YARN-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870044#comment-15870044 ] Varun Saxena edited comment on YARN-6069 at 2/16/17 2:35 PM: - bq. I think folks are forgetting one important thing that this JIRA is only for TimelineReader v.2 server which uses GET operation only. It is not necessarily required to worry about any custom configurations that controls other HTTP operations. Sorry didn't get you. Other than allowed methods configuration as this is only GET, which CORS related configuration is not relevant here. Also any idea what purpose max age serves. We do not make any check in the filter based on it. was (Author: varun_saxena): bq. I think folks are forgetting one important thing that this JIRA is only for TimelineReader v.2 server which uses GET operation only. It is not necessarily required to worry about any custom configurations that controls other HTTP operations. Sorry didn't get you. Other than allowed methods configuration, which CORS related configuration is not relevant here. Also any idea what purpose max age serves. We do not make any check in the filter based on it. > CORS support in timeline v2 > --- > > Key: YARN-6069 > URL: https://issues.apache.org/jira/browse/YARN-6069 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Sreenath Somarajapuram >Assignee: Rohith Sharma K S > Attachments: YARN-6069-YARN-5355.0001.patch, > YARN-6069-YARN-5355.0002.patch > > > By default the browser prevents accessing resources from multiple domains. In > most cases the UIs would be loaded form a domain different from that of > timeline server. Hence without CORS support, it would be difficult for the > UIs to load data from timeline v2. > YARN-2277 must provide more info on the implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6069) CORS support in timeline v2
[ https://issues.apache.org/jira/browse/YARN-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870044#comment-15870044 ] Varun Saxena commented on YARN-6069: bq. I think folks are forgetting one important thing that this JIRA is only for TimelineReader v.2 server which uses GET operation only. It is not necessarily required to worry about any custom configurations that controls other HTTP operations. Sorry didn't get you. Other than allowed methods configuration, which CORS related configuration is not relevant here. Also any idea what purpose max age serves. We do not make any check in the filter based on it. > CORS support in timeline v2 > --- > > Key: YARN-6069 > URL: https://issues.apache.org/jira/browse/YARN-6069 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Sreenath Somarajapuram >Assignee: Rohith Sharma K S > Attachments: YARN-6069-YARN-5355.0001.patch, > YARN-6069-YARN-5355.0002.patch > > > By default the browser prevents accessing resources from multiple domains. In > most cases the UIs would be loaded form a domain different from that of > timeline server. Hence without CORS support, it would be difficult for the > UIs to load data from timeline v2. > YARN-2277 must provide more info on the implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6197) CS Leaf queue am usage gets updated for unmanaged AM
[
https://issues.apache.org/jira/browse/YARN-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870013#comment-15870013
]
Bibin A Chundatt commented on YARN-6197:
[~sunilg]/[~varun_saxena]
For activating and checking {{amIfStarted}} should be consider
{{rmContext.getScheduler().getMinimumResourceCapability}}.
As per the current code
{code}
public FiCaSchedulerApp(ApplicationAttemptId applicationAttemptId,
String user, Queue queue, AbstractUsersManager abstractUsersManager,
RMContext rmContext, Priority appPriority, boolean isAttemptRecovering,
ActivitiesManager activitiesManager) {
if (rmApp == null || rmApp.getAMResourceRequest() == null) {
// the rmApp may be undefined (the resource manager checks for this too)
// and unmanaged applications do not provide an amResource request
// in these cases, provide a default using the scheduler
amResource = rmContext.getScheduler().getMinimumResourceCapability();
partition = CommonNodeLabelsManager.NO_LABEL;
} else {
amResource = rmApp.getAMResourceRequest().getCapability();
partition =
(rmApp.getAMResourceRequest().getNodeLabelExpression() == null)
? CommonNodeLabelsManager.NO_LABEL
: rmApp.getAMResourceRequest().getNodeLabelExpression();
}
{code}
Any info on why the minimum was considered instead of {{Resource.NONE}} ?? We
could have activated AM directly with out AM Limit check but following ordering
policy.
> CS Leaf queue am usage gets updated for unmanaged AM
>
>
> Key: YARN-6197
> URL: https://issues.apache.org/jira/browse/YARN-6197
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> {{LeafQueue#activateApplication()}} for unmanaged AM the am_usage is updated
> with scheduler minimum allocation size. Cluster resource/AM limit headroom
> for other apps in queue will get reduced .
> Solution: FicaScheduler unManagedAM flag can be used to check AM type.
> Based on flag the queueusage need to be updated during activation and removal
> Thoughts??
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6069) CORS support in timeline v2
[ https://issues.apache.org/jira/browse/YARN-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869957#comment-15869957 ] Varun Saxena commented on YARN-6069: Probably it did not come out clearly in my previous comment. I was alluding to the same thing. That a user may look at timeline service related CORS configurations and assume they are to be used for ATSv2 but till we document clearly what the behavior is, it should be fine. That is why I said I am fine with either of the 2 approaches. > CORS support in timeline v2 > --- > > Key: YARN-6069 > URL: https://issues.apache.org/jira/browse/YARN-6069 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Sreenath Somarajapuram >Assignee: Rohith Sharma K S > Attachments: YARN-6069-YARN-5355.0001.patch, > YARN-6069-YARN-5355.0002.patch > > > By default the browser prevents accessing resources from multiple domains. In > most cases the UIs would be loaded form a domain different from that of > timeline server. Hence without CORS support, it would be difficult for the > UIs to load data from timeline v2. > YARN-2277 must provide more info on the implementation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
