[jira] [Updated] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-09 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9820:

Fix Version/s: (was: 3.2.2)
   3.2.1

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.4
>
> Attachments: YARN-9820-001.patch, YARN-9820-002.patch, 
> YARN-9820-003.patch
>
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-09 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925567#comment-16925567
 ] 

Rohith Sharma K S commented on YARN-9820:
-

I back ported to 3.2.1 and updated fix version. 

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 2.10.0, 3.3.0, 3.2.1, 3.1.4
>
> Attachments: YARN-9820-001.patch, YARN-9820-002.patch, 
> YARN-9820-003.patch
>
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9821) NM hangs at serviceStop when ATSV2 Backend Hbase is Down

2019-09-09 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925486#comment-16925486
 ] 

Rohith Sharma K S commented on YARN-9821:
-

Only for branch-3.2. May be I can only back port it since it required for 
branch-3.2.1 as well.

> NM hangs at serviceStop when ATSV2 Backend Hbase is Down 
> -
>
> Key: YARN-9821
> URL: https://issues.apache.org/jira/browse/YARN-9821
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9821-001.patch, YARN-9821-002.patch
>
>
> NM hangs at serviceStop when ATSV2 Backend Hbase is Down.
> {code}
> "Thread-197" #302 prio=5 os_prio=0 tid=0x7f5f389ba000 nid=0x631d waiting 
> for monitor entry [0x7f5f1f29b000]
>java.lang.Thread.State: BLOCKED (on object monitor)
>   at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.close(BufferedMutatorImpl.java:249)
>   - waiting to lock <0x0006c834d148> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TypedBufferedMutator.close(TypedBufferedMutator.java:62)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.serviceStop(HBaseTimelineWriterImpl.java:636)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05808> (a java.lang.Object)
>   at 
> org.apache.hadoop.service.AbstractService.close(AbstractService.java:247)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:244)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.NodeTimelineCollectorManager.serviceStop(NodeTimelineCollectorManager.java:164)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05890> (a java.lang.Object)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService.serviceStop(PerNodeTimelineCollectorsAuxService.java:113)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c058f8> (a java.lang.Object)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStop(AuxServices.java:330)
>   - locked <0x0006c7c23400> (a java.util.Collections$SynchronizedMap)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c059a8> (a java.lang.Object)
>   at 
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
>   at 
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
>   at 
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05a98> (a java.lang.Object)
>   at 
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
>   at 
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
>   at 
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05c88> (a java.lang.Object)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager$1.run(NodeManager.java:552)
>   
>   
> "qtp183259297-76" #76 daemon prio=5 os_prio=0 tid=0x7f5f567ed000 
> nid=0x5fb7 in Object.wait() [0x7f5f23ad7000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
>   at java.lang.Object.wait(Native Method)
>   at java.lang.Object.wait(Object.java:460)
>   at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)
>   at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258)
>   - locked <0x000784ee8220> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
>   at 
> 

[jira] [Commented] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925358#comment-16925358
 ] 

Rohith Sharma K S commented on YARN-9820:
-

+1 lgtm as well.

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9820-001.patch, YARN-9820-002.patch, 
> YARN-9820-003.patch
>
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925326#comment-16925326
 ] 

Rohith Sharma K S commented on YARN-9820:
-

I agree with [~jhung] approach. We should send notifyApp flag so that 
RMstateStore decide to trigger an event or not. 



> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Critical
> Attachments: YARN-9820-001.patch, YARN-9820-002.patch
>
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9821) NM hangs at serviceStop when ATSV2 Backend Hbase is Down

2019-09-08 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925318#comment-16925318
 ] 

Rohith Sharma K S commented on YARN-9821:
-

patch looks reasonable to me.. +1. 

> NM hangs at serviceStop when ATSV2 Backend Hbase is Down 
> -
>
> Key: YARN-9821
> URL: https://issues.apache.org/jira/browse/YARN-9821
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9821-001.patch
>
>
> NM hangs at serviceStop when ATSV2 Backend Hbase is Down.
> {code}
> "Thread-197" #302 prio=5 os_prio=0 tid=0x7f5f389ba000 nid=0x631d waiting 
> for monitor entry [0x7f5f1f29b000]
>java.lang.Thread.State: BLOCKED (on object monitor)
>   at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.close(BufferedMutatorImpl.java:249)
>   - waiting to lock <0x0006c834d148> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TypedBufferedMutator.close(TypedBufferedMutator.java:62)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.serviceStop(HBaseTimelineWriterImpl.java:636)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05808> (a java.lang.Object)
>   at 
> org.apache.hadoop.service.AbstractService.close(AbstractService.java:247)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:244)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.NodeTimelineCollectorManager.serviceStop(NodeTimelineCollectorManager.java:164)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05890> (a java.lang.Object)
>   at 
> org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService.serviceStop(PerNodeTimelineCollectorsAuxService.java:113)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c058f8> (a java.lang.Object)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStop(AuxServices.java:330)
>   - locked <0x0006c7c23400> (a java.util.Collections$SynchronizedMap)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c059a8> (a java.lang.Object)
>   at 
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
>   at 
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
>   at 
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStop(ContainerManagerImpl.java:720)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05a98> (a java.lang.Object)
>   at 
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:54)
>   at 
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:102)
>   at 
> org.apache.hadoop.service.CompositeService.stop(CompositeService.java:158)
>   at 
> org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStop(NodeManager.java:526)
>   at 
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220)
>   - locked <0x0006c7c05c88> (a java.lang.Object)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.NodeManager$1.run(NodeManager.java:552)
>   
>   
> "qtp183259297-76" #76 daemon prio=5 os_prio=0 tid=0x7f5f567ed000 
> nid=0x5fb7 in Object.wait() [0x7f5f23ad7000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
>   at java.lang.Object.wait(Native Method)
>   at java.lang.Object.wait(Object.java:460)
>   at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)
>   at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.pollForSpecificCompletedTask(ResultBoundedCompletionService.java:258)
>   - locked <0x000784ee8220> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
>   at 
> 

[jira] [Updated] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9820:

Target Version/s:   (was: 3.2.2)

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9820:

Affects Version/s: (was: 3.2.1)

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9820:

Target Version/s: 3.2.2

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.2.1
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925135#comment-16925135
 ] 

Rohith Sharma K S commented on YARN-9820:
-

YARN-9438 cause triggering update event immediately after app submit. It is 
expecting event, then this need to be ignored in RMAppImpl. 
cc:/ [~jhung] [~haibochen]

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.2.1
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9820:

Affects Version/s: 3.2.1

> RM logs InvalidStateTransitionException when app is submitted
> -
>
> Key: YARN-9820
> URL: https://issues.apache.org/jira/browse/YARN-9820
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.2.1
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> It is observed that RM logs InvalidStateTransitionException. Not sure what is 
> the impact but its better to handle it. 
> {noformat}
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED 
> on event = LAUNCHED
> 2019-09-08 12:40:46,327 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
> launch time for applicationId: application_1567926390667_0001, attemptId: 
> appattempt_1567926390667_0001_01launchTime: 1567926646327
> 2019-09-08 12:40:46,328 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
> info for app: application_1567926390667_0001
> 2019-09-08 12:40:46,332 ERROR 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
> application_1567926390667_0001 can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
> APP_UPDATE_SAVED at ACCEPTED
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
> at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-9820) RM logs InvalidStateTransitionException when app is submitted

2019-09-08 Thread Rohith Sharma K S (Jira)
Rohith Sharma K S created YARN-9820:
---

 Summary: RM logs InvalidStateTransitionException when app is 
submitted
 Key: YARN-9820
 URL: https://issues.apache.org/jira/browse/YARN-9820
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Rohith Sharma K S


It is observed that RM logs InvalidStateTransitionException. Not sure what is 
the impact but its better to handle it. 

{noformat}
2019-09-08 12:40:46,327 INFO 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
appattempt_1567926390667_0001_01 State change from ALLOCATED to LAUNCHED on 
event = LAUNCHED
2019-09-08 12:40:46,327 INFO 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: update the 
launch time for applicationId: application_1567926390667_0001, attemptId: 
appattempt_1567926390667_0001_01launchTime: 1567926646327
2019-09-08 12:40:46,328 INFO 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating 
info for app: application_1567926390667_0001
2019-09-08 12:40:46,332 ERROR 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: App: 
application_1567926390667_0001 can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: 
APP_UPDATE_SAVED at ACCEPTED
at 
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
at 
org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
at 
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
at 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:881)
at 
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:116)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1030)
at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:1014)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:219)
at 
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:133)
at java.lang.Thread.run(Thread.java:748)
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8199) Logging fileSize of log files under NM Local Dir

2019-09-07 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925070#comment-16925070
 ] 

Rohith Sharma K S commented on YARN-8199:
-

[~snemeth] [~Prabhu Joseph] I don't see this JIRA ID in any of the commits in 
trunk/branches. 
Is this really committed or missed? 

> Logging fileSize of log files under NM Local Dir
> 
>
> Key: YARN-8199
> URL: https://issues.apache.org/jira/browse/YARN-8199
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation
>Affects Versions: 2.7.3
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>  Labels: supportability
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch, 
> YARN-8199-003.patch, YARN-8199-004.patch, YARN-8199-branch-3.1.001.patch, 
> YARN-8199-branch-3.2.001.patch
>
>
> Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir 
> by NodeManager before the cleanup will help to find the application which has 
> written too verbose.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9756) Create metric that sums total memory/vcores preempted per round

2019-09-07 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924771#comment-16924771
 ] 

Rohith Sharma K S commented on YARN-9756:
-

Updated correct fix version i.e 3.2.1

> Create metric that sums total memory/vcores preempted per round
> ---
>
> Key: YARN-9756
> URL: https://issues.apache.org/jira/browse/YARN-9756
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Eric Payne
>Assignee: Manikandan R
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.4
>
> Attachments: YARN-9756-branch-2.8.005.patch, 
> YARN-9756-branch-3.0.004.patch, YARN-9756-branch-3.2.003.patch, 
> YARN-9756.001.patch, YARN-9756.002.patch, YARN-9756.WIP.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9756) Create metric that sums total memory/vcores preempted per round

2019-09-07 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9756:

Fix Version/s: (was: 3.2.2)
   3.2.1

> Create metric that sums total memory/vcores preempted per round
> ---
>
> Key: YARN-9756
> URL: https://issues.apache.org/jira/browse/YARN-9756
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Eric Payne
>Assignee: Manikandan R
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.4
>
> Attachments: YARN-9756-branch-2.8.005.patch, 
> YARN-9756-branch-3.0.004.patch, YARN-9756-branch-3.2.003.patch, 
> YARN-9756.001.patch, YARN-9756.002.patch, YARN-9756.WIP.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7055) YARN Timeline Service v.2: beta 1 / GA

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923445#comment-16923445
 ] 

Rohith Sharma K S commented on YARN-7055:
-

Thanks to [~abmodi] [~Prabhu Joseph] [~prabham] [~vrushalic] [~Sushil-K-S]j 
[~haibochen] [~varun_saxena] [~sunilg] [~vinodkv] and bunch of folks who 
helping in reviews - patch/design, guidance, suggestion. Finally, we are 
announcing ATSv2 as GA. 

> YARN Timeline Service v.2: beta 1 / GA
> --
>
> Key: YARN-7055
> URL: https://issues.apache.org/jira/browse/YARN-7055
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Priority: Major
> Fix For: 3.2.1
>
> Attachments: TSv2 next steps.pdf
>
>
> This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service 
> v.2.
> YARN-2928 was alpha1, YARN-5355 was alpha2. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-7055) YARN Timeline Service v.2: beta 1 / GA

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-7055.
-
Fix Version/s: 3.2.1
 Release Note: Application Timeline Server v2 is ready for production. It 
is GA from 3.2.1 release on wards. 
   Resolution: Fixed

> YARN Timeline Service v.2: beta 1 / GA
> --
>
> Key: YARN-7055
> URL: https://issues.apache.org/jira/browse/YARN-7055
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Priority: Major
> Fix For: 3.2.1
>
> Attachments: TSv2 next steps.pdf
>
>
> This is an umbrella JIRA for the beta 1 milestone for YARN Timeline Service 
> v.2.
> YARN-2928 was alpha1, YARN-5355 was alpha2. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Reopened] (YARN-9388) HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase tablenames

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reopened YARN-9388:
-

Reopening the issue to mark as duplicate!

> HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase 
> tablenames
> -
>
> Key: YARN-9388
> URL: https://issues.apache.org/jira/browse/YARN-9388
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0, 3.1.2, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> HBaseTimelineSchemaCreator provides option to configure custom table names 
> for timelineservice tables. The option skipExistingTable ignores the tables 
> already created with custom tablenames and recreates the tables with default 
> table names. The custom table names has to be persisted so that 
> skipExistingTable, HBaseTimelineWriterImpl, HBaseTimelineReaderImpl can use 
> this. It currently expects to pass all custom table names every time 
> TimelineSchemaCreator is used.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-9388) HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase tablenames

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-9388.
-
Resolution: Duplicate

This is fixed as part of YARN-9387. Closing as duplicate

> HBaseTimelineSchemaCreator skipExistingTable should honor custom hbase 
> tablenames
> -
>
> Key: YARN-9388
> URL: https://issues.apache.org/jira/browse/YARN-9388
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0, 3.1.2, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> HBaseTimelineSchemaCreator provides option to configure custom table names 
> for timelineservice tables. The option skipExistingTable ignores the tables 
> already created with custom tablenames and recreates the tables with default 
> table names. The custom table names has to be persisted so that 
> skipExistingTable, HBaseTimelineWriterImpl, HBaseTimelineReaderImpl can use 
> this. It currently expects to pass all custom table names every time 
> TimelineSchemaCreator is used.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8499) ATS v2 Generic TimelineStorageMonitor

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-8499:

Fix Version/s: 3.2.1

> ATS v2 Generic TimelineStorageMonitor
> -
>
> Key: YARN-8499
> URL: https://issues.apache.org/jira/browse/YARN-8499
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Reporter: Sunil Govindan
>Assignee: Prabhu Joseph
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-8499-001.patch, YARN-8499-002.patch, 
> YARN-8499-003.patch, YARN-8499-004.patch, YARN-8499-005.patch, 
> YARN-8499-006.patch, YARN-8499-007.patch, YARN-8499-008.patch, 
> YARN-8499-009.patch, YARN-8499-010.patch, YARN-8499-011.patch, 
> YARN-8499-012.patch
>
>
> Post YARN-8302, Hbase connection issues are handled in ATSv2. However this 
> could be made general by introducing an api in storage interface and 
> implementing in each of the storage as per the store semantics.
>  
> cc [~rohithsharma] [~vinodkv] [~vrushalic]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9374) HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9374:

Fix Version/s: 3.2.1

> HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage 
> down
> 
>
> Key: YARN-9374
> URL: https://issues.apache.org/jira/browse/YARN-9374
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9374-001.patch, YARN-9374-002.patch, 
> YARN-9374-003.patch, YARN-9374-004.patch, YARN-9374-005.patch, 
> YARN-9374-006.patch, YARN-9374-007.patch
>
>
> HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage 
> is down. Currently we check if hbase storage is down in TimelineReader before 
> reading entities and fail immediately in YARN-8302. Similar fix is needed for 
> write. Async is handled in YARN-9335.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8567) Fetching yarn logs fails for long running application if it is not present in timeline store

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-8567:

Fix Version/s: 3.2.1

> Fetching yarn logs fails for long running application if it is not present in 
> timeline store
> 
>
> Key: YARN-8567
> URL: https://issues.apache.org/jira/browse/YARN-8567
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 2.7.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
>  Labels: log-aggregation
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-8567.001.patch, YARN-8567.002.patch
>
>
> Using yarn logs command for a long running application which has been running 
> longer than the configured timeline service ttl 
> {{yarn.timeline-service.ttl-ms }} fails with the following exception.
> {code:java}
> Exception in thread "main" 
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: The entity 
> for application application_152347939332_1 doesn't exist in the timeline 
> store
> at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplication(ApplicationHistoryManagerOnTimelineStore.java:670)
> at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainers(ApplicationHistoryManagerOnTimelineStore.java:219)
> at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.getContainers(ApplicationHistoryClientService.java:211)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationHistoryProtocolPBServiceImpl.getContainers(ApplicationHistoryProtocolPBServiceImpl.java:172)
> at 
> org.apache.hadoop.yarn.proto.ApplicationHistoryProtocol$ApplicationHistoryProtocolService$2.callBlockingMethod(ApplicationHistoryProtocol.java:201)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2309)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at 
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at 
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationHistoryProtocolPBClientImpl.getContainers(ApplicationHistoryProtocolPBClientImpl.java:183)
> at 
> org.apache.hadoop.yarn.client.api.impl.AHSClientImpl.getContainers(AHSClientImpl.java:151)
> at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:720)
> at 
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerReportsFromRunningApplication(LogsCLI.java:1089)
> at 
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainersLogRequestForRunningApplication(LogsCLI.java:1064)
> at 
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:976)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:300)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:107)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:327)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8567) Fetching yarn logs fails for long running application if it is not present in timeline store

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923399#comment-16923399
 ] 

Rohith Sharma K S commented on YARN-8567:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Fetching yarn logs fails for long running application if it is not present in 
> timeline store
> 
>
> Key: YARN-8567
> URL: https://issues.apache.org/jira/browse/YARN-8567
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 2.7.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
>  Labels: log-aggregation
> Fix For: 3.3.0
>
> Attachments: YARN-8567.001.patch, YARN-8567.002.patch
>
>
> Using yarn logs command for a long running application which has been running 
> longer than the configured timeline service ttl 
> {{yarn.timeline-service.ttl-ms }} fails with the following exception.
> {code:java}
> Exception in thread "main" 
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: The entity 
> for application application_152347939332_1 doesn't exist in the timeline 
> store
> at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getApplication(ApplicationHistoryManagerOnTimelineStore.java:670)
> at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerOnTimelineStore.getContainers(ApplicationHistoryManagerOnTimelineStore.java:219)
> at 
> org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryClientService.getContainers(ApplicationHistoryClientService.java:211)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationHistoryProtocolPBServiceImpl.getContainers(ApplicationHistoryProtocolPBServiceImpl.java:172)
> at 
> org.apache.hadoop.yarn.proto.ApplicationHistoryProtocol$ApplicationHistoryProtocolService$2.callBlockingMethod(ApplicationHistoryProtocol.java:201)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2309)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at 
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:101)
> at 
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationHistoryProtocolPBClientImpl.getContainers(ApplicationHistoryProtocolPBClientImpl.java:183)
> at 
> org.apache.hadoop.yarn.client.api.impl.AHSClientImpl.getContainers(AHSClientImpl.java:151)
> at 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getContainers(YarnClientImpl.java:720)
> at 
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerReportsFromRunningApplication(LogsCLI.java:1089)
> at 
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainersLogRequestForRunningApplication(LogsCLI.java:1064)
> at 
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchApplicationLogs(LogsCLI.java:976)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:300)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:107)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:327)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7537) [Atsv2] load hbase configuration from filesystem rather than URL

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7537:

Fix Version/s: 3.2.1

> [Atsv2] load hbase configuration from filesystem rather than URL
> 
>
> Key: YARN-7537
> URL: https://issues.apache.org/jira/browse/YARN-7537
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-7537-005.patch, YARN-7537-006.patch, 
> YARN-7537-03.patch, YARN-7537-04.patch, YARN-7537.01.patch, YARN-7537.02.patch
>
>
> Currently HBaseTimelineStorageUtils#getTimelineServiceHBaseConf loads hbase 
> configurations using URL if *yarn.timeline-service.hbase.configuration.file* 
> is configured. But it is restricted to URLs only. This need to be changed to 
> load from file system. In deployment, hbase configuration can be kept under 
> filesystem so that it be utilized by all the NodeManager and ResourceManager.
> cc :/ [~vrushalic] [~varun_saxena]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9335) [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9335:

Fix Version/s: 3.2.1

> [atsv2] Restrict the number of elements held in timeline collector when 
> backend is unreachable for async calls
> --
>
> Key: YARN-9335
> URL: https://issues.apache.org/jira/browse/YARN-9335
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: atvs
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9335.001.patch, YARN-9335.002.patch, 
> YARN-9335.003.patch, YARN-9335.004.patch
>
>
> For ATSv2 , if the backend is unreachable, the number/size of data held in 
> timeline collector's memory increases significantly. This is not good for the 
> NM memory. 
> Filing jira to set a limit on how many/much should be retained by the 
> timeline collector in memory in case the backend is not reachable.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8499) ATS v2 Generic TimelineStorageMonitor

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923397#comment-16923397
 ] 

Rohith Sharma K S commented on YARN-8499:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> ATS v2 Generic TimelineStorageMonitor
> -
>
> Key: YARN-8499
> URL: https://issues.apache.org/jira/browse/YARN-8499
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Reporter: Sunil Govindan
>Assignee: Prabhu Joseph
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0
>
> Attachments: YARN-8499-001.patch, YARN-8499-002.patch, 
> YARN-8499-003.patch, YARN-8499-004.patch, YARN-8499-005.patch, 
> YARN-8499-006.patch, YARN-8499-007.patch, YARN-8499-008.patch, 
> YARN-8499-009.patch, YARN-8499-010.patch, YARN-8499-011.patch, 
> YARN-8499-012.patch
>
>
> Post YARN-8302, Hbase connection issues are handled in ATSv2. However this 
> could be made general by introducing an api in storage interface and 
> implementing in each of the storage as per the store semantics.
>  
> cc [~rohithsharma] [~vinodkv] [~vrushalic]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7537) [Atsv2] load hbase configuration from filesystem rather than URL

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-7537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923396#comment-16923396
 ] 

Rohith Sharma K S commented on YARN-7537:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> [Atsv2] load hbase configuration from filesystem rather than URL
> 
>
> Key: YARN-7537
> URL: https://issues.apache.org/jira/browse/YARN-7537
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-7537-005.patch, YARN-7537-006.patch, 
> YARN-7537-03.patch, YARN-7537-04.patch, YARN-7537.01.patch, YARN-7537.02.patch
>
>
> Currently HBaseTimelineStorageUtils#getTimelineServiceHBaseConf loads hbase 
> configurations using URL if *yarn.timeline-service.hbase.configuration.file* 
> is configured. But it is restricted to URLs only. This need to be changed to 
> load from file system. In deployment, hbase configuration can be kept under 
> filesystem so that it be utilized by all the NodeManager and ResourceManager.
> cc :/ [~vrushalic] [~varun_saxena]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9374) HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage down

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923398#comment-16923398
 ] 

Rohith Sharma K S commented on YARN-9374:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage 
> down
> 
>
> Key: YARN-9374
> URL: https://issues.apache.org/jira/browse/YARN-9374
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9374-001.patch, YARN-9374-002.patch, 
> YARN-9374-003.patch, YARN-9374-004.patch, YARN-9374-005.patch, 
> YARN-9374-006.patch, YARN-9374-007.patch
>
>
> HBaseTimelineWriterImpl sync writes has to avoid thread blocking if storage 
> is down. Currently we check if hbase storage is down in TimelineReader before 
> reading entities and fail immediately in YARN-8302. Similar fix is needed for 
> write. Async is handled in YARN-9335.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9335) [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923395#comment-16923395
 ] 

Rohith Sharma K S commented on YARN-9335:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> [atsv2] Restrict the number of elements held in timeline collector when 
> backend is unreachable for async calls
> --
>
> Key: YARN-9335
> URL: https://issues.apache.org/jira/browse/YARN-9335
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: atvs
> Fix For: 3.3.0
>
> Attachments: YARN-9335.001.patch, YARN-9335.002.patch, 
> YARN-9335.003.patch, YARN-9335.004.patch
>
>
> For ATSv2 , if the backend is unreachable, the number/size of data held in 
> timeline collector's memory increases significantly. This is not good for the 
> NM memory. 
> Filing jira to set a limit on how many/much should be retained by the 
> timeline collector in memory in case the backend is not reachable.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-3841:

Fix Version/s: 3.2.1

> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: YARN-5355, atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch, 
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch, 
> YARN-3841.005.patch, YARN-3841.006.patch, YARN-3841.007.patch, 
> YARN-3841.008.patch, YARN-3841.009.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable. 
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily 
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the 
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of 
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-3879:

Fix Version/s: 3.2.1

> [Storage implementation] Create HDFS backing storage implementation for ATS 
> reads
> -
>
> Key: YARN-3879
> URL: https://issues.apache.org/jira/browse/YARN-3879
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: YARN-5355, YARN-7055
> Fix For: 2.10.0, 3.3.0, 3.2.1
>
> Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, 
> YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, 
> YARN-3879.005.patch, YARN-3879.006.patch
>
>
> Reader version of YARN-3841



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9389) FlowActivity and FlowRun table prefix is wrong

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9389:

Fix Version/s: 3.2.1

> FlowActivity and FlowRun table prefix is wrong
> --
>
> Key: YARN-9389
> URL: https://issues.apache.org/jira/browse/YARN-9389
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9389-001.patch
>
>
> FlowActivity and FlowRun table prefix configuration has double dots 
> yarn.timeline-service..flowactivity.table.name and 
> yarn.timeline-service..flowrun.table.name 
>   private static final String PREFIX =
>   YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowrun";
>   private static final String PREFIX =
>   YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowactivity";



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9387) Update document for ATS HBase Custom tablenames (-entityTableName)

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9387:

Fix Version/s: 3.2.1

> Update document for ATS HBase Custom tablenames (-entityTableName)
> --
>
> Key: YARN-9387
> URL: https://issues.apache.org/jira/browse/YARN-9387
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.1.2, 3.3.0, 3.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: Screen Shot 2019-03-15 at 1.21.21 PM.png, 
> YARN-9387-001.patch
>
>
> {{HbaseTimelineSchemaCreator}} provides option to provide custom table name 
> and it creates properly. But The {{HBaseTimelineWriterImpl / 
> HBaseTimelineReaderImpl}} does not know the custom name and uses the table 
> with default name leading to data loss.
> NM {{TimelineCollector}} inserts to default table name 
> '{{prod.timelineservice.entity' }} which won;t be exist.
> {code}
> 2019-03-14 15:37:10,739 WARN 
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 
> 20 actions: Table 'prod.timelineservice.entity' was not found, got: 
> prod.timelineservice.domain.: 20 times,
> at 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:197)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923253#comment-16923253
 ] 

Rohith Sharma K S commented on YARN-9418:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
> 
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch, 
> YARN-9418-003.patch, YARN-9418-branch-3.2.001.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s 
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
>  | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID": 
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID": 
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this 
> is not shown in above output. Found NM container entities are set with 
> entityIdPrefix as inverted container starttime whereas RM container entities 
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM 
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9389) FlowActivity and FlowRun table prefix is wrong

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923254#comment-16923254
 ] 

Rohith Sharma K S commented on YARN-9389:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> FlowActivity and FlowRun table prefix is wrong
> --
>
> Key: YARN-9389
> URL: https://issues.apache.org/jira/browse/YARN-9389
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9389-001.patch
>
>
> FlowActivity and FlowRun table prefix configuration has double dots 
> yarn.timeline-service..flowactivity.table.name and 
> yarn.timeline-service..flowrun.table.name 
>   private static final String PREFIX =
>   YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowrun";
>   private static final String PREFIX =
>   YarnConfiguration.TIMELINE_SERVICE_PREFIX + ".flowactivity";



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9418:

Fix Version/s: 3.2.1

> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
> 
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch, 
> YARN-9418-003.patch, YARN-9418-branch-3.2.001.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s 
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
>  | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID": 
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID": 
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this 
> is not shown in above output. Found NM container entities are set with 
> entityIdPrefix as inverted container starttime whereas RM container entities 
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM 
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923256#comment-16923256
 ] 

Rohith Sharma K S commented on YARN-3841:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: YARN-5355, atsv2
> Fix For: 3.3.0
>
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch, 
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch, 
> YARN-3841.005.patch, YARN-3841.006.patch, YARN-3841.007.patch, 
> YARN-3841.008.patch, YARN-3841.009.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable. 
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily 
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the 
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of 
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9387) Update document for ATS HBase Custom tablenames (-entityTableName)

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923255#comment-16923255
 ] 

Rohith Sharma K S commented on YARN-9387:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Update document for ATS HBase Custom tablenames (-entityTableName)
> --
>
> Key: YARN-9387
> URL: https://issues.apache.org/jira/browse/YARN-9387
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.1.2, 3.3.0, 3.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: Screen Shot 2019-03-15 at 1.21.21 PM.png, 
> YARN-9387-001.patch
>
>
> {{HbaseTimelineSchemaCreator}} provides option to provide custom table name 
> and it creates properly. But The {{HBaseTimelineWriterImpl / 
> HBaseTimelineReaderImpl}} does not know the custom name and uses the table 
> with default name leading to data loss.
> NM {{TimelineCollector}} inserts to default table name 
> '{{prod.timelineservice.entity' }} which won;t be exist.
> {code}
> 2019-03-14 15:37:10,739 WARN 
> org.apache.hadoop.yarn.webapp.GenericExceptionHandler: INTERNAL_SERVER_ERROR
> javax.ws.rs.WebApplicationException: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 
> 20 actions: Table 'prod.timelineservice.entity' was not found, got: 
> prod.timelineservice.domain.: 20 times,
> at 
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:197)
> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923257#comment-16923257
 ] 

Rohith Sharma K S commented on YARN-3879:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> [Storage implementation] Create HDFS backing storage implementation for ATS 
> reads
> -
>
> Key: YARN-3879
> URL: https://issues.apache.org/jira/browse/YARN-3879
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: YARN-5355, YARN-7055
> Fix For: 2.10.0, 3.3.0
>
> Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, 
> YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch, 
> YARN-3879.005.patch, YARN-3879.006.patch
>
>
> Reader version of YARN-3841



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9303) Username splits won't help timelineservice.app_flow table

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9303:

Fix Version/s: 3.2.1

> Username splits won't help timelineservice.app_flow table
> -
>
> Key: YARN-9303
> URL: https://issues.apache.org/jira/browse/YARN-9303
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>  Labels: atsv2, atsv2-hbase
> Fix For: 3.3.0, 3.2.1
>
> Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch
>
>
> timelineservice.app_flow hbase table uses pre split logic based on username 
> whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All 
> data will go to the last region and remaining regions will never be inserted. 
> Need to choose right split or use auto-split.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923251#comment-16923251
 ] 

Rohith Sharma K S commented on YARN-9382:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Publish container killed, paused and resumed events to ATSv2.
> -
>
> Key: YARN-9382
> URL: https://issues.apache.org/jira/browse/YARN-9382
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9382.001.patch, YARN-9382.002.patch, 
> YARN-9382.003.patch
>
>
> There are some events missing in container lifecycle. We need to add support 
> for adding events for when container gets killed, paused and resumed. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9382) Publish container killed, paused and resumed events to ATSv2.

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9382:

Fix Version/s: 3.2.1

> Publish container killed, paused and resumed events to ATSv2.
> -
>
> Key: YARN-9382
> URL: https://issues.apache.org/jira/browse/YARN-9382
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9382.001.patch, YARN-9382.002.patch, 
> YARN-9382.003.patch
>
>
> There are some events missing in container lifecycle. We need to add support 
> for adding events for when container gets killed, paused and resumed. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9303) Username splits won't help timelineservice.app_flow table

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923252#comment-16923252
 ] 

Rohith Sharma K S commented on YARN-9303:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Username splits won't help timelineservice.app_flow table
> -
>
> Key: YARN-9303
> URL: https://issues.apache.org/jira/browse/YARN-9303
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>  Labels: atsv2, atsv2-hbase
> Fix For: 3.3.0
>
> Attachments: Only_Last_Region_Used.png, YARN-9303-001.patch
>
>
> timelineservice.app_flow hbase table uses pre split logic based on username 
> whereas the rowkeys starts with inverted timestamp (Long.MAX_VALUE - ts). All 
> data will go to the last region and remaining regions will never be inserted. 
> Need to choose right split or use auto-split.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923204#comment-16923204
 ] 

Rohith Sharma K S commented on YARN-9418:
-

I am cherry picking this. Appears it easy to cherry pick.. I will back port 
using cherry pick!

> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
> 
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch, 
> YARN-9418-003.patch, YARN-9418-branch-3.2.001.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s 
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
>  | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID": 
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID": 
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this 
> is not shown in above output. Found NM container entities are set with 
> entityIdPrefix as inverted container starttime whereas RM container entities 
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM 
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8871) Document behavior of YARN-5742

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-8871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-8871:

Fix Version/s: 3.2.1

> Document behavior of YARN-5742
> --
>
> Key: YARN-8871
> URL: https://issues.apache.org/jira/browse/YARN-8871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-8871.1.patch, YARN-8871.2.patch, YARN-8871.3.patch
>
>
> YARN-5742 allows for serving aggregated logs of historical apps from timeline 
> service v2. Need the documentation updates for that. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6149) Allow port range to be specified while starting NM Timeline collector manager.

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-6149:

Fix Version/s: 3.2.1

> Allow port range to be specified while starting NM Timeline collector manager.
> --
>
> Key: YARN-6149
> URL: https://issues.apache.org/jira/browse/YARN-6149
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Abhishek Modi
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-6149.001.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6735) Have a way to turn off container metrics from NMs

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-6735:

Fix Version/s: 3.2.1

> Have a way to turn off container metrics from NMs
> -
>
> Key: YARN-6735
> URL: https://issues.apache.org/jira/browse/YARN-6735
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-6735.001.patch, YARN-6735.002.patch, 
> YARN-6735.003.patch
>
>
> Have a way to turn off emitting system metrics from NMs



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7754) [Atsv2] Update document for running v1 and v2 TS

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7754:

Fix Version/s: 3.2.1

> [Atsv2] Update document for running v1 and v2 TS
> 
>
> Key: YARN-7754
> URL: https://issues.apache.org/jira/browse/YARN-7754
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-7754.1.patch, YARN-7754.2.patch
>
>
> Post YARN-6736, RM can publish events into both v1 and v2 TS. Newer 
> configuration need to be updated in document. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5336:

Fix Version/s: 3.2.1

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355, atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-5336.001.patch, YARN-5336.002.patch, 
> YARN-5336.003.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9034) ApplicationCLI should have option to take clusterId

2019-09-05 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9034:

Fix Version/s: 3.2.1

> ApplicationCLI should have option to take clusterId
> ---
>
> Key: YARN-9034
> URL: https://issues.apache.org/jira/browse/YARN-9034
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0, 3.2.1
>
> Attachments: YARN-9034.01.patch, YARN-9034.02.patch, 
> YARN-9034.03.patch, YARN-9034.04.patch
>
>
> Post YARN-8303, LogsCLI provide an option to input clusterid which could be 
> used for fetching data from atsv2.  ApplicationCLI also should have this 
> option.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923128#comment-16923128
 ] 

Rohith Sharma K S commented on YARN-5336:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355, atsv2
> Fix For: 3.3.0
>
> Attachments: YARN-5336.001.patch, YARN-5336.002.patch, 
> YARN-5336.003.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6735) Have a way to turn off container metrics from NMs

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923127#comment-16923127
 ] 

Rohith Sharma K S commented on YARN-6735:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Have a way to turn off container metrics from NMs
> -
>
> Key: YARN-6735
> URL: https://issues.apache.org/jira/browse/YARN-6735
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0
>
> Attachments: YARN-6735.001.patch, YARN-6735.002.patch, 
> YARN-6735.003.patch
>
>
> Have a way to turn off emitting system metrics from NMs



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9034) ApplicationCLI should have option to take clusterId

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923125#comment-16923125
 ] 

Rohith Sharma K S commented on YARN-9034:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> ApplicationCLI should have option to take clusterId
> ---
>
> Key: YARN-9034
> URL: https://issues.apache.org/jira/browse/YARN-9034
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
>  Labels: atsv2
> Fix For: 3.3.0
>
> Attachments: YARN-9034.01.patch, YARN-9034.02.patch, 
> YARN-9034.03.patch, YARN-9034.04.patch
>
>
> Post YARN-8303, LogsCLI provide an option to input clusterid which could be 
> used for fetching data from atsv2.  ApplicationCLI also should have this 
> option.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-6149) Allow port range to be specified while starting NM Timeline collector manager.

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923126#comment-16923126
 ] 

Rohith Sharma K S commented on YARN-6149:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Allow port range to be specified while starting NM Timeline collector manager.
> --
>
> Key: YARN-6149
> URL: https://issues.apache.org/jira/browse/YARN-6149
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Abhishek Modi
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-6149.001.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7754) [Atsv2] Update document for running v1 and v2 TS

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923124#comment-16923124
 ] 

Rohith Sharma K S commented on YARN-7754:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> [Atsv2] Update document for running v1 and v2 TS
> 
>
> Key: YARN-7754
> URL: https://issues.apache.org/jira/browse/YARN-7754
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-7754.1.patch, YARN-7754.2.patch
>
>
> Post YARN-6736, RM can publish events into both v1 and v2 TS. Newer 
> configuration need to be updated in document. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-8871) Document behavior of YARN-5742

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-8871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923123#comment-16923123
 ] 

Rohith Sharma K S commented on YARN-8871:
-

back ported to branch-3.2. updated the fix version as 3.2.1

> Document behavior of YARN-5742
> --
>
> Key: YARN-8871
> URL: https://issues.apache.org/jira/browse/YARN-8871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Suma Shivaprasad
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-8871.1.patch, YARN-8871.2.patch, YARN-8871.3.patch
>
>
> YARN-5742 allows for serving aggregated logs of historical apps from timeline 
> service v2. Need the documentation updates for that. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9418) ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics

2019-09-05 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923091#comment-16923091
 ] 

Rohith Sharma K S commented on YARN-9418:
-

[~Prabhu Joseph] why this isn't back ported to branch-3.2?

> ATSV2 /apps/appId/entities/YARN_CONTAINER rest api does not show metrics
> 
>
> Key: YARN-9418
> URL: https://issues.apache.org/jira/browse/YARN-9418
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-9418-001.patch, YARN-9418-002.patch, 
> YARN-9418-003.patch
>
>
> ATSV2 entities rest api does not show the metrics
> {code:java}
> [hbase@yarn-ats-3 centos]$ curl -s 
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553685341603_0006/entities/YARN_CONTAINER/container_e18_1553685341603_0006_01_01?user.name=hbase=METRICS;
>  | jq .
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553695002014,
> "idprefix": 0,
> "type": "YARN_CONTAINER",
> "id": "container_e18_1553685341603_0006_01_01",
> "info": {
> "UID": 
> "ats!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01",
> "FROM_ID": 
> "ats!hbase!QuasiMonteCarlo!1553695001394!application_1553685341603_0006!YARN_CONTAINER!0!container_e18_1553685341603_0006_01_01"
> },
> "configs": {},
> "isrelatedto": {},
> "relatesto": {}
> }{code}
> NodeManager puts YARN_CONTAINER entities with CPU and MEMORY metrics but this 
> is not shown in above output. Found NM container entities are set with 
> entityIdPrefix as inverted container starttime whereas RM container entities 
> are set with default 0. TimelineReader fetches only RM container entries.
> Confirmed with setting NM container entities entityIdPrefix to 0 same as RM 
> (for testing purpose) and found metrics are shown.
> {code:java}
> "metrics": [
> {
> "type": "SINGLE_VALUE",
> "id": "MEMORY",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 490430464
> }
> },
> {
> "type": "SINGLE_VALUE",
> "id": "CPU",
> "aggregationOp": "NOP",
> "values": {
> "1553774981355": 5
> }
> }
> ]{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9804) Update ATSv2 document for latest feature supports

2019-09-04 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923012#comment-16923012
 ] 

Rohith Sharma K S commented on YARN-9804:
-

committing shortly

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch, YARN-9804.02.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9804) Update ATSv2 document for latest feature supports

2019-09-03 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921488#comment-16921488
 ] 

Rohith Sharma K S commented on YARN-9804:
-

[~vrushalic] Would you like to take a look at the doc? Otherwise I will commit 
it later of today!

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch, YARN-9804.02.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9804) Update ATSv2 document for latest feature supports

2019-09-02 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921169#comment-16921169
 ] 

Rohith Sharma K S commented on YARN-9804:
-

Updated the patch fixing review comments. [~abmodi] can you take a look at 
updated patch, please?

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch, YARN-9804.02.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9804) Update ATSv2 document for latest feature supports

2019-09-02 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9804:

Attachment: YARN-9804.02.patch

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch, YARN-9804.02.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9804) Update ATSv2 document for latest feature supports

2019-09-02 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921158#comment-16921158
 ] 

Rohith Sharma K S commented on YARN-9804:
-

Thanks [~abmodi] for the review. I will update the patch as per the comments.

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9804) Update ATSv2 document for latest feature supports

2019-08-30 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919602#comment-16919602
 ] 

Rohith Sharma K S commented on YARN-9804:
-

[~abmodi] [~vrushalic] [~Sushil-K-S] [~prabham] Could you please take a look at 
the doc update? 

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-9804) Update ATSv2 document for latest feature supports

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-9804:

Attachment: YARN-9804.01.patch

> Update ATSv2 document for latest feature supports
> -
>
> Key: YARN-9804
> URL: https://issues.apache.org/jira/browse/YARN-9804
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Blocker
> Attachments: YARN-9804.01.patch
>
>
> Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9802) YARN Timeline Service v2 (post GA features)

2019-08-30 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919281#comment-16919281
 ] 

Rohith Sharma K S commented on YARN-9802:
-

Cleaned up and moved all the outstanding subtask from YARN-7055. Feel free to 
assign from non-active community members.
cc:/ [~prabham] [~Sushil-K-S] [~abmodi] [~Prabhu Joseph] [~vrushalic]


> YARN Timeline Service v2 (post GA features)
> ---
>
> Key: YARN-9802
> URL: https://issues.apache.org/jira/browse/YARN-9802
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Vrushali C
>Priority: Major
>
> Epic jira to track new feature enhancements or issues after the next release 
> of Timeline Service v2



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9797) LeafQueue#activateApplications should use resourceCalculator#fitsIn

2019-08-30 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919276#comment-16919276
 ] 

Rohith Sharma K S commented on YARN-9797:
-

cc:/ [~sunilg] [~leftnoteasy]

> LeafQueue#activateApplications should use resourceCalculator#fitsIn
> ---
>
> Key: YARN-9797
> URL: https://issues.apache.org/jira/browse/YARN-9797
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9797-001.patch, YARN-9797-002.patch, 
> YARN-9797-003.patch, YARN-9797-004.patch, YARN-9797-005.patch
>
>
> Dominant resource calculator compare function check for dominant resource is 
> lessThan.
> Incase case of AM limit we should activate application only when all the 
> resourceValues are less than the AM limit.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics

2019-08-30 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919272#comment-16919272
 ] 

Rohith Sharma K S commented on YARN-3881:
-

[~prabham] feel free to assign your self as there is not response for longer 
period 

> Writing RM cluster-level metrics
> 
>
> Key: YARN-3881
> URL: https://issues.apache.org/jira/browse/YARN-3881
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Major
>  Labels: YARN-5355
> Attachments: metrics.json
>
>
> RM has a bunch of metrics that we may want to write into the timeline backend 
> to. I attached the metrics.json that I've crawled via 
> {{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to 
> three groups of metrics:
> 1. QueueMetrics
> 2. JvmMetrics
> 3. ClusterMetrics
> The problem is that unlike other metrics belongs to a single application, 
> these ones belongs to RM or cluster-wide. Therefore, current write path is 
> not going to work for these metrics because they don't have the associated 
> user/flow/app context info. We need to rethink of modeling cross-app metrics 
> and the api to handle them.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-4069) For long running apps (> 2 days), populate flow activity table

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-4069.
-
Resolution: Won't Fix

No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA 
which are not going implement. Hence, closing the JIRA as Won't Fix. 
Feel free to reopen if still same design approach exists.  

> For long running apps (> 2 days), populate flow activity table
> --
>
> Key: YARN-4069
> URL: https://issues.apache.org/jira/browse/YARN-4069
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Varun Saxena
>Priority: Major
>  Labels: YARN-5355
>
> YARN-4063 / YARN-3901 contain the work being done to populate the flow 
> activity and flow run tables.
> The flow activity table is updated each time a yarn application is created 
> and finishes. So if an application runs for more than 3 days, day1 has an 
> entry for the flow for start time, day3 has an entry for the flow for end 
> time but day2 has no entry for that flow. 
> Filing the jira to ensure that for long running apps, the flow activity table 
> does get a snapshot time entered for each day that an application is running 
> in that flow.
> It may be the case that for ALL apps (long running or not) the same update 
> may be done in the flow activity table.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-4343) Need to support Application History Server on ATSV2

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-4343.
-
Resolution: Duplicate

This functionality exist in latest trunk code. I don't remember which JIRA but 
[~abmodi] did it. 

If ATSv2 is enabled, then completed application which is not present in RM is 
fetched from ATSv2. 


> Need to support Application History Server on ATSV2
> ---
>
> Key: YARN-4343
> URL: https://issues.apache.org/jira/browse/YARN-4343
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>Priority: Major
>  Labels: YARN-5355
>
> AHS is used by the CLI and Webproxy(REST), if the application related 
> information is not found in RM then it tries to fetch from AHS and show



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-3865) Backward compatibility of reader with ATSv1

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-3865.
-
Resolution: Won't Fix

No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA 
which are not going implement. Hence, closing the JIRA as Won't Fix. 
Feel free to reopen if still same design approach exists.  

> Backward compatibility of reader with ATSv1
> ---
>
> Key: YARN-3865
> URL: https://issues.apache.org/jira/browse/YARN-3865
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Major
>  Labels: YARN-5355
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-5071) address HBase compatibility issues with trunk

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-5071.
-
Resolution: Won't Fix

No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA 
which are not going implement. Hence, closing the JIRA as Won't Fix. 
Feel free to reopen if still same design approach exists.  

Currently we support only hbase-1.2.4/5 hbase-2 versions. We have separate 
packages for both the versions. I guess this shouldn't be much issue as per 
current code in trunk.

> address HBase compatibility issues with trunk
> -
>
> Key: YARN-5071
> URL: https://issues.apache.org/jira/browse/YARN-5071
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
>  Labels: YARN-5355
>
> The trunk is now adding or planning to add more and more 
> backward-incompatible changes. Some examples include
> - remove v.1 metrics classes (HADOOP-12504)
> - update jersey version (HADOOP-9613)
> - target java 8 by default (HADOOP-11858)
> This poses big challenges for the timeline service v.2 as we have a 
> dependency on hbase which depends on an older version of hadoop.
> We need to find a way to solve/contain/manage these risks before it is too 
> late.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-4261) fix the order of timelinereader in yarn/yarn.cmd

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-4261.
-
Resolution: Won't Fix

No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA 
which are not going implement. Hence, closing the JIRA as Won't Fix. 
Feel free to reopen if still same design approach exists.  

> fix the order of timelinereader in yarn/yarn.cmd
> 
>
> Key: YARN-4261
> URL: https://issues.apache.org/jira/browse/YARN-4261
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Trivial
>  Labels: YARN-5355
>
> The order of the timelinereader command is not correct in yarn/yarn.cmd.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-3981) offline collector: support timeline clients not associated with an application

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-3981:
---

Assignee: (was: Rohith Sharma K S)

> offline collector: support timeline clients not associated with an application
> --
>
> Key: YARN-3981
> URL: https://issues.apache.org/jira/browse/YARN-3981
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-3981- offline-collector-draft.pdf
>
>
> In the current v.2 design, all timeline writes must belong in a 
> flow/application context (cluster + user + flow + flow run + application).
> But there are use cases that require writing data outside the context of an 
> application. One such example is a higher level client (e.g. tez client or 
> hive/oozie/cascading client) writing flow-level data that spans multiple 
> applications. We need to find a way to support them.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-5751) Support UNIT for TimelineMetric

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-5751:
---

Assignee: (was: Rohith Sharma K S)

> Support UNIT for TimelineMetric
> ---
>
> Key: YARN-5751
> URL: https://issues.apache.org/jira/browse/YARN-5751
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2
>Reporter: Rohith Sharma K S
>Priority: Major
>
> ATSv2 allows users to write its metrics using TimelineMetric. But, there is 
> no field to tell what is the UNIT of published metric. This is very difficult 
> when metrics are read. 
> I propose to add UNIT for TimelineMetric so that once user can use this field 
> to tell what is the unit of published metric.  May be this can be optional 
> for few kind or metrics where unit is not required say CPU. But definitely 
> there should be a way to set units while publishing the entities. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-6058) Support for listing all applications i.e /apps

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-6058:
---

Assignee: (was: Rohith Sharma K S)

> Support for listing all applications i.e /apps
> --
>
> Key: YARN-6058
> URL: https://issues.apache.org/jira/browse/YARN-6058
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Priority: Critical
>
> Primary use case for /apps is many execution engines runs on top of YARN 
> example, Tez, MR. These engines will have their own UI's which list specific 
> type of entities which are published by them Ex: DAG entities. 
> But, these UI's do not aware of either userName or flowName or applicationId 
> which are submitted by these engines.
> Currently, given that user do not aware of user, flownName, and 
> applicationId, then he can not retrieve any entities. 
> By supporting /apps with filters, user can list of application with given 
> ApplicationType. These applications can be used for retrieving engine 
> specific entities like DAG. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-5627) [Atsv2] Support streaming reader API to fetch entities

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-5627:
---

Assignee: (was: Rohith Sharma K S)

> [Atsv2] Support streaming reader API to fetch entities
> --
>
> Key: YARN-5627
> URL: https://issues.apache.org/jira/browse/YARN-5627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Priority: Major
>
> There is no limit for size of TimelineEntitie object. It can be varied from 
> KB's to MB. While reading entities list, it would be an potential issue that 
> TimeLineReder would go into OOM situation based on the entity size and limit. 
> Proposal is to support an streaming API to read entities list. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-7932) [atsv2 read acls] Add a API TimelineV2Client#putDomain to publish domain information.

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-7932:
---

Assignee: (was: Rohith Sharma K S)

> [atsv2 read acls] Add a API TimelineV2Client#putDomain to publish domain 
> information.
> -
>
> Key: YARN-7932
> URL: https://issues.apache.org/jira/browse/YARN-7932
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Priority: Major
> Attachments: YARN-7932.001.patch
>
>
>  
> Need to have a TimelineDomain class in atsv2 that can contain the acls info 
> for entities. TimelineEntity will contain TimelineDomain as a member 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-7272) Enable timeline collector fault tolerance

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S reassigned YARN-7272:
---

Assignee: (was: Rohith Sharma K S)

> Enable timeline collector fault tolerance
> -
>
> Key: YARN-7272
> URL: https://issues.apache.org/jira/browse/YARN-7272
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Priority: Major
> Attachments: YARN-7272-wip.patch
>
>
> If a NM goes down and along with it the timeline collector aux service for a 
> running yarn app, we would like that yarn app to re-establish connection with 
> a new timeline collector. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-4504) Retrospect on defaults for created time while querying

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-4504.
-
Resolution: Won't Fix

No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA 
which are not going implement. Hence, closing the JIRA as Won't Fix. 
Feel free to reopen if still same design approach exists.  

> Retrospect on defaults for created time while querying
> --
>
> Key: YARN-4504
> URL: https://issues.apache.org/jira/browse/YARN-4504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-4504) Retrospect on defaults for created time while querying

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4504:

Target Version/s:   (was: YARN-2928)

> Retrospect on defaults for created time while querying
> --
>
> Key: YARN-4504
> URL: https://issues.apache.org/jira/browse/YARN-4504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-4504) Retrospect on defaults for created time while querying

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-4504:

Labels:   (was: YARN-5355)

> Retrospect on defaults for created time while querying
> --
>
> Key: YARN-4504
> URL: https://issues.apache.org/jira/browse/YARN-4504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-5760) [ATSv2] Create HBase connection only if an app collector is publishing from NM

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S resolved YARN-5760.
-
Resolution: Won't Fix

No activity for long time. In ATSv2 weekly call, discussed for cleaning up JIRA 
which are not going implement. Hence, closing the JIRA as Won't Fix. 
Feel free to reopen if still same design approach exists.  

> [ATSv2] Create HBase connection only if an app collector is publishing from NM
> --
>
> Key: YARN-5760
> URL: https://issues.apache.org/jira/browse/YARN-5760
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Major
>
> Irrespective of NM handling an app or not, we initialize 
> HBaseTimelineWriterImpl in TimelineCollectorManager.
> This in turn calls ConnectionFactory#createConnection to manage connections 
> with HBase. 
> But it seems this opens up a connection with Zookeeper (i.e. as soon as NM 
> starts up) instead of opening connection when atleast one app arrives for 
> publishing and closing it if no apps are being published from this NM.
> This leads to unnecessary connections to Zookeeper.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-4821) Have a separate NM timeline publishing-interval

2019-08-30 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919256#comment-16919256
 ] 

Rohith Sharma K S commented on YARN-4821:
-

[~abmodi] Feel free to assign yourself as no activity for long time other 
community members

> Have a separate NM timeline publishing-interval
> ---
>
> Key: YARN-4821
> URL: https://issues.apache.org/jira/browse/YARN-4821
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Naganarasimha G R
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-4821-YARN-2928.v1.001.patch
>
>
> Currently the interval with which NM publishes container CPU and memory 
> metrics is tied to {{yarn.nodemanager.resource-monitor.interval-ms}} whose 
> default is 3 seconds. This is too aggressive.
> There should be a separate configuration that controls how often 
> {{NMTimelinePublisher}} publishes container metrics.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-9804) Update ATSv2 document for latest feature supports

2019-08-30 Thread Rohith Sharma K S (Jira)
Rohith Sharma K S created YARN-9804:
---

 Summary: Update ATSv2 document for latest feature supports
 Key: YARN-9804
 URL: https://issues.apache.org/jira/browse/YARN-9804
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Rohith Sharma K S
Assignee: Rohith Sharma K S


Revisit ATSv2 documents and update for GA features. And also for the road map.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-9785) Fix DominantResourceCalculator when one resource is zero

2019-08-30 Thread Rohith Sharma K S (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919250#comment-16919250
 ] 

Rohith Sharma K S commented on YARN-9785:
-

[~sunilg]/[~leftnoteasy] Does this patch good to go?

> Fix DominantResourceCalculator when one resource is zero
> 
>
> Key: YARN-9785
> URL: https://issues.apache.org/jira/browse/YARN-9785
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Blocker
> Attachments: YARN-9785-001.patch, YARN-9785.002.patch, 
> YARN-9785.wip.patch
>
>
> Configure below property in resource-types.xml
> {quote}
>  yarn.resource-types
>  yarn.io/gpu
>  
> {quote}
> Submit applications even after AM limit for a queue is reached. Applications 
> get activated even after limit is reached
> !queue.png!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6105) Support for new REST end point /clusterids

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-6105:

Parent: YARN-9802  (was: YARN-7055)

> Support for new REST end point /clusterids
> --
>
> Key: YARN-6105
> URL: https://issues.apache.org/jira/browse/YARN-6105
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Sushil Ks
>Priority: Major
>
> As discussed in YARN-5378 and YARN-6095, it is required to have */clusterids* 
> that returns list of clusterids that back end has is useful. 
> Use case : In cloud, clusters are arbitrarily spin up and destroyed. Each 
> cluster has its own clusterId which UI never knows about it. To all those 
> newly spin up cluster, same ATS server has been used. And sam web UI has been 
> used. Admin can select the clusterId and navigate to any pages. So, it is 
> worth to list ClusterId's from ATS



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7272) Enable timeline collector fault tolerance

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7272:

Parent: YARN-9802  (was: YARN-7055)

> Enable timeline collector fault tolerance
> -
>
> Key: YARN-7272
> URL: https://issues.apache.org/jira/browse/YARN-7272
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-7272-wip.patch
>
>
> If a NM goes down and along with it the timeline collector aux service for a 
> running yarn app, we would like that yarn app to re-establish connection with 
> a new timeline collector. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-5357) Timeline service v2 integration with Federation

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5357:

Parent: YARN-9802  (was: YARN-7055)

> Timeline service v2 integration with Federation 
> 
>
> Key: YARN-5357
> URL: https://issues.apache.org/jira/browse/YARN-5357
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>
> Jira to note the discussion points from an initial chat about integrating 
> Timeline Service v2 with Federation (YARN-2915).
> cc [~subru] [~curino] 
> For Federation:
> - all entities that belong to the same flow run should have the same cluster 
> name
> - app id in the same flow run strongly ordered in time
> - need a logical cluster name and physical cluster name
> - a possibility to implement the Application TimelineCollector as an 
> interceptor in the AMRMProxyService.
> For Timeline Service:
> - need to store physical cluster id and logical cluster id so that we don't 
> lose information at any level (flow/app/entity etc)
> - add a  new table app id to cluster mapping table
> - need a different entity table/some table to store node level metrics for 
> physical cluster stats. Once we get to node-level rollup, we probably have to 
> store something in a dc, cluster, rack, node hierarchy. In that case a 
> physical cluster makes sense, but we'd still need some way to tie physical 
> and logical together in order to make automatic error detection etc that 
> we're envisioning feasible within a federated setup.
> For the Cluster Naming convention:
> - three situations for cluster name:
> > app submitted to router should take federated (aka logical) cluster name
> > app submitted directly to RM should take physical cluster name
> > Info about the physical cluster  in entities?
> - suggestion to set the cluster name as yarn tag at the router level (in the 
> app submission context) 
> Other points to note:
> - for federation to work smoothly in environments that use HDFS some 
> additional considerations are needed, and possibly some solution like what is 
> being used at Twitter with the nFly approach.
> Email thread context:
> {code}
> -- Forwarded message --
> From: Joep Rottinghuis 
> Date: Fri, Jul 8, 2016 at 1:22 PM
> Subject: Re: Federation -Timeline Service meeting notes
> To: Subramaniam Venkatraman Krishnan 
> Cc: Sangjin Lee, Vrushali Channapattan , Carlo Curino
> Thanks for the notes.
> I think that for federation to work smoothly in environments that use HDFS 
> some additional considerations are needed, and possibly some solution like 
> what we're using at Twitter with our nFly approach.
> bq. - need a different entity table/some table to store node level metrics 
> for physical cluster stats
> Once we get to node-level rollup, we probably have to store something in a 
> dc, cluster, rack, node hierarchy. In that case a physical cluster makes 
> sense, but we'd still need some way to tie physical and logical together in 
> order to make automatic error detection etc that we're envisioning feasible 
> within a federated setup.
> Cheers,
> Joep
> On Fri, Jul 8, 2016 at 1:00 PM, Subramaniam Venkatraman Krishnan  wrote:
> Thanks Vrushali for crisply capturing the essential from our rambling 
> discussion J.
>  
> Sangjin, I just want to add one comment to yours – we want to retain the 
> physical cluster name (possibly as a new entity type) so that we don’t lose 
> information & we can cluster level rollups even if they are not efficient.
>  
> Additionally, based on the walkthrough of Federation design:
> · There was general agreement with the proposed approach.
> · There is a possibility to implement the Application 
> TimelineCollector as an interceptor in the AMRMProxyService.
> · Joep raised the concern that it would be better if the RMs 
> obtain the epoch from FederationStateStore. This is not currently in the 
> roadmap of our MVP but we definitely plan to address this in future.
>  
> Regards,
> Subru
>  
> From: Sangjin Lee
> Sent: Thursday, July 07, 2016 6:22 PM
> To: Vrushali Channapattan 
> Cc: Joep Rottinghuis; Carlo Curino; Subramaniam Venkatraman Krishnan 
> Subject: Re: Federation -Timeline Service meeting notes
>  
> Thanks for the summary Vrushali!
>  
> Just so that we're on the same page regarding the terminology, I 
> understand we're using the terms "logical cluster" and "federated cluster" 
> interchangeably.
>  
> Also, between using the federated cluster name and the home cluster name 
> as a solution, I think we were leaning towards the federated cluster name 
> (although not 

[jira] [Updated] (YARN-7271) Add a yarn application cost calculation framework in TimelineService v2

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7271:

Parent: YARN-9802  (was: YARN-7055)

> Add a yarn application cost calculation framework in TimelineService v2
> ---
>
> Key: YARN-7271
> URL: https://issues.apache.org/jira/browse/YARN-7271
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>
> Timeline Service v2 captures information about a yarn application. From this 
> info, we would like to calculate the "cost" of an yarn application. This 
> would be rolled up to the flow level  as well (and user and queue level 
> eventually).
> We need a way to accept machine cost (TCO per day) and enable this 
> calculation. This will help in chargeback for yarn apps. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8087) Allow YARN ATSv2 ACLs to be disabled

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-8087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-8087:

Parent: YARN-9802  (was: YARN-7055)

> Allow YARN ATSv2 ACLs to be disabled
> 
>
> Key: YARN-8087
> URL: https://issues.apache.org/jira/browse/YARN-8087
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineservice
>Affects Versions: 3.1.0
>Reporter: Haibo Chen
>Priority: Major
>
> YARN-3895 supports ACLs in ATSv2. We should allow admins to disable ACLs if 
> they decide that they do not need such feature



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7060) [ATSv2 Security] Consider using delegation token for publishing entities from NM

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7060:

Parent: YARN-9802  (was: YARN-7055)

> [ATSv2 Security] Consider using delegation token for publishing entities from 
> NM
> 
>
> Key: YARN-7060
> URL: https://issues.apache.org/jira/browse/YARN-7060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Assignee: Abhishek Modi
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8313) [atsv2 read acls] Update NM to include ACLs for system entities

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-8313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-8313:

Parent: YARN-9802  (was: YARN-7055)

> [atsv2 read acls] Update NM to include ACLs for system entities 
> 
>
> Key: YARN-8313
> URL: https://issues.apache.org/jira/browse/YARN-8313
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Prabhu Joseph
>Priority: Major
>
> NM should be updated to include the Application ACLs for system entities 
> while writing to the backend for timeline service. This will be used for read 
> authorization of system entities when querying information from timeline 
> service. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7979) Do ACLs check for application entities

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7979:

Parent: YARN-9802  (was: YARN-7055)

> Do ACLs check for application entities
> --
>
> Key: YARN-7979
> URL: https://issues.apache.org/jira/browse/YARN-7979
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
>
> Verify ACLs for application entities



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6070) Support substring match in filters

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-6070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-6070:

Parent: YARN-9802  (was: YARN-7055)

> Support substring match in filters
> --
>
> Key: YARN-6070
> URL: https://issues.apache.org/jira/browse/YARN-6070
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Sreenath Somarajapuram
>Assignee: Prabhu Joseph
>Priority: Major
>
> Current filter functionalities are good, but could be better if it supports 
> substring match. 
> Checked with the HBase guys, and they were of the opinion that its feasible.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7980) Do ACLs check for generic entities

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7980:

Parent: YARN-9802  (was: YARN-7055)

> Do ACLs check for generic entities
> --
>
> Key: YARN-7980
> URL: https://issues.apache.org/jira/browse/YARN-7980
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Haibo Chen
>Priority: Major
>
> This JIRA is to verify ACLs for generic entities.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-6699) [ATSv2 Security] Delegation token support for Timeline reader

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-6699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-6699:

Parent: YARN-9802  (was: YARN-7055)

> [ATSv2 Security] Delegation token support for Timeline reader
> -
>
> Key: YARN-6699
> URL: https://issues.apache.org/jira/browse/YARN-6699
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7932) [atsv2 read acls] Add a API TimelineV2Client#putDomain to publish domain information.

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7932:

Parent: YARN-9802  (was: YARN-7055)

> [atsv2 read acls] Add a API TimelineV2Client#putDomain to publish domain 
> information.
> -
>
> Key: YARN-7932
> URL: https://issues.apache.org/jira/browse/YARN-7932
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-7932.001.patch
>
>
>  
> Need to have a TimelineDomain class in atsv2 that can contain the acls info 
> for entities. TimelineEntity will contain TimelineDomain as a member 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-8314) [atsv2 read acls] Update timeline collector to include domain for entities

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-8314:

Parent: YARN-9802  (was: YARN-7055)

> [atsv2 read acls] Update timeline collector to include domain for entities
> --
>
> Key: YARN-8314
> URL: https://issues.apache.org/jira/browse/YARN-8314
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>
> The timeline collector should be updated to add the domain info for entities 
> that do not contain it. The putdomain call will be invoked first by the AM so 
> that the collector & backend are aware of this domain information for that 
> application. From then on, the collector will ensure appending of this 
> information to entities written to timeline service.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7977) Do ACLs check for flow activity entities

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7977:

Parent: YARN-9802  (was: YARN-7055)

> Do ACLs check for flow activity entities
> 
>
> Key: YARN-7977
> URL: https://issues.apache.org/jira/browse/YARN-7977
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
>
> Verify ACLs while retrieving flow activity entities



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7061) [ATSv2 Security] Use token for communication between NM and node collector

2019-08-30 Thread Rohith Sharma K S (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-7061:

Parent: YARN-9802  (was: YARN-7055)

> [ATSv2 Security] Use token for communication between NM and node collector
> --
>
> Key: YARN-7061
> URL: https://issues.apache.org/jira/browse/YARN-7061
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Varun Saxena
>Priority: Major
>
> We can consider using token for communication between NM and node collector 
> once collector runs outside NM.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



  1   2   3   4   5   6   7   8   9   10   >