[jira] [Commented] (YARN-7376) YARN top ACLs
[ https://issues.apache.org/jira/browse/YARN-7376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213759#comment-16213759 ] Hadoop QA commented on YARN-7376: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 43s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 45s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 20s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 95m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:ca8ddc6 | | JIRA Issue | YARN-7376 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12893394/YARN-7376.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 3bf92bd7770b 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revisi
[jira] [Created] (YARN-7378) Documentation changes post branch-2 merge
Varun Saxena created YARN-7378: -- Summary: Documentation changes post branch-2 merge Key: YARN-7378 URL: https://issues.apache.org/jira/browse/YARN-7378 Project: Hadoop YARN Issue Type: Sub-task Reporter: Varun Saxena -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5102) timeline service build fails with java 8
[ https://issues.apache.org/jira/browse/YARN-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5102: --- Fix Version/s: 2.9.0 > timeline service build fails with java 8 > > > Key: YARN-5102 > URL: https://issues.apache.org/jira/browse/YARN-5102 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Blocker > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5102-YARN-2928.01.patch > > > The build fails with java 8: > {noformat} > [WARNING] > Dependency convergence error for jdk.tools:jdk.tools:1.8 paths to dependency > are: > +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT > +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT > +-jdk.tools:jdk.tools:1.8 > and > +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT > +-org.apache.hbase:hbase-common:1.0.1 > +-org.apache.hbase:hbase-annotations:1.0.1 > +-jdk.tools:jdk.tools:1.7 > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence > failed with message: > Failed while enforcing releasability the error(s) are [ > Dependency convergence error for jdk.tools:jdk.tools:1.8 paths to dependency > are: > +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT > +-org.apache.hadoop:hadoop-annotations:3.0.0-SNAPSHOT > +-jdk.tools:jdk.tools:1.8 > and > +-org.apache.hadoop:hadoop-yarn-server-timelineservice:3.0.0-SNAPSHOT > +-org.apache.hbase:hbase-common:1.0.1 > +-org.apache.hbase:hbase-annotations:1.0.1 > +-jdk.tools:jdk.tools:1.7 > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3862) Support for fetching specific configs and metrics based on prefixes
[ https://issues.apache.org/jira/browse/YARN-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3862: --- Fix Version/s: 2.9.0 > Support for fetching specific configs and metrics based on prefixes > --- > > Key: YARN-3862 > URL: https://issues.apache.org/jira/browse/YARN-3862 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3862-YARN-2928.wip.01.patch, > YARN-3862-YARN-2928.wip.02.patch, YARN-3862-feature-YARN-2928.005.patch, > YARN-3862-feature-YARN-2928.04.patch, YARN-3862-feature-YARN-2928.wip.03.patch > > > Currently, we will retrieve all the contents of the field if that field is > specified in the query API. In case of configs and metrics, this can become a > lot of data even though the user doesn't need it. So we need to provide a way > to query only a set of configs or metrics. > As a comma spearated list of configs/metrics to be returned will be quite > cumbersome to specify, we have to support either of the following options : > # Prefix match > # Regex > # Group the configs/metrics and query that group. > We also need a facility to specify a metric time window to return metrics in > a that window. This may be useful in plotting graphs -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3461) Consolidate flow name/version/run defaults
[ https://issues.apache.org/jira/browse/YARN-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3461: --- Fix Version/s: 2.9.0 > Consolidate flow name/version/run defaults > -- > > Key: YARN-3461 > URL: https://issues.apache.org/jira/browse/YARN-3461 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3461-YARN-2928.01.patch, > YARN-3461-YARN-2928.02.patch, YARN-3461-YARN-2928.03.patch > > > In YARN-3391, it's not resolved what should be the defaults for flow > name/version/run. Let's continue the discussion here and unblock YARN-3391 > from moving forward. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3993) Change to use the AM flag in ContainerContext determine AM container
[ https://issues.apache.org/jira/browse/YARN-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3993: --- Fix Version/s: 2.9.0 > Change to use the AM flag in ContainerContext determine AM container > > > Key: YARN-3993 > URL: https://issues.apache.org/jira/browse/YARN-3993 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Zhijie Shen >Assignee: Sunil G > Labels: newbie > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-3993-YARN-2928.patch, > YARN-3993-YARN-2928.0001.patch > > > After YARN-3116, we will have a flag in ContainerContext to determine if the > container is AM or not in aux service. We need to change accordingly to make > use of this feature instead of depending on container ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4711) NM is going down with NPE's due to single thread processing of events by Timeline client
[ https://issues.apache.org/jira/browse/YARN-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4711: --- Fix Version/s: 2.9.0 > NM is going down with NPE's due to single thread processing of events by > Timeline client > > > Key: YARN-4711 > URL: https://issues.apache.org/jira/browse/YARN-4711 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: 4711Analysis.txt, YARN-4711-YARN-2928.v1.001.patch, > YARN-4711-YARN-2928.v1.002.patch > > > After YARN-3367, while testing the latest 2928 branch came across few NPEs > due to which NM is shutting down. > {code} > 2016-02-21 23:19:54,078 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: > Error in dispatcher thread > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:306) > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ContainerEventHandler.handle(NMTimelinePublisher.java:296) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109) > at java.lang.Thread.run(Thread.java:745) > {code} > {code} > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:213) > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.publishContainerFinishedEvent(NMTimelinePublisher.java:192) > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.access$400(NMTimelinePublisher.java:63) > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:289) > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$ApplicationEventHandler.handle(NMTimelinePublisher.java:280) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:183) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109) > at java.lang.Thread.run(Thread.java:745) > {code} > On analysis found that the there was delay in processing of events, as after > YARN-3367 all the events were getting processed by a single thread inside the > timeline client. > Additionally found one scenario where there is possibility of NPE: > * TimelineEntity.toString() when {{real}} is not null -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7038) [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled
[ https://issues.apache.org/jira/browse/YARN-7038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7038: --- Fix Version/s: 2.9.0 > [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when > service authorization is enabled > -- > > Key: YARN-7038 > URL: https://issues.apache.org/jira/browse/YARN-7038 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-5355 >Reporter: Rohith Sharma K S >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-7038-YARN-5355.01.patch, > YARN-7038-YARN-5355.02.patch, YARN-7038-YARN-5355.03.patch > > > Below error appears in the log when authorization is enabled. > {noformat} > 2017-08-17 11:16:40,664 ERROR collector.NodeTimelineCollectorManager > (NodeTimelineCollectorManager.java:doPostPut(227)) - Failed to communicate > with NM Collector Service for application_1502964541476_0001 > 2017-08-17 11:16:40,665 WARN containermanager.AuxServices > (AuxServices.java:logWarningWhenAuxServiceThrowExceptions(283)) - The > auxService name is timeline_collector and it got an error at event: > CONTAINER_INIT > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.security.authorize.AuthorizationException: Protocol > interface org.apache.hadoop.yarn.server.api.CollectorNodemanagerProtocolPB is > not known. > at > org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.putIfAbsent(TimelineCollectorManager.java:146) > at > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5109) timestamps are stored unencoded causing parse errors
[ https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5109: --- Fix Version/s: 2.9.0 > timestamps are stored unencoded causing parse errors > > > Key: YARN-5109 > URL: https://issues.apache.org/jira/browse/YARN-5109 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Blocker > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5109-YARN-2928.003.patch, > YARN-5109-YARN-2928.01.patch, YARN-5109-YARN-2928.02.patch, > YARN-5109-YARN-2928.03.patch, YARN-5109-YARN-2928.04.patch, > YARN-5109-YARN-2928.05.patch, YARN-5109-YARN-2928.06.patch, > YARN-5109-YARN-2928.07.patch, YARN-5109-YARN-2928.08.patch > > > When we store timestamps (for example as part of the row key or part of the > column name for an event), the bytes are used as is without any encoding. If > the byte value happens to contain a separator character we use (e.g. "!" or > "="), it causes a parse failure when we read it. > I came across this while looking into this error in the timeline reader: > {noformat} > 2016-05-17 21:28:38,643 WARN > org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils: > incorrectly formatted column name: it will be discarded > {noformat} > I traced the data that was causing this, and the column name (for the event) > was the following: > {noformat} > i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST > {noformat} > Note that the column name is supposed to be of the format (event > id)=(timestamp)=(event info key). However, observe the timestamp portion: > {noformat} > \x7F\xFF\xFE\xABDY=\x99 > {noformat} > The presence of the separator ("=") causes the parse error. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5142) fix findbugs warnings/errors for hadoop-yarn-server-timelineservice-hbase-tests
[ https://issues.apache.org/jira/browse/YARN-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5142: --- Fix Version/s: 2.9.0 > fix findbugs warnings/errors for > hadoop-yarn-server-timelineservice-hbase-tests > --- > > Key: YARN-5142 > URL: https://issues.apache.org/jira/browse/YARN-5142 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5142-YARN-2928.01.patch > > > Fix the errors/warnings reported for > hadoop-yarn-server-timelineservice-hbase-tests once YARN-5138 is in -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6146) Add Builder methods for TimelineEntityFilters
[ https://issues.apache.org/jira/browse/YARN-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6146: --- Fix Version/s: 2.9.0 > Add Builder methods for TimelineEntityFilters > - > > Key: YARN-6146 > URL: https://issues.apache.org/jira/browse/YARN-6146 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6146-YARN-5355.01.patch, > YARN-6146-YARN-5355.02.patch, YARN-6146-YARN-5355.03.patch, > YARN-6146.01.patch, YARN-6146.02.patch, YARN-6146.03.patch > > > The timeline filters are evolving and can be add more and more filters. It is > better to start using Builder methods rather than changing constructor every > time for adding new filters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6555) Store application flow context in NM state store for work-preserving restart
[ https://issues.apache.org/jira/browse/YARN-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6555: --- Fix Version/s: 2.9.0 > Store application flow context in NM state store for work-preserving restart > > > Key: YARN-6555 > URL: https://issues.apache.org/jira/browse/YARN-6555 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 >Reporter: Vrushali C >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6555.001.patch, YARN-6555.002.patch, > YARN-6555.003.patch > > > If timeline service v2 is enabled and NM is restarted with recovery enabled, > then NM fails to start and throws an error as "flow context can't be null". > This is happening because the flow context did not exist before but now that > timeline service v2 is enabled, ApplicationImpl expects it to exist. > This would also happen even if flow context existed before but since we are > not persisting it / reading it during > ContainerManagerImpl#recoverApplication, it does not get passed in to > ApplicationImpl. > full stack trace > {code} > 2017-05-03 21:51:52,178 FATAL > org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting > NodeManager > java.lang.IllegalArgumentException: flow context cannot be null > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:104) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:90) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:318) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:280) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:267) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:276) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:588) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:649) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3816) [Aggregation] App-level aggregation and accumulation for YARN system metrics
[ https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3816: --- Fix Version/s: 2.9.0 > [Aggregation] App-level aggregation and accumulation for YARN system metrics > > > Key: YARN-3816 > URL: https://issues.apache.org/jira/browse/YARN-3816 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Li Lu > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Application Level Aggregation of Timeline Data.pdf, > YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch, > YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch, > YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.1.patch, > YARN-3816-YARN-2928-v3.patch, YARN-3816-YARN-2928-v4.patch, > YARN-3816-YARN-2928-v5.patch, YARN-3816-YARN-2928-v6.patch, > YARN-3816-YARN-2928-v7.patch, YARN-3816-YARN-2928-v8.patch, > YARN-3816-YARN-2928-v9.patch, YARN-3816-feature-YARN-2928.v4.1.patch, > YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch > > > We need application level aggregation of Timeline data: > - To present end user aggregated states for each application, include: > resource (CPU, Memory) consumption across all containers, number of > containers launched/completed/failed, etc. We need this for apps while they > are running as well as when they are done. > - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be > aggregated to show details of states in framework level. > - Other level (Flow/User/Queue) aggregation can be more efficient to be based > on Application-level aggregations rather than raw entity-level data as much > less raws need to scan (with filter out non-aggregated entities, like: > events, configurations, etc.). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6134) [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active
[ https://issues.apache.org/jira/browse/YARN-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6134: --- Fix Version/s: 2.9.0 > [ATSv2 Security] Regenerate delegation token for app just before token > expires if app collector is active > - > > Key: YARN-6134 > URL: https://issues.apache.org/jira/browse/YARN-6134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6134-YARN-5355.01.patch, > YARN-6134-YARN-5355.02.patch, YARN-6134-YARN-5355.03.patch, > YARN-6134-YARN-5355.04.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3125) [Event producers] Change distributed shell to use new timeline service
[ https://issues.apache.org/jira/browse/YARN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3125: --- Fix Version/s: 2.9.0 > [Event producers] Change distributed shell to use new timeline service > -- > > Key: YARN-3125 > URL: https://issues.apache.org/jira/browse/YARN-3125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3125.patch, YARN-3125_UT-022615.patch, > YARN-3125_UT-022715.patch, YARN-3125v2.patch, YARN-3125v3.patch > > > We can start with changing distributed shell to use new timeline service once > the framework is completed, in which way we can quickly verify the next gen > is working fine end-to-end. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3901) Populate flow run data in the flow_run & flow activity tables
[ https://issues.apache.org/jira/browse/YARN-3901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3901: --- Fix Version/s: 2.9.0 > Populate flow run data in the flow_run & flow activity tables > - > > Key: YARN-3901 > URL: https://issues.apache.org/jira/browse/YARN-3901 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3901-YARN-2928.1.patch, > YARN-3901-YARN-2928.10.patch, YARN-3901-YARN-2928.2.patch, > YARN-3901-YARN-2928.3.patch, YARN-3901-YARN-2928.4.patch, > YARN-3901-YARN-2928.5.patch, YARN-3901-YARN-2928.6.patch, > YARN-3901-YARN-2928.7.patch, YARN-3901-YARN-2928.8.patch, > YARN-3901-YARN-2928.9.patch > > > As per the schema proposed in YARN-3815 in > https://issues.apache.org/jira/secure/attachment/12743391/hbase-schema-proposal-for-aggregation.pdf > filing jira to track creation and population of data in the flow run table. > Some points that are being considered: > - Stores per flow run information aggregated across applications, flow version > RM’s collector writes to on app creation and app completion > - Per App collector writes to it for metric updates at a slower frequency > than the metric updates to application table > primary key: cluster ! user ! flow ! flow run id > - Only the latest version of flow-level aggregated metrics will be kept, even > if the entity and application level keep a timeseries. > - The running_apps column will be incremented on app creation, and > decremented on app completion. > - For min_start_time the RM writer will simply write a value with the tag for > the applicationId. A coprocessor will return the min value of all written > values. - > - Upon flush and compactions, the min value between all the cells of this > column will be written to the cell without any tag (empty tag) and all the > other cells will be discarded. > - Ditto for the max_end_time, but then the max will be kept. > - Tags are represented as #type:value. The type can be not set (0), or can > indicate running (1) or complete (2). In those cases (for metrics) only > complete app metrics are collapsed on compaction. > - The m! values are aggregated (summed) upon read. Only when applications are > completed (indicated by tag type 2) can the values be collapsed. > - The application ids that have completed and been aggregated into the flow > numbers are retained in a separate column for historical tracking: we don’t > want to re-aggregate for those upon replay > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4455) Support fetching metrics by time range
[ https://issues.apache.org/jira/browse/YARN-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4455: --- Fix Version/s: 2.9.0 > Support fetching metrics by time range > -- > > Key: YARN-4455 > URL: https://issues.apache.org/jira/browse/YARN-4455 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-4455-YARN-5355.01.patch, > YARN-4455-YARN-5355.02.patch, YARN-4455-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes
[ https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4765: --- Fix Version/s: 2.9.0 > Split TestHBaseTimelineStorage into multiple test classes > - > > Key: YARN-4765 > URL: https://issues.apache.org/jira/browse/YARN-4765 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, atsv2-hbase, oct16-medium > Fix For: 2.9.0, 3.0.0-alpha2, YARN-5355 > > Attachments: YARN-4765-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6159) Documentation changes for TimelineV2Client
[ https://issues.apache.org/jira/browse/YARN-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6159: --- Fix Version/s: 2.9.0 > Documentation changes for TimelineV2Client > -- > > Key: YARN-6159 > URL: https://issues.apache.org/jira/browse/YARN-6159 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Reporter: Varun Saxena >Assignee: Naganarasimha G R >Priority: Minor > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: TimelineServiceV2.html, YARN-6159.v1.001.patch, > YARN-6159.v1.002.patch, YARN-6159.v1.003.patch, YARN-6159.v1.004.patch > > > Make documentation changes for TimelineV2Client i.e. to reflect changes made > in client API in YARN-4675. > Also in TimelineServiceV2.md, under section Publishing application specific > data, we have the following code snippet. Here, > {{timelineClient.putEntitiesAsync(entity);}} should be > {{client.putEntitiesAsync(entity);}} instead. > {code} > // Create and start the Timeline client v.2 > TimelineClient client = TimelineClient.createTimelineClient(appId); > client.init(conf); > client.start(); > try { > TimelineEntity myEntity = new TimelineEntity(); > myEntity.setEntityType("MY_APPLICATION"); > myEntity.setEntityId("MyApp1") > // Compose other entity info > // Blocking write > client.putEntities(entity); > TimelineEntity myEntity2 = new TimelineEntity(); > // Compose other info > // Non-blocking write > timelineClient.putEntitiesAsync(entity); > } catch (IOException e) { > // Handle the exception > } catch (RuntimeException e) { > {code} > Below can also be changed to client to keep it consistent. > {code} > amRMClient.registerTimelineClient(timelineClient); > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3041: --- Fix Version/s: 2.9.0 > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, > YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5156) YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state
[ https://issues.apache.org/jira/browse/YARN-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5156: --- Fix Version/s: 2.9.0 > YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state > - > > Key: YARN-5156 > URL: https://issues.apache.org/jira/browse/YARN-5156 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Vrushali C > Labels: YARN-5355 > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1 > > Attachments: YARN-5156-YARN-2928.01.patch, > YARN-5156-YARN-5355.01.patch, YARN-5156-YARN-5355.02.patch > > > On container finished, we're reporting "YARN_CONTAINER_STATE: "RUNNING"". Do > we design this deliberately or it's a bug? > {code} > { > metrics: [ ], > events: [ > { > id: "YARN_CONTAINER_FINISHED", > timestamp: 1464213765890, > info: { > YARN_CONTAINER_EXIT_STATUS: 0, > YARN_CONTAINER_STATE: "RUNNING", > YARN_CONTAINER_DIAGNOSTICS_INFO: "" > } > }, > { > id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED", > timestamp: 1464213761133, > info: { } > }, > { > id: "YARN_CONTAINER_CREATED", > timestamp: 1464213761132, > info: { } > }, > { > id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED", > timestamp: 1464213761132, > info: { } > } > ], > id: "container_e15_1464213707405_0001_01_18", > type: "YARN_CONTAINER", > createdtime: 1464213761132, > info: { > YARN_CONTAINER_ALLOCATED_PRIORITY: "20", > YARN_CONTAINER_ALLOCATED_VCORE: 1, > YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS: "10.22.16.164:0", > UID: > "yarn_cluster!application_1464213707405_0001!YARN_CONTAINER!container_e15_1464213707405_0001_01_18", > YARN_CONTAINER_ALLOCATED_HOST: "10.22.16.164", > YARN_CONTAINER_ALLOCATED_MEMORY: 1024, > SYSTEM_INFO_PARENT_ENTITY: { > type: "YARN_APPLICATION_ATTEMPT", > id: "appattempt_1464213707405_0001_01" > }, > YARN_CONTAINER_ALLOCATED_PORT: 64694 > }, > configs: { }, > isrelatedto: { }, > relatesto: { } > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6237) Move UID constant to TimelineReaderUtils
[ https://issues.apache.org/jira/browse/YARN-6237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6237: --- Fix Version/s: 2.9.0 > Move UID constant to TimelineReaderUtils > > > Key: YARN-6237 > URL: https://issues.apache.org/jira/browse/YARN-6237 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: newbie > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6237-YARN-5355.0001.patch > > > UID constant is kept in TimelineReader Manager. This can be moved to > TimelineReaderUtils which can keep track of all reader constants. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4409) Fix javadoc and checkstyle issues in timelineservice code
[ https://issues.apache.org/jira/browse/YARN-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4409: --- Fix Version/s: 2.9.0 > Fix javadoc and checkstyle issues in timelineservice code > - > > Key: YARN-4409 > URL: https://issues.apache.org/jira/browse/YARN-4409 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4409-YARN-2928.01.patch, > YARN-4409-YARN-2928.02.patch, YARN-4409-YARN-2928.03.patch > > > There are a large number of javadoc and checkstyle issues currently open in > timelineservice code. We need to fix them before we merge it into trunk. > Refer to > https://issues.apache.org/jira/browse/YARN-3862?focusedCommentId=15035267&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15035267 > We still have 94 open checkstyle issues and javadocs failing for Java 8. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3276) Refactor and fix null casting in some map cast for TimelineEntity (old and new) and fix findbug warnings
[ https://issues.apache.org/jira/browse/YARN-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3276: --- Fix Version/s: 2.9.0 > Refactor and fix null casting in some map cast for TimelineEntity (old and > new) and fix findbug warnings > > > Key: YARN-3276 > URL: https://issues.apache.org/jira/browse/YARN-3276 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3276-YARN-2928.v3.patch, > YARN-3276-YARN-2928.v4.patch, YARN-3276-YARN-2928.v5-fix-checkstyle.patch, > YARN-3276-YARN-2928.v5.patch, YARN-3276-YARN-2928.v6.patch, > YARN-3276-v2.patch, YARN-3276-v3.patch, YARN-3276.patch > > > Per discussion in YARN-3087, we need to refactor some similar logic to cast > map to hashmap and get rid of NPE issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4297) TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 branch
[ https://issues.apache.org/jira/browse/YARN-4297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4297: --- Fix Version/s: 2.9.0 > TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 > branch > --- > > Key: YARN-4297 > URL: https://issues.apache.org/jira/browse/YARN-4297 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4297-YARN-2928.01.patch, > YARN-4297-feature-YARN-2928.02.patch, YARN-4297-feature-YARN-2928.03.patch > > > {noformat} > Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.09 sec > <<< FAILURE! - in > org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler > testTimelineEventHandling(org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler) > Time elapsed: 0.11 sec <<< ERROR! > java.lang.ClassCastException: > org.apache.hadoop.mapreduce.v2.app.AppContext$$EnhancerByMockitoWithCGLIB$$95d3ddbe > cannot be cast to > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$RunningAppContext > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceInit(JobHistoryEventHandler.java:271) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler.testTimelineEventHandling(TestJobHistoryEventHandler.java:495) > {noformat} > {noformat} > testRMContainerAllocatorResendsRequestsOnRMRestart(org.apache.hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator) > Time elapsed: 2.649 sec <<< ERROR! > java.lang.ClassCastException: > org.apache.hadoop.mapreduce.v2.app.AppContext$$EnhancerByMockitoWithCGLIB$$8e08559a > cannot be cast to > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$RunningAppContext > at > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:802) > at > org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:269) > Tests in error: > TestRMContainerAllocator.testExcessReduceContainerAssign:669 » ClassCast > org.a... > TestRMContainerAllocator.testReportedAppProgress:970 » NullPointer > TestRMContainerAllocator.testBlackListedNodesWithSchedulingToThatNode:1578 > » ClassCast > TestRMContainerAllocator.testBlackListedNodes:1292 » ClassCast > org.apache.hado... > TestRMContainerAllocator.testAMRMTokenUpdate:2691 » ClassCast > org.apache.hadoo... > TestRMContainerAllocator.testMapReduceAllocationWithNodeLabelExpression:722 > » ClassCast > TestRMContainerAllocator.testReducerRampdownDiagnostics:443 » ClassCast > org.ap... > TestRMContainerAllocator.testReportedAppProgressWithOnlyMaps:1118 » > NullPointer > TestRMContainerAllocator.testMapReduceScheduling:819 » ClassCast > org.apache.ha... > TestRMContainerAllocator.testResource:390 » ClassCast > org.apache.hadoop.mapred... > TestRMContainerAllocator.testUpdatedNodes:1190 » ClassCast > org.apache.hadoop.m... > TestRMContainerAllocator.testCompletedTasksRecalculateSchedule:2249 » > ClassCast > TestRMContainerAllocator.testConcurrentTaskLimits:2779 » ClassCast > org.apache > TestRMContainerAllocator.testSimple:219 » ClassCast > org.apache.hadoop.mapreduc... > > TestRMContainerAllocator.testIgnoreBlacklisting:1378->getContainerOnHost:1511 > » ClassCast > TestRMContainerAllocator.testMapNodeLocality:310 » ClassCast > org.apache.hadoop... > > TestRMContainerAllocator.testRMContainerAllocatorResendsRequestsOnRMRestart:2489 > » ClassCast > Tests run: 26, Failures: 0, Errors: 17, Skipped: 0 > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6424) TimelineCollector is not stopped when an app finishes in RM
[ https://issues.apache.org/jira/browse/YARN-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6424: --- Fix Version/s: 2.9.0 > TimelineCollector is not stopped when an app finishes in RM > --- > > Key: YARN-6424 > URL: https://issues.apache.org/jira/browse/YARN-6424 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha2 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Critical > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6424.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3390) Reuse TimelineCollectorManager for RM
[ https://issues.apache.org/jira/browse/YARN-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3390: --- Fix Version/s: 2.9.0 > Reuse TimelineCollectorManager for RM > - > > Key: YARN-3390 > URL: https://issues.apache.org/jira/browse/YARN-3390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3390.1.patch, YARN-3390.2.patch, YARN-3390.3.patch, > YARN-3390.4.patch > > > RMTimelineCollector should have the context info of each app whose entity > has been put -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3908) Bugs in HBaseTimelineWriterImpl
[ https://issues.apache.org/jira/browse/YARN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3908: --- Fix Version/s: 2.9.0 > Bugs in HBaseTimelineWriterImpl > --- > > Key: YARN-3908 > URL: https://issues.apache.org/jira/browse/YARN-3908 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3908-YARN-2928.001.patch, > YARN-3908-YARN-2928.002.patch, YARN-3908-YARN-2928.003.patch, > YARN-3908-YARN-2928.004.patch, YARN-3908-YARN-2928.004.patch, > YARN-3908-YARN-2928.005.patch > > > 1. In HBaseTimelineWriterImpl, the info column family contains the basic > fields of a timeline entity plus events. However, entity#info map is not > stored at all. > 2 event#timestamp is also not persisted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5378) Accommodate app-id->cluster mapping
[ https://issues.apache.org/jira/browse/YARN-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5378: --- Fix Version/s: 2.9.0 > Accommodate app-id->cluster mapping > --- > > Key: YARN-5378 > URL: https://issues.apache.org/jira/browse/YARN-5378 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Joep Rottinghuis >Assignee: Sangjin Lee > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-5378-YARN-5355.01.patch, > YARN-5378-YARN-5355.02.patch, YARN-5378-YARN-5355.03.patch > > > In discussion with [~sjlee0], [~vrushalic], [~subru], and [~curino] a > use-case came up to be able to map from application-id to cluster-id in > context of federation for Yarn. > What happens is that a "random" cluster in the federation is asked to > generate an app-id and then potentially a different cluster can be the "home" > cluster for the AM. Furthermore, tasks can then run in yet other clusters. > In order to be able to pull up the logical home cluster on which the > application ran, there needs to be a mapping from application-id to > cluster-id. This mapping is available in the federated Yarn case only during > the active live of the application. > A similar situation is common in our larger production environment. Somebody > will complain about a slow job, some failure or whatever. If we're lucky we > have an application-id. When we ask the user which cluster they ran on, > they'll typically answer with the machine from where they launched the job > (many users are unaware of the underlying physical clusters). This leaves us > to spelunk through various RM ui's to find a matching epoch in the > application ID. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5170) Eliminate singleton converters and static method access
[ https://issues.apache.org/jira/browse/YARN-5170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5170: --- Fix Version/s: 2.9.0 > Eliminate singleton converters and static method access > --- > > Key: YARN-5170 > URL: https://issues.apache.org/jira/browse/YARN-5170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5170-YARN-2928.01.patch, > YARN-5170-YARN-2928.02.patch, YARN-5170-YARN-2928.03.patch, > YARN-5170-YARN-2928.04.patch, YARN-5170-YARN-2928.05.patch, > YARN-5170-YARN-2928.06.patch, YARN-5170-YARN-2928.07.patch, > YARN-5170-YARN-2928.08.patch, YARN-5170-YARN-2928.09.patch, > YARN-5170-YARN-2928.10.patch, YARN-5170-YARN-2928.11.patch, > YARN-5170-YARN-2928.12.patch, YARN-5170-YARN-2928.13.patch > > > As part of YARN-5109 we introduced several KeyConverter classes. > To stay consistent with the existing LongConverter in the sample patch I > created I made these other converter classes singleton as well. > In conversation with [~sjlee0] who has a general dislike of singletons, we > discussed it is best to get rid of these singletons and make them simply > instance variables. > There are other classes where the keys have static methods referring to a > singleton converter. > Moreover, it turns out that due to code evolution we end up creating the same > keys several times. > So general approach is to not re-instantiate rowkeys, converters when not > needed. > I would like to create the byte[] rowKey in the RowKey classes their > constructor, but that would leak an incomplete object to the converter. > There are a few method in TimelineStorageUtils that are used only once, or > only by one class, as part of this refactor I'll move these to keep the > "Utils" class as small as possible and keep them for truly generally used > utils that don't really belong anywhere else. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4622) TestDistributedShell fails for v2 test cases after modifications for 1.5
[ https://issues.apache.org/jira/browse/YARN-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4622: --- Fix Version/s: 2.9.0 > TestDistributedShell fails for v2 test cases after modifications for 1.5 > > > Key: YARN-4622 > URL: https://issues.apache.org/jira/browse/YARN-4622 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: test > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4622-YARN-2928.v1.001.patch > > > TestDistributedShell fails for v2 test cases : > *testDSShellWithoutDomainV2DefaultFlow and > testDSShellWithoutDomainV2CustomizedFlow* after trunk rebase with > modifications for 1.5, > {code} > Caused by: > org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): > java.lang.NullPointerException > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:187) > at com.google.common.base.Joiner.toString(Joiner.java:532) > at com.google.common.base.Joiner.appendTo(Joiner.java:124) > at com.google.common.base.Joiner.appendTo(Joiner.java:181) > at com.google.common.base.Joiner.join(Joiner.java:237) > at com.google.common.base.Joiner.join(Joiner.java:226) > at com.google.common.base.Joiner.join(Joiner.java:253) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.constructResURI(TimelineClientImpl.java:726) > at > org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceStart(TimelineClientImpl.java:336) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.createAndStartTimelineClient(ApplicationImpl.java:149) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:113) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainerInternal(ContainerManagerImpl.java:971) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:830) > at > org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:65) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3264) [Storage implementation] Create backing storage write interface and a POC only file based storage implementation
[ https://issues.apache.org/jira/browse/YARN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3264: --- Fix Version/s: 2.9.0 > [Storage implementation] Create backing storage write interface and a POC > only file based storage implementation > - > > Key: YARN-3264 > URL: https://issues.apache.org/jira/browse/YARN-3264 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3264.001.patch, YARN-3264.002.patch, > YARN-3264.003.patch, YARN-3264.004.patch, YARN-3264.005.patch, > YARN-3264.006.patch, YARN-3264.007.patch, YARN-3264.008.patch > > > For the PoC, need to create a backend impl for file based storage of entities -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5243) fix several rebase and other miscellaneous issues before merge
[ https://issues.apache.org/jira/browse/YARN-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5243: --- Fix Version/s: 2.9.0 > fix several rebase and other miscellaneous issues before merge > -- > > Key: YARN-5243 > URL: https://issues.apache.org/jira/browse/YARN-5243 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5243-YARN-2928.01.patch, > YARN-5243-YARN-2928.02.patch, YARN-5243-YARN-2928.03.patch > > > I have come across a couple of miscellaneous issues while inspecting the > diffs against the trunk. > We also need to review one last time (probably after the final rebase) to > ensure the timeline services v.2 leaves no impact when disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5052) [Documentation] Update timeline service v2 documentation to capture information about filters
[ https://issues.apache.org/jira/browse/YARN-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5052: --- Fix Version/s: 2.9.0 > [Documentation] Update timeline service v2 documentation to capture > information about filters > - > > Key: YARN-5052 > URL: https://issues.apache.org/jira/browse/YARN-5052 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Apache Hadoop 3.0.0-SNAPSHOT – The YARN Timeline Service > v.pdf, Hierarchy.png, The YARN Timeline Service v2.02.pdf, > YARN-5052-YARN-2928.01.patch, YARN-5052-YARN-2928.02.patch > > > Since YARN-4447 has gone in, we can update our documentation to capture > information about usage of filters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4712) CPU Usage Metric is not captured properly in YARN-2928
[ https://issues.apache.org/jira/browse/YARN-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4712: --- Fix Version/s: 2.9.0 > CPU Usage Metric is not captured properly in YARN-2928 > -- > > Key: YARN-4712 > URL: https://issues.apache.org/jira/browse/YARN-4712 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4712-YARN-2928.v1.001.patch, > YARN-4712-YARN-2928.v1.002.patch, YARN-4712-YARN-2928.v1.003.patch, > YARN-4712-YARN-2928.v1.004.patch, YARN-4712-YARN-2928.v1.005.patch, > YARN-4712-YARN-2928.v1.006.patch > > > There are 2 issues with CPU usage collection > * I was able to observe that that many times CPU usage got from > {{pTree.getCpuUsagePercent()}} is > ResourceCalculatorProcessTree.UNAVAILABLE(i.e. -1) but ContainersMonitor do > the calculation i.e. {{cpuUsageTotalCoresPercentage = cpuUsagePercentPerCore > /resourceCalculatorPlugin.getNumProcessors()}} because of which UNAVAILABLE > check in {{NMTimelinePublisher.reportContainerResourceUsage}} is not > encountered. so proper checks needs to be handled > * {{EntityColumnPrefix.METRIC}} uses always LongConverter but > ContainerMonitor is publishing decimal values for the CPU usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4102) Add a "skip existing table" mode for timeline schema creator
[ https://issues.apache.org/jira/browse/YARN-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4102: --- Fix Version/s: 2.9.0 > Add a "skip existing table" mode for timeline schema creator > > > Key: YARN-4102 > URL: https://issues.apache.org/jira/browse/YARN-4102 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4102-YARN-2928.001.patch, > YARN-4102-YARN-2928.002.patch, YARN-4102-YARN-2928.003.patch, > YARN-4102-YARN-2928.004.patch > > > When debugging timeline POCs, we may need to create hbase tables that are > added in some ongoing patches. Right now, our schema creator will exit when > it hits one existing table. While this is a correct behavior with end users, > this introduces much trouble in debugging POCs: every time we have to disable > all existing tables, drop them, run the schema creator to generate all > tables, and regenerate all test data. > Maybe we'd like to add an "incremental" mode so that the creator will only > create non-existing tables? This is pretty handy in deploying our POCs. Of > course, consistency has to be kept in mind across tables. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3706) Generalize native HBase writer for additional tables
[ https://issues.apache.org/jira/browse/YARN-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3706: --- Fix Version/s: 2.9.0 > Generalize native HBase writer for additional tables > > > Key: YARN-3706 > URL: https://issues.apache.org/jira/browse/YARN-3706 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3706-YARN-2928.001.patch, > YARN-3706-YARN-2928.010.patch, YARN-3706-YARN-2928.011.patch, > YARN-3706-YARN-2928.012.patch, YARN-3706-YARN-2928.013.patch, > YARN-3706-YARN-2928.014.patch, YARN-3706-YARN-2928.015.patch, > YARN-3726-YARN-2928.002.patch, YARN-3726-YARN-2928.003.patch, > YARN-3726-YARN-2928.004.patch, YARN-3726-YARN-2928.005.patch, > YARN-3726-YARN-2928.006.patch, YARN-3726-YARN-2928.007.patch, > YARN-3726-YARN-2928.008.patch, YARN-3726-YARN-2928.009.patch > > > When reviewing YARN-3411 we noticed that we could change the class hierarchy > a little in order to accommodate additional tables easily. > In order to get ready for benchmark testing we left the original layout in > place, as performance would not be impacted by the code hierarchy. > Here is a separate jira to address the hierarchy. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6064) Support fromId for flowRuns and flow/flowRun apps REST API's
[ https://issues.apache.org/jira/browse/YARN-6064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6064: --- Fix Version/s: 2.9.0 > Support fromId for flowRuns and flow/flowRun apps REST API's > > > Key: YARN-6064 > URL: https://issues.apache.org/jira/browse/YARN-6064 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6064-YARN-5355.0001.patch, > YARN-6064-YARN-5355.0002.patch, YARN-6064-YARN-5355.0003.patch, > YARN-6064-YARN-5355.0004.patch, YARN-6064-YARN-5355.addendum.patch > > > Splitting out JIRA YARN-6027 for pagination support for flowRuns, flow apps > and flow run apps. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3034) [Collector wireup] Implement RM starting its timeline collector
[ https://issues.apache.org/jira/browse/YARN-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3034: --- Fix Version/s: 2.9.0 > [Collector wireup] Implement RM starting its timeline collector > --- > > Key: YARN-3034 > URL: https://issues.apache.org/jira/browse/YARN-3034 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Naganarasimha G R > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3024.20150324-1.patch, YARN-3034-20150312-1.patch, > YARN-3034.20150205-1.patch, YARN-3034.20150316-1.patch, > YARN-3034.20150318-1.patch, YARN-3034.20150320-1.patch > > > Per design in YARN-2928, implement resource managers starting their own ATS > writers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5095) flow activities and flow runs are populated with wrong timestamp when RM restarts w/ recovery enabled
[ https://issues.apache.org/jira/browse/YARN-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5095: --- Fix Version/s: 2.9.0 > flow activities and flow runs are populated with wrong timestamp when RM > restarts w/ recovery enabled > - > > Key: YARN-5095 > URL: https://issues.apache.org/jira/browse/YARN-5095 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5095-YARN-2928.01.patch, > YARN-5095-YARN-2928.02.patch, YARN-5095-YARN-2928.03.patch > > > I have the RM recovery enabled. I see that upon restart the RM populates > records into flow activity and flow runs but with *wrong* timestamps. What I > mean by the timestamp is the part of the row key: > - flow activity: row created with the day of the RM restart > - flow run: row created with the RM start time as the "run id" > The following illustrates an example flow run: > {noformat} > metrics: [ ], > events: [ ], > id: "sjlee@Sleep job/1463433569917", > type: "YARN_FLOW_RUN", > createdtime: 1463422860987, > info: { > UID: "yarn_cluster!sjlee!Sleep job!1463433569917", > SYSTEM_INFO_FLOW_RUN_ID: 1463433569917, > SYSTEM_INFO_FLOW_NAME: "Sleep job", > SYSTEM_INFO_FLOW_RUN_END_TIME: 1463422865033, > SYSTEM_INFO_USER: "sjlee" > }, > isrelatedto: { }, > relatesto: { } > {noformat} > The created time and the end time are correct (i.e. original time), whereas > the timestamp in the row key (= run id: 1463433569917) is actually later than > the end time and coincides with the RM restart. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3721) build is broken on YARN-2928 branch due to possible dependency cycle
[ https://issues.apache.org/jira/browse/YARN-3721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3721: --- Fix Version/s: 2.9.0 > build is broken on YARN-2928 branch due to possible dependency cycle > > > Key: YARN-3721 > URL: https://issues.apache.org/jira/browse/YARN-3721 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Li Lu >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3721-YARN-2928.001.patch, > YARN-3721-YARN-2928.002.patch, YARN-3721-YARN-2928.002.patch > > > The build is broken on the YARN-2928 branch at the > hadoop-yarn-server-timelineservice module. It's been broken for a while, but > we didn't notice it because the build happens to work despite this if the > maven local cache is not cleared. > To reproduce, remove all hadoop (3.0.0-SNAPSHOT) artifacts from your maven > local cache and build it. > Almost certainly it was introduced by YARN-3529. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3049) [Storage Implementation] Implement storage reader interface to fetch raw data from HBase backend
[ https://issues.apache.org/jira/browse/YARN-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3049: --- Fix Version/s: 2.9.0 > [Storage Implementation] Implement storage reader interface to fetch raw data > from HBase backend > > > Key: YARN-3049 > URL: https://issues.apache.org/jira/browse/YARN-3049 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3049-WIP.1.patch, YARN-3049-WIP.2.patch, > YARN-3049-WIP.3.patch, YARN-3049-YARN-2928.2.patch, > YARN-3049-YARN-2928.3.patch, YARN-3049-YARN-2928.4.patch, > YARN-3049-YARN-2928.5.patch, YARN-3049-YARN-2928.6.patch, > YARN-3049-YARN-2928.7.patch > > > Implement existing ATS queries with the new ATS reader design. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4350) TestDistributedShell fails for V2 scenarios
[ https://issues.apache.org/jira/browse/YARN-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4350: --- Fix Version/s: 2.9.0 > TestDistributedShell fails for V2 scenarios > --- > > Key: YARN-4350 > URL: https://issues.apache.org/jira/browse/YARN-4350 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Naganarasimha G R > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4350-feature-YARN-2928.001.patch, > YARN-4350-feature-YARN-2928.002.patch, YARN-4350-feature-YARN-2928.003.patch > > > Currently TestDistributedShell does not pass on the feature-YARN-2928 branch. > There seem to be 2 distinct issues. > (1) testDSShellWithoutDomainV2* tests fail sporadically > These test fail more often than not if tested by themselves: > {noformat} > testDSShellWithoutDomainV2DefaultFlow(org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell) > Time elapsed: 30.998 sec <<< FAILURE! > java.lang.AssertionError: Application created event should be published > atleast once expected:<1> but was:<0> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at > org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.checkTimelineV2(TestDistributedShell.java:451) > at > org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShell(TestDistributedShell.java:326) > at > org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithoutDomainV2DefaultFlow(TestDistributedShell.java:207) > {noformat} > They start happening after YARN-4129. I suspect this might have to do with > some timing issue. > (2) the whole test times out > If you run the whole TestDistributedShell test, it times out without fail. > This may or may not have to do with the port change introduced by YARN-2859 > (just a hunch). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3906) split the application table from the entity table
[ https://issues.apache.org/jira/browse/YARN-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3906: --- Fix Version/s: 2.9.0 > split the application table from the entity table > - > > Key: YARN-3906 > URL: https://issues.apache.org/jira/browse/YARN-3906 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3906-YARN-2928.001.patch, > YARN-3906-YARN-2928.002.patch, YARN-3906-YARN-2928.003.patch, > YARN-3906-YARN-2928.004.patch, YARN-3906-YARN-2928.005.patch, > YARN-3906-YARN-2928.006.patch, YARN-3906-YARN-2928.007.patch > > > Per discussions on YARN-3815, we need to split the application entities from > the main entity table into its own table (application). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6801) NPE in RM while setting collectors map in NodeHeartbeatResponse
[ https://issues.apache.org/jira/browse/YARN-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6801: --- Fix Version/s: 2.9.0 > NPE in RM while setting collectors map in NodeHeartbeatResponse > --- > > Key: YARN-6801 > URL: https://issues.apache.org/jira/browse/YARN-6801 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355, YARN-5355-branch-2 >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6801-YARN-5355.001.patch > > > Null Pointer Exception seen in > ResourceTrackerService#setAppCollectorsMapToResponse call > {code} > 2017-06-22 22:24:01,437 WARN org.apache.hadoop.ipc.Server: IPC Server handler > 49 on 8031, call > org.apache.hadoop.yarn.server.api.ResourceTrackerPB.nodeHeartbeat from > 10.35.172.116:44399 Call#3929 Retry#0 > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.setAppCollectorsMapToResponse(ResourceTrackerService.java:467) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(ResourceTrackerService.java:447) > at > org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceTrackerPBServiceImpl.nodeHeartbeat(ResourceTrackerPBServiceImpl.java:68) > at > org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$2.callBlockingMethod(ResourceTracker.java:81) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2084) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2080) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1645) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2078) > {code} > It correlates to RM invoking setAppCollectorsMapToResponse and calling > {code} > AppCollectorData appCollectorData = > rmApps.get(appId).getCollectorData(); > {code} > If the app object is not present in the list of running app ids, then this > will throw NPE. > Filing jira to fix it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3367) Replace starting a separate thread for post entity with event loop in TimelineClient
[ https://issues.apache.org/jira/browse/YARN-3367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3367: --- Fix Version/s: 2.9.0 > Replace starting a separate thread for post entity with event loop in > TimelineClient > > > Key: YARN-3367 > URL: https://issues.apache.org/jira/browse/YARN-3367 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Naganarasimha G R > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3367-YARN-2928.v1.005.patch, > YARN-3367-YARN-2928.v1.006.patch, YARN-3367-YARN-2928.v1.007.patch, > YARN-3367-YARN-2928.v1.008.patch, YARN-3367-YARN-2928.v1.009.patch, > YARN-3367-YARN-2928.v1.010.patch, YARN-3367-YARN-2928.v1.011.patch, > YARN-3367-YARN-2928.v1.012.patch, YARN-3367-YARN-2928.v1.013.patch, > YARN-3367-YARN-2928.v1.014.patch, YARN-3367-feature-YARN-2928.003.patch, > YARN-3367-feature-YARN-2928.v1.002.patch, > YARN-3367-feature-YARN-2928.v1.004.patch, YARN-3367.YARN-2928.001.patch, > sjlee-suggestion.patch > > > Since YARN-3039, we add loop in TimelineClient to wait for > collectorServiceAddress ready before posting any entity. In consumer of > TimelineClient (like AM), we are starting a new thread for each call to get > rid of potential deadlock in main thread. This way has at least 3 major > defects: > 1. The consumer need some additional code to wrap a thread before calling > putEntities() in TimelineClient. > 2. It cost many thread resources which is unnecessary. > 3. The sequence of events could be out of order because each posting > operation thread get out of waiting loop randomly. > We should have something like event loop in TimelineClient side, > putEntities() only put related entities into a queue of entities and a > separated thread handle to deliver entities in queue to collector via REST > call. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6850) Ensure that supplemented timestamp is stored only for flow run metrics
[ https://issues.apache.org/jira/browse/YARN-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6850: --- Fix Version/s: 2.9.0 > Ensure that supplemented timestamp is stored only for flow run metrics > -- > > Key: YARN-6850 > URL: https://issues.apache.org/jira/browse/YARN-6850 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Varun Saxena > Labels: atsv2-hbase, yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6850-YARN-5355.01.patch > > > In timeline service v2, ColumnHelper#getPutTimestamp supplements the > timestamp and is called by ColumnHelper#store. This is not conditional and > called for every put. > We need to ensure that the cell timestamps for metrics in entity and > application (and sub application) tables are "correct" timestamps since we > will be enabling TTLs for these cells. > The supplemented timestamp is to be used only in the flow run table by the > coprocessor which intercepts all reads & writes to cells in this table. It > looks at the supplemented timestamp to figure out which app id this > particular cell belongs to. This is done in order to ensure no collision > occurs when two apps belonging to same flow run write the same metric at the > same timestamp. > Discovered in the discussion in YARN-4455 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5638) Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery
[ https://issues.apache.org/jira/browse/YARN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5638: --- Fix Version/s: 2.9.0 > Introduce a collector timestamp to uniquely identify collectors creation > order in collector discovery > - > > Key: YARN-5638 > URL: https://issues.apache.org/jira/browse/YARN-5638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1 > > Attachments: YARN-5638-YARN-5355.v4.patch, > YARN-5638-YARN-5355.v5.patch, YARN-5638-trunk.v1.patch, > YARN-5638-trunk.v2.patch, YARN-5638-trunk.v3.patch > > > As discussed in YARN-3359, we need to further identify timeline collectors' > creation order to rebuild collector discovery data in the RM. This JIRA > proposes to use to order collectors > for each application in the RM. This timestamp can then be used when a > standby RM becomes active and rebuild collector discovery data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5699) Retrospect yarn entity fields which are publishing in events info fields.
[ https://issues.apache.org/jira/browse/YARN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5699: --- Fix Version/s: 2.9.0 > Retrospect yarn entity fields which are publishing in events info fields. > - > > Key: YARN-5699 > URL: https://issues.apache.org/jira/browse/YARN-5699 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: 0001-YARN-5699.YARN-5355.patch, 0001-YARN-5699.patch, > 0002-YARN-5699.YARN-5355.patch, 0002-YARN-5699.patch, > 0003-YARN-5699.YARN-5355.patch, 0003-YARN-5699.patch > > > Currently, all the container information are published at 2 places. Some of > them are at entity info(top-level) and some are at event info. > For containers, some of the event info should be published at container info > level. For example : container exist status, container state, createdTime, > finished time. These are general information to container required for > container-report. So it is better to publish at top level info field. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7041) Nodemanager NPE running jobs with security off
[ https://issues.apache.org/jira/browse/YARN-7041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-7041: --- Fix Version/s: 2.9.0 > Nodemanager NPE running jobs with security off > -- > > Key: YARN-7041 > URL: https://issues.apache.org/jira/browse/YARN-7041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-5355 >Reporter: Aaron Gresch >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-7041-YARN-5355.01.patch > > > 2017-08-17 09:47:18,127 ERROR [Node Status Updater] > nodemanager.NodeStatusUpdaterImpl: Caught exception in status-updater > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.convertToProtoFormat(NodeHeartbeatRequestPBImpl.java:326) > at > org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.addRegisteringCollectorsToProto(NodeHeartbeatRequestPBImpl.java:174) > at > org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.mergeLocalToBuilder(NodeHeartbeatRequestPBImpl.java:122) > at > org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.mergeLocalToProto(NodeHeartbeatRequestPBImpl.java:183) > at > org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.NodeHeartbeatRequestPBImpl.getProto(NodeHeartbeatRequestPBImpl.java:77) > at > org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.nodeHeartbeat(ResourceTrackerPBClientImpl.java:83) > at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:411) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:348) > at com.sun.proxy.$Proxy77.nodeHeartbeat(Unknown Source) > at > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl$1.run(NodeStatusUpdaterImpl.java:799) > at java.lang.Thread.run(Thread.java:745) > Aug 17, 2017 9:47:18 AM com.sun.jersey.api.core.PackagesResourceConfig init > INFO: Scanning for root resource and provider classes in the packages: > org.apache.hadoop.yarn.server.timelineservice.collector > org.apache.hadoop.yarn.webapp > I believe this is due to YARN-6130, commented on that JIRA. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6658) Remove columnFor() methods of Columns in HBaseTimeline backend
[ https://issues.apache.org/jira/browse/YARN-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6658: --- Fix Version/s: 2.9.0 > Remove columnFor() methods of Columns in HBaseTimeline backend > --- > > Key: YARN-6658 > URL: https://issues.apache.org/jira/browse/YARN-6658 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Haibo Chen > Labels: atsv2-hbase > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6658.00.patch > > > The columnFor() methods are all dead. We can remove them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3634) TestMRTimelineEventHandling and TestApplication are broken
[ https://issues.apache.org/jira/browse/YARN-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3634: --- Fix Version/s: 2.9.0 > TestMRTimelineEventHandling and TestApplication are broken > -- > > Key: YARN-3634 > URL: https://issues.apache.org/jira/browse/YARN-3634 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3634-YARN-2928.001.patch, > YARN-3634-YARN-2928.002.patch, YARN-3634-YARN-2928.003.patch, > YARN-3634-YARN-2928.004.patch > > > TestMRTimelineEventHandling is broken. Relevant error message: > {noformat} > 2015-05-12 06:28:56,415 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 0 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:28:57,416 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 1 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:28:58,416 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 2 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:28:59,417 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 3 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:00,418 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 4 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:01,419 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 5 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:02,420 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 6 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:03,420 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 7 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:04,421 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 8 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:05,422 INFO [AsyncDispatcher event handler] ipc.Client > (Client.java:handleConnectionFailure(882)) - Retrying connect to server: > asf904.gq1.ygridcore.net/67.195.81.148:0. Already tried 9 time(s); retry > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 > MILLISECONDS) > 2015-05-12 06:29:05,424 ERROR [AsyncDispatcher event handler] > collector.NodeTimelineCollectorManager > (NodeTimelineCollectorManager.java:postPut(121)) - Failed to communicate with > NM Collector Service for application_1431412130291_0001 > 2015-05-12 06:29:05,425 WARN [AsyncDispatcher event handler] > containermanager.AuxServices > (AuxServices.java:logWarningWhenAuxServiceThrowExceptions(261)) - The > auxService name is timeline_collector and it got an error at event: > CONTAINER_INIT > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > org.apache.hadoop.yarn.exceptions.YarnRuntimeException: > java.net.ConnectException: Call From asf904.gq1.ygridcore.net/67.195.81.148 > to asf904.gq1.ygridcore.net:0 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see:
[jira] [Updated] (YARN-6455) Enhance the timelinewriter.flush() race condition fix
[ https://issues.apache.org/jira/browse/YARN-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6455: --- Fix Version/s: 2.9.0 > Enhance the timelinewriter.flush() race condition fix > - > > Key: YARN-6455 > URL: https://issues.apache.org/jira/browse/YARN-6455 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.0.0-alpha2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6455.00.patch > > > YARN-6376 fixes timelinewriter.flush() race condition among concurrent > putEntities() calls and periodical flush by TimelineCollectorManager by > synchronizing on the writer object. > Synchronizing on the writer is still a little brittle there, because there is > a getWriter method which lets callers access the writer without synchronizing > on it. AppLevelTimelineCollector#AppLevelAggregator#agregate() does this in > line 152: getWriter().write(...) In this case it doesn't flush, but if that > were to be added, that would re-introduce the race fixed in YARN-6376. > Instead of exposing the writer, perhaps it would be better to have the > sub-classes call #putEntities instead. It defers to the private > writeTimelineEntities which does the same work to get the context: > TimelineCollectorContext context = getTimelineEntityContext(); -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3949) ensure timely flush of timeline writes
[ https://issues.apache.org/jira/browse/YARN-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3949: --- Fix Version/s: 2.9.0 > ensure timely flush of timeline writes > -- > > Key: YARN-3949 > URL: https://issues.apache.org/jira/browse/YARN-3949 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3949-YARN-2928.001.patch, > YARN-3949-YARN-2928.002.patch, YARN-3949-YARN-2928.002.patch, > YARN-3949-YARN-2928.003.patch, YARN-3949-YARN-2928.004.patch, > YARN-3949-YARN-2928.004.patch > > > Currently flushing of timeline writes is not really handled. For example, > {{HBaseTimelineWriterImpl}} relies on HBase's {{BufferedMutator}} to batch > and write puts asynchronously. However, {{BufferedMutator}} may not flush > them to HBase unless the internal buffer fills up. > We do need a flush functionality first to ensure that data are written in a > reasonably timely manner, and to be able to ensure some critical writes are > done synchronously (e.g. key lifecycle events). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4447) Provide a mechanism to represent complex filters and parse them at the REST layer
[ https://issues.apache.org/jira/browse/YARN-4447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4447: --- Fix Version/s: 2.9.0 > Provide a mechanism to represent complex filters and parse them at the REST > layer > -- > > Key: YARN-4447 > URL: https://issues.apache.org/jira/browse/YARN-4447 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Timeline-Filters.pdf, YARN-4447-YARN-2928.01.patch, > YARN-4447-YARN-2928.02.patch, YARN-4447-YARN-2928.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6874) Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite
[ https://issues.apache.org/jira/browse/YARN-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6874: --- Fix Version/s: 2.9.0 > Supplement timestamp for min start/max end time columns in flow run table to > avoid overwrite > > > Key: YARN-6874 > URL: https://issues.apache.org/jira/browse/YARN-6874 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Vrushali C > Labels: atsv2-hbase > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6874-YARN-5355.0001.patch > > > Following test case is failing in YARN-5355 branch. > This is coming because we are not supplementing the timestamp for > FlowRunColumn i.e. min_start_time and max_end_time columns, post YARN-6850 > which can lead to a clash, if 2 writes for app created events happen at the > same time, which is true for this test case. > To fix this, we need to pass true flag into ColumnHelper constructor. I did > encounter this failure once earlier too. > {noformat} > testWriteFlowRunMinMax(org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun) > Time elapsed: 0.088 sec <<< FAILURE! > java.lang.AssertionError: expected:<142502690> but was:<1425026901000> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun.testWriteFlowRunMinMax(TestHBaseStorageFlowRun.java:237) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3984) Rethink event column key issue
[ https://issues.apache.org/jira/browse/YARN-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3984: --- Fix Version/s: 2.9.0 > Rethink event column key issue > -- > > Key: YARN-3984 > URL: https://issues.apache.org/jira/browse/YARN-3984 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3984-YARN-2928.001.patch > > > Currently, the event column key is event_id?info_key?timestamp, which is not > so friendly to fetching all the events of an entity and sorting them in a > chronologic order. IMHO, timestamp?event_id?info_key may be a better key > schema. I open this jira to continue the discussion about it which was > commented on YARN-3908. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4174) Fix javadoc warnings floating up from hbase
[ https://issues.apache.org/jira/browse/YARN-4174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4174: --- Fix Version/s: 2.9.0 > Fix javadoc warnings floating up from hbase > > > Key: YARN-4174 > URL: https://issues.apache.org/jira/browse/YARN-4174 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vrushali C >Assignee: Sangjin Lee >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > > As part of the patch for YARN-3901, [~sjlee0] observed some (~200) javadoc > warnings that are coming from hbase classes. > We tried a bunch of things like making the FlowRunCoprocessor class non > public and excluding the package from the pom. If the class in made non > public, the table creation has an exception. > {code} > 206 warnings > [WARNING] Javadoc Warnings > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestWALObserver.class): > warning: Cannot find annotation method 'value()' in type 'Category': class > file for org.junit.experimental.categories.Category not found > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerObserver.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.class): > warning: Cannot find annotation method 'timeout()' in type 'Test': class > file for org.junit.Test not found > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.class): > warning: Cannot find annotation method 'timeout()' in type 'Test' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.class): > warning: Cannot find annotation method 'timeout()' in type 'Test' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorEndpoint.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class): > warning: Cannot find annotation method 'value()' in type 'Category' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class): > warning: Cannot find annotation method 'timeout()' in type 'Test' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class): > warning: Cannot find annotation method 'timeout()' in type 'Test' > [WARNING] > /Users/username/.m2/repository/org/apache/hbase/hbase-server/1.0.1/hbase-server-1.0.1-tests.jar(org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.class): > warning: Cannot find annotation method 'timeout()' in type
[jira] [Updated] (YARN-6170) TimelineReaderServer should wait to join with HttpServer2
[ https://issues.apache.org/jira/browse/YARN-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6170: --- Fix Version/s: 2.9.0 > TimelineReaderServer should wait to join with HttpServer2 > - > > Key: YARN-6170 > URL: https://issues.apache.org/jira/browse/YARN-6170 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Affects Versions: 3.0.0-alpha2, YARN-5355 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Fix For: 2.9.0, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6170.01.patch > > > While I was backporting YARN-5355-branch-2 to a 2.6.0-based code branch, I > noticed that the timeline reader daemon would promptly shut down upon start. > It turns out that in the 2.6.0 code line at least there are only daemon > threads left once the main method returns. That causes the JVM to shut down. > The right pattern to start an embedded jetty web server is to call > {{Server.start()}} followed by {{Server.join()}}. That way, the server stays > up reliably no matter what other threads get created. > It works on YARN-5355 only because there *happens* to be one other non-daemon > thread. We should add the {{join()}} call to be always correct. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3039) [Collector wireup] Implement timeline app-level collector service discovery
[ https://issues.apache.org/jira/browse/YARN-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3039: --- Fix Version/s: 2.9.0 > [Collector wireup] Implement timeline app-level collector service discovery > --- > > Key: YARN-3039 > URL: https://issues.apache.org/jira/browse/YARN-3039 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Service Binding for applicationaggregator of ATS > (draft).pdf, Service Discovery For Application Aggregator of ATS (v2).pdf, > YARN-3039-no-test.patch, YARN-3039-v2-incomplete.patch, > YARN-3039-v3-core-changes-only.patch, YARN-3039-v4.patch, YARN-3039-v5.patch, > YARN-3039-v6.patch, YARN-3039-v7.patch, YARN-3039-v8.patch, YARN-3039.9.patch > > > Per design in YARN-2928, implement ATS writer service discovery. This is > essential for off-node clients to send writes to the right ATS writer. This > should also handle the case of AM failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5229) Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl
[ https://issues.apache.org/jira/browse/YARN-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5229: --- Fix Version/s: 2.9.0 > Refactor #isApplicationEntity and #getApplicationEvent from > HBaseTimelineWriterImpl > --- > > Key: YARN-5229 > URL: https://issues.apache.org/jira/browse/YARN-5229 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928, YARN-5355 >Reporter: Joep Rottinghuis >Assignee: Vrushali C >Priority: Minor > Labels: YARN-5355, atsv2-hbase > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1 > > Attachments: YARN-229-YARN-5355.01.patch, > YARN-5229-YARN-2928.01.patch, YARN-5229-YARN-2928.02.patch, > YARN-5229-YARN-2928.03.patch, YARN-5229-YARN-2928.04.patch > > > As per [~gtCarrera9] commented in YARN-5170 > bq. In HBaseTimelineWriterImpl isApplicationEntity and getApplicationEvent > seem to be awkward. Looks more like something related to TimelineEntity or > ApplicationEntity > In YARN-5170 we just made the method private, and in this separate jira we > can refactor these methods to TimelineEntity or ApplicationEntity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6604) Allow metric TTL for Application table to be specified through cmd
[ https://issues.apache.org/jira/browse/YARN-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6604: --- Fix Version/s: 2.9.0 > Allow metric TTL for Application table to be specified through cmd > -- > > Key: YARN-6604 > URL: https://issues.apache.org/jira/browse/YARN-6604 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.0.0-alpha2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Labels: atsv2-hbase > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6604.00.patch > > > We should allow metrics TTL in application table to be specified in schema > cmd, as we do with metrics TTL in entity table -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6342) Make TimelineV2Client's drain timeout after stop configurable
[ https://issues.apache.org/jira/browse/YARN-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6342: --- Fix Version/s: 2.9.0 > Make TimelineV2Client's drain timeout after stop configurable > - > > Key: YARN-6342 > URL: https://issues.apache.org/jira/browse/YARN-6342 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Haibo Chen > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6342.00.patch, YARN-6342.01.patch, > YARN-6342.02.patch > > > Found these with [~rohithsharma] while browsing the code > - In stop: it calls shutdownNow which doens't wait for pending tasks, should > it use shutdown instead ? > {code} > public void stop() { > LOG.info("Stopping TimelineClient."); > executor.shutdownNow(); > try { > executor.awaitTermination(DRAIN_TIME_PERIOD, TimeUnit.MILLISECONDS); > } catch (InterruptedException e) { > {code} > - In TimelineClientImpl#createRunnable: > If any exception happens when publish one entity > (publishWithoutBlockingOnQueue), the thread exists. I think it should try > best effort to continue publishing the timeline entities, one failure should > not cause all followup entities not published. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5070) upgrade HBase version for first merge
[ https://issues.apache.org/jira/browse/YARN-5070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5070: --- Fix Version/s: 2.9.0 > upgrade HBase version for first merge > - > > Key: YARN-5070 > URL: https://issues.apache.org/jira/browse/YARN-5070 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Vrushali C >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5070-YARN-2928.01.patch, > YARN-5070-YARN-2928.02.patch, YARN-5070-YARN-2928.03.patch, > YARN-5070-YARN-2928.04.patch, YARN-5070-YARN-2928.05.patch, > YARN-5070-YARN-2928.06.patch, YARN-5070-YARN-2928.07.patch > > > Currently we set the HBase version for the timeline service storage to 1.0.1. > This is a fairly old version, and there are reasons to upgrade to a newer > version. We should upgrade it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3240) [Data Mode] Implement client API to put generic entities
[ https://issues.apache.org/jira/browse/YARN-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3240: --- Fix Version/s: 2.9.0 > [Data Mode] Implement client API to put generic entities > > > Key: YARN-3240 > URL: https://issues.apache.org/jira/browse/YARN-3240 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3240.1.patch, YARN-3240.2.patch, YARN-3240.3.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4450) TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail
[ https://issues.apache.org/jira/browse/YARN-4450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4450: --- Fix Version/s: 2.9.0 > TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail > - > > Key: YARN-4450 > URL: https://issues.apache.org/jira/browse/YARN-4450 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 > Environment: jenkins >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4450-feature-YARN-2928.01.patch > > > When I run the unit tests against the current branch, > TestTimelineAuthenticationFilter and TestYarnConfigurationFields fail: > {noformat} > TestTimelineAuthenticationFilter.testDelegationTokenOperations:251 » > NullPointer > TestTimelineAuthenticationFilter.testDelegationTokenOperations:251 » > NullPointer > > TestYarnConfigurationFields>TestConfigurationFieldsBase.testCompareConfigurationClassAgainstXml:429 > class org.apache.hadoop.yarn.conf.YarnConfiguration has 1 variables missing > in yarn-default.xml > {noformat} > The latter failure is caused by YARN-4356 (when we deprecated > RM_SYSTEM_METRICS_PUBLISHER_ENABLED), and the former an older issue that was > caused when a later use of field {{resURI}} was added in trunk. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6563) ConcurrentModificationException in TimelineCollectorManager while stopping RM
[ https://issues.apache.org/jira/browse/YARN-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6563: --- Fix Version/s: 2.9.0 > ConcurrentModificationException in TimelineCollectorManager while stopping RM > - > > Key: YARN-6563 > URL: https://issues.apache.org/jira/browse/YARN-6563 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Rohith Sharma K S >Assignee: Haibo Chen > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6563.00.patch > > > It is seen that ConcurrentModificationException while stopping RM when ATSv2 > enabled. > {noformat} > 2017-05-05 15:04:11,563 WARN org.apache.hadoop.service.CompositeService: When > stopping the service > org.apache.hadoop.yarn.server.resourcemanager.timelineservice.RMTimelineCollectorManager > : java.util.ConcurrentModificationException > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$ValueIterator.next(HashMap.java:1466) > at > org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorManager.serviceStop(TimelineCollectorManager.java:222) > at > org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) > at > org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52) > at > org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80) > at > org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157) > at > org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStop(ResourceManager.java:1285) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3431) Sub resources of timeline entity needs to be passed to a separate endpoint.
[ https://issues.apache.org/jira/browse/YARN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3431: --- Fix Version/s: 2.9.0 > Sub resources of timeline entity needs to be passed to a separate endpoint. > --- > > Key: YARN-3431 > URL: https://issues.apache.org/jira/browse/YARN-3431 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3431.1.patch, YARN-3431.2.patch, YARN-3431.3.patch, > YARN-3431.4.patch, YARN-3431.5.patch, YARN-3431.6.patch, YARN-3431.7.patch > > > We have TimelineEntity and some other entities as subclass that inherit from > it. However, we only have a single endpoint, which consume TimelineEntity > rather than sub-classes and this endpoint will check the incoming request > body contains exactly TimelineEntity object. However, the json data which is > serialized from sub-class object seems not to be treated as an TimelineEntity > object, and won't be deserialized into the corresponding sub-class object > which cause deserialization failure as some discussions in YARN-3334 : > https://issues.apache.org/jira/browse/YARN-3334?focusedCommentId=14391059&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14391059. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6074) FlowRunEntity does not deserialize long values correctly
[ https://issues.apache.org/jira/browse/YARN-6074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6074: --- Fix Version/s: 2.9.0 > FlowRunEntity does not deserialize long values correctly > > > Key: YARN-6074 > URL: https://issues.apache.org/jira/browse/YARN-6074 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Fix For: 2.9.0, 3.0.0-alpha2, YARN-5355, YARN-5355-branch-2 > > Attachments: YARN-6074.patch > > > I see that FlowRunEntity methods *getRunId()* and *getMaxEndTime()* does not > deserialize in efficient way which causes class cast exception based on the > number. > {code} > public long getRunId() { > Object runId = getInfo().get(FLOW_RUN_ID_INFO_KEY); > return runId == null ? 0L : (Long) runId; > } > {code} > and > {code} > public long getMaxEndTime() { > Object time = getInfo().get(FLOW_RUN_END_TIME); > return time == null ? 0L : (Long)time; > } > {code} > The reason for class caste exception is Json has data type Number which > includes all java primitive types. So, if number with in the range of Integer > max, then Object is converted to Integer which fails to type cast to Long. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4062) Add the flush and compaction functionality via coprocessors and scanners for flow run table
[ https://issues.apache.org/jira/browse/YARN-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4062: --- Fix Version/s: 2.9.0 > Add the flush and compaction functionality via coprocessors and scanners for > flow run table > --- > > Key: YARN-4062 > URL: https://issues.apache.org/jira/browse/YARN-4062 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4062-YARN-2928.04.patch, > YARN-4062-YARN-2928.05.patch, YARN-4062-YARN-2928.06.patch, > YARN-4062-YARN-2928.07.patch, YARN-4062-YARN-2928.08.patch, > YARN-4062-YARN-2928.09.patch, YARN-4062-YARN-2928.1.patch, > YARN-4062-feature-YARN-2928.01.patch, YARN-4062-feature-YARN-2928.02.patch, > YARN-4062-feature-YARN-2928.03.patch > > > As part of YARN-3901, coprocessor and scanner is being added for storing into > the flow_run table. It also needs a flush & compaction processing in the > coprocessor and perhaps a new scanner to deal with the data during flushing > and compaction stages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6638) [ATSv2 Security] Timeline reader side changes for loading auth filters and principals
[ https://issues.apache.org/jira/browse/YARN-6638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6638: --- Fix Version/s: 2.9.0 > [ATSv2 Security] Timeline reader side changes for loading auth filters and > principals > - > > Key: YARN-6638 > URL: https://issues.apache.org/jira/browse/YARN-6638 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6638-YARN-5355.01.patch, > YARN-6638-YARN-5355.02.patch, YARN-6638-YARN-5355.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6820: --- Fix Version/s: 2.9.0 > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch, > YARN-6820-YARN-5355_branch_2.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6323) Rolling upgrade/config change is broken on timeline v2.
[ https://issues.apache.org/jira/browse/YARN-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6323: --- Fix Version/s: 2.9.0 > Rolling upgrade/config change is broken on timeline v2. > > > Key: YARN-6323 > URL: https://issues.apache.org/jira/browse/YARN-6323 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355_branch2 > > Attachments: YARN-6323-YARN-5355.0001.patch, > YARN-6323-YARN-5355.0002.patch, YARN-6323.001.patch > > > Found this issue when deploying on real clusters. If there are apps running > when we enable timeline v2 (with work preserving restart enabled), node > managers will fail to start due to missing app context data. We should > probably assign some default names to these "left over" apps. I believe it's > suboptimal to let users clean up the whole cluster before enabling timeline > v2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6414) ATSv2 HBase related tests fail due to guava version upgrade
[ https://issues.apache.org/jira/browse/YARN-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6414: --- Fix Version/s: 2.9.0 > ATSv2 HBase related tests fail due to guava version upgrade > --- > > Key: YARN-6414 > URL: https://issues.apache.org/jira/browse/YARN-6414 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha4 > Environment: Ubuntu 14.04 > x86, ppc64le > $ java -version > openjdk version "1.8.0_111" > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14) > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) >Reporter: Sonia Garudi >Assignee: Haibo Chen > Labels: ppc64le, x86_64 > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6414.00.patch, YARN-6414.01.patch > > > Test failures seen in Hadoop YARN Timeline Service HBase tests project with > following error : > {code} > java.lang.NoClassDefFoundError: com/google/common/io/LimitInputStream > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:223) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:913) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:899) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:722) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:660) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:751) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:735) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:998) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:869) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:704) > at > org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:642) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:590) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:987) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:868) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:862) > at > org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:806) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setup(TestTimelineReaderWebServicesHBaseStorage.java:88) > {code} > Below are the failing tests : > {code} > TestTimelineReaderWebServicesHBaseStorage.setup:88 » NoClassDefFound > com/googl... > TestHBaseTimelineStorageEntities.setupBeforeClass:100 » NoClassDefFound > com/go... > TestHBaseStorageFlowActivity.setupBeforeClass:72 » NoClassDefFound > com/google/... > TestHBaseStorageFlowRunCompaction.setupBeforeClass:84 » NoClassDefFound > com/go... > TestHBaseStorageFlowRun.setupBeforeClass:86 » NoClassDefFound > com/google/commo... > TestHBaseTimelineStorageApps.setupBeforeClass:93 » NoClassDefFound > com/google/... > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5097) NPE in Separator.joinEncoded()
[ https://issues.apache.org/jira/browse/YARN-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5097: --- Fix Version/s: 2.9.0 > NPE in Separator.joinEncoded() > -- > > Key: YARN-5097 > URL: https://issues.apache.org/jira/browse/YARN-5097 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Vrushali C >Priority: Critical > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5097-YARN-2928.01.patch, > YARN-5097-YARN-2928.02.patch, YARN-5097-YARN-2928.03.patch, > YARN-5097-YARN-2928.04.patch > > > Both in the RM log and the NM log, I see the following exception thrown. > First for RM, > {noformat} > 2016-05-16 14:19:29,930 ERROR > org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector: > Error aggregating timeline metrics > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.timelineservice.storage.common.Separator.joinEncoded(Separator.java:249) > at > org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKey(ApplicationRowKey.java:110) > at > org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.write(HBaseTimelineWriterImpl.java:131) > at > org.apache.hadoop.yarn.server.timelineservice.collector.AppLevelTimelineCollector$AppLevelAggregator.run(AppLevelTimelineCollector.java:136) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > {noformat} > In the NM log, I see a similar exception: > {noformat} > 2016-05-16 14:54:23,116 ERROR > org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector: > Error aggregating timeline metrics > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.timelineservice.storage.common.Separator.joinEncoded(Separator.java:249) > at > org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKey(ApplicationRowKey.java:110) > at > org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.write(HBaseTimelineWriterImpl.java:131) > at > org.apache.hadoop.yarn.server.timelineservice.collector.AppLevelTimelineCollector$AppLevelAggregator.run(AppLevelTimelineCollector.java:136) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4460) [Bug fix] RM fails to start when SMP is enabled
[ https://issues.apache.org/jira/browse/YARN-4460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4460: --- Fix Version/s: 2.9.0 > [Bug fix] RM fails to start when SMP is enabled > --- > > Key: YARN-4460 > URL: https://issues.apache.org/jira/browse/YARN-4460 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Li Lu >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4460-feature-YARN-2928.001.patch > > > When SMP is enabled, the RM starts with the following fatal message: > {code} > FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(189)) - Error in > dispatcher thread true > java.lang.Exception: No handler for registered for class > org.apache.hadoop.yarn.server.resourcemanager.metrics.AbstractSystemMetricsPubli > sher$SystemMetricsEventType >at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:185) >at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109) >at java.lang.Thread.run(Thread.java:745) > {code} > We should register event handlers in service init stage in > TimelineServiceV2Publisher to fix this problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4445) Unify the term flowId and flowName in timeline v2 codebase
[ https://issues.apache.org/jira/browse/YARN-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4445: --- Fix Version/s: 2.9.0 > Unify the term flowId and flowName in timeline v2 codebase > -- > > Key: YARN-4445 > URL: https://issues.apache.org/jira/browse/YARN-4445 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Zhan Zhang > Labels: refactor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4445-feature-YARN-2928.001.patch, YARN-4445.patch > > > Flow names are not sufficient to identify a flow. I noticed we used both > "flowName" and "flowId" to point to the same thing. We need to unify them to > flowName. Otherwise, front end users may think flow id is a top level concept > and try to directly locate a flow by its flow id. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5980) Update documentation for single node hbase deploy
[ https://issues.apache.org/jira/browse/YARN-5980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5980: --- Fix Version/s: 2.9.0 > Update documentation for single node hbase deploy > - > > Key: YARN-5980 > URL: https://issues.apache.org/jira/browse/YARN-5980 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: atsv2-hbase, yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1 > > Attachments: YARN-5980.001.patch, YARN-5980.002.patch, > YARN-5980.003.patch, YARN-5980.004.patch > > > Per HBASE-17272, a single node hbase deployment (single jvm running daemons + > hdfs writes) will be added to hbase shortly. > We should update the timeline service documentation in the setup/deployment > context accordingly, this will help users who are a bit wary of hbase > deployments help get started with timeline service more easily. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4210) HBase reader throws NPE if Get returns no rows
[ https://issues.apache.org/jira/browse/YARN-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4210: --- Fix Version/s: 2.9.0 > HBase reader throws NPE if Get returns no rows > -- > > Key: YARN-4210 > URL: https://issues.apache.org/jira/browse/YARN-4210 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4210-YARN-2928.01.patch, > YARN-4210-YARN-2928.02.patch, YARN-4210-YARN-2928.03.patch > > > This JIRA handles multiple issues. > * If HBase Get does not fetch any rows for the query, we still try to parse > the result and read fields. This leads to NPE while reading metrics. We > should not attempt to read anything if no row is returned i.e. result is > empty. > Found during web UI poc testing. > {noformat} > 2015-09-29 20:22:32,027 ERROR [95336304@qtp-1814206058-0] > reader.TimelineReaderWebServices > (TimelineReaderWebServices.java:handleException(199)) - Error while > processing REST request > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.readResultsWithTimestamps(ColumnHelper.java:176) > at > org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunColumnPrefix.readResultsWithTimestamps(FlowRunColumnPrefix.java:182) > at > org.apache.hadoop.yarn.server.timelineservice.storage.TimelineEntityReader.readMetrics(TimelineEntityReader.java:212) > at > org.apache.hadoop.yarn.server.timelineservice.storage.FlowRunEntityReader.parseEntity(FlowRunEntityReader.java:136) > at > org.apache.hadoop.yarn.server.timelineservice.storage.TimelineEntityReader.readEntity(TimelineEntityReader.java:137) > at > org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.getEntity(HBaseTimelineReaderImpl.java:72) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderManager.getEntity(TimelineReaderManager.java:93) > at > org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getFlowRun(TimelineReaderWebServices.java:403) > {noformat} > * ResultScanner is not closed in HBase Reader. > * Exception encountered while reading start and end time in > FlowRunEntityReader -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5291) Store node information for finished containers in timeline v2
[ https://issues.apache.org/jira/browse/YARN-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5291: --- Fix Version/s: 2.9.0 > Store node information for finished containers in timeline v2 > - > > Key: YARN-5291 > URL: https://issues.apache.org/jira/browse/YARN-5291 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu > Fix For: 2.9.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3134) [Storage implementation] Exploiting the option of using Phoenix to access HBase backend
[ https://issues.apache.org/jira/browse/YARN-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3134: --- Fix Version/s: 2.9.0 > [Storage implementation] Exploiting the option of using Phoenix to access > HBase backend > --- > > Key: YARN-3134 > URL: https://issues.apache.org/jira/browse/YARN-3134 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Li Lu > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: SettingupPhoenixstorageforatimelinev2end-to-endtest.pdf, > YARN-3134-040915_poc.patch, YARN-3134-041015_poc.patch, > YARN-3134-041415_poc.patch, YARN-3134-042115.patch, YARN-3134-042715.patch, > YARN-3134-YARN-2928.001.patch, YARN-3134-YARN-2928.002.patch, > YARN-3134-YARN-2928.003.patch, YARN-3134-YARN-2928.004.patch, > YARN-3134-YARN-2928.005.patch, YARN-3134-YARN-2928.006.patch, > YARN-3134-YARN-2928.007.patch, YARN-3134DataSchema.pdf, > hadoop-zshen-nodemanager-d-128-95-184-84.dhcp4.washington.edu.out > > > Quote the introduction on Phoenix web page: > {code} > Apache Phoenix is a relational database layer over HBase delivered as a > client-embedded JDBC driver targeting low latency queries over HBase data. > Apache Phoenix takes your SQL query, compiles it into a series of HBase > scans, and orchestrates the running of those scans to produce regular JDBC > result sets. The table metadata is stored in an HBase table and versioned, > such that snapshot queries over prior versions will automatically use the > correct schema. Direct use of the HBase API, along with coprocessors and > custom filters, results in performance on the order of milliseconds for small > queries, or seconds for tens of millions of rows. > {code} > It may simply our implementation read/write data from/to HBase, and can > easily build index and compose complex query. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3551) Consolidate data model change according to the backend implementation
[ https://issues.apache.org/jira/browse/YARN-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3551: --- Fix Version/s: 2.9.0 > Consolidate data model change according to the backend implementation > - > > Key: YARN-3551 > URL: https://issues.apache.org/jira/browse/YARN-3551 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3551-YARN-2928.4.patch, > YARN-3551-YARN-2928.5.patch, YARN-3551.1.patch, YARN-3551.2.patch, > YARN-3551.3.patch > > > Based on the comments on > [YARN-3134|https://issues.apache.org/jira/browse/YARN-3134?focusedCommentId=14512080&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512080] > and > [YARN-3411|https://issues.apache.org/jira/browse/YARN-3411?focusedCommentId=14512098&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14512098], > we need to change the data model to restrict the data type of > info/config/metric section. > 1. Info: the value could be all kinds object that is able to be > serialized/deserialized by jackson. > 2. Config: the value will always be assumed as String. > 3. Metric: single data or time series value have to be number for aggregation. > Other than that, info/start time/finish time of metric seem not to be > necessary for storage. They should be removed. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3726) Fix TestHBaseTimelineWriterImpl unit test failure by fixing it's test data
[ https://issues.apache.org/jira/browse/YARN-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3726: --- Fix Version/s: 2.9.0 > Fix TestHBaseTimelineWriterImpl unit test failure by fixing it's test data > -- > > Key: YARN-3726 > URL: https://issues.apache.org/jira/browse/YARN-3726 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3726-YARN-2928.001.patch > > > There is a very fascinating bug that was introduced by the test data in the > metrics time series check in the unit test in TestHBaseTimelineWriterImpl in > YARN-3411. > The unit test failure seen is > {code} > Error Message > expected:<1> but was:<6> > Stacktrace > java.lang.AssertionError: expected:<1> but was:<6> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineWriterImpl.checkMetricsTimeseries(TestHBaseTimelineWriterImpl.java:219) > at > org.apache.hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineWriterImpl.testWriteEntityToHBase(TestHBaseTimelineWriterImpl.java:204) > {code} > The test data had 6 timestamps that belonged to 22nd April 2015. When the > patch in YARN-3411 was submitted and tested by Hadoop QA on May 19th, the > unit test was working fine. Fast forward a few more days and the test started > failing. There has been no relevant code change or package version change > interim. The change that is triggering the unit test failure is the passage > of time. > The reason for test failure is that the metrics time series data lives in a > column family which has a TTL set to 30 days. Metrics time series data was > written to the mini hbase cluster with cell timestamps set to April 22nd. > Based on the column family configuration, hbase started deleting the data > that was older than 30 days and the test started failing. The last value is > retained, hence there is one value fetched from hbase. > Will submit a patch with the test case fixed shortly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4644) TestRMRestart fails and findbugs issue in YARN-2928 branch
[ https://issues.apache.org/jira/browse/YARN-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4644: --- Fix Version/s: 2.9.0 > TestRMRestart fails and findbugs issue in YARN-2928 branch > -- > > Key: YARN-4644 > URL: https://issues.apache.org/jira/browse/YARN-4644 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4644-YARN-2928.01.patch, > YARN-4644-YARN-2928.02.patch > > > This was reported by YARN-4238 QA report. Refer to > https://builds.apache.org/job/PreCommit-YARN-Build/10389/testReport/ > Error reported is as under : > {noformat} > org.mockito.exceptions.verification.TooManyActualInvocations: > noOpSystemMetricPublisher.appCreated( > , > > ); > Wanted 3 times: > -> at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955) > But was 6 times. Undesired invocation: > -> at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1274) > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testRMRestartGetApplicationList(TestRMRestart.java:955) > {noformat} > Failing because in {{RMAppImpl#recover}}, {{sendATSCreateEvent}} has been > called twice. > Has been introduced during rebase I guess. > After removing the duplicate call, the test passes. > There is a *findbugs warning* in resourcemanager in YARN-2928 branch as well. > Fix it as part of this JIRA itself. > {noformat} > DLS Dead store to keepAliveApps in > org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest) > Bug type DLS_DEAD_LOCAL_STORE (click for details) > In class org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService > In method > org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService.nodeHeartbeat(NodeHeartbeatRequest) > Local variable named keepAliveApps > At ResourceTrackerService.java:[line 486] > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3334) [Event Producers] NM TimelineClient container metrics posting to new timeline service.
[ https://issues.apache.org/jira/browse/YARN-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3334: --- Fix Version/s: 2.9.0 > [Event Producers] NM TimelineClient container metrics posting to new timeline > service. > -- > > Key: YARN-3334 > URL: https://issues.apache.org/jira/browse/YARN-3334 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: YARN-2928 >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3334-demo.patch, YARN-3334-v1.patch, > YARN-3334-v2.patch, YARN-3334-v3.patch, YARN-3334-v4.patch, > YARN-3334-v5.patch, YARN-3334-v6.patch, YARN-3334-v8.patch, YARN-3334.7.patch > > > After YARN-3039, we have service discovery mechanism to pass app-collector > service address among collectors, NMs and RM. In this JIRA, we will handle > service address setting for TimelineClients in NodeManager, and put container > metrics to the backend storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6094) Update the coprocessor to be a dynamically loaded one
[ https://issues.apache.org/jira/browse/YARN-6094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6094: --- Fix Version/s: 2.9.0 > Update the coprocessor to be a dynamically loaded one > - > > Key: YARN-6094 > URL: https://issues.apache.org/jira/browse/YARN-6094 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: atsv2-hbase, yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1 > > Attachments: YARN-6094-YARN-5355.001.patch, > YARN-6094-YARN-5355.002.patch, YARN-6094-YARN-5355.003.patch, > YARN-6094-YARN-5355.004.patch, YARN-6094.001.patch > > > The timeline service v2 code base on yarn now uses hbase 1.2.4 after > YARN-5976. > With this version of hbase, system classes (starting with org.apache.hadoop) > can be loaded as table level coprocessors. Hence we should update the > timeline service coprocessor to be a dynamically loaded one instead of static > loading. > This involves code changes as well as documentation updates for deployment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6027) Support fromid(offset) filter for /flows API
[ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6027: --- Fix Version/s: 2.9.0 > Support fromid(offset) filter for /flows API > > > Key: YARN-6027 > URL: https://issues.apache.org/jira/browse/YARN-6027 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6027-YARN-5355-branch-2.01.patch, > YARN-6027-YARN-5355.0001.patch, YARN-6027-YARN-5355.0002.patch, > YARN-6027-YARN-5355.0003.patch, YARN-6027-YARN-5355.0004.patch, > YARN-6027-YARN-5355.0005.patch, YARN-6027-YARN-5355.0006.patch, > YARN-6027-YARN-5355.0007.patch, YARN-6027-YARN-5355.0008.patch > > > In YARN-5585 , fromId is supported for retrieving entities. We need similar > filter for flows/flowRun apps and flow run and flow as well. > Along with supporting fromId, this JIRA should also discuss following points > * Should we throw an exception for entities/entity retrieval if duplicates > found? > * TimelieEntity : > ** Should equals method also check for idPrefix? > ** Does idPrefix is part of identifiers? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3051) [Storage abstraction] Create backing storage read interface for ATS readers
[ https://issues.apache.org/jira/browse/YARN-3051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3051: --- Fix Version/s: 2.9.0 > [Storage abstraction] Create backing storage read interface for ATS readers > --- > > Key: YARN-3051 > URL: https://issues.apache.org/jira/browse/YARN-3051 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3051-YARN-2928.003.patch, > YARN-3051-YARN-2928.03.patch, YARN-3051-YARN-2928.04.patch, > YARN-3051-YARN-2928.05.patch, YARN-3051-YARN-2928.06.patch, > YARN-3051-YARN-2928.07.patch, YARN-3051-YARN-2928.08.patch, > YARN-3051.Reader_API.patch, YARN-3051.Reader_API_1.patch, > YARN-3051.Reader_API_2.patch, YARN-3051.Reader_API_3.patch, > YARN-3051.Reader_API_4.patch, YARN-3051.wip.02.YARN-2928.patch, > YARN-3051.wip.patch, YARN-3051_temp.patch > > > Per design in YARN-2928, create backing storage read interface that can be > implemented by multiple backing storage implementations. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3333) rename TimelineAggregator etc. to TimelineCollector
[ https://issues.apache.org/jira/browse/YARN-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-: --- Fix Version/s: 2.9.0 > rename TimelineAggregator etc. to TimelineCollector > --- > > Key: YARN- > URL: https://issues.apache.org/jira/browse/YARN- > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.9.0 > > Attachments: YARN--unit-tests-fixes.patch, YARN-.001.patch, > YARN-.002.patch > > > Per discussions on YARN-2928, let's rename TimelineAggregator, etc. to > TimelineCollector, etc. > There are also several minor issues on the current branch, which can be fixed > as part of this: > - fixing some imports > - missing license in TestTimelineServerClientIntegration.java > - whitespaces > - missing direct dependency -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4238) correctly set createdTime and remove modifiedTime when publishing entities
[ https://issues.apache.org/jira/browse/YARN-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4238: --- Fix Version/s: 2.9.0 > correctly set createdTime and remove modifiedTime when publishing entities > -- > > Key: YARN-4238 > URL: https://issues.apache.org/jira/browse/YARN-4238 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4238-YARN-2928.01.patch, > YARN-4238-YARN-2928.04.patch, YARN-4238-YARN-2928.05.patch, > YARN-4238-feature-YARN-2928.002.patch, YARN-4238-feature-YARN-2928.003.patch > > > While publishing entities from RM and elsewhere we are not sending created > time. For instance, created time in TimelineServiceV2Publisher class and for > other entities in other such similar classes is not updated. We can easily > update created time when sending application created event. Likewise for > modification time on every write. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5096) timelinereader has a lot of logging that's not useful
[ https://issues.apache.org/jira/browse/YARN-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5096: --- Fix Version/s: 2.9.0 > timelinereader has a lot of logging that's not useful > - > > Key: YARN-5096 > URL: https://issues.apache.org/jira/browse/YARN-5096 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Minor > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5096-YARN-2928.01.patch > > > After running about a dozen or so requests, the timelinereader log is filled > with the following logging entries: > {noformat} > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > 2016-05-16 15:59:13,364 INFO > org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper: > null prefix was specified; returning all columns > {noformat} > There were some ~ 3,000 such logging entries. It's too excessive. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5189) Make HBaseTimeline[Reader|Writer]Impl default and move FileSystemTimeline*Impl
[ https://issues.apache.org/jira/browse/YARN-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5189: --- Fix Version/s: 2.9.0 > Make HBaseTimeline[Reader|Writer]Impl default and move FileSystemTimeline*Impl > -- > > Key: YARN-5189 > URL: https://issues.apache.org/jira/browse/YARN-5189 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5189-YARN-2928.01.patch, > YARN-5189-YARN-2928.02.patch, YARN-5189-YARN-2928.03.patch, > YARN-5189-YARN-2928.04.patch, YARN-5189-YARN-2928.05.patch, > YARN-5189-YARN-2928.06.patch > > > [~naganarasimha...@apache.org] questioned whether it made sense to default to > an implementation that doesn't support all functionality. > [~sjlee0] opened YARN-5174 to track updating the documentation for ATS to > reflect the default shifting to the fully functional HBase implementation. > It makes sense to remove a partial implementation, but on the other hand it > is still handing in testing. Hence this jira to move the file based > implementations to the test package and to make the HBase impls the default. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6734) Ensure sub-application user is extracted & sent to timeline service
[ https://issues.apache.org/jira/browse/YARN-6734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6734: --- Fix Version/s: 2.9.0 > Ensure sub-application user is extracted & sent to timeline service > --- > > Key: YARN-6734 > URL: https://issues.apache.org/jira/browse/YARN-6734 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Rohith Sharma K S > Labels: atsv2-subapp > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1, YARN-5355-branch-2 > > Attachments: YARN-6734-YARN-5355.001.patch, > YARN-6734-YARN-5355.002.patch, YARN-6734-YARN-5355.003.patch, > YARN-6734-YARN-5355.004.patch > > > After a discussion with Tez folks, we have been thinking over introducing a > table to store sub-application information. YARN-6733 > For example, if a Tez session runs for a certain period as User X and runs a > few AMs. These AMs accept DAGs from other users. Tez will execute these dags > with a doAs user. ATSv2 should store this information in a new table perhaps > called as "sub_application" table. > YARN-6733 tracks the code changes needed for table schema creation. > This jira tracks writing to that table, updating the user name fields to > include sub-application user etc. This would mean adding a field to Flow > Context which can store an additional user -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3045) [Event producers] Implement NM writing container lifecycle events to ATS
[ https://issues.apache.org/jira/browse/YARN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3045: --- Fix Version/s: 2.9.0 > [Event producers] Implement NM writing container lifecycle events to ATS > > > Key: YARN-3045 > URL: https://issues.apache.org/jira/browse/YARN-3045 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Naganarasimha G R > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-3045-YARN-2928.002.patch, > YARN-3045-YARN-2928.003.patch, YARN-3045-YARN-2928.004.patch, > YARN-3045-YARN-2928.005.patch, YARN-3045-YARN-2928.006.patch, > YARN-3045-YARN-2928.007.patch, YARN-3045-YARN-2928.008.patch, > YARN-3045-YARN-2928.009.patch, YARN-3045-YARN-2928.010.patch, > YARN-3045-YARN-2928.011.patch, YARN-3045.20150420-1.patch > > > Per design in YARN-2928, implement NM writing container lifecycle events and > container system metrics to ATS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4179) [reader implementation] support flow activity queries based on time
[ https://issues.apache.org/jira/browse/YARN-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4179: --- Fix Version/s: 2.9.0 > [reader implementation] support flow activity queries based on time > --- > > Key: YARN-4179 > URL: https://issues.apache.org/jira/browse/YARN-4179 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Varun Saxena >Priority: Minor > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-4179-YARN-2928.01.patch, > YARN-4179-YARN-2928.02.patch, YARN-4179-YARN-2928.03.patch > > > This came up as part of YARN-4074 and YARN-4075. > Currently the only query pattern that's supported on the flow activity table > is by cluster only. But it might be useful to support queries by cluster and > certain date or dates. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5014) Ensure non-metric values are returned as is for flow run table from the coprocessor
[ https://issues.apache.org/jira/browse/YARN-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5014: --- Fix Version/s: 2.9.0 > Ensure non-metric values are returned as is for flow run table from the > coprocessor > --- > > Key: YARN-5014 > URL: https://issues.apache.org/jira/browse/YARN-5014 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > > Presently the FlowScanner class presumes existence of NumericValueConverter > in it's emitCells function. This causes an exception when we try to retrieve > non-numeric values from this table. > Exception is seen as: > {code} > java.lang.ClassCastException: > org.apache.hadoop.yarn.server.timelineservice.storage.common.GenericConverter > cannot be cast to > org.apache.hadoop.yarn.server.timelineservice.storage.common.NumericValueConverter > at > org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.nextInternal(FlowScanner.java:246) > at > org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.nextRaw(FlowScanner.java:125) > at > org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowScanner.nextRaw(FlowScanner.java:119) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2117) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31443) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) > at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) > at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6253) FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp
[ https://issues.apache.org/jira/browse/YARN-6253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6253: --- Fix Version/s: 2.9.0 > FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp > --- > > Key: YARN-6253 > URL: https://issues.apache.org/jira/browse/YARN-6253 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Labels: atsv2-hbase, yarn-5355-merge-blocker > Fix For: 2.9.0, YARN-5355, 3.0.0-beta1 > > Attachments: YARN-6253.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6069) CORS support in timeline v2
[ https://issues.apache.org/jira/browse/YARN-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-6069: --- Fix Version/s: 2.9.0 > CORS support in timeline v2 > --- > > Key: YARN-6069 > URL: https://issues.apache.org/jira/browse/YARN-6069 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Sreenath Somarajapuram >Assignee: Rohith Sharma K S > Fix For: 2.9.0, YARN-5355, YARN-5355-branch-2, 3.0.0-alpha4 > > Attachments: YARN-6069-YARN-5355.0001.patch, > YARN-6069-YARN-5355.0002.patch, YARN-6069-YARN-5355.0003.patch, > YARN-6069-YARN-5355.0004.patch, YARN-6069-YARN-5355.0005.patch > > > By default the browser prevents accessing resources from multiple domains. In > most cases the UIs would be loaded form a domain different from that of > timeline server. Hence without CORS support, it would be difficult for the > UIs to load data from timeline v2. > YARN-2277 must provide more info on the implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4675) Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2
[ https://issues.apache.org/jira/browse/YARN-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-4675: --- Fix Version/s: 2.9.0 > Reorganize TimelineClient and TimelineClientImpl into separate classes for > ATSv1.x and ATSv2 > > > Key: YARN-4675 > URL: https://issues.apache.org/jira/browse/YARN-4675 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Labels: YARN-5355, yarn-5355-merge-blocker > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-4675-YARN-2928.v1.001.patch, > YARN-4675-YARN-5355.v2.011.patch, YARN-4675.v2.002.patch, > YARN-4675.v2.003.patch, YARN-4675.v2.004.patch, YARN-4675.v2.005.patch, > YARN-4675.v2.006.patch, YARN-4675.v2.007.patch, YARN-4675.v2.008.patch, > YARN-4675.v2.009.patch, YARN-4675.v2.010.patch, YARN-4675.v2.011.patch > > > We need to reorganize TimeClientImpl into TimeClientV1Impl , > TimeClientV2Impl and if required a base class, so that its clear which part > of the code belongs to which version and thus better maintainable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5050) Code cleanup for TestDistributedShell
[ https://issues.apache.org/jira/browse/YARN-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-5050: --- Fix Version/s: 2.9.0 > Code cleanup for TestDistributedShell > - > > Key: YARN-5050 > URL: https://issues.apache.org/jira/browse/YARN-5050 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Li Lu > Labels: yarn-2928-1st-milestone > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: YARN-5050-YARN-2928.001.patch, > YARN-5050-YARN-2928.002.patch > > > We introduced some small errors after yesterday's rebase. Also, some timeout > settings for timeline v2 tests are deprecated since we introduced global time > out settings in YARN-4545. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org