[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589264#comment-15589264 ] Hudson commented on YARN-5561: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10638 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10638/]) YARN-5561. [Atsv2] : Support for ability to retrieve (sjlee: rev e9c4616b5e47e9c616799abc532269572ab24e6e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/storage/TestFileSystemTimelineReaderImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServices.java > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Fix For: YARN-5355 > > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.04.patch, YARN-5561.05.patch, YARN-5561.patch, > YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589197#comment-15589197 ] Sangjin Lee commented on YARN-5561: --- +1. Committing shortly. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.04.patch, YARN-5561.05.patch, YARN-5561.patch, > YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588451#comment-15588451 ] Varun Saxena commented on YARN-5561: +1 on the latest patch. [~sjlee0], I guess patch is fine for you as well. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.04.patch, YARN-5561.05.patch, YARN-5561.patch, > YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587603#comment-15587603 ] Hadoop QA commented on YARN-5561: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 10 new + 19 unchanged - 0 fixed = 29 total (was 19) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s {color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 18m 7s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12834094/YARN-5561.05.patch | | JIRA Issue | YARN-5561 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 044a7d1f23da 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4bca385 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13437/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13437/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13437/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST >
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586207#comment-15586207 ] Varun Saxena commented on YARN-5561: A small nit again in javadoc. Sorry for nitpicking. "Return a set of containers belongs to given application attempt id" should be "Return a set of container entities belonging to given application attempt id" Other than that +1 from my side too. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.04.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586195#comment-15586195 ] Sangjin Lee commented on YARN-5561: --- LGTM. [~varun_saxena]? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.04.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586109#comment-15586109 ] Hadoop QA commented on YARN-5561: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 32s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 10 new + 19 unchanged - 0 fixed = 29 total (was 19) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 46s {color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 22s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12833990/YARN-5561.04.patch | | JIRA Issue | YARN-5561 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 2ece49e7ffb3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d26a1bb | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13423/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13423/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13423/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > ---
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582903#comment-15582903 ] Varun Saxena commented on YARN-5561: Few nits. # "Return a set of application-attempts entities" should be "Return a set of application-attempt entities" in javadoc # Return a single application-attempt entity of the given Id => Return a single application-attempt entity for the given attempt Id in javadoc # Return a set of containers entities belongs to given application attempt => Return a set of container entities belonging to given application attempt in javadoc # Above getContainer method javadoc says " Return a single application-attempt entity of the given Id" which is incorrect. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582866#comment-15582866 ] Sangjin Lee commented on YARN-5561: --- [~rohithsharma], could you kindly update v.03 of the patch to address several checkstyle issues (the ones other than the number of arguments violations)? Then we can go ahead with this. Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582827#comment-15582827 ] Varun Saxena commented on YARN-5561: That should be fine. We can keep a JIRA open till next drop on trunk and update documentation via it just like we did before first drop on trunk. There can be multiple contributors to the JIRA. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582088#comment-15582088 ] Rohith Sharma K S commented on YARN-5561: - I would prefer to take it up as separate JIRA as consolidated documentation update. And also I suspect and pretty sure, there are couple of more changes will come which require to modify documentations. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582036#comment-15582036 ] Varun Saxena commented on YARN-5561: [~rohithsharma], should we update documentation for this ? We can also do it in a separate JIRA though because some of the query params seem to be repeated in the documentation. Probably we can consolidate these query params at a single place in that JIRA and add these endpoints as well. Or you can add the endpoints here itself. As you wish. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581949#comment-15581949 ] Varun Saxena commented on YARN-5561: Yeah what I meant is a separate endpoint may be required for things like that. Not sure what we name it though. Or we can just put everything under the hood of ws/v2/timeline. Anyways maybe a separate JIRA should be filed for that disucssion. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15581429#comment-15581429 ] Rohith Sharma K S commented on YARN-5561: - Log serving we can launch in separate REST namespace with in the TimelineReader. May it so called as YarnTimelineUtilService. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576357#comment-15576357 ] Varun Saxena commented on YARN-5561: One question though. From where we will serve aggregated logs of historical apps from ? We do it currently from AHS. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575569#comment-15575569 ] Rohith Sharma K S commented on YARN-5561: - During weekly sync up, we had discussion regarding for providing new REST URL i.e */ws/v2/applicationhistory*. One of the point discussed is what is the advantage is having separate REST end points which returns is *Report format? It will become a another version existing data with different format. But ATSv2 TimelineEntity is super set of all where in user might required information which are not in *Report. So, if required information are published under first class entity info level like YARN-5699 doing, then it is more than sufficient. The consensus are # we publish the yarn entities important information at first class entity info level so that user can make use of these information to query using infofilters. # We will be going ahead with adding new REST end points(patch already attached i.e YARN-5561.03.patch) which are enhancement of existing getEntities. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572419#comment-15572419 ] Varun Saxena commented on YARN-5561: I was also thinking that we can have an endpoint like /ws/v2/applicationhistory. This can be used to not only serve app attempt.container reports but also serve use cases like serving aggregated logs of historical apps. We would need to serve aggregated logs from somewhere too. This will be useful for UI as well. Regarding metrics, well we can always extend ContainerInfo to carry metrics as well. Right ? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570985#comment-15570985 ] Rohith Sharma K S commented on YARN-5561: - Thanks for clarifications.. Though utility class is very useful for converting reports, one major concern is number of REST calls to be invoked by user. Especially in Web, this will become 2 REST calls which decrease the performance. Basically, me and Li lu was thinking to embedded TimelineYARNEntityReaderWebService to timelinereader daemon with REST path */ws/v2/applicationhistory*. These REST end points would take care of retrieving entities from storage and convert to required YARN reports such as ApplicationAttemptReport or ContainerReport etc. bq. Also, please note that what's contained in the current REST output would likely to be a superset of *Report; I agree that current *Report would be subset of TimelineEntity object. So , this we can solve by defining new *Report which compatible with ATSv2 either by extending current *reports since metrics are required. If any new fields published to ATS, then It is contract between publisher and reader that these information fields should added in *Report. New *report class is only wrapper over TimelineRntity object for YARN entities. And at any point of time, if user thinks he need more informations, he can query from ATSv2 which is always open. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570893#comment-15570893 ] Sangjin Lee commented on YARN-5561: --- bq. Couple of more doubts just to be in sync, Does translation layer is with-in-the-reader service or separate ? What is the format of report object? Is this report object is general or only yarn entities? I was thinking of something like utility classes that can create and return specific report types. For example, {code} ApplicationAttemptReport getApplicationAttemptReport(TimelineEntity appAttemptEntity); ContainerReport getContainerReport(TimelineEntity containerEntity); {code} These classes can be (and probably would need to be) in the yarn common module so any REST client can invoke it to get a translated object back. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570688#comment-15570688 ] Rohith Sharma K S commented on YARN-5561: - bq. Could not get the intention behind pulling it out to a separate daemon though. Can you elaborate on that ? bq. could you elaborate more on the concrete use cases for the need of a separate daemon for now? I meant say, Initial idea is combine YARN entity reader along with TimelineReaderWebService. In case folks do not agree then next proposal is to pulling out as separate daemon. Even I am decline for pulling out from timelinereader service but if other folks have any concern for combining along with timelinereader then it is an option. bq. As we briefly discussed during the call the other day, what we need is a translation layer that can create a Report object out of the timeline entity. If we implement such a translation layer, would it satisfy this? Yes. Couple of more doubts just to be in sync, Does translation layer is with-in-the-reader service or separate ? What is the format of report object? Is this report object is general or only yarn entities? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569553#comment-15569553 ] Sangjin Lee commented on YARN-5561: --- I recognize the pain point you're mentioning in terms of wanting the data in the form of {{ApplicationAttemptReport}} or {{ContainerReport}}. That said, adding a new daemon should not be done lightly. I am in the opinion that the bar above which a new component is added to the system should be high. I'm not sure if this meets that bar. Also, please note that what's contained in the current REST output would likely to be a *superset* of {{*Report}}; in other words, there are things in the entity API that are not in the {{*Report}}. Things like the uid and the currently discussed entity id prefix come to mind. So we cannot simply replace the return type, or things will be crippled. As we briefly discussed during the call the other day, what we need is a translation layer that can create a Report object out of the timeline entity. If we implement such a translation layer, would it satisfy this? That way, the client gets the full timeline entity information, and it can convert it into a report object fairly easily without writing much code. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569539#comment-15569539 ] Li Lu commented on YARN-5561: - I'm generally OK with option 1, as it's quite consistent with my original thought (see the comment on Aug. 30). The only thing I would like to mention here is, we gave up this idea before because we thought when web UI requesting data, it needs to contact two different endpoints. I'm not sure if with current web UI use cases this problem is alleviated. bq. This approach can be modified/enhanced to pulling out of ATSv2 to separate daemon service OR can be start new service in RM itself with application history. I'd incline not to go this far as of now. Sure, architecturally we can keep this flexibility, but could you elaborate more on the concrete use cases for the need of a separate daemon for now? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569415#comment-15569415 ] Varun Saxena commented on YARN-5561: Similar to option 1 in terms of having separate endpoints is what we had in AHS/ATSv1 too i.e. in the form of AHSWebServices. That will clearly delink APIs'. What do we name it though ? {{ws/v2/applicationhistory}} ? We can further add things like serving container logs from this endpoint too. And this can act as one stop destination for fetching YARN specific history data. Could not get the intention behind pulling it out to a separate daemon though. Can you elaborate on that ? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568429#comment-15568429 ] Rohith Sharma K S commented on YARN-5561: - Thanks [~sjlee0] [~gtCarrera9] [~varun_saxena] for reviewing patch. On second thought, ultimate goal for YARN-5561 and YARN-5699 is to provide an easy way for YARN Web UI. It would be much helpful for YARN Web UI and also for the users who wants query containers/app-attempts from CLI if app-attempts/containers are in the form of ApplicationAttemptReport and ContainerReport format. Basically overall intention is to provide flexible REST API to users with which they can directly make use from response output rather than allowing them to parse the ATSv2 REST output. I think it would be good to take early decision rather than struggling at final stages which leads to many code changes in dependencies especialy YARN Web UI. Couple of approaches for providing a facility to convert TimelineEntity objects to respective YARN reports are # Launch a new web service i.e TimelineYARNEntityReaderWebServices along with TimelineReaderWebServices. This service would run as part of TimelienReader daemon. This new service is just an translator to converting TimelineEntity object to corresponding YARN reports such as ApplicationAttemptReport or ContainerReport. And this service would contains new YARN specific entities which are exposed REST end points in this patch. Note that these REST end points return type are corresponding reports like AppAttemptsInfo. This approach can be modified/enhanced to pulling out of ATSv2 to separate daemon service OR can be start new service in RM itself with application history. # For existing TimelineReaderWebServices, *change the return type* for newly added REST end points such as getAppAttempts/getAppAttempt/getContainers/getContainer with AppAttemptsInfo/AppAttemptInfo/ContainersInfo/ContainerInfo respectively. Cons to add in TimelienReader service is it becomes bit combination of ATS+YARN specific details. Thoughts.? This will bring down many of the JIRA like YARN-5561 YARN-5699 etc can be combined and avoided altogether . cc:/ [~vinodkv] > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.03.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568136#comment-15568136 ] Hadoop QA commented on YARN-5561: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 30s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 14 new + 19 unchanged - 0 fixed = 33 total (was 19) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 46s {color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 39s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832851/YARN-5561.03.patch | | JIRA Issue | YARN-5561 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 211dcbcf4f9d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 6476934 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13356/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13356/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13356/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST >
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563483#comment-15563483 ] Li Lu commented on YARN-5561: - I don't have a strong preference on those query parameters either, but I think adding them is totally fine. The latest patch generally LGTM. [~rohithsharma] would you like to rebase it to trunk and fix the few "longer than 80 chars" warnings so that we can commit this? Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1831#comment-1831 ] Sangjin Lee commented on YARN-5561: --- {quote} That said, these REST end points are enhancement of getEntitties and getEntity. So, do not want to change any of the query params associated with app-attempts and container. In future, we never know what metrics/configs/relationships would be stored in attempts/containers. {quote} Regarding whether to retain things that are not needed for app attempts and so on, I don't have a strong opinion either way, but I might lean slightly towards leaving them in. Even with the existing queries for them today, we don't strongly reject them even if they are specified for app attempts but simply return empty data, right? Also, leaving them in might be slightly more future-proof. Again these are not strong preferences. in terms of documenting, let's just be consistent and add the javadoc. Being consistent is probably a good thing here. If we decide later that javadoc is superfluous, we could take them out in bulk. [~rohithsharma], does this need to go into trunk or can it stay in YARN-5355? If it is needed on trunk (which is the impression I got), then let's change it to a top level JIRA and create a patch based on the trunk. Once it is committed to the trunk, we can cherry-pick it onto YARN-5355 (and its associated branch-2 branch). > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554982#comment-15554982 ] Varun Saxena commented on YARN-5561: bq. That said, these REST end points are enhancement of getEntitties and getEntity. So, do not want to change any of the query params associated with app-attempts and container. In future, we never know what metrics/configs/relationships would be stored in attempts/containers. Well the intention behind my comment was that we will document these REST endpoints separately. So for the user its a distinct endpoint. The fact that we in turn redirect this call to entity table is internal implementation. And if we indicate that you can apply config filters etc., it may indicate that we store them for containers, app attempts which we don't. And if there is something which we don't publish from RM/NM, I thought its pointless to have filter for it. Your thoughts on the same ? We currently do not restrict clients to publish even YARN specific entities so they can potentially publish configs for it but I remember we had a JIRA related to that i.e. do not allow arbitrary clients to publish YARN entities. I do not have a very strong opinion on this though. Let us see what others think. bq. This is one of point of concern for code readability. May be we can capture in document, but not in java class file. Thought? We had initially decided we will add both javadoc and documentation. We however want to trim down the documentation a bit because some query params are repeated. Let us keep it consistent no matter what we do. We can probably add non-javadoc comments over REST methods so that developer can read them. As such REST endpoints are already captured in documentation for outside users of ATS. I remember [~rohithsharma] you had a JIRA for it. Maybe we can handle it there if the team reaches a consensus on this. [~sjlee0], your opinion on this ? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15554926#comment-15554926 ] Rohith Sharma K S commented on YARN-5561: - bq We do not store configurations at container and app attempt level. So we can leave out query params associated with it, bq. Similarly no metrics are stored at app attempt level. Same goes for relationships. We can add them when we support them, if not for anything but just to reduce lines in code and documentation That said, these REST end points are enhancement of getEntitties and getEntity. So, do not want to change any of the query params associated with app-attempts and container. In future, we never know what metrics/configs/relationships would be stored in attempts/containers. Lets keep it as it, if user query configs/metrics then he gets result in empty. It indicates , there is no metrics or configs published. bq. Javadoc will have to be updated I think otherwise we will get a -1 in QA report. Yes this will unnecessarily increase code size but then there is no other way out. This is one of point of concern for code readability. May be we can capture in document, but not in java class file. Thought? bq. Update tests for app attempts too ? Let me update the tests and will update new patch. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552387#comment-15552387 ] Hadoop QA commented on YARN-5561: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 59s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} YARN-5355 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 13 new + 19 unchanged - 0 fixed = 32 total (was 19) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 44s {color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 17m 14s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12831977/0001-YARN-5561.YARN-5355.patch | | JIRA Issue | YARN-5561 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 4ee54e96c52d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | YARN-5355 / 5d7ad39 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13309/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13309/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13309/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552370#comment-15552370 ] Varun Saxena commented on YARN-5561: Thanks [~rohithsharma] for the patch. # We do not store configurations at container and app attempt level. So we can leave out query params associated with it, # Similarly no metrics are stored at app attempt level. Same goes for relationships. We can add them when we support them, if not for anything but just to reduce lines in code and documentation :) # Javadoc will have to be updated I think otherwise we will get a -1 in QA report. Yes this will unnecessarily increase code size but then there is no other way out. # Update tests for app attempts too ? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: 0001-YARN-5561.YARN-5355.patch, YARN-5561.02.patch, > YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546350#comment-15546350 ] Li Lu commented on YARN-5561: - Seems like we're reaching much agreement on the general proposal of this JIRA. [~rohithsharma] would you please update the patch so that we can move forward with the rest process? Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.02.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522168#comment-15522168 ] Rohith Sharma K S commented on YARN-5561: - bq. Rohith, I think you're saying you are OK without having that API since we have a query that returns all apps for a flow RIght, I am fine with all apps for a flow. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.02.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517898#comment-15517898 ] Sangjin Lee commented on YARN-5561: --- Sorry it took me a while to get to this. I'm generally fine with the newly proposed REST end points (as they seem to be more like short-handed API for existing ones). Regarding the need for listing all apps (in a cluster), I agree with others that it would be quite problematic. If I read this thread correctly, Rohith, I think you're saying you are OK without having that API since we have a query that returns all apps for a flow (and one that returns all flow activities). > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.02.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490259#comment-15490259 ] Hadoop QA commented on YARN-5561: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 11s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 31s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 18 new + 19 unchanged - 0 fixed = 37 total (was 19) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 35s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 13s {color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 46s {color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 13m 14s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12828437/YARN-5561.02.patch | | JIRA Issue | YARN-5561 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux a1099601e42f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ea0c2b8 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13105/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/13105/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13105/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13105/console | | Powered by | Apache Yetus 0.3.0 http://y
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15490212#comment-15490212 ] Rohith Sharma K S commented on YARN-5561: - As of now /apps cab be dropped since apps can be retrieved using flows. Based on the use case from UI let us decide the /apps at cluster level. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.02.patch, YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485249#comment-15485249 ] Li Lu commented on YARN-5561: - Thanks [~rohithsharma], the two proposed APIs make the app - attempt - container APIs more comprehensive. LGTM. bq. are looking for a complete applications page where all applications which were running/completed had to be listed. This is something I'm not sure if we'd like to support on the timeline level. This information is pretty much available on the RM side via the state store. Why do we want to encourage an expensive operation on the timeline store for this data? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483258#comment-15483258 ] Varun Saxena commented on YARN-5561: bq. are looking for a complete applications page where all applications which were running/completed had to be listed. For this purpose, I think we need the api as suggested by Rohith. Being said this, We will also be showing hierarchy from flows too. So you plan to have 2 app pages. One from a specific flow run and other a list of all the apps in a cluster. Right ? [~rohithsharma], how do you plan to support fetching all apps within a cluster ? Probably you can adopt the approach I had suggested. Because otherwise it would lead to full table scan. bq. New API's required. Thoughts? We should have them for the sake of completeness. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483122#comment-15483122 ] Rohith Sharma K S commented on YARN-5561: - Thanks [~sunilg] for quick response. I will update patch accordingly. Along with below API's additionally 2 more API require I think. Thoughts? {noformat} 2. (cluster - )app - app_attempt - container sequence (6) - /clusters/{clusterid}/apps/{appid}/appattempts/{appattemptid}/containers - /clusters/{clusterid}/apps/{appid}/appattempts - /clusters/{clusterid}/apps/{appid}/ - /apps/{appid}/appattempts/{appattemptid}/containers - /apps/{appid}/appattempts - /apps/{appid}/ New API's required. Thoughts? - /clusters/{clusterid}/apps/{appid}/appattempts/{appattemptid} - /clusters/{clusterid}/apps/{appid}/appattempts/{appattemptid}/containers/{contianer-id} {noformat} > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482969#comment-15482969 ] Sunil G commented on YARN-5561: --- Hi [~rohithsharma] Yes. We are looking for a complete applications page where all applications which were running/completed had to be listed. For this purpose, I think we need the api as suggested by Rohith. Being said this, We will also be showing hierarchy from flows too. Once end use land on this applications page, various filters/views could be derived. Hence we could also cover or show details of flows. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482923#comment-15482923 ] Rohith Sharma K S commented on YARN-5561: - In *Li lu*'s REST endpoint list, there is one end point which is missed i.e listing all the apps for given flow name */users/{userid}/flows/{flowname}/apps/*. bq. And based on use case of Rohith maybe list all apps within a cluster as well. This is one of the objective of this JIRA when I reported. But there is another end point from which apps can be retrieved i.e at flow name, the client can get all the flows in the cluster and for each flow, client can retrieve the apps using above specified path. So, let me confirm with Sunil does new UI has such use case to get all the apps per cluster. If so, I will refresh the patch. cc:/ [~sunilg] > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475482#comment-15475482 ] Li Lu commented on YARN-5561: - bq. We can follow approach suggested by you as well. I do not have a strong opinion on either. So I will leave it as it is. Lets see what others think. Fine with me. Once we have a clear view about the current rest end points and they're not confusing we're good. bq. And based on use case of Rohith maybe list all apps within a cluster as well. However in my personal opinion that may not be necessary. Nice catch. [~rohithsharma] if you feel we need this endpoint, please feel free to refresh the patch. Generally LGTM. I'll wait ~24 hrs and then commit the patch. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472930#comment-15472930 ] Varun Saxena commented on YARN-5561: bq. So the new addition looks fine to me. Looks fine to me as well. bq. Do we want to reorganize the code in a way consistent with this list? Well the current organization is based on what we are retrieving that is all endpoints for fetching entities are together, for fetching apps are together and so on. We can follow approach suggested by you as well. I do not have a strong opinion on either. So I will leave it as it is. Lets see what others think. bq. discuss the endpoints that marked as weird These endpoints were added to get all apps belonging to a flow so we skip the flow run section. There were use cases to fetch all apps within a flow in case run id is not known. Refer to [~vrushalic]'s [comment on YARN-3864 | https://issues.apache.org/jira/browse/YARN-3864?focusedCommentId=14940489&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14940489] We also plan to list all apps for a user or queue in future as well. And based on use case of Rohith maybe list all apps within a cluster as well. However in my personal opinion that may not be necessary. You can check with New Web UI folks. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472330#comment-15472330 ] Li Lu commented on YARN-5561: - Sequence 2 in this list seems to be replacing the majority part of the old AHS APIs, except for container logs. We may use them as a starting point to support AHS-like use cases in timeline v2. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472318#comment-15472318 ] Li Lu commented on YARN-5561: - OK I revisited the patch and our reader REST APIs. Right now we have 26 (!) APIs for reader server. I tried to organize them in a way easier to understand: {code} 0. cluster activity (2) - /flows/ - /clusters/{clusterid}/flows/ 1. (cluster - )user - flow - run - app - entity_type - entity_id sequence 1.1 hierarchical (12) - /clusters/{clusterid}/users/{userid}/flows/{flowname}/runs/{flowrunid}/apps/ - /clusters/{clusterid}/users/{userid}/flows/{flowname}/runs/{flowrunid}/ - /clusters/{clusterid}/users/{userid}/flows/{flowname}/runs/ - /users/{userid}/flows/{flowname}/runs/{flowrunid}/apps/ - /users/{userid}/flows/{flowname}/runs/{flowrunid}/ - /users/{userid}/flows/{flowname}/runs/ (user, flow, run information omitted since an application is unique in a cluster) - /clusters/{clusterid}/apps/{appid}/entities/{entitytype}/{entityid}/ - /clusters/{clusterid}/apps/{appid}/entities/{entitytype} - /apps/{appid}/entities/{entitytype}/{entityid}/ - /apps/{appid}/entities/{entitytype} - /clusters/{clusterid}/users/{userid}/flows/{flowname}/apps/ (looks weird, jumping levels) - /users/{userid}/flows/{flowname}/apps/ (looks weird, jumping levels) 1.2 uid (6) - /flow-uid/{uid}/runs/ - /run-uid/{uid}/ - /run-uid/{uid}/apps - /app-uid/{uid}/ - /app-uid/{uid}/entities/{entitytype} (entity type looks weird) - /entity-uid/{uid}/ 2. (cluster - )app - app_attempt - container sequence (6) - /clusters/{clusterid}/apps/{appid}/appattempts/{appattemptid}/containers - /clusters/{clusterid}/apps/{appid}/appattempts - /clusters/{clusterid}/apps/{appid}/ - /apps/{appid}/appattempts/{appattemptid}/containers - /apps/{appid}/appattempts - /apps/{appid}/ {code} So the new addition looks fine to me. Do we want to reorganize the code in a way consistent with this list? Right now the code seems to be a little bit messy. We can do it in this JIRA, or we can open a new JIRA to reorganize these APIs and discuss the endpoints that marked as weird? Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457045#comment-15457045 ] Li Lu commented on YARN-5561: - I've got offline discussions with several folks on if we should have concepts like "app-attempt" and "container". From a designer's perspective, app-attempts and containers should not be included in timeline APIs, but from YARN users perspective, requesting app-attempt and container level information seems to be very natural operations, especially since both concepts are top level concepts in YARN. So I'm relatively fine with having terms like "containers" and "app-attempts" exposed in timeline APIs, but we may want to be very careful to not to give an impression that attempts and containers are on the hierarchical order as flows and flowruns. So how about having two different hierarchical orders: Order 1, native timeline order: cluster, user, flow, flow-run, application, entity Order 2, YARN application order: application, app-attempt, container Once we're not mixing the two orders in APIs, the logic should be clear. Thoughts? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454304#comment-15454304 ] Rohith Sharma K S commented on YARN-5561: - Separating out YARN specific details is good idea similar to v1. Here is my vote will be 50-50 for this approach. In v1, entities were fully separated out from yarn specific details. But in v2, Apart from the entities, *Query Apps for a Flow* and *Query Apps for a Flow Run* and other details are in TimelineReaderWebService. These are belongs to YARN specific details nevertheless of any underlying storage schema. All the entities are published under application scope which makes decision harder for devs to adding a new REST YARN specific end points. >From the user perspective, I want to share you that say for retrieving all >the apps with flow/flowrun uses path /ws/v2/timeline, but for retrieving >attempts uses path /ws/v2/applicationhistory would lead to big question for >users why there are 2 different Path for same application details!!!. May be we can takes other folks thoughts too on this. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452972#comment-15452972 ] Li Lu commented on YARN-5561: - OK let me clarify: IMO the reader API of YARN timeline service should focus on serving timeline entities according to caller's request, but not on how to serve YARN specific use cases. To the storage layer of timeline service, requesting "container info" should be similar to requesting distributed shell application information or Tez job information. I noticed that in this patch, we're passing some predefined constants, like: {code} String entityType = TimelineEntityType.YARN_CONTAINER.toString(); {code} This will query for a specific type of timeline entities. We may want to provide a different endpoint (like /ws/v2/applicationhistory) to support this YARN specific use case. In v1, we have AHSWebServices to support YARN specific application history information. Maybe we would like to keep the same way? This is my own (and subjective) idea. Feel free to let me know if you noticed some critical things I'm missing... Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451966#comment-15451966 ] Rohith Sharma K S commented on YARN-5561: - bq. One concern I have is, shall we expose the semantics of "containers" and "appattempts" in timeline reader API level? Sorry could not understand this fully. I assume that you are asking to create another web service with different path like AHSWebService which internally redirect the calls to getEntites in TimeLineReaderWebApp. So there will be two @Path i.e */ws/v2/timeline* and */ws/v2/applicationhistory* exposed in ATSv2. Is it ? bq. Similar to the past AHS APIs, maybe we'd like to abstract a separate layer of APIs for YARN to read data from timeline service? Sorry I could not get it. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450303#comment-15450303 ] Hadoop QA commented on YARN-5561: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice: The patch generated 12 new + 19 unchanged - 0 fixed = 31 total (was 19) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 43s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s {color} | {color:red} hadoop-yarn-server-timelineservice in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 49s {color} | {color:green} hadoop-yarn-server-timelineservice in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 57s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12826160/YARN-5561.patch | | JIRA Issue | YARN-5561 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 5dabe98111ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 9dcbdbd | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12953/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/12953/artifact/patchprocess/whitespace-eol.txt | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/12953/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12953/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450294#comment-15450294 ] Li Lu commented on YARN-5561: - Thanks [~rohithsharma]! I took a deeper look into the patch. The overall approach LGTM. One concern I have is, shall we expose the semantics of "containers" and "appattempts" in timeline reader API level? To me, the timeline API is independent from the actual use cases: it merely provides concrete ways to retrieve timeline entities. Similar to the past AHS APIs, maybe we'd like to abstract a separate layer of APIs for YARN to read data from timeline service? Of course we can integrate the new "AHS REST API" layer in the timeline reader server, but code-wise shall we separate the new APIs from TimelineReaderWebService? BTW, I set this JIRA to be patch available so that Jenkins can find the patch and test it. If you want something otherwise please feel free to change it back. Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448961#comment-15448961 ] Rohith Sharma K S commented on YARN-5561: - I created new JIRA YARN-5585 for adding new filter *fromId* in REST endpoints. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.patch, YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447776#comment-15447776 ] Rohith Sharma K S commented on YARN-5561: - bq. Currently not supported. What's the use case for this by the way ? This is required for pagination. And more importantly fromId is must for support for pagination. And also limit & fromId should be there in all the REST end points. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446709#comment-15446709 ] Li Lu commented on YARN-5561: - Thanks! That's fine. Once we keep this behavior consistent we're good. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446698#comment-15446698 ] Varun Saxena commented on YARN-5561: bq. Is there a way to specify "unlimit" Not as such. Max value of long can be specified. Will return everything > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446641#comment-15446641 ] Li Lu commented on YARN-5561: - Ah I just checked the v0 patch, the current approach to keep default limit to be 100 unless specified seems fine. Is there a way to specify "unlimit" (although not encouraged)? Thanks! > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446573#comment-15446573 ] Varun Saxena commented on YARN-5561: bq. FromId is something which filter need to support so from where to retrieve the apps(not there now). For example , app-1 to app-10 are there. So if fromId=app-5 then apps app-6 to app-10 should be retrieved. Is there any other way to achieve this now? Currently not supported. What's the use case for this by the way ? This can be supported easily though, within the scope of a flow. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446553#comment-15446553 ] Li Lu commented on YARN-5561: - One potential problem of returning all apps in a cluster/containers in an app attempt is that they may easily cause severe server slowness if not used properly. In the past this became a great pain when Apache Ambari called the default endpoint of AHS, when the total number of applications is extremely large. I agree it is quite desirable to have the ability to "list all", but at the same time we may want to consider restrict the number of returned entities unless specified by the caller, or support paging, or both? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446476#comment-15446476 ] Rohith Sharma K S commented on YARN-5561: - Attached a patch for supporting legacy API i.e retrieving containers for an attempt-id i.e {{/apps/\{appid\}/appattempts/\{appattemptid\}/containers}}. Basically, REST calls to getContainers method construct an infofilters internally and retrieve the entities. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446467#comment-15446467 ] Rohith Sharma K S commented on YARN-5561: - It is required to get all the apps in cluster. REST client will ask for limit and fromId. FromId is something which filter need to support so from where to retrieve the apps(not there now). For example , app-1 to app-10 are there. So if fromId=app-5 then apps app-6 to app-10 should be retrieved. Is there any other way to achieve this now? bq. If merely list of apps within a cluster is required(without any other information) we can probably serve this from app to flow table. Not only app id's but also other information required. And currently the way I see is need to get all the flows first and for each flows, retrieve apps. This is something where client need to lot of REST calls constructing for each flows. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-5561.v0.patch > > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446415#comment-15446415 ] Varun Saxena commented on YARN-5561: If merely list of apps within a cluster is required(without any other information) we can probably serve this from app to flow table. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15446353#comment-15446353 ] Varun Saxena commented on YARN-5561: Copying my comment over from YARN-5571. If we have to support fetching all the apps within a cluster, upto a limit, this currently is not advisable in HBase implementation as it would lead to a full table scan or a very large scan. Because the API guarantees returning entities in a descending order by created time. This would mean that we will go through each row to determine the result set. It is about taking one of the choices. Do we want to drill down to apps via flows or list all the apps in a cluster. We can discuss further in the team on this, depending on the use case. And see if we can have some workaround. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15445032#comment-15445032 ] Rohith Sharma K S commented on YARN-5561: - Other option is publish the entities as string rather than object. The below one is info value holds as object. {code} "SYSTEM_INFO_PARENT_ENTITY": { "type": "YARN_APPLICATION", "id": "application_1471931266232_0024" } {code} This can split into below which is very much useful in filtering. {code} "SYSTEM_INFO_PARENT_ENTITY_TYPE": "YARN_APPLICATION" "SYSTEM_INFO_PARENT_ENTITY_ID":"application_1471931266232_0024" {code} > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15445004#comment-15445004 ] Rohith Sharma K S commented on YARN-5561: - Looking into code base for reader for supporting these REST end points. I see that ApplicationTable schema is defined with row key *clusterId!userName!flowName!flowRunId!AppId*. # */apps* without userId and flowName should be able to read. Currently it is mandatory that to read apps to provide userId and flowName. # */apps/\{app-id\}/app-attempts* can be redirect to existing getEntity for YARN-APPLICATIOIN-ENTITY. # */apps/\{app-id\}/app-attempt/\{app-attempt-id\}/containers*. As per design, YARN_APPLICATION_ATTEMPT and YARN_CONTAINERS are stored as general entities. Reading will be same as others user entities. I think there should be another hbase table that stores YARN specific entities. Otherwise it is big overhead for clients to parse output to know about its parent entity. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437394#comment-15437394 ] Varun Saxena commented on YARN-5561: Sorry not really a full table scan. But a large scan. [~rohithsharma], do you have any use case for number 4 ? > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437380#comment-15437380 ] Varun Saxena commented on YARN-5561: Number 4 I think should be avoided. This would lead to full table scan and EntityTable can grow quite large. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5561) [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and entities via REST
[ https://issues.apache.org/jira/browse/YARN-5561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437375#comment-15437375 ] Vrushali C commented on YARN-5561: -- + 1 on adding more rest endpoints. It makes things easier to query/script for. As such in the list above, perhaps 2 and 3 can be combined such that 3 becomes query params in 2. As discussed in today's call, this jira might help with YARN-4343. > [Atsv2] : Support for ability to retrieve apps/app-attempt/containers and > entities via REST > --- > > Key: YARN-5561 > URL: https://issues.apache.org/jira/browse/YARN-5561 > Project: Hadoop YARN > Issue Type: Bug > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > > ATSv2 model lacks retrieval of {{list-of-all-apps}}, > {{list-of-all-app-attempts}} and {{list-of-all-containers-per-attempt}} via > REST API's. And also it is required to know about all the entities in an > applications. > It is pretty much highly required these URLs for Web UI. > New REST URL would be > # GET {{/ws/v2/timeline/apps}} > # GET {{/ws/v2/timeline/apps/\{app-id\}/appattempts}}. > # GET > {{/ws/v2/timeline/apps/\{app-id\}/appattempts/\{attempt-id\}/containers}} > # GET {{/ws/v2/timeline/apps/\{app id\}/entities}} should display list of > entities that can be queried. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org