[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369739#comment-15369739 ] Hudson commented on YARN-3864: -- SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10074/]) YARN-3864. Implement support for querying single app and all apps for a (sjlee: rev 0f44b5508d2ffcae08f130b6535a9832d37e2b38) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/application/ApplicationRowKey.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServicesHBaseStorage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderWebServices.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/TimelineEntityReaderFactory.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/TestTimelineReaderWebServicesFlowRun.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/TimelineEntityReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/GenericEntityReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/ApplicationEntityReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FlowActivityEntityReader.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/storage/FlowRunEntityReader.java > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Fix For: YARN-2928 > > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch, > YARN-3864-YARN-2928.04.patch, YARN-3864-addendum-appaggregation.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943702#comment-14943702 ] Varun Saxena commented on YARN-3864: [~sjlee0], thanks for the review. Will address your comments and update a patch shortly. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch, > YARN-3864-addendum-appaggregation.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943771#comment-14943771 ] Sangjin Lee commented on YARN-3864: --- The latest patch (v.4) LGTM. Unless there are additional comments, and with jenkins passing, I'll commit it soon. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch, > YARN-3864-YARN-2928.04.patch, YARN-3864-addendum-appaggregation.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943791#comment-14943791
]
Hadoop QA commented on YARN-3864:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 9s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 17s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 7s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 53s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 2m 57s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 41m 3s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12765030/YARN-3864-YARN-2928.04.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / a95b8f5 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9350/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9350/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9350/console |
This message was automatically generated.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch,
> YARN-3864-YARN-2928.04.patch, YARN-3864-addendum-appaggregation.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943645#comment-14943645
]
Sangjin Lee commented on YARN-3864:
---
I kicked off another jenkins build.
I have reviewed the latest patch (v.3), and it looks good to me for the most
part. I have only a few minor comments.
(TimelineReaderWebServices.java)
- l.540: nit: let's use a normal Java style: {{req.getQueryString() == null}}
- l.575: If we're calling this end point "flowrunapps", then shouldn't the
method be called {{getFlowRunApps}}? The latter one seems to be named that.
- Both for /flowrunapps and /flowapps, I understand it will return the most
recent N apps if item is specified, correct? Then it should be stated in the
javadoc.
If you could address those, and with jenkins passing, I'd like to go ahead and
commit the patch. Do let me know if you have other comments. Thanks!
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch,
> YARN-3864-addendum-appaggregation.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943613#comment-14943613
]
Hadoop QA commented on YARN-3864:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764952/YARN-3864-addendum-appaggregation.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b925cf1 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9348/console |
This message was automatically generated.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch,
> YARN-3864-addendum-appaggregation.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943648#comment-14943648 ] Sangjin Lee commented on YARN-3864: --- Should have refreshed the page first. :) The jenkins is failing because it's testing the addendum patch. Built the patch locally, ran all the tests and findbugs. All seem fine. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch, > YARN-3864-addendum-appaggregation.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942793#comment-14942793
]
Hadoop QA commented on YARN-3864:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764952/YARN-3864-addendum-appaggregation.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3b85bd7 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9344/console |
This message was automatically generated.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch,
> YARN-3864-addendum-appaggregation.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942175#comment-14942175 ] Varun Saxena commented on YARN-3864: Test failure unrelated. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941624#comment-14941624
]
Sangjin Lee commented on YARN-3864:
---
The design doc says
{quote}
Flow runs for a given flow must have unique and totally ordered run identifiers.
{quote}
It should be safe to assume that the flow run id is sequential *for a given
flow*.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941663#comment-14941663
]
Hadoop QA commented on YARN-3864:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 34s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 23s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 17s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 42s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 55s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 3m 21s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 42m 53s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764824/YARN-3864-YARN-2928.02.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / a95b8f5 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9333/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9333/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9333/console |
This message was automatically generated.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941606#comment-14941606 ] Varun Saxena commented on YARN-3864: Sorry I mean flow run ids' cant be guaranteed to be sorted as per run start time. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941619#comment-14941619 ] Sangjin Lee commented on YARN-3864: --- Thanks for updating the patch [~varun_saxena]. I'm now in transit, and it will take a little time before I can go over the patch in more detail. I'd like to summarize a couple of issues, and like us to tackle them after this. I don't think they need to hold this JIRA up. (1) the REST API At some point, let's take a look at the REST API (as discussed earlier), and see if we need to make them more consistent with the overall REST best practices and more importantly hadoop's REST style (YARN, etc.). It's not terribly urgent, and we just need to review it some time soon. (2) storing user in the app-to-flow table [~vrushalic] brought up the point of the user. And I'm realizing that we may need to store the user in the app-to-flow table. Was there a reason that we didn't store the user info in the app-to-flow table? The issue is that the user is a critical piece of the context (cluster/user/flow_id/flow_run_id/app_id), and the app-to-flow lookup should have it. Right now in the REST call, the user is optional, and if it is not provided it is deduced from the caller UGI. But IMO the caller UGI is a poor/incorrect choice. This would work only if the user/client that's executing the REST call is the owner of that YARN app. Rather, the user should be always the user of the app, regardless of who executes the REST call (authorization is a separate topic). I propose that we store the user id in the app-to-flow table, and when we recover the context on the read path, we use that user if the user is not provided by the caller. We don't have to do it as part of this JIRA, but we should fix this shortly after. What are your thoughts? > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941631#comment-14941631 ] Varun Saxena commented on YARN-3864: [~sjlee0], bq. At some point, let's take a look at the REST API (as discussed earlier), and see if we need to make them more consistent with the overall REST best practices and more importantly hadoop's REST style (YARN, etc.). It's not terribly urgent, and we just need to review it some time soon. We can do so. For user facing APIs' the form and the naming is very important. Everyone can chime in on this one. bq. storing user in the app-to-flow table Even I was wondering why its not there. In FS implementation, it was kept in the mapping. I think this should be fixed. Should I raise a new JIRA for this and fix it ? bq. I propose that we store the user id in the app-to-flow table, and when we recover the context on the read path, we use that user if the user is not provided by the caller. But for certain queries such as querying flows and flowruns or all apps for a flow/flow run, we will not have context information. Because context is for cluster and app. And for these queries, a single app has no meaning. I think for these queries, we can take UGI and for others take it from context. But this can make it confusing for user. So one option is to disregard UGI altogether and force user to be supplied for queries for which context cant be retrieved. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941673#comment-14941673 ] Varun Saxena commented on YARN-3864: bq. Yes, let's open a new JIRA. I can get to this next week, or you can take it on if you prefer. Ok. Will open a new JIRA. I can take it up. Will create a patch on top of this one. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941662#comment-14941662
]
Sangjin Lee commented on YARN-3864:
---
{quote}
Even I was wondering why its not there. In FS implementation, it was kept in
the mapping.
I think this should be fixed. Should I raise a new JIRA for this and fix it ?
{quote}
Yes, let's open a new JIRA. I can get to this next week, or you can take it on
if you prefer. Let me know. I'm fine either way. For the record, I don't think
this needs to be done by the POC time. The same goes for the REST API review.
{quote}
But for certain queries such as querying flows and flowruns or all apps for a
flow/flow run, we will not have context information. Because context is for
cluster and app. And for these queries, a single app has no meaning. I think
for these queries, we can take UGI and for others take it from context. But
this can make it confusing for user. So one option is to disregard UGI
altogether and force user to be supplied for queries for which context cant be
retrieved.
{quote}
Yes, I'm specifically talking about the existing app-to-flow lookup case which
applies only to generic entities and apps. Sorry if it was not clear. I agree
the flow run case is a little different. I do think the UGI is not very
meaningful in that case either (if the caller user does not match the right
user, it will return nothing anyway). I'm OK with making the user required.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941634#comment-14941634 ] Varun Saxena commented on YARN-3864: Ohh...I missed that part. I was assuming logically it will be in sequence but we were not enforcing it. But if this is documented this should be fine. Will change in next patch. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941646#comment-14941646 ] Varun Saxena commented on YARN-3864: I think REST API refactoring we can do after web UI poc. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941684#comment-14941684 ] Varun Saxena commented on YARN-3864: [~sjlee0], filed YARN-4221. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941586#comment-14941586 ] Varun Saxena commented on YARN-3864: [~gtCarrera9], the endpoint for web UI poc will now be /flowrunapps instead of /apps > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941727#comment-14941727 ] Li Lu commented on YARN-3864: - OK, I can update my local version accordingly. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch, > YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941839#comment-14941839
]
Hadoop QA commented on YARN-3864:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 32s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 10m 18s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 19s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 19s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 49s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 49s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 10s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 19m 34s | Tests failed in
hadoop-yarn-server-timelineservice. |
| | | 67m 35s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage |
| |
hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity |
| |
hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl
|
| |
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
|
| | hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764841/YARN-3864-YARN-2928.03.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / a95b8f5 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9336/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9336/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9336/console |
This message was automatically generated.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch,
> YARN-3864-YARN-2928.02.patch, YARN-3864-YARN-2928.03.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939960#comment-14939960 ] Varun Saxena commented on YARN-3864: Updated an initial patch. This can act as base for Web UI work to proceed. This should be enough but I will add some tests for filters later. In the patch I have renamed TestTimelineReaderWebServicesFlowRun to TestTimelineReaderWebServicesHBaseStorage. As mentioned before, we will make TestTimelineReaderWebServices use HBase and at that time both these classes will be merged. Added 5 test cases. [~djp] wanted some tests added for aggregation flow but as YARN-3816 has not yet gone in. Cannot do here. Will probably add an addendum to this patch on top of 3816. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940018#comment-14940018
]
Hadoop QA commented on YARN-3864:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 42s | Findbugs (version ) appears to
be broken on YARN-2928. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 2s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 16s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 4s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 3m 11s | Tests passed in
hadoop-yarn-server-timelineservice. |
| | | 40m 49s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12764627/YARN-3864-YARN-2928.01.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | YARN-2928 / a95b8f5 |
| hadoop-yarn-server-timelineservice test log |
https://builds.apache.org/job/PreCommit-YARN-Build/9325/artifact/patchprocess/testrun_hadoop-yarn-server-timelineservice.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/9325/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/9325/console |
This message was automatically generated.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940278#comment-14940278 ] Varun Saxena commented on YARN-3864: [~gtCarrera9], bq. As a big picture I assume there is an "applications" section in our flowrun endpoint's return value, which can be similar to the "flowruns" section in our /flows endpoint No we do not have that. Current flow run table does not store application IDs' belonging to a flow. Although it is possible to do that but would require it for Web UI ? I guess we can get flow runs from /flows endpoint and then for each flow run query the apps using /apps endpoint. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940420#comment-14940420 ] Sangjin Lee commented on YARN-3864: --- Thanks for the patch [~varun_saxena]! I'm away from the desk, and I'll communicate quick feedback. The URL patterns of the new endpoints seem bit unnatural. For example, "/apps" comes before the flow id and the flow run id. Does this conform to the normal REST style? IIUC, they usually follow the direction of the narrowing scope, right? For example, the current YARN REST API follows that style (app -> app attempt -> container). Should we make our REST API consistent with that style? (TimelineEntityReader.java) - l.82: nit: I think sortedKeys can/should be private > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940344#comment-14940344 ] Li Lu commented on YARN-3864: - bq. I guess we can get flow runs from /flows endpoint and then for each flow run query the apps using /apps endpoint. Yes, this is my current plan. Thanks for letting me know this! > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940381#comment-14940381 ] Sangjin Lee commented on YARN-3864: --- Yes, the apps query (and therefore the application table) is the most natural place to query for all apps for a flow run. The flow run table is mainly for handling real-time aggregation, and should be considered secondary for this purpose. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940551#comment-14940551 ] Varun Saxena commented on YARN-3864: Yeah but this is similar to what we do while querying entity. These optional params, if provided can preclude the need to look up flow context from app to flow mapping table. And user id would be required anyways because it cannot be retrieved from app to flow mapping table. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940460#comment-14940460 ] Varun Saxena commented on YARN-3864: bq. I think sortedKeys can/should be private Ok. bq. The URL patterns of the new endpoints seem bit unnatural. For example, "/apps" comes before the flow id and the flow run id. apps indicates what we want to query and is a static path param. And flow id and flow run id are dynamic path params. In path, because they are mandatory for the query. This follows the style for previous endpoints as well. Even for entities endpoint for instance, we have entities in front(static part of URL) and then cluster id, app id etc. The format we are following for URL endpoint is as under : # A fixed static part in URL indicating what this query is meant for i.e. app, apps, entity, entities, flowrun, flows, etc. You can say this defines what table to peek into. # Followed by dynamic mandatory parameters in URL path. Mainly parts which define the row key and need to be given by client mandatorily. # Other things which are not mandatory for query are given via query parameters. We can change it to have dynamic parts in front to follow an order of cluster->flow->flowrun->app->entity. But this is not specific to this endpoint. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[
https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940520#comment-14940520
]
Varun Saxena commented on YARN-3864:
[~vrushalic]
bq. For the /app endpoint, I think we need only the app id as the path param,
none of the others are needed.
Thats what we have.
We have two endpoints for app - /app/{appid} and /app/{clusterid}/{appid}/ .
In first one, cluster id will be taken from config and in the next one supplied
by client. Do you mean something else ?
bq. for the /apps endpoint, do you think we can call it flowApps or appsflow or
something so that it is clear that we are querying for all applications in this
particular in this endpoint.
Makes sense. Lets make it flowapps.
bq.I think this endpoint should be also available without requiring the flow
run id.
Ok. Will do it. In that case we can have an endpoint like /apps/{flowid} and
/apps/{clusterid}/{flowid} and flow run id can be an optional query param.
> Implement support for querying single app and all apps for a flow run
> -
>
> Key: YARN-3864
> URL: https://issues.apache.org/jira/browse/YARN-3864
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Blocker
> Attachments: YARN-3864-YARN-2928.01.patch
>
>
> This JIRA will handle support for querying all apps for a flow run in HBase
> reader implementation.
> And also REST API implementation for single app and multiple apps.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940537#comment-14940537 ] Vrushali C commented on YARN-3864: -- bq. In first one, cluster id will be taken from config and in the next one supplied by client. Do you mean something else ? Yes, I got that. I meant, lines 517 - 519 the query params like flow, flow run id and user id are redundant in the context of cluster+appid. We don't need these query params for the /app endpoint. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940489#comment-14940489 ] Vrushali C commented on YARN-3864: -- Thanks [~varun_saxena]! - For the /app endpoint, I think we need only the app id as the path param, none of the others are needed. For a given cluster, an application id will be unique. What do you think - for the /apps endpoint, do you think we can call it flowApps or appsflow or something so that it is clear that we are querying for all applications in this particular in this endpoint. As such, later we may want to add/enhance an endpoint for apps such that we can return all apps for a given user or all apps in a given queue or some other criteria. For this endpoint in this jira, I believe we are considering only the applications in a given flow run. - I think this endpoint should be also available without requiring the flow run id. For instance, if I want to see all the apps for a particular pig job (say default is last run), then I may not know the run id but I know only the pig job name. In that case, the reader should scan the applications table for that flow and return the latest run id's applications. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > Attachments: YARN-3864-YARN-2928.01.patch > > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938775#comment-14938775 ] Vrushali C commented on YARN-3864: -- Thank you! > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939136#comment-14939136 ] Li Lu commented on YARN-3864: - This issue becomes a major blocker for the web UI POC since we cannot list applications within one flowrun. Meanwhile, could anyone remind me where did we store this information in the HBase storage? As a big picture I assume there is an "applications" section in our flowrun endpoint's return value, which can be similar to the "flowruns" section in our /flows endpoint, but let's make sure we do have that information first (and hopefully do not need to set up coprocessors). > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939309#comment-14939309 ] Vrushali C commented on YARN-3864: -- cc [~sjlee0] bq. This issue becomes a major blocker for the web UI POC since we cannot list applications within one flowrun. [~gtCarrera] Perhaps it will be a good idea if we all can know the scope of the UI PoC. When jiras suddenly become blockers, it is hard to plan vs prioritize work. Also, are the main landing page and flow details page of the UI PoC done? Would love to know more about the UI PoC current status and planned work. If there are any jiras that already contain this info, please point me to those! thanks > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3864) Implement support for querying single app and all apps for a flow run
[ https://issues.apache.org/jira/browse/YARN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939312#comment-14939312 ] Li Lu commented on YARN-3864: - Sure, I'll send some recent update about the UI work soon. Sorry for suddenly raising this into a blocker. If there are any bandwidth problems I can take care of this work. > Implement support for querying single app and all apps for a flow run > - > > Key: YARN-3864 > URL: https://issues.apache.org/jira/browse/YARN-3864 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Blocker > > This JIRA will handle support for querying all apps for a flow run in HBase > reader implementation. > And also REST API implementation for single app and multiple apps. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
