[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369692#comment-15369692
]
Sangjin Lee commented on YARN-2928:
---
I think we're good.
I think the test failure with {{TestMRTimelineEventHandling}} was caused by
concurrent builds. The test uses a fixed timeline service data location which
doesn't bode well. Furthermore, it uses "/" as the separator which needs to be
fixed too. {{TestDistributedShell}} has the same problem. I'll file a JIRA to
fix those tests after merging this.
I'll merge it shortly.
> YARN Timeline Service: Next generation
> --
>
> Key: YARN-2928
> URL: https://issues.apache.org/jira/browse/YARN-2928
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf,
> ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN
> Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning
> - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf,
> YARN-2928.01.patch, YARN-2928.02.patch, YARN-2928.03.patch,
> timeline_service_v2_next_milestones.pdf
>
>
> We have the application timeline server implemented in yarn per YARN-1530 and
> YARN-321. Although it is a great feature, we have recognized several critical
> issues and features that need to be addressed.
> This JIRA proposes the design and implementation changes to address those.
> This is phase 1 of this effort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369623#comment-15369623
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 0s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 35s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
30s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 43s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 0s {color}
| {color:red} root generated 2 new + 708 unchanged - 0 fixed = 710 total (was
708) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 30s
{color} | {color:red} root: The patch generated 103 new + 3267 unchanged - 128
fixed = 3370 total (was 3395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 17s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 6621
unchanged - 1 fixed = 6621 total (was 6622) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} had
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369415#comment-15369415 ] Sangjin Lee commented on YARN-2928: --- The latest run still produces a number of known unit test failures. TestMRTimelineEventHandling also failed, and I suspect this might have something to do with multiple builds that ran at the same time. I just kicked off another jenkins run to see if we can get a cleaner run. At any rate, I think we should go ahead and merge it soon unless there is a clear indication there is an issue. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, YARN-2928.02.patch, YARN-2928.03.patch, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369413#comment-15369413
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 0s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
55s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 29s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 54s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m 20s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 20s {color}
| {color:red} root generated 2 new + 708 unchanged - 0 fixed = 710 total (was
708) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 30s
{color} | {color:red} root: The patch generated 103 new + 3267 unchanged - 128
fixed = 3370 total (was 3395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 20s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 14m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 6621
unchanged - 1 fixed = 6621 total (was 6622) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{col
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369396#comment-15369396
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 0s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
54s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 41s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 41s {color}
| {color:red} root generated 2 new + 708 unchanged - 0 fixed = 710 total (was
708) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 33s
{color} | {color:red} root: The patch generated 103 new + 3267 unchanged - 128
fixed = 3370 total (was 3395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 18s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 14m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s
{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 6621
unchanged - 1 fixed = 6621 total (was 6622) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369287#comment-15369287
]
Sangjin Lee commented on YARN-2928:
---
More analysis:
(1) mvninstall failure (hadoop-mapreduce-client-app)
I cannot reproduce this locally. It looks as though the
hadoop-mapreduce-client-app build picked up an older version of
hadoop-mapreduce-client-core that does not have this new code. It appears
somehow jenkins is picking up wrong jars.
(2) javac errors
Both are from files and code that we did not touch.
(3) javadoc
Fixed.
(4) checkstyle
Fixed 3 more. The remaining checkstyle violations are not related to code
changes by us, not practical to fix as part of this (e.g. adding javadoc to a
large number of existing classes), or else. I think we have fixed it as much as
possible without burdening this JIRA more with a lot of unrelated changes.
(5) unit test failures
- TestGangliaMetrics: known issue (HADOOP-12588)
- TestZKFailoverController: appears that it encountered "connection refused"
errors (environment?)
- TestYarnClient: known issue (YARN-4202, YARN-4954)
- TestLogsCLI: known issue (YARN-5313)
- TestContainerManagerSecurity: known issue (YARN-4342)
- TestMiniYarnClusterNodeUtilization: known issue (YARN-4453)
That leaves
{{TestMRTimelineEventHandling#testMRNewTimelineServiceEventHandling}}. This is
a new test we added. I tried to reproduce this locally, but it is not
reproducible on mine (tried a number of times). I'm not exactly sure how this
could fail other than it failed to create the directory. The desired directory
is correct. I'm going to see if this failure persists with the next run.
> YARN Timeline Service: Next generation
> --
>
> Key: YARN-2928
> URL: https://issues.apache.org/jira/browse/YARN-2928
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf,
> ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN
> Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning
> - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf,
> YARN-2928.01.patch, YARN-2928.02.patch, YARN-2928.03.patch,
> timeline_service_v2_next_milestones.pdf
>
>
> We have the application timeline server implemented in yarn per YARN-1530 and
> YARN-321. Although it is a great feature, we have recognized several critical
> issues and features that need to be addressed.
> This JIRA proposes the design and implementation changes to address those.
> This is phase 1 of this effort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369258#comment-15369258 ] Sangjin Lee commented on YARN-2928: --- The latest one run on patch v.2 seems more promising, although that one still seems strange. Will look into that one. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, YARN-2928.02.patch, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369164#comment-15369164
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 1s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
0s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 16m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
12s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m 43s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 54s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 31s
{color} | {color:red} root generated 2 new + 709 unchanged - 0 fixed = 711
total (was 709) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 51s
{color} | {color:red} root: The patch generated 108 new + 3268 unchanged - 127
fixed = 3376 total (was 3395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 7m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
16s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 20s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 15m
33s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 36s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 6621
unchanged - 1 fixed = 6622 total (was 6622) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client
generated 1 new + 155 unchanged - 1 fixed = 156 total (was 156) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369159#comment-15369159 ] Varun Saxena commented on YARN-2928: The test result is weird. javadoc seems to be double of previous number of errors. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, YARN-2928.02.patch, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369154#comment-15369154
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 0s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 54s
{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 57s
{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
32s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-yarn-api in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 42s
{color} | {color:red} hadoop-yarn-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-yarn-server-nodemanager in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 54s
{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 35s
{color} | {color:red} hadoop-mapreduce-client-app in trunk failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
50s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 11s
{color} | {color:red} branch/hadoop-common-project/hadoop-common no findbugs
output file (hadoop-common-project/hadoop-common/target/findbugsXml.xml)
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 29s
{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/findbugsXml.xml)
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 40s
{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-mapreduce-client-app in trunk failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m 4s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
46s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m 24s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 5m 24s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 24s {color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 11s
{color} | {color:red} root: The patch generated 6771 new + 3395 unchanged - 0
fixed = 10166 total (was 3395) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m 2s
{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 7m
5s {color} | {color:green} the patch passed {color} |
| {color:
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15369054#comment-15369054
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 1s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
42s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 2s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 49s
{color} | {color:red} hadoop-yarn-server-timelineservice-hbase-tests in the
patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 24s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 24s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 24s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
50s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 2m 32s
{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 1m 31s
{color} | {color:red} hadoop-yarn-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 39s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 5m 28s
{color} | {color:red} hadoop-yarn-server-timelineservice-hbase-tests in the
patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 8m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
13s {color} | {color:green} The patch generated 0 new + 74 unchanged - 1 fixed
= 74 total (was 75) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 19s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368976#comment-15368976 ] Sangjin Lee commented on YARN-2928: --- I'm fixing javac warnings, javadoc warnings, and checkstyle violations. I'll kick off another build when I'm done with that, which should be shortly. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368972#comment-15368972 ] Varun Saxena commented on YARN-2928: Should we invoke the build again ? Tests like TestQueuingContainerManager should not really fail on trunk. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368866#comment-15368866 ] Sangjin Lee commented on YARN-2928: --- There are also strange compilation failures that are reported as mvninstall failures and some "unit test failures". I would have thought with dockers you would not have interference from concurrently running builds? I'm analyzing javac warnings, unit test failures, and javadoc errors. Findbugs is clean. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368851#comment-15368851
]
Joep Rottinghuis commented on YARN-2928:
License part is bogus. Files reported w/o license are:
{code}
!?
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/output/_SUCCESS
!?
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/output/part-0
!?
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/input/part-0
!?
/testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/input/part-1
{code}
Jenkins URLs don't seem to resolve, so I cannot check the other items right now.
> YARN Timeline Service: Next generation
> --
>
> Key: YARN-2928
> URL: https://issues.apache.org/jira/browse/YARN-2928
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf,
> ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN
> Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning
> - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf,
> YARN-2928.01.patch, timeline_service_v2_next_milestones.pdf
>
>
> We have the application timeline server implemented in yarn per YARN-1530 and
> YARN-321. Although it is a great feature, we have recognized several critical
> issues and features that need to be addressed.
> This JIRA proposes the design and implementation changes to address those.
> This is phase 1 of this effort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368825#comment-15368825
]
Hadoop QA commented on YARN-2928:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 1s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 84 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 52s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 46s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
36s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 5s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 53s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 53s {color}
| {color:red} root generated 13 new + 708 unchanged - 0 fixed = 721 total (was
708) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 40s
{color} | {color:red} root: The patch generated 493 new + 3285 unchanged - 114
fixed = 3778 total (was 3399) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 6m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
12s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 18s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
5s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 35s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 19 new + 6622
unchanged - 0 fixed = 6641 total (was 6622) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated
2 new + 156 unchanged - 0 fixed = 158 total (was 156) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common
generated 7 new + 457
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346892#comment-15346892 ] Joep Rottinghuis commented on YARN-2928: Note the https://issues.apache.org/jira/secure/attachment/12811409/timeline_service_v2_next_milestones.pdf attachment in this jira with the outcome of initial discussion what the next milestones could look like. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340115#comment-15340115 ] Joep Rottinghuis commented on YARN-2928: email thread on yarn-dev "[DISCUSS] merging YARN-2928 (Timeline Service v.2) to trunk": http://markmail.org/thread/bnpwpjhkbs6wsn7z > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262963#comment-15262963 ] Sangjin Lee commented on YARN-2928: --- For those who are following this ticket, we are nearing the first merge to trunk milestone: http://markmail.org/message/27uk4iwqvihs335e Please check out the WIP documentation on YARN-3150. Thanks! > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[
https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14579305#comment-14579305
]
James Taylor commented on YARN-2928:
bq. I'm wondering that, when adding the dynamic columns into a view, do I still
need to explicitly claim those dynamic columns (I assume yes but would like to
double check)?
Yes - instead of building up the SQL string with the dynamic column name, you'd
execute the following:
{code}
ALTER VIEW my_view ADD IF NOT EXISTS
{code}
Then, when you query, you no longer need to use dynamic columns, but instead
can select all of them:
{code}
SELECT * FROM my_view;
{code}
As far as APIs, we'll be happy to give you the ones you need, [~gtCarrera9].
The higher up in the stack you hook in, the easier it'll be. For reading from
HBase, you can always fallback to creating a read-only view over your HBase
table. We should work through a couple of examples, though, as if you're
storing multiple pieces of information in your row key, we'll want to make sure
it's compatible with the way Phoenix expects it to be structured.
For writing to HBase, I think it'd be good to re-test the Phoenix write path
once I finish PHOENIX-2028 (with your KeyPrefixRegionSplitPolicy installed on
the table). If it's still not fast enough, then there are a number of options:
- Use PDataType.toBytes() to get the KeyValue value bytes
- Use PhoenixRuntime APIs to create the row key if they encapsulate multiple
pieces of information
- Create new APIs as needed
> YARN Timeline Service: Next generation
> --
>
> Key: YARN-2928
> URL: https://issues.apache.org/jira/browse/YARN-2928
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal
> v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx,
> TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf
>
>
> We have the application timeline server implemented in yarn per YARN-1530 and
> YARN-321. Although it is a great feature, we have recognized several critical
> issues and features that need to be addressed.
> This JIRA proposes the design and implementation changes to address those.
> This is phase 1 of this effort.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577891#comment-14577891 ] Li Lu commented on YARN-2928: - Oh one more thing [~jamestaylor], are there any plans to make the PDataTypes APIs to be public and/or stable, or, at least make it limited public to YARN? I believe that will be very helpful for us. Thanks! > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577885#comment-14577885 ] Li Lu commented on YARN-2928: - Hi [~jamestaylor], thank you very much for your great help! Some clarifications on my questions... bq. For your configuration/metric key-value pair, how are they named? Do you know the possible set of key values in advance? Or are they known more-or-less on-the-fly? For our use case they're completely on-the-fly. For each timeline entity, we plan to store each of its configuration/metric in one dynamic column. It is possible that different entities may have completely different configs/metrics. For example, a mapreduce job may have a completely different set of configs to a tez job. Therefore, we need to generate all columns for configs/metrics dynamically. I'm wondering that, when adding the dynamic columns into a view, do I still need to explicitly claim those dynamic columns (I assume yes but would like to double check)? bq. Are you thinking to have a secondary table that's a rollup aggregation of more raw data? Is that required, or is it more of a convenience for the user? If the raw data is Phoenix-queryable, then I think you have a lot of options. Can you point me to some more info on your design? Yes, we are considering to have multiple levels of aggregation tables, each with a different granularity. For example, now we're planning to do the first level (application level) aggregation from an HBase table to a Phoenix table. Then, we can aggregate flow level information based on our application level aggregation (since each application belongs to and only belongs to one flow). In this way, we can temporarily get rid of the write throughput limitation of Phoenix, but still support SQL queries on aggregated data. If the Phoenix PDataTypes are stable, then is it possible for us to do the following two things? # Use HBase API and PDataTypes to read a Phoenix table, and read dynamic columns iteratively. # Use HBase API and PDataTypes to write a Phoenix table, and write dynamic columns iteratively. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575490#comment-14575490 ] James Taylor commented on YARN-2928: Happy to help, [~gtCarrera9]. Thanks for the information. bq. If I understand this correctly, in this case, Phoenix will inherit pre-split settings from HBase? Will this alter the existing HBase table, including its schema and/or data inside? In general, if one runs CREATE TABLE IF NOT EXISTS or simply CREATE TABLE commands over a pre-split existing HBase table, will Phoenix simply accept the existing table as-is? If you create a table in Phoenix and the table already exists in HBase, Phoenix will accept the existing table as-is, adding any metadata it needs (i.e. it's coprocessors). If the table has existing data, then Phoenix will add an empty KeyValue to each row in the first column family referenced in the create table statement (or the default column family if there are no column families referenced). Phoenix needs this empty key value for a variety of reasons. The onus is on the user to ensure that the types in the create table statement match the actual means in which the data was serialized. For your configuration/metric key-value pair, how are they named? Do you know the possible set of key values in advance? Or are they known more-or-less on-the-fly? One way you could model this with views is to just dynamically add the column to the view when you need to. Adding a column to a view is a very light weight operation - corresponding to a few Puts to the SYSTEM.CATALOG table. Then you'd have a way of looping through all metrics for a given view using the metadata APIs. Think of a view as a set of explicitly named dynamic columns. You'd still need to generate the SQL statement, though. bq. One potential solution is to use HBase coprocessors to aggregate application data from the HBase storage, and then store them in a Phoenix aggregation table. I'm not following. Are you thinking to have a secondary table that's a rollup aggregation of more raw data? Is that required, or is it more of a convenience for the user? If the raw data is Phoenix-queryable, then I think you have a lot of options. Can you point me to some more info on your design? The stable APIs for Phoenix are the ones we expose through our public APIs: JDBC and our various integration modules (i.e. MapReduce, Pig, etc.). I'd say that our serialization format produced by PDataType is stable (it needs to be for us to meet our b/w compat guarantees) and the PDataType APIs are more stable than others. Also, we're looking to integrate with Apache Calcite, so we may have some other APIs that could be hooked into as well down the road. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575358#comment-14575358 ] Li Lu commented on YARN-2928: - Hi [~jamestaylor] Thank you very much for your suggestions and PHOENIX-2028! I wrote the experimental Phoenix writer code and currently have some follow up questions w.r.t your comments. bq. The easiest is probably to create the HBase table the same way (through code or using the HBase shell) with the KeyPrefixRegionSplitPolicy specified at create time. Then, in Phoenix you can issue a CREATE TABLE statement against the existing HBase table and it'll just map to it. Then you'll have your split policy for your benchmark in both write paths. If I understand this correctly, in this case, Phoenix will inherit pre-split settings from HBase? Will this alter the existing HBase table, including its schema and/or data inside? In general, if one runs CREATE TABLE IF NOT EXISTS or simply CREATE TABLE commands over a pre-split existing HBase table, will Phoenix simply accept the existing table as-is? bq. An alternative to dynamic columns is to define views over your Phoenix table (http://phoenix.apache.org/views.html). I once looked at views but I'm not sure if that fits our write path use case well. Let me briefly talk about our use case in YARN first. In general, we would like to dynamically store the configuration and metrics for each YARN timeline entity in a Phoenix database, such that our timeline reader apps or users can use SQL to query historical data. Phoenix view may make a perfect solution for the reader use cases. However, we are hitting problems on the writer side. We store each configuration/metric key-value pair in a dynamic column. This causes us two main troubles. First, we need to use a dynamically generated SQL statement to write to the Phoenix table which is cumbersome and error-prone. Second, when performing aggregations, we need to aggregate on all available metrics for an application (or a user, flow), but we cannot simply iterate on those dynamic columns because there is no such API. I'm not sure how to resolve these two problems via Phoenix view, or via existing Phoenix APIs. Actually, I suspect that if it's possible to fall back to the HBase-style APIs, our writer path would be much simpler. bq. If you do end up going with a direct HBase write path, I'd encourage you to use the Phoenix serialization format (through PDataType and derived classes) to ensure you can do adhoc querying on the data. We're currently looking into this method in the aggregation part. We're doing our best to support SQL on the aggregated data by using Phoenix. One potential solution is to use HBase coprocessors to aggregate application data from the HBase storage, and then store them in a Phoenix aggregation table. However, if we want to keep aggregating on the Phoenix table, can we also write a HBase coprocessor that read the Phoenix PDataTypes, and aggregate them into other Phoenix tables? If it's possible, are there any stable (or "safe") APIs for PDataTypes? A slightly more generalized question here is, is SQL the _only_ API for Phoenix, or there may be more? I ask this question because from a YARN timeline service perspective, Phoenix is a nice tool through which we can easily add SQL support to our final users, but we may not necessarily use SQL to program it all the time? Thank you very much for your comments and help from the Phoenix side. Our current Phoenix writer is more of an experimental version, but we really hope to have something for our aggregators and readers in near future. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575059#comment-14575059 ] Vrushali C commented on YARN-2928: -- Hi [~jamestaylor] Thank you for taking the time to look through the write up and for filing PHOENIX-2028. In the context of pre-splits, yes, we wanted to have both writers write to tables that were pre-split with the same presplit strategy. However, I believe the folks working on the Phoenix writer mentioned that the only way to achieve in Phoenix that was to use SPLIT ON substatement, which required that approach to rewrite the HBase presplitting strategy. Perhaps [~gtCarrera9] might be able to speak to this better. bq. I'd encourage you to use the Phoenix serialization format (through PDataType and derived classes) to ensure you can do adhoc querying on the data Okay, thanks, I will check that out. We are working on a whole set of enhancements for the base writer as well and I will look at this. bq. The most important aspect is how your row key is written and the separators you use if you're storing multiple values in the row key. You’ve hit the nail on the head. We do have multiple values with different datatypes in row key as well as in column names with and without prefixes, so we have different datatypes and bunch of separators. [~jrottinghuis] has been addressing these points in YARN-3706 , for e.g. dealing with storing and parsing byte representations of separators. The timeline service schema has more tables and we are considering storing aggregated values in these Phoenix based tables (current thinking is to have them populated via co-processors watching the basic entity table). Thanks for suggesting defining views on Phoenix tables, I will look up more details on that. Thanks once again, Vrushali > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14574991#comment-14574991 ] James Taylor commented on YARN-2928: Nice writeup, [~vrushalic]. For your benchmarks, if you're pre-splitting for the HBase direct write path but not for the Phoenix write path, you're not really comparing apples-to-apples. There are a number of ways you can install your KeyPrefixRegionSplitPolicy in Phoenix. The easiest is probably to create the HBase table the same way (through code or using the HBase shell) with the KeyPrefixRegionSplitPolicy specified at create time. Then, in Phoenix you can issue a CREATE TABLE statement against the existing HBase table and it'll just map to it. Then you'll have your split policy for your benchmark in both write paths. An alternative to dynamic columns is to define views over your Phoenix table (http://phoenix.apache.org/views.html). In each view, you could specify the set of columns it contains. Then you can use the regular JDBC metadata APIs to get the set of columns that define your view: http://docs.oracle.com/javase/7/docs/api/java/sql/DatabaseMetaData.html#getColumns%28java.lang.String,%20java.lang.String,%20java.lang.String,%20java.lang.String%29 Another interesting angle with views (not sure if this is relevant for your use case or not), but they're capable of being multi-tenant where the definition of the "tenant" is up to you (maybe it would map to a User?). In this case, each tenant can define their own derived view and add columns specific to their usage. You can even create secondary indexes over a view. This is the way Phoenix surfaces NoSQL in the SQL world. More here: http://phoenix.apache.org/multi-tenancy.html There is room for improvement in the Phoenix write path, though. I've filed PHOENIX-2028 and plan to work on that shortly. If you do end up going with a direct HBase write path, I'd encourage you to use the Phoenix serialization format (through PDataType and derived classes) to ensure you can do adhoc querying on the data. The most important aspect is how your row key is written and the separators you use if you're storing multiple values in the row key. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571765#comment-14571765 ] Li Lu commented on YARN-2928: - Thanks [~sjlee0], [~jrottinghuis], and [~vrushalic] for hosting the benchmark session. This is very helpful! > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571660#comment-14571660 ] Sangjin Lee commented on YARN-2928: --- Thanks [~vrushalic] for the summary! > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx, > TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2928) YARN Timeline Service: Next generation
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14511284#comment-14511284 ] Sangjin Lee commented on YARN-2928: --- We should make sure the timeline service v.2 does the right thing in this regard. > YARN Timeline Service: Next generation > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal > v1.pdf, Timeline Service Next Gen - Planning - ppt.pptx > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
