[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131881#comment-14131881
]
Hadoop QA commented on YARN-2032:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668402/YARN-2032-091114.patch
against trunk revision 3122daa.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.TestHBaseTimelineStoreUtil
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4930//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4930//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-applicationhistoryservice.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4930//console
This message is automatically generated.
Implement a scalable, available TimelineStore using HBase
-
Key: YARN-2032
URL: https://issues.apache.org/jira/browse/YARN-2032
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Li Lu
Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch,
YARN-2032-branch2-2.patch
As discussed on YARN-1530, we should pursue implementing a scalable,
available Timeline store using HBase.
One goal is to reuse most of the code from the levelDB Based store -
YARN-1635.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132031#comment-14132031 ] stack commented on YARN-2032: - What you need of hbase lads? Our next release undoes our dependency on HTTPServer (the coming 1.0, a 0.99.0 developer release is imminent). If you want us to change our sync method call, np, just say; now would be a good time to do it before 1.0 goes out. We are also well-practiced poking around with reflection looking for whatever the method that does hdfs sync'ing is called (smile). Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Li Lu Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132062#comment-14132062 ] Zhijie Shen commented on YARN-2032: --- [~stack], it's great to hear. Once HBase 1.0 comes out, we can think of upgrading our dependency. So far, I only see that HTTPServer and FSDataOutputStream.sync are two points of binary incompatibility against Hadoop trunk. It doesn't break compilation, but at runtime of the test cases, MiniHBaseCluster fails to start due to these two incompatible parts (see HADOOP-10995 for more details). Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Li Lu Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132300#comment-14132300 ] stack commented on YARN-2032: - This is our issue for running on hadoop3. I was able to get over the httpserver issue and it is about time we moved off sync and on to hsync given how long sync has been deprecated. Am stuck now on how to migrate over Job, JobConf, etc. Will be back. Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Li Lu Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132307#comment-14132307 ] Li Lu commented on YARN-2032: - Hi [~stack], thank you for your information! I agree to [~zjshen] that assists from the HBase side would be very helpful to us. Thanks for keeping us posted! Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Li Lu Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130935#comment-14130935
]
Hadoop QA commented on YARN-2032:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668169/YARN-2032-091114.patch
against trunk revision c656d7d.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 24 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1265 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.mover.TestStorageMover
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4895//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/4895//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4895//console
This message is automatically generated.
Implement a scalable, available TimelineStore using HBase
-
Key: YARN-2032
URL: https://issues.apache.org/jira/browse/YARN-2032
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch,
YARN-2032-branch2-2.patch
As discussed on YARN-1530, we should pursue implementing a scalable,
available Timeline store using HBase.
One goal is to reuse most of the code from the levelDB Based store -
YARN-1635.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130972#comment-14130972 ] Li Lu commented on YARN-2032: - Oh sorry, it should be HDFS-6584. Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130971#comment-14130971 ] Li Lu commented on YARN-2032: - The HDFS unit test failures are very confusing here. TestStorageMover is even not in trunk. As pointed by [~jingzhao], the result of this run may come from HDFS-6564. Apparently there were something wrong during this test run. Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130981#comment-14130981
]
Hadoop QA commented on YARN-2032:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668169/YARN-2032-091114.patch
against trunk revision 17ffbe0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4905//console
This message is automatically generated.
Implement a scalable, available TimelineStore using HBase
-
Key: YARN-2032
URL: https://issues.apache.org/jira/browse/YARN-2032
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch,
YARN-2032-branch2-2.patch
As discussed on YARN-1530, we should pursue implementing a scalable,
available Timeline store using HBase.
One goal is to reuse most of the code from the levelDB Based store -
YARN-1635.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131048#comment-14131048 ] Zhijie Shen commented on YARN-2032: --- [~gtCarrera9], is the compilation failure also related to HDFS-6584? Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131082#comment-14131082 ] Li Lu commented on YARN-2032: - Hi [~zjshen], I don't think so. I'm looking into that. Compilation was fine on my local machine, when reapplying this patch. Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Li Lu Attachments: YARN-2032-091114.patch, YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094694#comment-14094694
]
Hadoop QA commented on YARN-2032:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12661260/YARN-2032-branch2-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:
org.apache.hadoop.yarn.server.timeline.TestHBaseTimelineStore
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4606//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4606//console
This message is automatically generated.
Implement a scalable, available TimelineStore using HBase
-
Key: YARN-2032
URL: https://issues.apache.org/jira/browse/YARN-2032
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2032-branch-2-1.patch, YARN-2032-branch2-2.patch
As discussed on YARN-1530, we should pursue implementing a scalable,
available Timeline store using HBase.
One goal is to reuse most of the code from the levelDB Based store -
YARN-1635.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032167#comment-14032167
]
Hadoop QA commented on YARN-2032:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644451/YARN-2032-branch-2-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3992//console
This message is automatically generated.
Implement a scalable, available TimelineStore using HBase
-
Key: YARN-2032
URL: https://issues.apache.org/jira/browse/YARN-2032
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2032-branch-2-1.patch
As discussed on YARN-1530, we should pursue implementing a scalable,
available Timeline store using HBase.
One goal is to reuse most of the code from the levelDB Based store -
YARN-1635.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032226#comment-14032226 ] Zhijie Shen commented on YARN-2032: --- [~mayank_bansal], thanks for the patch! Here're some quick comment after the first glance. 1. Why does this patch target branch-2 instead of trunk? 2. The packages of the newly added classes need to be updated after YARN-2107. 3. TTL configure should be put in YarnConfiguration. Another concern is that the data retention policy is different between Hbase and Leveldb. In Leveldb, we determine whether an entity is old enough according to TTL, and then delete it as well as its events. However, in HBase impl, it seems that deletion depends on each column family's TTL individually. In this case, it is possible that the entity is deleted, but its events (or part of them) are still there. 4. fromId and fromTs not implemented seems not be implemented yet. 5. Why do ENTITY_TABLE and INDEX_TABLE have the same schema? If I remember it correctly, we only index against the primary filters only. 6. Query parameters need to fully function, such as the secondary filters. Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal Attachments: YARN-2032-branch-2-1.patch As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993864#comment-13993864 ] Vinod Kumar Vavilapalli commented on YARN-2032: --- Sure, as I can see, you already took it over while the mailing list is down :) Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Mayank Bansal As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[
https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995381#comment-13995381
]
Ted Yu commented on YARN-2032:
--
{code}
+ version0.98.0-hadoop2/version
{code}
0.98.2 is the latest relase for 0.98
Since SingleColumnValueFilter is used, you need to have HBASE-10850 which is in
0.98.2
{code}
+ protected void serviceInit(Configuration conf) throws Exception {
+HBaseAdmin hbase = initHBase(conf);
{code}
Looks like the HBaseAdmin instance is not closed upon leaving serviceInit().
Implement a scalable, available TimelineStore using HBase
-
Key: YARN-2032
URL: https://issues.apache.org/jira/browse/YARN-2032
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Mayank Bansal
Attachments: YARN-2032-branch-2-1.patch
As discussed on YARN-1530, we should pursue implementing a scalable,
available Timeline store using HBase.
One goal is to reuse most of the code from the levelDB Based store -
YARN-1635.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2032) Implement a scalable, available TimelineStore using HBase
[ https://issues.apache.org/jira/browse/YARN-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993000#comment-13993000 ] Mayank Bansal commented on YARN-2032: - Taking it over, As I am already working on it Thanks, Mayank Implement a scalable, available TimelineStore using HBase - Key: YARN-2032 URL: https://issues.apache.org/jira/browse/YARN-2032 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli As discussed on YARN-1530, we should pursue implementing a scalable, available Timeline store using HBase. One goal is to reuse most of the code from the levelDB Based store - YARN-1635. -- This message was sent by Atlassian JIRA (v6.2#6252)
