[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659272#comment-13659272 ] Hudson commented on HBASE-5930: --- Integrated in hbase-0.95-on-hadoop2 #101 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/101/]) HBASE-5930 Addendum with tests. (Davaraj) (Revision 1483037) Result = FAILURE larsh : Files : * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94-added-addendum-with-tests.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658903#comment-13658903 ] Hudson commented on HBASE-5930: --- Integrated in HBase-0.94-security #141 (See [https://builds.apache.org/job/HBase-0.94-security/141/]) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Davaraj and LarsH) (Revision 1483022) Result = FAILURE larsh : Files : * /hbase/branches/0.94/security/src/test/resources/hbase-site.xml * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/resources/hbase-default.xml * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * /hbase/branches/0.94/src/test/resources/hbase-site.xml Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94-added-addendum-with-tests.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658966#comment-13658966 ] Hudson commented on HBASE-5930: --- Integrated in hbase-0.95 #196 (See [https://builds.apache.org/job/hbase-0.95/196/]) HBASE-5930 Addendum with tests. (Davaraj) (Revision 1483037) Result = FAILURE larsh : Files : * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94-added-addendum-with-tests.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13659222#comment-13659222 ] Hudson commented on HBASE-5930: --- Integrated in HBase-0.94 #984 (See [https://builds.apache.org/job/HBase-0.94/984/]) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Davaraj and LarsH) (Revision 1483022) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/security/src/test/resources/hbase-site.xml * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/resources/hbase-default.xml * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * /hbase/branches/0.94/src/test/resources/hbase-site.xml Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94-added-addendum-with-tests.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657849#comment-13657849 ] Lars Hofhansl commented on HBASE-5930: -- This is the one for 0.94 :) Will commit tomorrow unless I hear objections. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94-added-addendum-with-tests.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656248#comment-13656248 ] Lars Hofhansl commented on HBASE-5930: -- Thanks DD (you beat me to the patch :) ). Was more thinking about not starting the chore at at all when disabled. Then again, we do have per table (or CF) configs now, right? In that case this approach would be the right one. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656585#comment-13656585 ] Lars Hofhansl commented on HBASE-5930: -- Nice. +1 on addendum. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-2.txt, 5930-0.94-added-addendum.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-addendum-for-disabling.trunk.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-addendum-for-disabling.trunk.with-tests.txt, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13655139#comment-13655139 ] Lars Hofhansl commented on HBASE-5930: -- Added release notes. I also would like add an option to disable this feature (for example by setting base.regionserver.optionalcacheflushinterval to a value = 0). Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-added-addendum.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651133#comment-13651133 ] Lars Hofhansl commented on HBASE-5930: -- Cool. If there are no objections I will commit this to 0.94 as well. (Should this be disabled by default in 0.94? I can see this both ways.) Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-added-addendum.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651139#comment-13651139 ] Enis Soztutar commented on HBASE-5930: -- +1 for this going to 0.94. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-added-addendum.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13651140#comment-13651140 ] Jean-Daniel Cryans commented on HBASE-5930: --- Would also be good to have a release note here since it's a new feature coming along with new configs. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94-added-addendum.txt, 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646404#comment-13646404 ] Devaraj Das commented on HBASE-5930: Sorry for the delay in getting back with a review on the 0.94 patch. I will get to it tomorrow. I am fine with a new jira for the 0.94 patch or using this one itself... Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646397#comment-13646397 ] Lars Hofhansl commented on HBASE-5930: -- Should I close this one and open a 0.94 backport patch? Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642684#comment-13642684 ] Hudson commented on HBASE-5930: --- Integrated in hbase-0.95 #163 (See [https://builds.apache.org/job/hbase-0.95/163/]) HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475991) HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475874) Result = FAILURE ddas : Files : * /hbase/branches/0.95/hbase-server/src/test/resources/hbase-site.xml ddas : Files : * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642798#comment-13642798 ] Hudson commented on HBASE-5930: --- Integrated in hbase-0.95-on-hadoop2 #81 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/81/]) HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475991) HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475874) Result = FAILURE ddas : Files : * /hbase/branches/0.95/hbase-server/src/test/resources/hbase-site.xml ddas : Files : * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642833#comment-13642833 ] Hudson commented on HBASE-5930: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #511 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/511/]) HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475990) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Revision 1475970) HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475872) Result = FAILURE ddas : Files : * /hbase/trunk/hbase-server/src/test/resources/hbase-site.xml eclark : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java ddas : Files : * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642165#comment-13642165 ] Lars Hofhansl commented on HBASE-5930: -- And we should have had a test too. Can you think something up DD? Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Priority: Minor Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642182#comment-13642182 ] Devaraj Das commented on HBASE-5930: Okay.. Will think of a test. Will look at the 0.94 patch. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Priority: Minor Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642231#comment-13642231 ] Devaraj Das commented on HBASE-5930: Thanks, Elliott for catching this. I was testing releaseaudit - randomly chose a java file and removed the header with an intent to make releaseaudit fail. Forgot to revert the change before my commit. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Priority: Minor Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642248#comment-13642248 ] Elliott Clark commented on HBASE-5930: -- Comitted the addendum and rat is clean again. Thanks [~devaraj] Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Priority: Minor Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[
https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642279#comment-13642279
]
Enis Soztutar commented on HBASE-5930:
--
This is causing the test at patch HBASE-2231 to fail. The reason is that we had
a residual configuration in hbase-site.xml from earlier:
{code}
namehbase.regionserver.optionalcacheflushinterval/name
value1000/value
{code}
I also was not able to understand when we put a delayed flush request to the
queue, and later another request with no delay comes, it seems we will just
ignore the latter one. Shouldn't we honor the no-delay one before waiting on
the delay?
Limits the amount of time an edit can live in the memstore.
---
Key: HBASE-5930
URL: https://issues.apache.org/jira/browse/HBASE-5930
Project: HBase
Issue Type: Improvement
Reporter: Lars Hofhansl
Assignee: Devaraj Das
Priority: Minor
Fix For: 0.98.0, 0.94.8, 0.95.1
Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch,
5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt,
5930-wip.patch, HBASE-5930-ADD-0.patch
A colleague of mine ran into an interesting issue.
He inserted some data with the WAL disabled, which happened to fit in the
aggregate Memstores memory.
Two weeks later he a had problem with the HDFS cluster, which caused the
region servers to abort. He found that his data was lost. Looking at the log
we found that the Memstores were not flushed at all during these two weeks.
Should we have an option to flush memstores periodically. There are obvious
downsides to this, like many small storefiles, etc.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642294#comment-13642294 ] Lars Hofhansl commented on HBASE-5930: -- 2nd addendum is good. Will add that to the 0.94 patch as well. Are we good with the patch as is, or should we revert and work through the details? Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642298#comment-13642298 ] Devaraj Das commented on HBASE-5930: bq. I also was not able to understand when we put a delayed flush request to the queue, and later another request with no delay comes, it seems we will just ignore the latter one. Shouldn't we honor the no-delay one before waiting on the delay? Makes sense to honor the no-delay one. I'll open an issue to take this into account. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642299#comment-13642299 ] Enis Soztutar commented on HBASE-5930: -- bq. Are we good with the patch as is, or should we revert and work through the details? Either that or new issue. Whichever is the cleanest. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642308#comment-13642308 ] Devaraj Das commented on HBASE-5930: Lars, I think the addendums fixes the issues to do with rat and the test, but I'd like to open a separate issue on the queuing of flush requests. I'll get a patch by today (worst case tomorrow) and will do a 0.94 patch for that as well.. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642315#comment-13642315 ] Lars Hofhansl commented on HBASE-5930: -- Sounds good. We have time until 0.94.8, so we're good, unless [~stack] want's to release 0.95.1 tomorrow. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642583#comment-13642583 ] Hudson commented on HBASE-5930: --- Integrated in HBase-TRUNK #4080 (See [https://builds.apache.org/job/HBase-TRUNK/4080/]) HBASE-5930. Removed a configuration that was causing unnecessary flushes in tests. (Revision 1475990) HBASE-5930 Limits the amount of time an edit can live in the memstore. (Revision 1475970) HBASE-5930. Limits the amount of time an edit can live in the memstore. (Revision 1475872) Result = FAILURE ddas : Files : * /hbase/trunk/hbase-server/src/test/resources/hbase-site.xml eclark : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java ddas : Files : * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedWriter.java Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642615#comment-13642615 ] Devaraj Das commented on HBASE-5930: bq. I'd like to open a separate issue on the queuing of flush requests After some deliberations and code browsing, it seems like we don't need to do this. The reasons being: 1. We already have the emergency flush procedure in place (MemStoreFlusher.flushRegion with an emergencyFlush boolean argument exists that is called when there is an urgent need to flush). 2. In the normal steady state of affairs, entries from the queue are processed asynchronously anyway. Flushes are done by a thread and there is no guarantee when the thread picks a flush request up. By adding a delay, this situation is not worsened much, and especially in combination with (1) above. 3. To address the issue raised, I was thinking that when a new flush request comes without a delay, we could remove an existing delayed-flush-request entry that was queued before, and queue a new one without a delay. However, there is a complication here - there is logic that queues up entries with delayed flushes in _MemStoreFlusher.flushRegion(final FlushRegionEntry)_. That would be impacted if I did what I said. All in all, this doesn't seem like an issue.. [~enis] also ack'ed this in our offline discussions. Limits the amount of time an edit can live in the memstore. --- Key: HBASE-5930 URL: https://issues.apache.org/jira/browse/HBASE-5930 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Devaraj Das Fix For: 0.98.0, 0.94.8, 0.95.1 Attachments: 5930-0.94.txt, 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, hbase-5930-addendum2.patch, hbase-5930-test-execution.log A colleague of mine ran into an interesting issue. He inserted some data with the WAL disabled, which happened to fit in the aggregate Memstores memory. Two weeks later he a had problem with the HDFS cluster, which caused the region servers to abort. He found that his data was lost. Looking at the log we found that the Memstores were not flushed at all during these two weeks. Should we have an option to flush memstores periodically. There are obvious downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
