[jira] [Updated] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[ https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-7942: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.95. Make use of the plumbing in HBASE-7932 to provide location hints when region files are created -- Key: HBASE-7942 URL: https://issues.apache.org/jira/browse/HBASE-7942 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.1 Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch, 7942-trunk-1.5.txt, 7942-trunk-1.6.txt This functionality is blocked by the fixes in HDFS. Reflection is going to be used to figure out whether the underlying Hadoop version has the new API. Things will behave the same way if the underlying HDFS version doesn't have the new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8504) HTable.getRegionsInRange() should provide a non-cached API
[ https://issues.apache.org/jira/browse/HBASE-8504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664916#comment-13664916 ] Hudson commented on HBASE-8504: --- Integrated in HBase-0.94-security #151 (See [https://builds.apache.org/job/HBase-0.94-security/151/]) HBASE-8504 HTable.getRegionsInRange() should provide a non-cached API (Revision 1485543) Result = FAILURE mbertozzi : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java HTable.getRegionsInRange() should provide a non-cached API -- Key: HBASE-8504 URL: https://issues.apache.org/jira/browse/HBASE-8504 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 0.94.7 Reporter: Alan Choi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.9 Attachments: HBASE-8504-v0.patch, HBASE-8504-v0.patch getRegionsInRange() calls getRegionLocation() without reloading it. It will return wrong result if the cache is outdated due to region split. If the cost of always reloading isn't significant, we should consider doing that by default. Otherwise, let's have an API for getRegionsInRange() that forces a reload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8555) FilterList correctness may be affected by random ordering of sub-filter(list)
[ https://issues.apache.org/jira/browse/HBASE-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664917#comment-13664917 ] Hudson commented on HBASE-8555: --- Integrated in HBase-0.94-security #151 (See [https://builds.apache.org/job/HBase-0.94-security/151/]) HBASE-8555 FilterList correctness may be affected by random ordering of sub-filter(list) (Liang Xie) (Revision 1485492) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java FilterList correctness may be affected by random ordering of sub-filter(list) - Key: HBASE-8555 URL: https://issues.apache.org/jira/browse/HBASE-8555 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.3 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Fix For: 0.98.0, 0.95.1, 0.94.9 Attachments: 8555-trunk-v1.txt, HBASE-8555-0.94.txt, HBASE-8555-0.94-v2.txt, HBASE-8555-0.94-v3.txt say, ther're 10 rows, column value is i%2: row0 0 row1 1 row2 0 row3 1 row4 0 row5 1 row6 0 row7 1 row8 0 row9 1 1: filter : row filter row4 === row5 row6 row7 row8 row9 2: subFilterList: row filter = row4 column==0=== row0 row2 row4 3.1 filterlist[expected] filter || subFilterList === row0 row2 row4 row5 row6 row7 row8 row9 3.2 filterlist[BUGON!] subFilterList || filter === row0 row1 row2 row3 row4 row5 row6 row7 row8 row9 (Please refer to the new testNestedFilterListWithSCVF case) It was found when i managed to transform the following SQL into HBase scan statement: select xxx from xxx where (pk = xxx and column1 = xxx) or pk xxx My finding is that we had an assumption for filter methods call sequence: e.g. filterRowKey() should be called before filterKeyValue(). and the orignial filterList.filterRowKey impl broke it due to fast short-circuit returning. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8564) TestMetricsRegionServer depends on test order
[ https://issues.apache.org/jira/browse/HBASE-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664930#comment-13664930 ] Hudson commented on HBASE-8564: --- Integrated in HBase-TRUNK #4138 (See [https://builds.apache.org/job/HBase-TRUNK/4138/]) HBASE-8564 ADDENDUM TestMetricsRegionServer depends on test order (Revision 1485525) Result = SUCCESS eclark : Files : * /hbase/trunk/hbase-it/pom.xml TestMetricsRegionServer depends on test order - Key: HBASE-8564 URL: https://issues.apache.org/jira/browse/HBASE-8564 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.0 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8564-0.patch, HBASE-8564-1.patch, HBASE-8564-ADD-0.patch, HBASE-8564-FIX-0.patch, HBASE-8564-FIX-1.patch TestMetricsRegionServer has failed a few times on trunk. It's passing depends upon test ordering so will fail sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8504) HTable.getRegionsInRange() should provide a non-cached API
[ https://issues.apache.org/jira/browse/HBASE-8504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664931#comment-13664931 ] Hudson commented on HBASE-8504: --- Integrated in HBase-TRUNK #4138 (See [https://builds.apache.org/job/HBase-TRUNK/4138/]) HBASE-8504 HTable.getRegionsInRange() should provide a non-cached API (Revision 1485540) Result = SUCCESS mbertozzi : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java HTable.getRegionsInRange() should provide a non-cached API -- Key: HBASE-8504 URL: https://issues.apache.org/jira/browse/HBASE-8504 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 0.94.7 Reporter: Alan Choi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.9 Attachments: HBASE-8504-v0.patch, HBASE-8504-v0.patch getRegionsInRange() calls getRegionLocation() without reloading it. It will return wrong result if the cache is outdated due to region split. If the cost of always reloading isn't significant, we should consider doing that by default. Otherwise, let's have an API for getRegionsInRange() that forces a reload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server
[ https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664932#comment-13664932 ] Hudson commented on HBASE-7649: --- Integrated in HBase-TRUNK #4138 (See [https://builds.apache.org/job/HBase-TRUNK/4138/]) HBASE-7649 ADDENDUM client retry timeout doesn't need to do x2 fallback when going to different server (Revision 1485513) Result = SUCCESS sershe : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java client retry timeout doesn't need to do x2 fallback when going to different server -- Key: HBASE-7649 URL: https://issues.apache.org/jira/browse/HBASE-7649 Project: HBase Issue Type: Improvement Components: Client, MTTR Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.95.1 Attachments: HBASE-7649-ADDENDUM-v0.patch, HBASE-7649-v0.patch, HBASE-7649-v1.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, HBASE-7649-v6.patch, HBASE-7649-v7.patch, HBASE-7649-v8.1.patch, HBASE-7649-v8.patch See HBASE-7520. When we go to server A, get a bunch of failures, then finally learn the region is on B it doesn't make sense to wait for 30 seconds before going to B. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8574) Add how to rename a table in the docbook
[
https://issues.apache.org/jira/browse/HBASE-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664933#comment-13664933
]
Hudson commented on HBASE-8574:
---
Integrated in HBase-TRUNK #4138 (See
[https://builds.apache.org/job/HBase-TRUNK/4138/])
HBASE-8574 Add how to rename a table in th docbook (addendum) (Revision
1485515)
Result = SUCCESS
mbertozzi :
Files :
* /hbase/trunk/src/main/docbkx/ops_mgt.xml
Add how to rename a table in the docbook
-
Key: HBASE-8574
URL: https://issues.apache.org/jira/browse/HBASE-8574
Project: HBase
Issue Type: Task
Components: documentation
Affects Versions: 0.94.7, 0.95.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.98.0, 0.94.8, 0.95.1
Add a section how to rename a table in the doc book.
The current easy solution without adding extra code in 94/95 is to use
snapshots
{code}
hbase shell disable 'tableName'
hbase shell snapshot 'tableName', 'tableSnapshot'
hbase shell clone 'tableSnapshot', 'newTableName'
hbase shell delete_snapshot 'tableSnapshot'
{code}
{code}
void rename(HBaseAdmin admin, String oldTableName, String newTableName) {
String snapshotName = randomName();
admin.snapshot(snapshotName, oldTableName);
admin.cloneSnapshot(snapshotName, newTableName);
admin.deleteSnapshot(snapshotName);
admin.deleteTable(oldTableName)
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664934#comment-13664934 ] Hudson commented on HBASE-8450: --- Integrated in HBase-TRUNK #4138 (See [https://builds.apache.org/job/HBase-TRUNK/4138/]) HBASE-8450 Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. (Revision 1485561) Result = SUCCESS stack : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8564) TestMetricsRegionServer depends on test order
[ https://issues.apache.org/jira/browse/HBASE-8564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664963#comment-13664963 ] Hudson commented on HBASE-8564: --- Integrated in hbase-0.95 #210 (See [https://builds.apache.org/job/hbase-0.95/210/]) HBASE-8564 ADDENDUM TestMetricsRegionServer depends on test order (Revision 1485523) Result = SUCCESS eclark : Files : * /hbase/branches/0.95/hbase-it/pom.xml TestMetricsRegionServer depends on test order - Key: HBASE-8564 URL: https://issues.apache.org/jira/browse/HBASE-8564 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.95.0 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Critical Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8564-0.patch, HBASE-8564-1.patch, HBASE-8564-ADD-0.patch, HBASE-8564-FIX-0.patch, HBASE-8564-FIX-1.patch TestMetricsRegionServer has failed a few times on trunk. It's passing depends upon test ordering so will fail sometimes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8504) HTable.getRegionsInRange() should provide a non-cached API
[ https://issues.apache.org/jira/browse/HBASE-8504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664964#comment-13664964 ] Hudson commented on HBASE-8504: --- Integrated in hbase-0.95 #210 (See [https://builds.apache.org/job/hbase-0.95/210/]) HBASE-8504 HTable.getRegionsInRange() should provide a non-cached API (Revision 1485541) Result = SUCCESS mbertozzi : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java HTable.getRegionsInRange() should provide a non-cached API -- Key: HBASE-8504 URL: https://issues.apache.org/jira/browse/HBASE-8504 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 0.94.7 Reporter: Alan Choi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.9 Attachments: HBASE-8504-v0.patch, HBASE-8504-v0.patch getRegionsInRange() calls getRegionLocation() without reloading it. It will return wrong result if the cache is outdated due to region split. If the cost of always reloading isn't significant, we should consider doing that by default. Otherwise, let's have an API for getRegionsInRange() that forces a reload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7649) client retry timeout doesn't need to do x2 fallback when going to different server
[ https://issues.apache.org/jira/browse/HBASE-7649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664965#comment-13664965 ] Hudson commented on HBASE-7649: --- Integrated in hbase-0.95 #210 (See [https://builds.apache.org/job/hbase-0.95/210/]) HBASE-7649 ADDENDUM client retry timeout doesn't need to do x2 fallback when going to different server (Revision 1485514) Result = SUCCESS sershe : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java client retry timeout doesn't need to do x2 fallback when going to different server -- Key: HBASE-7649 URL: https://issues.apache.org/jira/browse/HBASE-7649 Project: HBase Issue Type: Improvement Components: Client, MTTR Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.95.1 Attachments: HBASE-7649-ADDENDUM-v0.patch, HBASE-7649-v0.patch, HBASE-7649-v1.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v2.patch, HBASE-7649-v3.patch, HBASE-7649-v4.patch, HBASE-7649-v5.patch, HBASE-7649-v6.patch, HBASE-7649-v7.patch, HBASE-7649-v8.1.patch, HBASE-7649-v8.patch See HBASE-7520. When we go to server A, get a bunch of failures, then finally learn the region is on B it doesn't make sense to wait for 30 seconds before going to B. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8596) [docs] Add docs about Region server draining mode
[ https://issues.apache.org/jira/browse/HBASE-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664966#comment-13664966 ] Hudson commented on HBASE-8596: --- Integrated in hbase-0.95 #210 (See [https://builds.apache.org/job/hbase-0.95/210/]) HBASE-8596 [docs] Add docs about RegionServer draining mode (Revision 1485441) Result = SUCCESS jmhsieh : Files : * /hbase/branches/0.95/src/main/docbkx/ops_mgt.xml [docs] Add docs about Region server draining mode --- Key: HBASE-8596 URL: https://issues.apache.org/jira/browse/HBASE-8596 Project: HBase Issue Type: Bug Components: documentation, regionserver Affects Versions: 0.92.2, 0.98.0, 0.94.7, 0.95.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.98.0, 0.95.1 Attachments: hbase-8596.patch, hbase-8596.v2.patch HBASE-4298 introduced draining mode for region servers to optimize rolling restarts to allow for multiple RS's going down simultaneously. There is a good blog post from the original author. I've added highlights from and and a link to it in the Node Decommissioning section of the ref guide. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664967#comment-13664967 ] Hudson commented on HBASE-8450: --- Integrated in hbase-0.95 #210 (See [https://builds.apache.org/job/hbase-0.95/210/]) HBASE-8450 Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. (Revision 1485562) Result = SUCCESS stack : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8504) HTable.getRegionsInRange() should provide a non-cached API
[ https://issues.apache.org/jira/browse/HBASE-8504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665033#comment-13665033 ] Hudson commented on HBASE-8504: --- Integrated in HBase-0.94 #995 (See [https://builds.apache.org/job/HBase-0.94/995/]) HBASE-8504 HTable.getRegionsInRange() should provide a non-cached API (Revision 1485543) Result = FAILURE mbertozzi : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java HTable.getRegionsInRange() should provide a non-cached API -- Key: HBASE-8504 URL: https://issues.apache.org/jira/browse/HBASE-8504 Project: HBase Issue Type: New Feature Components: Client Affects Versions: 0.94.7 Reporter: Alan Choi Assignee: Matteo Bertozzi Fix For: 0.98.0, 0.95.2, 0.94.9 Attachments: HBASE-8504-v0.patch, HBASE-8504-v0.patch getRegionsInRange() calls getRegionLocation() without reloading it. It will return wrong result if the cache is outdated due to region split. If the cost of always reloading isn't significant, we should consider doing that by default. Otherwise, let's have an API for getRegionsInRange() that forces a reload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8555) FilterList correctness may be affected by random ordering of sub-filter(list)
[ https://issues.apache.org/jira/browse/HBASE-8555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665034#comment-13665034 ] Hudson commented on HBASE-8555: --- Integrated in HBase-0.94 #995 (See [https://builds.apache.org/job/HBase-0.94/995/]) HBASE-8555 FilterList correctness may be affected by random ordering of sub-filter(list) (Liang Xie) (Revision 1485492) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java FilterList correctness may be affected by random ordering of sub-filter(list) - Key: HBASE-8555 URL: https://issues.apache.org/jira/browse/HBASE-8555 Project: HBase Issue Type: Bug Components: Filters Affects Versions: 0.94.3 Reporter: Liang Xie Assignee: Liang Xie Priority: Critical Fix For: 0.98.0, 0.95.1, 0.94.9 Attachments: 8555-trunk-v1.txt, HBASE-8555-0.94.txt, HBASE-8555-0.94-v2.txt, HBASE-8555-0.94-v3.txt say, ther're 10 rows, column value is i%2: row0 0 row1 1 row2 0 row3 1 row4 0 row5 1 row6 0 row7 1 row8 0 row9 1 1: filter : row filter row4 === row5 row6 row7 row8 row9 2: subFilterList: row filter = row4 column==0=== row0 row2 row4 3.1 filterlist[expected] filter || subFilterList === row0 row2 row4 row5 row6 row7 row8 row9 3.2 filterlist[BUGON!] subFilterList || filter === row0 row1 row2 row3 row4 row5 row6 row7 row8 row9 (Please refer to the new testNestedFilterListWithSCVF case) It was found when i managed to transform the following SQL into HBase scan statement: select xxx from xxx where (pk = xxx and column1 = xxx) or pk xxx My finding is that we had an assumption for filter methods call sequence: e.g. filterRowKey() should be called before filterKeyValue(). and the orignial filterList.filterRowKey impl broke it due to fast short-circuit returning. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8570) CompactSplitThread logs a CompactSplitThread$CompactionRunner but it does not have a toString
[ https://issues.apache.org/jira/browse/HBASE-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] samar updated HBASE-8570: - Attachment: HBASE-8570_1.patch toString overridden . used getRequest because CompactionRequest#toString has lot of details for the compaction CompactSplitThread logs a CompactSplitThread$CompactionRunner but it does not have a toString - Key: HBASE-8570 URL: https://issues.apache.org/jira/browse/HBASE-8570 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.98.0 Reporter: Nicolas Liochon Priority: Trivial Attachments: HBASE-8570_1.patch 2013-05-17 13:51:35,664 ERROR [regionserver60020-smallCompactions-1368795988827] org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner@1a7abea0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4811) Support reverse Scan
[ https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-4811: - Attachment: HBase-4811-0.94-v2.txt a tentative rebase against 0.94 branch(it passed runAllTests in our internal 0.94.3 codebase) Support reverse Scan Key: HBASE-4811 URL: https://issues.apache.org/jira/browse/HBASE-4811 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.20.6 Reporter: John Carrino Attachments: HBase-4811-0.94.3modified.txt, HBase-4811-0.94-v2.txt All the documentation I find about HBase says that if you want forward and reverse scans you should just build 2 tables and one be ascending and one descending. Is there a fundamental reason that HBase only supports forward Scan? It seems like a lot of extra space overhead and coding overhead (to keep them in sync) to support 2 tables. I am assuming this has been discussed before, but I can't find the discussions anywhere about it or why it would be infeasible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4811) Support reverse Scan
[ https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-4811: - Assignee: Liang Xie Affects Version/s: 0.94.7 Status: Patch Available (was: Open) Support reverse Scan Key: HBASE-4811 URL: https://issues.apache.org/jira/browse/HBASE-4811 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.7, 0.20.6 Reporter: John Carrino Assignee: Liang Xie Attachments: HBase-4811-0.94.3modified.txt, HBase-4811-0.94-v2.txt All the documentation I find about HBase says that if you want forward and reverse scans you should just build 2 tables and one be ascending and one descending. Is there a fundamental reason that HBase only supports forward Scan? It seems like a lot of extra space overhead and coding overhead (to keep them in sync) to support 2 tables. I am assuming this has been discussed before, but I can't find the discussions anywhere about it or why it would be infeasible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665095#comment-13665095
]
Hadoop QA commented on HBASE-4811:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584478/HBase-4811-0.94-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5796//console
This message is automatically generated.
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: Liang Xie
Attachments: HBase-4811-0.94.3modified.txt, HBase-4811-0.94-v2.txt
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to
keep them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the
discussions anywhere about it or why it would be infeasible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4811) Support reverse Scan
[ https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-4811: - Attachment: (was: HBase-4811-0.94-v2.txt) Support reverse Scan Key: HBASE-4811 URL: https://issues.apache.org/jira/browse/HBASE-4811 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.20.6, 0.94.7 Reporter: John Carrino Assignee: Liang Xie Attachments: HBase-4811-0.94.3modified.txt All the documentation I find about HBase says that if you want forward and reverse scans you should just build 2 tables and one be ascending and one descending. Is there a fundamental reason that HBase only supports forward Scan? It seems like a lot of extra space overhead and coding overhead (to keep them in sync) to support 2 tables. I am assuming this has been discussed before, but I can't find the discussions anywhere about it or why it would be infeasible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4811) Support reverse Scan
[ https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-4811: - Attachment: HBase-4811-0.94-v2.txt repost Support reverse Scan Key: HBASE-4811 URL: https://issues.apache.org/jira/browse/HBASE-4811 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.20.6, 0.94.7 Reporter: John Carrino Assignee: Liang Xie Attachments: HBase-4811-0.94.3modified.txt, HBase-4811-0.94-v2.txt All the documentation I find about HBase says that if you want forward and reverse scans you should just build 2 tables and one be ascending and one descending. Is there a fundamental reason that HBase only supports forward Scan? It seems like a lot of extra space overhead and coding overhead (to keep them in sync) to support 2 tables. I am assuming this has been discussed before, but I can't find the discussions anywhere about it or why it would be infeasible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665121#comment-13665121
]
Hadoop QA commented on HBASE-4811:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584481/HBase-4811-0.94-v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 19 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5797//console
This message is automatically generated.
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: Liang Xie
Attachments: HBase-4811-0.94.3modified.txt, HBase-4811-0.94-v2.txt
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to
keep them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the
discussions anywhere about it or why it would be infeasible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[ https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665152#comment-13665152 ] Hudson commented on HBASE-7942: --- Integrated in hbase-0.95 #211 (See [https://builds.apache.org/job/hbase-0.95/211/]) HBASE-7942. Makes the RegionServer aware of favored-nodes. The regionserver can now specify where to create the region files if the underlying HDFS supports the corresponding create(..favorednodes..) API. (Revision 1485587) Result = SUCCESS ddas : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FavoredNodesForRegion.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionFavoredNodes.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java Make use of the plumbing in HBASE-7932 to provide location hints when region files are created -- Key: HBASE-7942 URL: https://issues.apache.org/jira/browse/HBASE-7942 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.1 Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch, 7942-trunk-1.5.txt, 7942-trunk-1.6.txt This functionality is blocked by the fixes in HDFS. Reflection is going to be used to figure out whether the underlying Hadoop version has the new API. Things will behave the same way if the underlying HDFS version doesn't have the new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[ https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665179#comment-13665179 ] Hudson commented on HBASE-7942: --- Integrated in HBase-TRUNK #4139 (See [https://builds.apache.org/job/HBase-TRUNK/4139/]) HBASE-7942. Makes the RegionServer aware of favored-nodes. The regionserver can now specify where to create the region files if the underlying HDFS supports the corresponding create(..favorednodes..) API. (Revision 1485584) Result = SUCCESS ddas : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FavoredNodesForRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionFavoredNodes.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java Make use of the plumbing in HBASE-7932 to provide location hints when region files are created -- Key: HBASE-7942 URL: https://issues.apache.org/jira/browse/HBASE-7942 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.1 Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch, 7942-trunk-1.5.txt, 7942-trunk-1.6.txt This functionality is blocked by the fixes in HDFS. Reflection is going to be used to figure out whether the underlying Hadoop version has the new API. Things will behave the same way if the underlying HDFS version doesn't have the new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8570) CompactSplitThread logs a CompactSplitThread$CompactionRunner but it does not have a toString
[ https://issues.apache.org/jira/browse/HBASE-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-8570: - Assignee: samar CompactSplitThread logs a CompactSplitThread$CompactionRunner but it does not have a toString - Key: HBASE-8570 URL: https://issues.apache.org/jira/browse/HBASE-8570 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.98.0 Reporter: Nicolas Liochon Assignee: samar Priority: Trivial Attachments: HBASE-8570_1.patch 2013-05-17 13:51:35,664 ERROR [regionserver60020-smallCompactions-1368795988827] org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner@1a7abea0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8336) PooledHTable may be returned multiple times to the same pool
[ https://issues.apache.org/jira/browse/HBASE-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-8336: - Assignee: samar (was: Nikolai Grigoriev) PooledHTable may be returned multiple times to the same pool Key: HBASE-8336 URL: https://issues.apache.org/jira/browse/HBASE-8336 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.95.0 Reporter: Nikolai Grigoriev Assignee: samar Priority: Minor Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8336_1.patch, HBASE-8336_2.patch, HBASE-8336_3.patch I have recently observed a very strange issue with an application using HBase and HTablePool. After an investigation I have found that the root cause was the piece of code that was calling close() twice on the same HTableInterface instance retrieved from HTablePool (created with default policy). A closer look at the code revealed that PooledHTable.close() calls returnTable(), which, in turn, places the table back into the QUEUE of the pooled tables. No checking of any kind is done so it is possible to call it multiple times and place multiple references to the same HTable into the same pool. This creates a number of negative effects: - pool grows on each close() call and eventually gets filled up with the references to the same HTable. From this moment the pool stops working as pool. - multiple callers will get the same instance of HTable while expecting to have unique instances - once the pool is full, next call to close() will result to the call to the real close() method of HTable. This will make HTable unusable as close() call may shutdown() the internal thread pool. From this moment other attempts to use this HTable will fail with RejectedExecutionException. And since the HTablePool will have additional references to that HTable, other users of the pool will just start failing on any call that leads to flushCommits() The problem was, obviously, triggered by bad code on our side. But I think the pool has to be protected. Probably the best way to fix it would be to implement a flag in PooledHTable that represent its state (leased/returned) and once close() is called, it would be returned. From this moment any operations on this PooledHTable would result in something like IllegalStateException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665268#comment-13665268 ] Hudson commented on HBASE-8450: --- Integrated in hbase-0.95-on-hadoop2 #109 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/109/]) HBASE-8450 Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. (Revision 1485562) Result = FAILURE stack : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/branches/0.95/hbase-common/src/main/resources/hbase-default.xml * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[ https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665269#comment-13665269 ] Hudson commented on HBASE-7942: --- Integrated in hbase-0.95-on-hadoop2 #109 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/109/]) HBASE-7942. Makes the RegionServer aware of favored-nodes. The regionserver can now specify where to create the region files if the underlying HDFS supports the corresponding create(..favorednodes..) API. (Revision 1485587) Result = FAILURE ddas : Files : * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FavoredNodesForRegion.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionFavoredNodes.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java Make use of the plumbing in HBASE-7932 to provide location hints when region files are created -- Key: HBASE-7942 URL: https://issues.apache.org/jira/browse/HBASE-7942 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.1 Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch, 7942-trunk-1.5.txt, 7942-trunk-1.6.txt This functionality is blocked by the fixes in HDFS. Reflection is going to be used to figure out whether the underlying Hadoop version has the new API. Things will behave the same way if the underlying HDFS version doesn't have the new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8235) Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-8235: -- Resolution: Fixed Fix Version/s: 0.95.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and 0.95 Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool --- Key: HBASE-8235 URL: https://issues.apache.org/jira/browse/HBASE-8235 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8235_1.patch, HBASE-8235_2.patch, HBASE-8235.patch Adding the inmemory CF attribute to the LoadTestTool and PerfEvaluation tool will make it extensible to test such inmemory scenarios also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665363#comment-13665363 ] Hudson commented on HBASE-8450: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #540 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/540/]) HBASE-8450 Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. (Revision 1485561) Result = FAILURE stack : Files : * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/trunk/hbase-common/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestEncodedSeekers.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSeekOptimizations.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7942) Make use of the plumbing in HBASE-7932 to provide location hints when region files are created
[ https://issues.apache.org/jira/browse/HBASE-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665364#comment-13665364 ] Hudson commented on HBASE-7942: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #540 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/540/]) HBASE-7942. Makes the RegionServer aware of favored-nodes. The regionserver can now specify where to create the region files if the underlying HDFS supports the corresponding create(..favorednodes..) API. (Revision 1485584) Result = FAILURE ddas : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FavoredNodesForRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionFavoredNodes.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java Make use of the plumbing in HBASE-7932 to provide location hints when region files are created -- Key: HBASE-7942 URL: https://issues.apache.org/jira/browse/HBASE-7942 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.1 Attachments: 7942-trunk-1.3.patch, 7942-trunk-1.4.patch, 7942-trunk-1.5.txt, 7942-trunk-1.6.txt This functionality is blocked by the fixes in HDFS. Reflection is going to be used to figure out whether the underlying Hadoop version has the new API. Things will behave the same way if the underlying HDFS version doesn't have the new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8014) Backport HBASE-6915 to 0.94.
[ https://issues.apache.org/jira/browse/HBASE-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665380#comment-13665380 ] Dave Latham commented on HBASE-8014: Are the changes in HBASE-6921 also relevant? Backport HBASE-6915 to 0.94. Key: HBASE-8014 URL: https://issues.apache.org/jira/browse/HBASE-8014 Project: HBase Issue Type: Bug Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Critical Fix For: 0.94.7 Attachments: HBASE-8014-v0-0.94.patch JDK 1.7 changed some data size. Goal of this JIRA is to backport HBASE-6915 to 0.94 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7896) make rename_table working in 92/94
[ https://issues.apache.org/jira/browse/HBASE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7896: --- Attachment: rename_table.rb make rename_table working in 92/94 -- Key: HBASE-7896 URL: https://issues.apache.org/jira/browse/HBASE-7896 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.2, 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.92.3 Attachments: rename_table.rb, rename_table.rb The rename_table function is very useful for our customers. However, rename_table.rb does not work for 92/94. It has several bugs. It will be useful to fix them so that users can solve their problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7896) make rename_table working in 92/94
[ https://issues.apache.org/jira/browse/HBASE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7896: --- Attachment: (was: rename_table.rb) make rename_table working in 92/94 -- Key: HBASE-7896 URL: https://issues.apache.org/jira/browse/HBASE-7896 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.2, 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.92.3 Attachments: rename_table.rb, rename_table.rb The rename_table function is very useful for our customers. However, rename_table.rb does not work for 92/94. It has several bugs. It will be useful to fix them so that users can solve their problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7379) Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk
[ https://issues.apache.org/jira/browse/HBASE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-7379: - Assignee: (was: Ted Yu) Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk --- Key: HBASE-7379 URL: https://issues.apache.org/jira/browse/HBASE-7379 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Priority: Minor Attachments: 7379-trunk.txt, 7379-trunk-v2.txt HBASE-6728 ported '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to 0.94 branch This issue is to port it to trunk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7841) Parallelize offline snapshot in DisabledTableSnapshotHandler
[
https://issues.apache.org/jira/browse/HBASE-7841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-7841:
--
Status: Open (was: Patch Available)
Parallelize offline snapshot in DisabledTableSnapshotHandler
Key: HBASE-7841
URL: https://issues.apache.org/jira/browse/HBASE-7841
Project: HBase
Issue Type: Sub-task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 7841.txt, 7841-v2.txt, 7841-v3.txt, 7841-v4.txt
In DisabledTableSnapshotHandler, there is TODO:
{code}
// TODO consider parallelizing these operations since they are independent.
Right now its just
// easier to keep them serial though
@Override
public void snapshotRegions(ListPairHRegionInfo, ServerName
regionsAndLocations) throws IOException,
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7543) TestSnapshotDescriptionUtils#testValidateGlobalSnapshotDescriptor fails
[ https://issues.apache.org/jira/browse/HBASE-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665402#comment-13665402 ] Ted Yu commented on HBASE-7543: --- This test didn't fail for quite a while. Will resolve if there is no objection. TestSnapshotDescriptionUtils#testValidateGlobalSnapshotDescriptor fails --- Key: HBASE-7543 URL: https://issues.apache.org/jira/browse/HBASE-7543 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Attachments: 7543.txt testValidateGlobalSnapshotDescriptor(org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils): Message missing required fields: name -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8344) Improve the assignment when node failures happen to choose the secondary RS as the new primary RS
[ https://issues.apache.org/jira/browse/HBASE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-8344: --- Attachment: hbase-8344-2.1.txt I should have run 'mvn clean' after some changes I did. This patch addresses the failure. I should add that that this patch deletes a unit test from TestRegionPlacement - the reason being two new tests that I added subsumes that one. One more thing I forgot to mention earlier is that, the favorednodes are now stored without the startcode (set to -1). In reality, we only care about the hostname for the favorednodes stuff. Improve the assignment when node failures happen to choose the secondary RS as the new primary RS - Key: HBASE-8344 URL: https://issues.apache.org/jira/browse/HBASE-8344 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.2 Attachments: hbase-8344-1.txt, hbase-8344-2.1.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665411#comment-13665411 ] Sergey Shelukhin commented on HBASE-8497: - +1 Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665417#comment-13665417 ] Ted Yu commented on HBASE-8497: --- bq. lastPositionToRead is pointing to 392 From the log: bq. After reading the trailer: lastPositionToRead: 372 Should the two numbers be the same ? Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7543) TestSnapshotDescriptionUtils#testValidateGlobalSnapshotDescriptor fails
[ https://issues.apache.org/jira/browse/HBASE-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh resolved HBASE-7543. --- Resolution: Not A Problem sound good. closed at ted's suggestion. TestSnapshotDescriptionUtils#testValidateGlobalSnapshotDescriptor fails --- Key: HBASE-7543 URL: https://issues.apache.org/jira/browse/HBASE-7543 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Ted Yu Attachments: 7543.txt testValidateGlobalSnapshotDescriptor(org.apache.hadoop.hbase.snapshot.TestSnapshotDescriptionUtils): Message missing required fields: name -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[
https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665436#comment-13665436
]
Himanshu Vashishtha commented on HBASE-8497:
Oh I messed up with numbers:)
{code}
2013-05-22 17:11:23,400 DEBUG wal.ProtobufLogReader
(ProtobufLogReader.java:initInternal(118)) - After reading the trailer:
lastPositionToRead: 372, fileLength: 380, trailerPresent: true
{code}
So, it should read:
Note, magic is 4 bytes [377-380], trailer size is 4 bytes [373-376]. The
trailer size is 0; so last byte of the WALEdit is at 372, and
lastPositionToRead is pointing to 372 only.
Also note the lastPositionToRead variable is changed to walEditsStopOffset, but
that's a different point.
Protobuf WAL also needs a trailer
--
Key: HBASE-8497
URL: https://issues.apache.org/jira/browse/HBASE-8497
Project: HBase
Issue Type: Sub-task
Components: Protobufs, wal
Affects Versions: 0.95.1
Reporter: Enis Soztutar
Assignee: Himanshu Vashishtha
Fix For: 0.98.0, 0.95.1
Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch,
HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch
New Protobuf WAL has a header, but we will probably need a trailer as well,
reserved for later usage.
Right now, we can we just serialize an empty trailer, but putting more
metadata there, like range of sequence_id's, region names, table names etc
might be needed in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8599) HLogs in ZK are not cleaned up when replication lag is minimal
Varun Sharma created HBASE-8599:
---
Summary: HLogs in ZK are not cleaned up when replication lag is
minimal
Key: HBASE-8599
URL: https://issues.apache.org/jira/browse/HBASE-8599
Project: HBase
Issue Type: Improvement
Components: Replication
Affects Versions: 0.94.7
Reporter: Varun Sharma
On a cluster with very low replication lag (as measured by ageOfLastShippedOp
on source), we found HLogs accumulating and not being cleaned up as new WAL(s)
are rolled.
Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever
the current WAL is not being written to any more - as suggested by
currentWALBeingWrittenTo being false. However, when lags are small, we may hit
the following block first and continue onto the next WAL without clearing the
old WAL(s)...
ReplicationSource::run() {
if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo =
false)) {
// If we are here, then we advance to the next WAL without any cleaning
// and close existing WAL
continue;
}
// Ship some edits and call logPositionAndCleanOldLogs
}
If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older
logs are not cleaned out and persist in the zookeeper node since we simply call
continue and skip the subsequent logPositionAndCleanOldLogs call - if its
called more than once, we do end up clearing the old logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-8517:
-
Attachment: HBASE-8517-5.patch
Cleaned up most of the nits. This should be good to go in assuming that the
+1's stand.
Stochastic Loadbalancer isn't finding steady state on real clusters
---
Key: HBASE-8517
URL: https://issues.apache.org/jira/browse/HBASE-8517
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.1
Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch,
HBASE-8517-2.patch, HBASE-8517-3.patch, HBASE-8517-4.patch, HBASE-8517-5.patch
I have a cluster that runs IT tests. Last night after all tests were done I
noticed that the balancer was thrashing regions around.
The number of regions on each machine is not correct.
The balancer seems to value the cost of moving a region way too little.
{code}
2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 5367ms to try 8910
different iterations. Found a solution that moves 37 regions; Going from a
computed cost of 56.50254222730425 to a new cost of 11.214035466575254
2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4735ms to try 8910
different iterations. Found a solution that moves 38 regions; Going from a
computed cost of 56.612624531830996 to a new cost of 11.275763861636982
2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4502ms to try 8910
different iterations. Found a solution that moves 39 regions; Going from a
computed cost of 56.50048461413552 to a new cost of 11.225352339003237
{code}
Each of those balancer runs were triggered when there was no load on the
cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8599) HLogs in ZK are not cleaned up when replication lag is minimal
[
https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Sharma updated HBASE-8599:
Attachment: 8599-0.94.patch
HLogs in ZK are not cleaned up when replication lag is minimal
--
Key: HBASE-8599
URL: https://issues.apache.org/jira/browse/HBASE-8599
Project: HBase
Issue Type: Improvement
Components: Replication
Affects Versions: 0.94.7
Reporter: Varun Sharma
Attachments: 8599-0.94.patch
On a cluster with very low replication lag (as measured by ageOfLastShippedOp
on source), we found HLogs accumulating and not being cleaned up as new
WAL(s) are rolled.
Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever
the current WAL is not being written to any more - as suggested by
currentWALBeingWrittenTo being false. However, when lags are small, we may
hit the following block first and continue onto the next WAL without clearing
the old WAL(s)...
ReplicationSource::run() {
if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo =
false)) {
// If we are here, then we advance to the next WAL without any
cleaning
// and close existing WAL
continue;
}
// Ship some edits and call logPositionAndCleanOldLogs
}
If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older
logs are not cleaned out and persist in the zookeeper node since we simply
call continue and skip the subsequent logPositionAndCleanOldLogs call - if
its called more than once, we do end up clearing the old logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8599) HLogs in ZK are not cleaned up when replication lag is minimal
[
https://issues.apache.org/jira/browse/HBASE-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665445#comment-13665445
]
Varun Sharma commented on HBASE-8599:
-
Attached a patch for 0.94 to clean out logs from ZK whenever we close a WAL in
ReplicationSource and choose to advance to the next WAL.
HLogs in ZK are not cleaned up when replication lag is minimal
--
Key: HBASE-8599
URL: https://issues.apache.org/jira/browse/HBASE-8599
Project: HBase
Issue Type: Improvement
Components: Replication
Affects Versions: 0.94.7
Reporter: Varun Sharma
Attachments: 8599-0.94.patch
On a cluster with very low replication lag (as measured by ageOfLastShippedOp
on source), we found HLogs accumulating and not being cleaned up as new
WAL(s) are rolled.
Each time, we call logPositionAndCleanOldLogs() to clean older logs whenever
the current WAL is not being written to any more - as suggested by
currentWALBeingWrittenTo being false. However, when lags are small, we may
hit the following block first and continue onto the next WAL without clearing
the old WAL(s)...
ReplicationSource::run() {
if (readAllEntriesToReplicateOrNextFile(currentWALisBeingWrittenTo =
false)) {
// If we are here, then we advance to the next WAL without any
cleaning
// and close existing WAL
continue;
}
// Ship some edits and call logPositionAndCleanOldLogs
}
If we hit readAllEntriesToReplicateOrNextFile(false) only once - then older
logs are not cleaned out and persist in the zookeeper node since we simply
call continue and skip the subsequent logPositionAndCleanOldLogs call - if
its called more than once, we do end up clearing the old logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8449: - Attachment: 8449v2.txt Here is a patch w/ a first test. Will add more in subsequent tests. Also tries to use isFileClosed if available, hbase-8394. Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-8573) Store last flushed sequence id for each store of region for Distributed Log Replay
[ https://issues.apache.org/jira/browse/HBASE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong reassigned HBASE-8573: Assignee: Jeffrey Zhong Store last flushed sequence id for each store of region for Distributed Log Replay -- Key: HBASE-8573 URL: https://issues.apache.org/jira/browse/HBASE-8573 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Jeffrey Zhong HBASE-7006 stores last flushed sequence id of the region in zookeeper. To prevent deleted data from appearing again, we should store last flushed sequence id for each store of region in zookeeper. See discussion here: https://issues.apache.org/jira/browse/HBASE-7006?focusedCommentId=13660428page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13660428 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2231) Compaction events should be written to HLog
[
https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665476#comment-13665476
]
Jean-Daniel Cryans commented on HBASE-2231:
---
The new INFO logging seems kinda chatty (and it shows up on multiple lines):
{noformat}
2013-05-23 11:38:12,955 INFO
[RegionServer:0;172.21.3.117,57009,1369334164639-smallCompactions-1369334211720]
org.apache.hadoop.hbase.regionserver.wal.HLogUtil: Appended compaction marker
tableName: TestTable
encodedRegionName: 338447e35d9c2b5b0a38e982c08cbf09
familyName: info
compactionInput: 515e42ec2ef746fa86a5ca7acbc58aed
compactionInput: 8abc6ae8c4754484b2628a4f4909342c
compactionInput: 303b7f1767b24408a55813142a5618c8
compactionInput: 0f93806c7ecf43a487f0c5dcc2de911f
compactionOutput: 1bff72a0b8664a0b8072049a1899e908
storeHomeDir: info
{noformat}
I'm not sure what the value is.
Compaction events should be written to HLog
---
Key: HBASE-2231
URL: https://issues.apache.org/jira/browse/HBASE-2231
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Todd Lipcon
Assignee: stack
Priority: Blocker
Labels: moved_from_0_20_5
Fix For: 0.98.0, 0.95.1
Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt,
2231-testcase_v3.txt, 2231v2.txt, 2231v3.txt, 2231v4.txt,
hbase-2231-testcase.txt, hbase-2231.txt, hbase-2231_v5.patch,
hbase-2231_v6.patch, hbase-2231_v7-0.95.patch, hbase-2231_v7.patch,
hbase-2231_v7.patch
The sequence for a compaction should look like this:
# Compact region to new files
# Write a Compacted Region entry to the HLog
# Delete old files
This deals with a case where the RS has paused between step 1 and 2 and the
regions have since been reassigned.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2231) Compaction events should be written to HLog
[ https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665477#comment-13665477 ] Prakash Khemani commented on HBASE-2231: Hi, Prakash Khemani is no longer at Facebook so this email address is no longer being monitored. If you need assistance, please contact another person who is currently at the company. Compaction events should be written to HLog --- Key: HBASE-2231 URL: https://issues.apache.org/jira/browse/HBASE-2231 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Todd Lipcon Assignee: stack Priority: Blocker Labels: moved_from_0_20_5 Fix For: 0.98.0, 0.95.1 Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt, 2231-testcase_v3.txt, 2231v2.txt, 2231v3.txt, 2231v4.txt, hbase-2231-testcase.txt, hbase-2231.txt, hbase-2231_v5.patch, hbase-2231_v6.patch, hbase-2231_v7-0.95.patch, hbase-2231_v7.patch, hbase-2231_v7.patch The sequence for a compaction should look like this: # Compact region to new files # Write a Compacted Region entry to the HLog # Delete old files This deals with a case where the RS has paused between step 1 and 2 and the regions have since been reassigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8344) Improve the assignment when node failures happen to choose the secondary RS as the new primary RS
[
https://issues.apache.org/jira/browse/HBASE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665490#comment-13665490
]
Hadoop QA commented on HBASE-8344:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584531/hbase-8344-2.1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5798//console
This message is automatically generated.
Improve the assignment when node failures happen to choose the secondary RS
as the new primary RS
-
Key: HBASE-8344
URL: https://issues.apache.org/jira/browse/HBASE-8344
Project: HBase
Issue Type: Sub-task
Reporter: Devaraj Das
Assignee: Devaraj Das
Priority: Critical
Fix For: 0.95.2
Attachments: hbase-8344-1.txt, hbase-8344-2.1.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665524#comment-13665524
]
Hadoop QA commented on HBASE-8517:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584536/HBASE-8517-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5799//console
This message is automatically generated.
Stochastic Loadbalancer isn't finding steady state on real clusters
---
Key: HBASE-8517
URL: https://issues.apache.org/jira/browse/HBASE-8517
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.1
Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch,
HBASE-8517-2.patch, HBASE-8517-3.patch, HBASE-8517-4.patch, HBASE-8517-5.patch
I have a cluster that runs IT tests. Last night after all tests were done I
noticed that the balancer was thrashing regions around.
The number of regions on each machine is not correct.
The balancer seems to value the cost of moving a region way too little.
{code}
2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 5367ms to try 8910
different iterations. Found a solution that moves 37 regions; Going from a
computed cost of 56.50254222730425 to a new cost of 11.214035466575254
2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4735ms to try 8910
different iterations. Found a solution that moves 38 regions; Going from a
computed cost of 56.612624531830996 to a new cost of 11.275763861636982
2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4502ms to try 8910
different iterations. Found a solution that moves 39 regions; Going from a
computed cost of 56.50048461413552 to a new cost of 11.225352339003237
{code}
Each of those balancer runs were triggered when there was no load on the
cluster.
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665531#comment-13665531 ] Nicolas Liochon commented on HBASE-8450: [~jdcryans] bq. I agree with you that setting that down to 10 would currently mean that we might retry timeouts on the region servers. What I'm wondering now is how we can make this better. Should we also tweak the default operation timeout lower? Maybe it set it higher for MR jobs? HBASE-7590 helps here, because you can set a high timeout without beeing stuck on dead servers. May be we could have different timeouts for read writes, as read can be reexecuted without risks). But on the long term I think that the best option is around the approach of HBASE-7590: just wait, if the server is dead we will tell you... Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc. Key: HBASE-8450 URL: https://issues.apache.org/jira/browse/HBASE-8450 Project: HBase Issue Type: Task Components: Usability Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8450.txt, 8450v2.txt, 8450v3.txt, 8450v5.txt, 8450v5.txt This is a critical task we need to do before we release; review our defaults. On cursory review, there are configs in hbase-default.xml that no longer have matching code; there are some that should be changed because we know better now and others that should be amended because hardware and deploys are bigger than they used to be. We could also move stuff around so that the must-edit stuff is near the top (zk quorum config. is mid-way down the page) and beef up the descriptions -- especially since these descriptions shine through in the refguide. Lastly, I notice that our tgz does not include an hbase-default.xml other than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665543#comment-13665543 ] Ted Yu commented on HBASE-8497: --- +1 Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8449: - Attachment: 8449v3.txt Here is a patch. Has tests. Will try and make use of isFileClosed if available. Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8449: - Assignee: stack Status: Patch Available (was: Open) Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.95.0, 0.94.7 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8449: - Attachment: 8449v4.txt Fix my test (oops) Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665588#comment-13665588
]
Sergey Shelukhin commented on HBASE-8517:
-
+1... one question I have remaining is why we have 50% swap, 50% move chance?
Stochastic Loadbalancer isn't finding steady state on real clusters
---
Key: HBASE-8517
URL: https://issues.apache.org/jira/browse/HBASE-8517
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.1
Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch,
HBASE-8517-2.patch, HBASE-8517-3.patch, HBASE-8517-4.patch, HBASE-8517-5.patch
I have a cluster that runs IT tests. Last night after all tests were done I
noticed that the balancer was thrashing regions around.
The number of regions on each machine is not correct.
The balancer seems to value the cost of moving a region way too little.
{code}
2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 5367ms to try 8910
different iterations. Found a solution that moves 37 regions; Going from a
computed cost of 56.50254222730425 to a new cost of 11.214035466575254
2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4735ms to try 8910
different iterations. Found a solution that moves 38 regions; Going from a
computed cost of 56.612624531830996 to a new cost of 11.275763861636982
2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4502ms to try 8910
different iterations. Found a solution that moves 39 regions; Going from a
computed cost of 56.50048461413552 to a new cost of 11.225352339003237
{code}
Each of those balancer runs were triggered when there was no load on the
cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8497: --- Attachment: HBASE-8497-v6.patch attaching the last one from rb here. Thanks for the reviews. Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch, HBASE-8497-v6.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8522: -- Attachment: 8522-trunk-v3.patch Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster -- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.95.2, 0.94.9 Attachments: 8522-trunk-v3.patch, HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665613#comment-13665613 ] Nicolas Liochon commented on HBASE-8449: Increase hbase.lease.recovery.timeout default to 15 minutes, i.e. more than a standard hdfs recovery. hbase.lease.recovery.dfs.timeout: it should not be less than 10s imho. It's not only a question of dfs timeout, it's as well that it seems that the NN seems not to like multiple calls to the recoverLease. I tested again multiple calls, the datanodes logs were complaining about situation that should never occurs. Ok, it was with multi calls with an interval of 1 second, but it seems to be all luck. + * 1. Call recoverLease. + * 2. If it returns true, break. + * 3. If it returns false, wait a few seconds and then call it again. + * 4. If it returns true, break. + * 5. If it returns false, wait for what we think the datanode socket timeout is + * (configurable) and then try again. + * 6. If it returns true, break. + * 7. If it returns false, repeat starting at step 5. above. I would propose: the master - if HDFS-4754 is there, the master marks the node as stale as the first step of the recovery. - The master calls recover lease as a part of the distributed split. We can enhance it in an other jira to give higher priority to closed wals vs. wals being recovered. the region server: - calls isFileCLosed, if it's there. if true returns - Calls recoverLease, if true, return - if isFileCLosed is available, loop on it with a 1s sleep - else loops on 70s (configurable) sleep with recover lease Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2231) Compaction events should be written to HLog
[ https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665623#comment-13665623 ] Enis Soztutar commented on HBASE-2231: -- bq. The new INFO logging seems kinda chatty Yeah, I also noticed that today. Let me do an addendum to this to change to debug. Compaction events should be written to HLog --- Key: HBASE-2231 URL: https://issues.apache.org/jira/browse/HBASE-2231 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Todd Lipcon Assignee: stack Priority: Blocker Labels: moved_from_0_20_5 Fix For: 0.98.0, 0.95.1 Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt, 2231-testcase_v3.txt, 2231v2.txt, 2231v3.txt, 2231v4.txt, hbase-2231-testcase.txt, hbase-2231.txt, hbase-2231_v5.patch, hbase-2231_v6.patch, hbase-2231_v7-0.95.patch, hbase-2231_v7.patch, hbase-2231_v7.patch The sequence for a compaction should look like this: # Compact region to new files # Write a Compacted Region entry to the HLog # Delete old files This deals with a case where the RS has paused between step 1 and 2 and the regions have since been reassigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665625#comment-13665625 ] Matteo Bertozzi commented on HBASE-8522: +1 on v3 that fixes the long line issue Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster -- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.95.2, 0.94.9 Attachments: 8522-trunk-v3.patch, HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8559) increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor
[ https://issues.apache.org/jira/browse/HBASE-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665630#comment-13665630 ] Ted Yu commented on HBASE-8559: --- Should table.close() be called in after() ? increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor -- Key: HBASE-8559 URL: https://issues.apache.org/jira/browse/HBASE-8559 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.94.8, 0.95.2 Reporter: Ivan A. Veselovsky Attachments: HBASE-8559-0.94--N2.patch, HBASE-8559-trunk--N2.patch increase unit-test coverage of package org.apache.hadoop.hbase.coprocessor up to 80%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8597) compaction record (probably) can block WAL cleanup forever if region is closed without edits
[ https://issues.apache.org/jira/browse/HBASE-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665633#comment-13665633 ] Sergey Shelukhin commented on HBASE-8597: - bq. Why is the @param for htd removed above ? bq. Please call HTable.close() at the end of the test. fixed bq. I wonder if we don't need to introduce the new boolean for log.append(): I looked at the code and it seems that meta-family is already hacky enough, I'd rather have an explicit flag. Let me look at the test... compaction record (probably) can block WAL cleanup forever if region is closed without edits Key: HBASE-8597 URL: https://issues.apache.org/jira/browse/HBASE-8597 Project: HBase Issue Type: Bug Affects Versions: 0.95.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Critical Fix For: 0.95.1 Attachments: HBASE-8597-v0.patch A region is opened by a server, major compaction is performed, that triggers a split, and the region is closed and split. There's no indication of memstore flush for this region. After that, LogRoller repeatedly tries to request the flush of this region due to large number of HLogs, but fails to flush it for hours because the region is not in online regions. It seems that what's happening is that when we append entries to WAL we add the first entry after we flush/open some region to earliest unflushed seqNums per region map in FSHLog. However, compaction now adds compaction record to WAL, which also affects this map. If the compaction record is the first entry for this region to go into some WAL, and there are no writes to the region after that, there will be no memstore flush and the entry will never be removed. In fact flushing for compaction record doesn't make sense, there's no preservation of the record outside WAL; so, we probably should not add it to latest unflushed map. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8527) clean up code around compaction completion in HStore
[ https://issues.apache.org/jira/browse/HBASE-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8527: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) clean up code around compaction completion in HStore Key: HBASE-8527 URL: https://issues.apache.org/jira/browse/HBASE-8527 Project: HBase Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HBASE-8527-v0.patch, HBASE-8527-v1.patch The methods completeCompaction and it's caller are too long. Something I changed while doing something else there; putting in a separate easy-to-review JIRA to make the other future change smaller. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665646#comment-13665646
]
Elliott Clark commented on HBASE-8517:
--
We have that because as the balancer gets closer to a final solution, it has
probably already evened up the number of regions and swapping them will be the
only way to get a better balance (in one step). I intend to make that part a
little more intelligent (swap more as things become more balanced, try swapping
regions that have been moved in this balancer run, etc). But this has been
working pretty well.
Stochastic Loadbalancer isn't finding steady state on real clusters
---
Key: HBASE-8517
URL: https://issues.apache.org/jira/browse/HBASE-8517
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.1
Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch,
HBASE-8517-2.patch, HBASE-8517-3.patch, HBASE-8517-4.patch, HBASE-8517-5.patch
I have a cluster that runs IT tests. Last night after all tests were done I
noticed that the balancer was thrashing regions around.
The number of regions on each machine is not correct.
The balancer seems to value the cost of moving a region way too little.
{code}
2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 5367ms to try 8910
different iterations. Found a solution that moves 37 regions; Going from a
computed cost of 56.50254222730425 to a new cost of 11.214035466575254
2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4735ms to try 8910
different iterations. Found a solution that moves 38 regions; Going from a
computed cost of 56.612624531830996 to a new cost of 11.275763861636982
2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4502ms to try 8910
different iterations. Found a solution that moves 39 regions; Going from a
computed cost of 56.50048461413552 to a new cost of 11.225352339003237
{code}
Each of those balancer runs were triggered when there was no load on the
cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665659#comment-13665659
]
stack commented on HBASE-8517:
--
+1 after skimming the patch and presuming you have tried it on cluster. Good
on you Elliott.
Stochastic Loadbalancer isn't finding steady state on real clusters
---
Key: HBASE-8517
URL: https://issues.apache.org/jira/browse/HBASE-8517
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.1
Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch,
HBASE-8517-2.patch, HBASE-8517-3.patch, HBASE-8517-4.patch, HBASE-8517-5.patch
I have a cluster that runs IT tests. Last night after all tests were done I
noticed that the balancer was thrashing regions around.
The number of regions on each machine is not correct.
The balancer seems to value the cost of moving a region way too little.
{code}
2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 5367ms to try 8910
different iterations. Found a solution that moves 37 regions; Going from a
computed cost of 56.50254222730425 to a new cost of 11.214035466575254
2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4735ms to try 8910
different iterations. Found a solution that moves 38 regions; Going from a
computed cost of 56.612624531830996 to a new cost of 11.275763861636982
2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4502ms to try 8910
different iterations. Found a solution that moves 39 regions; Going from a
computed cost of 56.50048461413552 to a new cost of 11.225352339003237
{code}
Each of those balancer runs were triggered when there was no load on the
cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2231) Compaction events should be written to HLog
[ https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-2231: - Attachment: hbase-2231_v7-addendum.patch Compaction events should be written to HLog --- Key: HBASE-2231 URL: https://issues.apache.org/jira/browse/HBASE-2231 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Todd Lipcon Assignee: stack Priority: Blocker Labels: moved_from_0_20_5 Fix For: 0.98.0, 0.95.1 Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt, 2231-testcase_v3.txt, 2231v2.txt, 2231v3.txt, 2231v4.txt, hbase-2231-testcase.txt, hbase-2231.txt, hbase-2231_v5.patch, hbase-2231_v6.patch, hbase-2231_v7-0.95.patch, hbase-2231_v7-addendum.patch, hbase-2231_v7.patch, hbase-2231_v7.patch The sequence for a compaction should look like this: # Compact region to new files # Write a Compacted Region entry to the HLog # Delete old files This deals with a case where the RS has paused between step 1 and 2 and the regions have since been reassigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2231) Compaction events should be written to HLog
[ https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665662#comment-13665662 ] Enis Soztutar commented on HBASE-2231: -- How about this addendum? Compaction events should be written to HLog --- Key: HBASE-2231 URL: https://issues.apache.org/jira/browse/HBASE-2231 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Todd Lipcon Assignee: stack Priority: Blocker Labels: moved_from_0_20_5 Fix For: 0.98.0, 0.95.1 Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt, 2231-testcase_v3.txt, 2231v2.txt, 2231v3.txt, 2231v4.txt, hbase-2231-testcase.txt, hbase-2231.txt, hbase-2231_v5.patch, hbase-2231_v6.patch, hbase-2231_v7-0.95.patch, hbase-2231_v7-addendum.patch, hbase-2231_v7.patch, hbase-2231_v7.patch The sequence for a compaction should look like this: # Compact region to new files # Write a Compacted Region entry to the HLog # Delete old files This deals with a case where the RS has paused between step 1 and 2 and the regions have since been reassigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[
https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665663#comment-13665663
]
Hadoop QA commented on HBASE-8449:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584562/8449v3.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5800//console
This message is automatically generated.
Refactor recoverLease retries and pauses informed by findings over in
hbase-8389
Key: HBASE-8449
URL: https://issues.apache.org/jira/browse/HBASE-8449
Project: HBase
Issue Type: Bug
Components: Filesystem Integration
Affects Versions: 0.94.7, 0.95.0
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.95.1
Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt
HBASE-8359 is an interesting issue that roams near and far. This issue is
about making use of the findings handily summarized on the end of hbase-8359
which have it that trunk needs refactor around how it does its recoverLease
handling (and that the patch committed against HBASE-8359 is not what we want
going forward).
This issue is about making a patch that adds a lag between recoverLease
invocations where the lag is related to dfs timeouts -- the hdfs-side dfs
timeout -- and optionally makes use of the isFileClosed API if it is
available (a facility that is not yet committed to a branch near you and
unlikely to be within your locality with a good while to come).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2231) Compaction events should be written to HLog
[ https://issues.apache.org/jira/browse/HBASE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665664#comment-13665664 ] stack commented on HBASE-2231: -- Trace level it on commit? +1 Compaction events should be written to HLog --- Key: HBASE-2231 URL: https://issues.apache.org/jira/browse/HBASE-2231 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Todd Lipcon Assignee: stack Priority: Blocker Labels: moved_from_0_20_5 Fix For: 0.98.0, 0.95.1 Attachments: 2231-testcase-0.94.txt, 2231-testcase_v2.txt, 2231-testcase_v3.txt, 2231v2.txt, 2231v3.txt, 2231v4.txt, hbase-2231-testcase.txt, hbase-2231.txt, hbase-2231_v5.patch, hbase-2231_v6.patch, hbase-2231_v7-0.95.patch, hbase-2231_v7-addendum.patch, hbase-2231_v7.patch, hbase-2231_v7.patch The sequence for a compaction should look like this: # Compact region to new files # Write a Compacted Region entry to the HLog # Delete old files This deals with a case where the RS has paused between step 1 and 2 and the regions have since been reassigned. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8344) Improve the assignment when node failures happen to choose the secondary RS as the new primary RS
[ https://issues.apache.org/jira/browse/HBASE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665666#comment-13665666 ] Nicolas Liochon commented on HBASE-8344: Seems ok to me. I have a theoretical question. What's happens if we lose 2 nodes out of the ensemble of 3? Will we have the 3 region sets on the last one remaining? Improve the assignment when node failures happen to choose the secondary RS as the new primary RS - Key: HBASE-8344 URL: https://issues.apache.org/jira/browse/HBASE-8344 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.2 Attachments: hbase-8344-1.txt, hbase-8344-2.1.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8600) Bunch of log lines from QosFunction: Marking normal priority after
Enis Soztutar created HBASE-8600:
Summary: Bunch of log lines from QosFunction: Marking normal
priority after
Key: HBASE-8600
URL: https://issues.apache.org/jira/browse/HBASE-8600
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Fix For: 0.98.0, 0.95.1
In tests, RS logs contain a lot of the following because of
RegionOpeningException or RegionMovedException.
{code}
2013-05-23 20:40:08,891 DEBUG [IPC Reader 3 on port 60020]
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority after
getting exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException:
Region is being opened: f8665bfb57c728262807429efc41fa49
2013-05-23 20:40:08,891 DEBUG [IPC Reader 3 on port 60020]
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority after
getting exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException:
Region is being opened: f8665bfb57c728262807429efc41fa49
2013-05-23 20:40:10,385 DEBUG [IPC Reader 4 on port 60020]
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority after
getting exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException:
Region is being opened: f8665bfb57c728262807429efc41fa49
2013-05-23 20:40:10,385 DEBUG [IPC Reader 4 on port 60020]
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority after
getting exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException:
Region is being opened: f8665bfb57c728262807429efc41fa49
2013-05-23 20:40:10,786 DEBUG [IPC Reader 5 on port 60020]
org.apache.hadoop.hbase.regionserver.QosFunction: Marking normal priority after
getting exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException:
Region is being opened: f8665bfb57c728262807429efc41fa49
2
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665670#comment-13665670 ] stack commented on HBASE-8449: -- bq. Increase hbase.lease.recovery.timeout default to 15 minutes, i.e. more than a standard hdfs recovery. I do not follow [~nkeywal] It is 15minutes at the moment (this is just a copy of what was there before). bq. hbase.lease.recovery.dfs.timeout: it should not be less than 10s imho. This is set to 61 seconds, what we think the time it will take the NN to timeout on the datanode (dfs.socket.timeout hopefully). bq. it's as well that it seems that the NN seems not to like multiple calls to the recoverLease. Yes, the aim w/ this patch is to not kill an ongoing lease recovery. Regards your proposal, it is built on a patch not yet committed to hdfs. I am trying to get something done now so that I can make a 0.95.1 release (what we have currently will do the scenario you mocked up where you were calling the namenode every second). bq. The master calls recover lease as a part of the distributed split. We can enhance it in an other jira to give higher priority to closed wals vs. wals being recovered. Yeah, that would be a good TODO for later. All of your proposal seems for later rather than now. You +1 on what I have here [~nkeywal]? Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[
https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665674#comment-13665674
]
Hadoop QA commented on HBASE-8449:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584566/8449v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5801//console
This message is automatically generated.
Refactor recoverLease retries and pauses informed by findings over in
hbase-8389
Key: HBASE-8449
URL: https://issues.apache.org/jira/browse/HBASE-8449
Project: HBase
Issue Type: Bug
Components: Filesystem Integration
Affects Versions: 0.94.7, 0.95.0
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.95.1
Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt
HBASE-8359 is an interesting issue that roams near and far. This issue is
about making use of the findings handily summarized on the end of hbase-8359
which have it that trunk needs refactor around how it does its recoverLease
handling (and that the patch committed against HBASE-8359 is not what we want
going forward).
This issue is about making a patch that adds a lag between recoverLease
invocations where the lag is related to dfs timeouts -- the hdfs-side dfs
timeout -- and optionally makes use of the isFileClosed API if it is
available (a facility that is not yet committed to a branch near you and
unlikely to be within your locality with a good while to come).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8344) Improve the assignment when node failures happen to choose the secondary RS as the new primary RS
[ https://issues.apache.org/jira/browse/HBASE-8344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665675#comment-13665675 ] Devaraj Das commented on HBASE-8344: bq. What's happens if we lose 2 nodes out of the ensemble of 3? Will we have the 3 region sets on the last one remaining? Nope. The last one will only be the current owner of the region and have one copy of the region files' blocks. On node failures the favored-nodes mapping is not updated (a comment is there in the code on this). The balancer, when it runs, would figure the best favored nodes for the regions. When the balancer figures the best nodes, the RegionServers in question would be made aware of the new mapping... Until the balancer runs, any RegionServer would continue to use the old favored nodes mapping for HDFS file creations and the NameNode in turn will allocate replica locations that are close (same rack for example) to the favored-nodes the RegionServer is passing as hints. Improve the assignment when node failures happen to choose the secondary RS as the new primary RS - Key: HBASE-8344 URL: https://issues.apache.org/jira/browse/HBASE-8344 Project: HBase Issue Type: Sub-task Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.95.2 Attachments: hbase-8344-1.txt, hbase-8344-2.1.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6222) Add per-KeyValue Security
[ https://issues.apache.org/jira/browse/HBASE-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665690#comment-13665690 ] Andrew Purtell commented on HBASE-6222: --- Should we commit the shadow column family based approach we have today and migrate later to something based on KV tags, or wait for tags proper? Add per-KeyValue Security - Key: HBASE-6222 URL: https://issues.apache.org/jira/browse/HBASE-6222 Project: HBase Issue Type: New Feature Components: security Affects Versions: 0.98.0, 0.95.2 Reporter: stack Assignee: Andrew Purtell Attachments: 6222-aclcf.patch, 6222.pdf, cell-acls-kv-tags-not-for-review.zip, HBaseCellRow-LevelSecurityDesignDoc.docx, HBaseCellRow-LevelSecurityPRD.docx Saw an interesting article: http://www.fiercegovernmentit.com/story/sasc-accumulo-language-pro-open-source-say-proponents/2012-06-14 The Senate Armed Services Committee version of the fiscal 2013 national defense authorization act (S. 3254) would require DoD agencies to foreswear the Accumulo NoSQL database after Sept. 30, 2013, unless the DoD CIO certifies that there exists either no viable commercial open source database with security features comparable to [Accumulo] (such as the HBase or Cassandra databases)... Not sure what a 'commercial open source database' is, and I'm not sure whats going on in the article, but tra-la-la'ing, if we had per-KeyValue 'security' like Accumulo's, we might put ourselves in the running for federal contributions? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665693#comment-13665693 ] Jonathan Hsieh commented on HBASE-8497: --- +1 Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch, HBASE-8497-v6.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8601) Make ROW bloom work w/ .META.
stack created HBASE-8601: Summary: Make ROW bloom work w/ .META. Key: HBASE-8601 URL: https://issues.apache.org/jira/browse/HBASE-8601 Project: HBase Issue Type: Bug Reporter: stack I just tried enabling ROW blooms globally but tests against meta were failing doing getClosestOrBefore. If I waited, they worked. Something odd is going on here. Would be good having blooms on on .META. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8601) Make ROW bloom work w/ .META.
[ https://issues.apache.org/jira/browse/HBASE-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8601: - Component/s: Performance Make ROW bloom work w/ .META. - Key: HBASE-8601 URL: https://issues.apache.org/jira/browse/HBASE-8601 Project: HBase Issue Type: Bug Components: Performance Reporter: stack I just tried enabling ROW blooms globally but tests against meta were failing doing getClosestOrBefore. If I waited, they worked. Something odd is going on here. Would be good having blooms on on .META. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665699#comment-13665699 ] Nicolas Liochon commented on HBASE-8449: bq. I do not follow Nicolas Liochon It is 15minutes at the moment (this is just a copy of what was there before). +value9/value That's 90 seconds not 900 :-) bq. Yeah, that would be a good TODO for later. All of your proposal seems for later rather than now. The master could call recoverLease with what we have now, so it's not fully true :-). Frankly, I feel more urgency on 0.94 (and the users don't have much flexibility on HDFS when they are in prod) than on 0.95. But anyway this patch is a step in the good direction, let's more forward. I will create a jira to implement my proposal once HDFS-4754 will have made it. +1 (with the 900s...) Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[
https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665706#comment-13665706
]
Hadoop QA commented on HBASE-8497:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584568/HBASE-8497-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestAccessController
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5802//console
This message is automatically generated.
Protobuf WAL also needs a trailer
--
Key: HBASE-8497
URL: https://issues.apache.org/jira/browse/HBASE-8497
Project: HBase
Issue Type: Sub-task
Components: Protobufs, wal
Affects Versions: 0.95.1
Reporter: Enis Soztutar
Assignee: Himanshu Vashishtha
Fix For: 0.98.0, 0.95.1
Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch,
HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch,
HBASE-8497-v6.patch
New Protobuf WAL has a header, but we will probably need a trailer as well,
reserved for later usage.
Right now, we can we just serialize an empty trailer, but putting more
metadata there, like range of sequence_id's, region names, table names etc
might be needed in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665709#comment-13665709 ] stack commented on HBASE-8449: -- oops. Thanks [~nkeywal]. Let me fix. Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8449: - Attachment: 8449v6.txt Address nkeywal find. Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt, 8449v6.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8014) Backport HBASE-6915 to 0.94.
[ https://issues.apache.org/jira/browse/HBASE-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665711#comment-13665711 ] Jean-Marc Spaggiari commented on HBASE-8014: Indeed, might be relevant to backport HBASE-6921 too... Mind to open a jira for that? Backport HBASE-6915 to 0.94. Key: HBASE-8014 URL: https://issues.apache.org/jira/browse/HBASE-8014 Project: HBase Issue Type: Bug Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Critical Fix For: 0.94.7 Attachments: HBASE-8014-v0-0.94.patch JDK 1.7 changed some data size. Goal of this JIRA is to backport HBASE-6915 to 0.94 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665715#comment-13665715
]
Elliott Clark commented on HBASE-8517:
--
Trying it on a cluster right now.
Stochastic Loadbalancer isn't finding steady state on real clusters
---
Key: HBASE-8517
URL: https://issues.apache.org/jira/browse/HBASE-8517
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.95.1
Reporter: Elliott Clark
Assignee: Elliott Clark
Priority: Blocker
Fix For: 0.95.1
Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch,
HBASE-8517-2.patch, HBASE-8517-3.patch, HBASE-8517-4.patch, HBASE-8517-5.patch
I have a cluster that runs IT tests. Last night after all tests were done I
noticed that the balancer was thrashing regions around.
The number of regions on each machine is not correct.
The balancer seems to value the cost of moving a region way too little.
{code}
2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 5367ms to try 8910
different iterations. Found a solution that moves 37 regions; Going from a
computed cost of 56.50254222730425 to a new cost of 11.214035466575254
2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4735ms to try 8910
different iterations. Found a solution that moves 38 regions; Going from a
computed cost of 56.612624531830996 to a new cost of 11.275763861636982
2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
computing new load balance plan. Computation took 4502ms to try 8910
different iterations. Found a solution that moves 39 regions; Going from a
computed cost of 56.50048461413552 to a new cost of 11.225352339003237
{code}
Each of those balancer runs were triggered when there was no load on the
cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8602) Ugly stack trace just because regionserver comes up before master....
stack created HBASE-8602:
Summary: Ugly stack trace just because regionserver comes up
before master
Key: HBASE-8602
URL: https://issues.apache.org/jira/browse/HBASE-8602
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 8602.txt
I see this in the logs on simple startup:
{code}
2013-03-20 10:20:00,990 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we are
up
com.google.protobuf.ServiceException: methodName=regionServerStartup
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:147)
at com.sun.proxy.$Proxy8.regionServerStartup(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1783)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:713)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is not
running yet
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is not
running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1855)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1317)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
... 4 more
2013-03-20 10:20:00,991 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
sleeping and then retrying.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8602) Ugly stack trace just because regionserver comes up before master....
[
https://issues.apache.org/jira/browse/HBASE-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8602:
-
Attachment: 8602.txt
Small change...
Ugly stack trace just because regionserver comes up before master
-
Key: HBASE-8602
URL: https://issues.apache.org/jira/browse/HBASE-8602
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 8602.txt
I see this in the logs on simple startup:
{code}
2013-03-20 10:20:00,990 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we
are up
com.google.protobuf.ServiceException: methodName=regionServerStartup
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:147)
at com.sun.proxy.$Proxy8.regionServerStartup(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1783)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:713)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1855)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1317)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
... 4 more
2013-03-20 10:20:00,991 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
sleeping and then retrying.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8602) Ugly stack trace just because regionserver comes up before master....
[
https://issues.apache.org/jira/browse/HBASE-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665724#comment-13665724
]
Enis Soztutar commented on HBASE-8602:
--
+1
Ugly stack trace just because regionserver comes up before master
-
Key: HBASE-8602
URL: https://issues.apache.org/jira/browse/HBASE-8602
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 8602.txt
I see this in the logs on simple startup:
{code}
2013-03-20 10:20:00,990 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we
are up
com.google.protobuf.ServiceException: methodName=regionServerStartup
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:147)
at com.sun.proxy.$Proxy8.regionServerStartup(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1783)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:713)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1855)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1317)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
... 4 more
2013-03-20 10:20:00,991 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
sleeping and then retrying.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[
https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665729#comment-13665729
]
Ted Yu commented on HBASE-8420:
---
{code}
+prefetchScanFuture.cancel(false);
+ }
+ prefetchScanFuture = null;
+ if (prefetchedResultSize 0) {
+globalPrefetchedResultSize.addAndGet(-prefetchedResultSize);
{code}
prefetchScanFuture.cancel() returns a boolean. I think the return value should
be checked so that we know whether prefetchedResultSize should be deducted from
globalPrefetchedResultSize.
{code}
+ return tmp; // Keep the prefetched results for late
{code}
'late' - 'later'
{code}
+// Reset rows for next prefetching
+this.rows = rows;
{code}
What if tmp.results.size + scanResult.results.size() this.rows ?
{code}
+Thread.currentThread().interrupt();
+throw new IOException(scan was interrupted, ie);
{code}
Please use InterruptedIOException above.
{code}
+} catch (LeaseStillHeldException e) {
+ LOG.warn(THIS SHOULD NOT HAPPEN, e);
{code}
Looks like the log should be at error level.
{code}
+class ScanResult {
{code}
Maybe the above class can reside in HRegionServer ?
Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
-
Key: HBASE-8420
URL: https://issues.apache.org/jira/browse/HBASE-8420
Project: HBase
Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch,
trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch,
trunk-8420_v4.patch, trunk-8420_v5.1.patch, trunk-8420_v5.patch,
trunk-8420_v6.patch
This should help scanner performance. We should have it in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8449) Refactor recoverLease retries and pauses informed by findings over in hbase-8389
[ https://issues.apache.org/jira/browse/HBASE-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665733#comment-13665733 ] Jimmy Xiang commented on HBASE-8449: +1. I tried to keep loading data to my cluster patched with v4, and killed regionserver/datanode randomly. I did see recoverLease in my hbase logs and some files took couple tries to recover. Refactor recoverLease retries and pauses informed by findings over in hbase-8389 Key: HBASE-8449 URL: https://issues.apache.org/jira/browse/HBASE-8449 Project: HBase Issue Type: Bug Components: Filesystem Integration Affects Versions: 0.94.7, 0.95.0 Reporter: stack Assignee: stack Priority: Critical Fix For: 0.95.1 Attachments: 8449.txt, 8449v2.txt, 8449v3.txt, 8449v4.txt, 8449v6.txt HBASE-8359 is an interesting issue that roams near and far. This issue is about making use of the findings handily summarized on the end of hbase-8359 which have it that trunk needs refactor around how it does its recoverLease handling (and that the patch committed against HBASE-8359 is not what we want going forward). This issue is about making a patch that adds a lag between recoverLease invocations where the lag is related to dfs timeouts -- the hdfs-side dfs timeout -- and optionally makes use of the isFileClosed API if it is available (a facility that is not yet committed to a branch near you and unlikely to be within your locality with a good while to come). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665737#comment-13665737 ] Ted Yu commented on HBASE-8497: --- Integrated to 0.95 and trunk. Thanks for the patch, Himanshu. Thanks for the reviews, Sergey and Jon. Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch, HBASE-8497-v6.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster
[
https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13665738#comment-13665738
]
Hadoop QA commented on HBASE-8522:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584571/8522-trunk-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5803//console
This message is automatically generated.
Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner,
LogCleaner in HMaster
--
Key: HBASE-8522
URL: https://issues.apache.org/jira/browse/HBASE-8522
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.3
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 0.95.2, 0.94.9
Attachments: 8522-trunk-v3.patch, HBASE-8522-0.94.patch,
HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch,
HBASE-8522-trunk-v2.patch
TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in
hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the
hfile to determine if it should be deleted. But, the modify time of the hdfs
file is time when its writer is closed. The rename op will not change the
modify time of the hfile. So the hfile may be deleted immediatly by
HFileCleaner after it is moved to archives. See log:
2013-05-08 08:15:46,053 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory:
hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc
2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call:
getFileInfo took 1ms
2013-05-08 08:15:46,055 DEBUG
org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567,
ttl:30, current:1367972146054, from: 1367932112487
2013-05-08 08:15:46,055 DEBUG
org.apache.hadoop.hbase.master.cleaner.CleanerChore:
Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc
from archive
The same to old hlogs.
And my solution is very simple: When hfiles and hlogs are archived, we set
the modify time of files after rename.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your
[jira] [Updated] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8497: -- Hadoop Flags: Reviewed Protobuf WAL also needs a trailer -- Key: HBASE-8497 URL: https://issues.apache.org/jira/browse/HBASE-8497 Project: HBase Issue Type: Sub-task Components: Protobufs, wal Affects Versions: 0.95.1 Reporter: Enis Soztutar Assignee: Himanshu Vashishtha Fix For: 0.98.0, 0.95.1 Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch, HBASE-8497-v3.patch, HBASE-8497-v4.patch, HBASE-8497-v5.patch, HBASE-8497-v6.patch New Protobuf WAL has a header, but we will probably need a trailer as well, reserved for later usage. Right now, we can we just serialize an empty trailer, but putting more metadata there, like range of sequence_id's, region names, table names etc might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8602) Ugly stack trace just because regionserver comes up before master....
[
https://issues.apache.org/jira/browse/HBASE-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8602:
-
Attachment: 8602.trunk.txt
What I committed to trunk.
Ugly stack trace just because regionserver comes up before master
-
Key: HBASE-8602
URL: https://issues.apache.org/jira/browse/HBASE-8602
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 8602.trunk.txt, 8602.txt
I see this in the logs on simple startup:
{code}
2013-03-20 10:20:00,990 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we
are up
com.google.protobuf.ServiceException: methodName=regionServerStartup
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:147)
at com.sun.proxy.$Proxy8.regionServerStartup(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1783)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:713)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1855)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1317)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
... 4 more
2013-03-20 10:20:00,991 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
sleeping and then retrying.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8602) Ugly stack trace just because regionserver comes up before master....
[
https://issues.apache.org/jira/browse/HBASE-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-8602.
--
Resolution: Fixed
Fix Version/s: 0.95.1
0.98.0
Assignee: stack
Committed to 0.95 and trunk.
Ugly stack trace just because regionserver comes up before master
-
Key: HBASE-8602
URL: https://issues.apache.org/jira/browse/HBASE-8602
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Fix For: 0.98.0, 0.95.1
Attachments: 8602.trunk.txt, 8602.txt
I see this in the logs on simple startup:
{code}
2013-03-20 10:20:00,990 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: error telling master we
are up
com.google.protobuf.ServiceException: methodName=regionServerStartup
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:147)
at com.sun.proxy.$Proxy8.regionServerStartup(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1783)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:713)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146)
... 4 more
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException:
org.apache.hadoop.hbase.exceptions.ServerNotRunningYetException: Server is
not running yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1855)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1317)
at
org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131)
... 4 more
2013-03-20 10:20:00,991 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
sleeping and then retrying.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8522: -- Summary: Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster (was: Archived hfiles and old hlogs may be deleted immediatly by HFileCleaner, LogCleaner in HMaster) Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster --- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.95.2, 0.94.9 Attachments: 8522-trunk-v3.patch, HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8522: -- Fix Version/s: (was: 0.95.2) 0.95.1 0.98.0 Hadoop Flags: Reviewed Integrated to 0.94, 0.95 and trunk. Thanks for the patch, Shaohui. Thanks for the review, Matteo. Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster --- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.95.1, 0.94.9 Attachments: 8522-trunk-v3.patch, HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8522) Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster
[ https://issues.apache.org/jira/browse/HBASE-8522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8522: -- Resolution: Fixed Status: Resolved (was: Patch Available) Archived hfiles and old hlogs may be deleted immediately by HFileCleaner, LogCleaner in HMaster --- Key: HBASE-8522 URL: https://issues.apache.org/jira/browse/HBASE-8522 Project: HBase Issue Type: Bug Affects Versions: 0.94.3 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.98.0, 0.95.1, 0.94.9 Attachments: 8522-trunk-v3.patch, HBASE-8522-0.94.patch, HBASE-8522-0.94-v2.patch, HBASE-8522-trunk.patch, HBASE-8522-trunk-v2.patch, HBASE-8522-trunk-v2.patch TimeToLiveHFileCleaner is configed to 'hbase.master.hfilecleaner.plugins' in hbase-default.xml. And timeToLiveHFileCleaner uses the modify time of the hfile to determine if it should be deleted. But, the modify time of the hdfs file is time when its writer is closed. The rename op will not change the modify time of the hfile. So the hfile may be deleted immediatly by HFileCleaner after it is moved to archives. See log: 2013-05-08 08:15:46,053 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Checking directory: hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine: Call: getFileInfo took 1ms 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner: Life:40033567, ttl:30, current:1367972146054, from: 1367932112487 2013-05-08 08:15:46,055 DEBUG org.apache.hadoop.hbase.master.cleaner.CleanerChore: Removing:hdfs://hbase/.archive/table/4e48ffc1ec089082c66e6d1b5f018fb5/M/729e8bc1430540cb9b2c147c90039cdc from archive The same to old hlogs. And my solution is very simple: When hfiles and hlogs are archived, we set the modify time of files after rename. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
