[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache
[ https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114883#comment-13114883 ] jirapos...@reviews.apache.org commented on HBASE-4482: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2051/ --- (Updated 2011-09-26 19:17:34.041967) Review request for hbase. Summary (updated) --- Bugfix, kills a race condition. Ignore r1, thats the wrong patch. This addresses bug HBASE-4482. https://issues.apache.org/jira/browse/HBASE-4482 Diffs - src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 3798a06 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java 91b1603 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java e021780 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 8dd5159 Diff: https://reviews.apache.org/r/2051/diff Testing --- Looped tests. Thanks, Li Race Condition Concerning Eviction in SlabCache --- Key: HBASE-4482 URL: https://issues.apache.org/jira/browse/HBASE-4482 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: hbase-4482v1.txt, hbase-4482v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache
[ https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114884#comment-13114884 ] jirapos...@reviews.apache.org commented on HBASE-4482: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2051/ --- (Updated 2011-09-26 19:16:31.129009) Review request for hbase. Changes --- patch v2. Summary --- Bugfix, kills a race condition. This addresses bug HBASE-4482. https://issues.apache.org/jira/browse/HBASE-4482 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 3798a06 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java 91b1603 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java e021780 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 8dd5159 Diff: https://reviews.apache.org/r/2051/diff Testing --- Looped tests. Thanks, Li Race Condition Concerning Eviction in SlabCache --- Key: HBASE-4482 URL: https://issues.apache.org/jira/browse/HBASE-4482 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: hbase-4482v1.txt, hbase-4482v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache
[ https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114885#comment-13114885 ] jirapos...@reviews.apache.org commented on HBASE-4482: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2051/ --- Review request for hbase. Summary --- Bugfix, kills a race condition. This addresses bug HBASE-4482. https://issues.apache.org/jira/browse/HBASE-4482 Diffs - src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fe25a45 Diff: https://reviews.apache.org/r/2051/diff Testing --- Looped tests. Thanks, Li Race Condition Concerning Eviction in SlabCache --- Key: HBASE-4482 URL: https://issues.apache.org/jira/browse/HBASE-4482 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: hbase-4482v1.txt, hbase-4482v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache
[ https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114880#comment-13114880 ] jirapos...@reviews.apache.org commented on HBASE-4482: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2051/#review2070 --- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/2051/#comment4666 This log would be expensive. - Ted On 2011-09-26 19:17:34, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2051/ bq. --- bq. bq. (Updated 2011-09-26 19:17:34) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Bugfix, kills a race condition. bq. bq. Ignore r1, thats the wrong patch. bq. bq. bq. This addresses bug HBASE-4482. bq. https://issues.apache.org/jira/browse/HBASE-4482 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 3798a06 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java 91b1603 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java e021780 bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 8dd5159 bq. bq. Diff: https://reviews.apache.org/r/2051/diff bq. bq. bq. Testing bq. --- bq. bq. Looped tests. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Race Condition Concerning Eviction in SlabCache --- Key: HBASE-4482 URL: https://issues.apache.org/jira/browse/HBASE-4482 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: hbase-4482v1.txt, hbase-4482v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache
[ https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114887#comment-13114887 ] jirapos...@reviews.apache.org commented on HBASE-4482: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2051/ --- (Updated 2011-09-26 19:16:53.815702) Review request for hbase. Changes --- patch v3 - ted yu wanted to compare between the two. Summary --- Bugfix, kills a race condition. This addresses bug HBASE-4482. https://issues.apache.org/jira/browse/HBASE-4482 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 3798a06 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java 91b1603 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java e021780 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 8dd5159 Diff: https://reviews.apache.org/r/2051/diff Testing --- Looped tests. Thanks, Li Race Condition Concerning Eviction in SlabCache --- Key: HBASE-4482 URL: https://issues.apache.org/jira/browse/HBASE-4482 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: hbase-4482v1.txt, hbase-4482v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4482) Race Condition Concerning Eviction in SlabCache
[ https://issues.apache.org/jira/browse/HBASE-4482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13114888#comment-13114888 ] jirapos...@reviews.apache.org commented on HBASE-4482: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2051/#review2071 --- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/2051/#comment4667 Can you comment here on the change? And should this use LOG instead of System.out? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/2051/#comment4668 add a reference to the JIRA # in this comment... and break this to two lines. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java https://reviews.apache.org/r/2051/#comment4669 remove from javadoc src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/2051/#comment4670 Do we need this? At the least should be a DEBUG (seems like RS logs will be filled with this though, is that intended?) - Jonathan On 2011-09-26 19:17:34, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2051/ bq. --- bq. bq. (Updated 2011-09-26 19:17:34) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Bugfix, kills a race condition. bq. bq. Ignore r1, thats the wrong patch. bq. bq. bq. This addresses bug HBASE-4482. bq. https://issues.apache.org/jira/browse/HBASE-4482 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 3798a06 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java fe8b95a bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java 91b1603 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java 0814f41 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java e021780 bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java 8dd5159 bq. bq. Diff: https://reviews.apache.org/r/2051/diff bq. bq. bq. Testing bq. --- bq. bq. Looped tests. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Race Condition Concerning Eviction in SlabCache --- Key: HBASE-4482 URL: https://issues.apache.org/jira/browse/HBASE-4482 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: hbase-4482v1.txt, hbase-4482v2.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115391#comment-13115391 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2083 --- Thanks for the cleanup. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4725 Back to future :-) src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4727 Nice. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4726 Extra so. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4730 Do we need to update now and pass to the helper methods ? The helper methods can easily figure out what now should be. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4729 I think would be enough. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4731 I am confused by the condition here. - Ted On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public to package private. bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed. bq.(getMetaServerConnection) Change from public to package private. bq.Use MetaReader to read the meta location in root rather than a bq.raw HRegionInterface so we get retrying. bq.(remaining, timedout) Added utility methods. bq.(waitForMetaServer) Changed from public to private. bq.(resetMetaLocation) Made it synchronized on metaAvailable. bq.Not all accesses were synchronized. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java bq.Refactor to use HTable instead of raw HRegionInterface so we get bq.retrying. For each operation we get an HTable, use it, then close it. bq.(putToMetaTable, putsToMetaTable, etc) Utility methods. bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own bq.class since these classes are for a one-time migration only. bq. bq. A
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115394#comment-13115394 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2084 --- src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4732 Log should be changed accordingly. - Ted On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public to package private. bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed. bq.(getMetaServerConnection) Change from public to package private. bq.Use MetaReader to read the meta location in root rather than a bq.raw HRegionInterface so we get retrying. bq.(remaining, timedout) Added utility methods. bq.(waitForMetaServer) Changed from public to private. bq.(resetMetaLocation) Made it synchronized on metaAvailable. bq.Not all accesses were synchronized. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java bq.Refactor to use HTable instead of raw HRegionInterface so we get bq.retrying. For each operation we get an HTable, use it, then close it. bq.(putToMetaTable, putsToMetaTable, etc) Utility methods. bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own bq.class since these classes are for a one-time migration only. bq. bq. A src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java bq.New class that holds all Meta* methods updating meta table used bq.doing the one-time migration done to meta on startup. This class bq.is marked deprecated because its going to be dropped in 0.94. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java bq.Retrofit methods in here to use fullScan methods with Visitor. bq.(getCatalogRegionInterface, getCatalogRegionNameForTable, bq. getCatalogRegionNameForRegion) Removed. bq.(fullScan) Cleaned up the fullScans. Fixed up wrong javadoc. bq.(fullScanOfResults) Renamed as fullScan override. bq.(fullScanOfRoot) Added as deprecated. We should be doing bq.this against zk. bq.(metaRowToRegionPair, getServerNameFromResult)
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115592#comment-13115592 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2097 --- src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4761 I was trying to minimize how many times we do System.currentTimeMillis src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4762 Agreed src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4764 Will fix w/ a comment. If timeout is 0, then we do not timeout. I should call it out explicitly. - Michael On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public to package private. bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed. bq.(getMetaServerConnection) Change from public to package private. bq.Use MetaReader to read the meta location in root rather than a bq.raw HRegionInterface so we get retrying. bq.(remaining, timedout) Added utility methods. bq.(waitForMetaServer) Changed from public to private. bq.(resetMetaLocation) Made it synchronized on metaAvailable. bq.Not all accesses were synchronized. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java bq.Refactor to use HTable instead of raw HRegionInterface so we get bq.retrying. For each operation we get an HTable, use it, then close it. bq.(putToMetaTable, putsToMetaTable, etc) Utility methods. bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own bq.class since these classes are for a one-time migration only. bq. bq. A src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java bq.New class that holds all Meta* methods updating meta table used bq.doing the one-time migration done to meta on startup. This class bq.is marked deprecated because its going to be dropped in 0.94. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java bq.Retrofit methods in here to use fullScan methods with
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115708#comment-13115708 ] jirapos...@reviews.apache.org commented on HBASE-3025: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2041/#review2077 --- Looks good. The majority of my comments have to do with inconsistent logging practice. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlFilter.java https://reviews.apache.org/r/2041/#comment4718 Could be stated better. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlLists.java https://reviews.apache.org/r/2041/#comment4719 No. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlLists.java https://reviews.apache.org/r/2041/#comment4720 Comment needs updating. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4721 Can we make this 1? security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4775 Debug logging should go to LOG not AUDITLOG security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4782 Should be INFO or TRACE level? TRACE makes more sense to me. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4776 Debug logging should go to LOG not AUDITLOG security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4779 Should be INFO or TRACE level? TRACE makes more sense to me. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4795 Should something go to AUDITLOG here? security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4797 Should hasFamilyQualifierPermission log to AUDITLOG? It is used in places to make decisions -- an exception is thrown directly or not. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4783 Another one of these was sent to AUDITLOG above. Do the same here? Should be INFO or TRACE level? TRACE makes more sense to me. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4786 Ultimately users should be allowed to enable or disable their own tables, but only after such operations don't carry as much systemic risk as they do currently. In that case, CREATE permission and an ownership check could follow the test for ADMIN permission. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4787 As above security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4791 Should be logged with ERROR? security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4799 Would it be clearer then to call permissionGranted() something like hasColumnsPermission() ? Just a random thought. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4803 Should this go to AUDITLOG? At INFO or TRACE level? My preference is TRACE. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java https://reviews.apache.org/r/2041/#comment4804 Should this go to AUDITLOG? At INFO or TRACE level? My preference is TRACE. security/src/main/java/org/apache/hadoop/hbase/security/rbac/Permission.java https://reviews.apache.org/r/2041/#comment4807 What if instead we check for version 0 and throw an IllegalArgumentException if so? Technically, it is an invalid request if it contains an unrecognizable action code. Skipping this check if version 0 would be a way to handle new perms while not accepting incorrect input otherwise. security/src/main/java/org/apache/hadoop/hbase/security/rbac/TableAuthManager.java https://reviews.apache.org/r/2041/#comment4813 Maybe we can call this .auth.? We don't really have an RBAC implementation yet. Likewise for the package name for all of this stuff? Just a random thought. security/src/main/java/org/apache/hadoop/hbase/security/rbac/TableAuthManager.java https://reviews.apache.org/r/2041/#comment4815 Isn't this an error?
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115728#comment-13115728 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2099 --- src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4835 The condition, now stopTime, is reversed for isTimedOut(). - Ted On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public to package private. bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed. bq.(getMetaServerConnection) Change from public to package private. bq.Use MetaReader to read the meta location in root rather than a bq.raw HRegionInterface so we get retrying. bq.(remaining, timedout) Added utility methods. bq.(waitForMetaServer) Changed from public to private. bq.(resetMetaLocation) Made it synchronized on metaAvailable. bq.Not all accesses were synchronized. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java bq.Refactor to use HTable instead of raw HRegionInterface so we get bq.retrying. For each operation we get an HTable, use it, then close it. bq.(putToMetaTable, putsToMetaTable, etc) Utility methods. bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own bq.class since these classes are for a one-time migration only. bq. bq. A src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java bq.New class that holds all Meta* methods updating meta table used bq.doing the one-time migration done to meta on startup. This class bq.is marked deprecated because its going to be dropped in 0.94. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java bq.Retrofit methods in here to use fullScan methods with Visitor. bq.(getCatalogRegionInterface, getCatalogRegionNameForTable, bq. getCatalogRegionNameForRegion) Removed. bq.(fullScan) Cleaned up the fullScans. Fixed up wrong javadoc. bq.(fullScanOfResults) Renamed as fullScan override. bq.(fullScanOfRoot) Added as deprecated. We should be doing bq.this against zk. bq.(metaRowToRegionPair,
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115804#comment-13115804 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2106 --- Only part way done, will finish in the afternoon. I like the idea though, good stuff stack. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4855 Supposed to read When meta is moved to zk? src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4857 this comment talks a lot about what is wrong but it's not clear to me what changes are actually made right now. i see you say server-side only, but what do you propose instead? (i imagine i will find out reading the rest of the diff) src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4858 update this javadoc a bit... it's missing conf and you might also add additional context to stuff like abortable (which now appears optional and falls back to the connection itself?) src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4859 when would one override this? src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4860 is the behavior of this method unchanged? i guess now it returns before it's verified? any specific reason for the name change? (its behavior is definitely different from the old getRootServerConnection()). is it to match the Meta method? src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4861 i thought no verification in CT? src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4862 this is public but one with specified timeout is private now? src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4863 eeek, good catch - Jonathan On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115818#comment-13115818 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2107 --- Thanks for reviews Ted and Jon. Will put up new patch when you fellas finish... src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4864 @Jon True. I opened another issue with suggested fix. I should at least reference it in here. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4865 Nah. Thats TODO. - Michael On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public to package private. bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed. bq.(getMetaServerConnection) Change from public to package private. bq.Use MetaReader to read the meta location in root rather than a bq.raw HRegionInterface so we get retrying. bq.(remaining, timedout) Added utility methods. bq.(waitForMetaServer) Changed from public to private. bq.(resetMetaLocation) Made it synchronized on metaAvailable. bq.Not all accesses were synchronized. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java bq.Refactor to use HTable instead of raw HRegionInterface so we get bq.retrying. For each operation we get an HTable, use it, then close it. bq.(putToMetaTable, putsToMetaTable, etc) Utility methods. bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own bq.class since these classes are for a one-time migration only. bq. bq. A src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java bq.New class that holds all Meta* methods updating meta table used bq.doing the one-time migration done to meta on startup. This class bq.is marked deprecated because its going to be dropped in 0.94. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java bq.Retrofit methods in here to use fullScan methods with Visitor. bq.(getCatalogRegionInterface, getCatalogRegionNameForTable, bq.
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115868#comment-13115868 ] jirapos...@reviews.apache.org commented on HBASE-3025: -- bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 98 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line98 bq. bq. Can we make this 1? sure bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 192 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line192 bq. bq. Debug logging should go to LOG not AUDITLOG The idea was that all authorization decisions should be separated into audit log. Here we're allowing access, so AUDITLOG seemed to make sense. I agree that this still needs to be cleaned up a lot. Maybe all audit logging should be done up in requirePermission() with authorization result? At the very least we need a consistent format and consistent logging levels for messages (trace, right?). bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 200 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line200 bq. bq. Should be INFO or TRACE level? TRACE makes more sense to me. Sure, can use trace for all audit log decisions. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 208 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line208 bq. bq. Debug logging should go to LOG not AUDITLOG This is an authorization decision since we're returning true below. We can make this trace level, and improve the format, but I think AUDITLOG (if enabled) should contain a single message per request on why the request was allowed or denied. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 274 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line274 bq. bq. Should be INFO or TRACE level? TRACE makes more sense to me. will change to trace. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 354 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line354 bq. bq. Should something go to AUDITLOG here? Failure should already have been recorded in AUDITLOG via logDenied(). Agree that moving AUDITLOG messages up here with consistent format would be clearer, but will require some restructuring of return value from permissionGranted() so that some context specific reason can be pulled back up for logging. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 366 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line366 bq. bq. Should hasFamilyQualifierPermission log to AUDITLOG? It is used in places to make decisions -- an exception is thrown directly or not. Yes, agree, we should either log to AUDITLOG at decision points here or consistently move the AUDITLOG logging up a level out of permissionGranted() and hasFamilyQualifierPermission(). bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 375 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line375 bq. bq. Another one of these was sent to AUDITLOG above. Do the same here? Should be INFO or TRACE level? TRACE makes more sense to me. Agree, should go to AUDITLOG at trace. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 590 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line590 bq. bq. Should be logged with ERROR? sure bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 856 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line856 bq. bq. Should this go to AUDITLOG? At INFO or TRACE level? My preference is TRACE. Yes, agree. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/Permission.java, line 174 bq. https://reviews.apache.org/r/2041/diff/1/?file=45406#file45406line174 bq. bq. What if instead we check for version 0 and throw an IllegalArgumentException if so?
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115869#comment-13115869 ] jirapos...@reviews.apache.org commented on HBASE-3025: -- bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. Looks good. The majority of my comments have to do with inconsistent logging practice. Thanks for the review. I'll post an update with some cleanups and some reworking of the AUDITLOG handling. - Gary --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2041/#review2077 --- On 2011-09-23 19:14:20, Gary Helmling wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2041/ bq. --- bq. bq. (Updated 2011-09-23 19:14:20) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This patch implements access control list based authorization of HBase operations. The patch depends on the currently posted patch for HBASE-2742 (secure RPC engine). bq. bq. Key parts of the implementation are: bq. bq. * AccessControlLists - encapsulates storage of permission grants in a metadata table (_acl_). This differs from previous implementation where the .META. table was used to store permissions. bq. bq. * AccessController - bq.- implements MasterObserver and RegionObserver, performing authorization checks in each of the preXXX() hooks. If authorization fails, an AccessDeniedException is thrown. bq.- implements AccessControllerProtocol as a coprocessor endpoint to provide RPC methods for granting, revoking and listing permissions. bq. bq. * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries and updates throughout the cluster nodes using ZK. ACL entries are stored in per-table znodes as /hbase/acl/tablename. bq. bq. * Additional ruby shell scripts providing the grant, revoke and user_permission commands bq. bq. * Support for a new OWNER attribute in HTableDescriptor. I could separate out this change into a new JIRA for discussion, but I don't see it as currently useful outside of security. Alternately, I could handle the OWNER attribute completely in AccessController without changing HTD, but that would make interaction via hbase shell a bit uglier. bq. bq. bq. This addresses bug HBASE-3025. bq. https://issues.apache.org/jira/browse/HBASE-3025 bq. bq. bq. Diffs bq. - bq. bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlFilter.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControlLists.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessControllerProtocol.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/Permission.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/TableAuthManager.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/TablePermission.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/UserPermission.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/ZKPermissionWatcher.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/rbac/SecureTestUtil.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestAccessControlFilter.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestAccessController.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestTablePermissions.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/rbac/TestZKPermissionsWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 46a1a3d bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 699a5f5 bq.src/main/resources/hbase-default.xml 2c8f44b bq.src/main/ruby/hbase.rb 4d27191 bq.src/main/ruby/hbase/admin.rb b244ffe bq.src/main/ruby/hbase/hbase.rb beb2450 bq.src/main/ruby/hbase/security.rb PRE-CREATION bq.src/main/ruby/shell.rb 9a47600 bq.src/main/ruby/shell/commands.rb a352c2e bq.src/main/ruby/shell/commands/grant.rb PRE-CREATION bq.src/main/ruby/shell/commands/revoke.rb PRE-CREATION bq.src/main/ruby/shell/commands/table_permission.rb PRE-CREATION bq.
[jira] [Commented] (HBASE-4498) HBase RPM/DEB packages attempt to setup ZooKeeper environment incorrectly
[ https://issues.apache.org/jira/browse/HBASE-4498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115895#comment-13115895 ] jirapos...@reviews.apache.org commented on HBASE-4498: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2075/ --- Review request for hbase. Summary --- Updated ZOOKEEPER_HOME location to default to /usr (default location of ZooKeeper rpm/deb package) This addresses bug HBASE-4498. https://issues.apache.org/jira/browse/HBASE-4498 Diffs - /src/packages/update-hbase-env.sh 1176602 Diff: https://reviews.apache.org/r/2075/diff Testing --- Thanks, Eric HBase RPM/DEB packages attempt to setup ZooKeeper environment incorrectly - Key: HBASE-4498 URL: https://issues.apache.org/jira/browse/HBASE-4498 Project: HBase Issue Type: Bug Components: build, scripts Affects Versions: 0.92.0 Environment: Java, Linux Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.92.0 Attachments: HBASE-4498.patch HBase RPM packaging was done prior to completion of ZooKeeper RPM packaging. In update-hbase-env.sh, it expects ZooKeeper environment script to exist in /etc/default/zookeeper-env.sh. After several revision of ZOOKEEPER-999, it was decided to remove /etc/default/zookeeper-env.sh by ZooKeeper community. Hence, update-hbase-env.sh should not depend on /etc/default/zookeeper-env.sh. Instead, update-hbase-env.sh should assume ZooKeeper exists in /usr for RPM/deb packages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13115897#comment-13115897 ] jirapos...@reviews.apache.org commented on HBASE-3025: -- bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/AccessController.java, line 192 bq. https://reviews.apache.org/r/2041/diff/1/?file=45404#file45404line192 bq. bq. Debug logging should go to LOG not AUDITLOG bq. bq. Gary Helmling wrote: bq. The idea was that all authorization decisions should be separated into audit log. Here we're allowing access, so AUDITLOG seemed to make sense. I agree that this still needs to be cleaned up a lot. Maybe all audit logging should be done up in requirePermission() with authorization result? At the very least we need a consistent format and consistent logging levels for messages (trace, right?). bq. Maybe all audit logging should be done up in requirePermission() with authorization result? Sounds good. bq. At the very least we need a consistent format and consistent logging levels for messages (trace, right?). I'd argue for TRACE bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/TableAuthManager.java, line 47 bq. https://reviews.apache.org/r/2041/diff/1/?file=45407#file45407line47 bq. bq. Maybe we can call this .auth.? We don't really have an RBAC implementation yet. Likewise for the package name for all of this stuff? Just a random thought. bq. bq. Gary Helmling wrote: bq. Yeah rbac here and in package name is a misnomer. How about using access instead? auth seems ambiguous to me as it could mean authentication or authorization. JDK uses auth in javax.security.auth and claims it's for both, but seems like that and sub-packages are more authentication related to me. Hadoop uses authorize for a similar package to this. access sounds good to me. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/TableAuthManager.java, line 84 bq. https://reviews.apache.org/r/2041/diff/1/?file=45407#file45407line84 bq. bq. Isn't this an error? bq. bq. Gary Helmling wrote: bq. Yes, and in this context a pretty bad one, as it probably means region server initiated RPCs won't work or will be denied. We should probably let the IOE escape here... Agree. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/Permission.java, line 174 bq. https://reviews.apache.org/r/2041/diff/1/?file=45406#file45406line174 bq. bq. What if instead we check for version 0 and throw an IllegalArgumentException if so? Technically, it is an invalid request if it contains an unrecognizable action code. Skipping this check if version 0 would be a way to handle new perms while not accepting incorrect input otherwise. bq. bq. Gary Helmling wrote: bq. Yeah, seems safer to throw an exception here than to ignore invalid input. What about throwing an IOException (to tie in to existing error handling)? bq. bq. We could potentially trap the VersionMismatchException from VersionedWritable to allow skip and continue when reading newer versions of Permission with potentially added Action codes. Would need to think about what kind of errors that would expose us to. bq. What about throwing an IOException (to tie in to existing error handling)? Throwing an IOE sounds good. bq. We could potentially trap the VersionMismatchException from VersionedWritable to allow skip and continue when reading newer versions of Permission with potentially added Action codes. I think that is reasonable, with something logged at WARN level. The idea here is to ride over a rolling restart. Would not see long term operation with mismatching versions. bq. On 2011-09-27 16:58:47, Andrew Purtell wrote: bq. security/src/main/java/org/apache/hadoop/hbase/security/rbac/ZKPermissionWatcher.java, line 59 bq. https://reviews.apache.org/r/2041/diff/1/?file=45410#file45410line59 bq. bq. I wonder if there is some way we can check if a secure variant of ZooKeeper is running, and refuse to initialize if not. bq. bq. Gary Helmling wrote: bq. My thinking has been to handle all secure ZooKeeper changes separately. So I'd prefer to handle any check here as part of that. bq. bq. I do think it's reasonable to run AccessController with only SIMPLE auth and no secure ZooKeeper. It's not secure but could still be useful (we currently use this setup for tests). bq. bq. We could complain loudly to give an indication that you have a security hole though. bq. I do think it's reasonable to run AccessController with only SIMPLE auth and
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116008#comment-13116008 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2111 --- There is a lot of excellence in here. I'm going to look at the code itself with this diff applied to try and understand where/how CT is now being used. I'm a little unclear between the lines you'd like to draw and the lines you actually draw in this diff. Great work! src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4873 maybe note here that you should not be synchronized on metaAvailable (and it will do so in the method)... the next method below is nicely clear in this regard src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/2065/#comment4874 verify the connection works, and also that the server is actually hosting the region we think it is... the comment makes me think this is looking up which server hosts the passed region but it's just verifying if we can connect to the server we think is hosting the region and verifies whether it's hosting it or not (so this fails if we can't connect or if the region is not on this server) src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java https://reviews.apache.org/r/2065/#comment4875 i'm still trying to understand exactly what you've changed and what is still a TODO, but this looks much nicer now! :) src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java https://reviews.apache.org/r/2065/#comment4876 same here! nice (old stuff looks ripe with race conditions) src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java https://reviews.apache.org/r/2065/#comment4877 missing copyright and year? src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java https://reviews.apache.org/r/2065/#comment4878 nice moving cruft to separate classes src/main/java/org/apache/hadoop/hbase/client/Result.java https://reviews.apache.org/r/2065/#comment4881 seems like this should be moved to static methods in a helper class rather than exposing to our client-side Result src/main/java/org/apache/hadoop/hbase/client/Result.java https://reviews.apache.org/r/2065/#comment4882 yeah, shouldn't this be in MetaReader or some such class? src/main/java/org/apache/hadoop/hbase/client/RetriesExhaustedException.java https://reviews.apache.org/r/2065/#comment4895 missed some whitespace src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java https://reviews.apache.org/r/2065/#comment4896 nice src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java https://reviews.apache.org/r/2065/#comment4897 this seems like an important public method. i like the rename and your additional comments, but maybe we should add more. default behavior is to use a cached location, if one is not found, it is looked up in a catalog. setting reload to true bypasses the cache and forces the lookup to a catalog. and then, under what cases do we get an exception? does this verify that the server is actually hosting the region? or it just looks up in the catalog (i guess failure there could cause IOE) and if it finds something, just returns a connection to that RS (w/ no verification)... correct? src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/2065/#comment4898 why do you remove the javadoc on this method? src/main/java/org/apache/hadoop/hbase/zookeeper/MetaNodeTracker.java https://reviews.apache.org/r/2065/#comment4899 not even necessary to put this method in here at all now (we're just using it for getting the node name at this point but it's probably still nice to have the name in stacks and such) src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java https://reviews.apache.org/r/2065/#comment4900 yay! 3 src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java https://reviews.apache.org/r/2065/#comment4901 huh? :) src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java https://reviews.apache.org/r/2065/#comment4902 awesome src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java https://reviews.apache.org/r/2065/#comment4903 30,000 ft desc? i guess test name is self descriptive? :) - Jonathan On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq.
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116019#comment-13116019 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. There is a lot of excellence in here. I'm going to look at the code itself with this diff applied to try and understand where/how CT is now being used. I'm a little unclear between the lines you'd like to draw and the lines you actually draw in this diff. bq. bq. Great work! Sorry about that. Let me get you better answer to your question. I think its not very clear because I myself was unclear on scope of CT when I started in. What this patch has here is an attempt at shutting down CT scope with subsequent work put off for HBASE-4495. bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 513 bq. https://reviews.apache.org/r/2065/diff/1/?file=45907#file45907line513 bq. bq. maybe note here that you should not be synchronized on metaAvailable (and it will do so in the method)... the next method below is nicely clear in this regard Will do bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 575 bq. https://reviews.apache.org/r/2065/diff/1/?file=45907#file45907line575 bq. bq. verify the connection works, and also that the server is actually hosting the region we think it is... the comment makes me think this is looking up which server hosts the passed region but it's just verifying if we can connect to the server we think is hosting the region and verifies whether it's hosting it or not (so this fails if we can't connect or if the region is not on this server) Good point. bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 194 bq. https://reviews.apache.org/r/2065/diff/1/?file=45908#file45908line194 bq. bq. i'm still trying to understand exactly what you've changed and what is still a TODO, but this looks much nicer now! :) In the above, we'd get the HRegionInterface and do the invocation on the actual Interface. The alternative steps back and asks an HTable instance to do the work. If an issue with former we'd just let the exception out. In the alternative, we'll do HTable retries before we let the exception out (and the retries are boosted in server-context). bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java, line 2 bq. https://reviews.apache.org/r/2065/diff/1/?file=45909#file45909line2 bq. bq. missing copyright and year? Turns out that copyright is not actually needed https://issues.apache.org/jira/browse/HBASE-3870 bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 568 bq. https://reviews.apache.org/r/2065/diff/1/?file=45915#file45915line568 bq. bq. seems like this should be moved to static methods in a helper class rather than exposing to our client-side Result OK. It was kinda nice being able to do result.getServerNameFromCatalogResult. I suppose it does pollute. I can move it back to MetaReader since that seems like next best place. You are right shouldn't be generally public stuff. Will fix. bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java, line 70 bq. https://reviews.apache.org/r/2065/diff/1/?file=45918#file45918line70 bq. bq. this seems like an important public method. i like the rename and your additional comments, but maybe we should add more. default behavior is to use a cached location, if one is not found, it is looked up in a catalog. setting reload to true bypasses the cache and forces the lookup to a catalog. and then, under what cases do we get an exception? does this verify that the server is actually hosting the region? or it just looks up in the catalog (i guess failure there could cause IOE) and if it finds something, just returns a connection to that RS (w/ no verification)... correct? Will look into this. bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java, lines 1041-1047 bq. https://reviews.apache.org/r/2065/diff/1/?file=45921#file45921line1041 bq. bq. why do you remove the javadoc on this method? Will look into this. bq. On 2011-09-27 23:36:00, Jonathan Gray wrote: bq. src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java, line 132 bq. https://reviews.apache.org/r/2065/diff/1/?file=45931#file45931line132 bq. bq. huh? :) Let me fix. bq. On 2011-09-27 23:36:00,
[jira] [Commented] (HBASE-3446) ProcessServerShutdown fails if META moves, orphaning lots of regions
[ https://issues.apache.org/jira/browse/HBASE-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116024#comment-13116024 ] jirapos...@reviews.apache.org commented on HBASE-3446: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2065/#review2124 --- src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java https://reviews.apache.org/r/2065/#comment4913 This doesn't seem right. - Ted On 2011-09-27 06:38:09, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2065/ bq. --- bq. bq. (Updated 2011-09-27 06:38:09) bq. bq. bq. Review request for hbase and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. Make the Meta* operations against meta retry. We do it by using HTable instances. bq. (HTable calls HConnection.getRegionServerWithRetries for get, put, scan etc). bq. In 0.89, we had special RetryableMetaOperation class that was a bq. subclass of Callable which reproduced the guts of HConnection.getRegionServerWithRetries bq. with its retry loop. Now we just use HTable instead (Costs some on setup but bq. otherwise, we avoid duplicating code). Upped the retries on serverside too. bq. bq. Had problem with CatalogJanitor. MetaReader and MetaEditor were relying bq. heavily on CT methods getting proxy connections to meta and root servers. bq. CT needs to be cut back. This patch closes down access on (unused) public bq. methods and removes being able to get an HRegionInterface on meta and root bq. -- this stuff is used internally to CT only now; use MetaEditor or bq. MetaReader if you want to update or read catalog tables. Opening new issue bq. to cutback CT use over the code base. bq. bq. A little off topic but couldn't help it since was in MetaReader and MetaEditor bq. trying to clean them up, I ended up moving meta migration code out to its bq. own class rather than have it in all inside in MetaEditor. bq. bq. Here is some detail to help reviews. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java bq.Clean up. Shutdown access on some of these unused methods. Don't bq.let out HRegionInterface instances in particular since we are going bq.away from raw HRI use to instead use a connection with retries: bq.i.e. HTable. bq. bq.Comments on state of this class. Javadoc edits. bq.getZooKeeperWatcher on HConnection is deprecated so don't use it bq.in constructor. Override MetaNodeTracker and on node delete bq.reset meta location (We used to do this over in MetaNodeTracker bq.but to do that we had to have a CatalogTracker over in zk package bq.which is silly -- bad package encapsulation). bq. bq.(waitForRootServer) Renamed getRootServerConnection and change it bq.from public to package private. bq.(waitForRootServerConnectionDefault, getRootServerConnection) Removed. bq.(getMetaServerConnection) Change from public to package private. bq.Use MetaReader to read the meta location in root rather than a bq.raw HRegionInterface so we get retrying. bq.(remaining, timedout) Added utility methods. bq.(waitForMetaServer) Changed from public to private. bq.(resetMetaLocation) Made it synchronized on metaAvailable. bq.Not all accesses were synchronized. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java bq.Refactor to use HTable instead of raw HRegionInterface so we get bq.retrying. For each operation we get an HTable, use it, then close it. bq.(putToMetaTable, putsToMetaTable, etc) Utility methods. bq.(updateRootWithMetaMigrationStatus, etc.) Moved out to own bq.class since these classes are for a one-time migration only. bq. bq. A src/main/java/org/apache/hadoop/hbase/catalog/MetaMigrationRemovingHTD.java bq.New class that holds all Meta* methods updating meta table used bq.doing the one-time migration done to meta on startup. This class bq.is marked deprecated because its going to be dropped in 0.94. bq. bq. M src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java bq.Retrofit methods in here to use fullScan methods with Visitor. bq.(getCatalogRegionInterface, getCatalogRegionNameForTable, bq. getCatalogRegionNameForRegion) Removed. bq.(fullScan) Cleaned up the fullScans. Fixed up wrong javadoc. bq.(fullScanOfResults) Renamed as fullScan override. bq.(fullScanOfRoot) Added as deprecated. We should be doing bq.this against zk. bq.(metaRowToRegionPair, getServerNameFromResult) Moved to Result
[jira] [Commented] (HBASE-4062) Multi-column scanner unit test
[ https://issues.apache.org/jira/browse/HBASE-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116075#comment-13116075 ] jirapos...@reviews.apache.org commented on HBASE-4062: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1012/ --- (Updated 2011-09-28 02:16:10.418401) Review request for hbase and Michael Stack. Changes --- Linking the HBASE-4062 JIRA to this review. Summary --- Adding a unit test for the multi-column scanner. We are using this this to test an optimization we are making to multi-column scans using row-column Bloom filters. The scanner creates multiple StoreFiles for a single column family, each containing a randomized set of columns with different timestamps, and then tests scanning through the whole region with all possible sets of columns specified in the query. Point deletes (deletes of a specific timestamp) are also tested. This addresses bug HBASE-4062. https://issues.apache.org/jira/browse/HBASE-4062 Diffs - src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java PRE-CREATION Diff: https://reviews.apache.org/r/1012/diff Testing --- Run the unit test. Break the store scanner and make sure the test breaks. Thanks, Mikhail Multi-column scanner unit test -- Key: HBASE-4062 URL: https://issues.apache.org/jira/browse/HBASE-4062 Project: HBase Issue Type: Improvement Components: regionserver, test Affects Versions: 0.94.0 Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Labels: test Attachments: test-multi-column-scanner.patch Adding a unit test for the multi-column scanner. We are using this this to test an optimization we are making to multi-column scans using row-column Bloom filters. The scanner creates multiple StoreFiles for a single column family, each containing a randomized set of columns with different timestamps, and then tests scanning through the whole region with all possible sets of columns specified in the query. Point deletes (deletes of a specific timestamp) are also tested. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116569#comment-13116569
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/
---
Review request for hbase.
Summary
---
Previously we only used row-column Bloom filters for scans that only requested
one column. We have seen production queries that request up to 200 columns, and
with say ~6 store files per store (region / column family combination) this
might have resulted in 1200 block read operations in the worst case. With this
diff we will be avoiding seeks on store files that we know don't contain the
row/column of interest when using an ExplicitColumnTracker. The performance
should remain the same for column range queries.
This addresses bug HBASE-2794.
https://issues.apache.org/jira/browse/HBASE-2794
Diffs
-
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ac2348e
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 9d9895c
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
Diff: https://reviews.apache.org/r/2084/diff
Testing
---
Existing unit tests. A new unit test (TestScanWithBloomError). Load testing
using HBaseTest.
Thanks,
Mikhail
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116578#comment-13116578
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2130
---
nice work mikhail! i will let someone else give the +1 though
src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2084/#comment4946
method doesn't actually take a KeyValue... this is to create the last KV
the on row and column for the KeyValue this is called on?
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
https://reviews.apache.org/r/2084/#comment4947
got it. maybe add a comment on this method to explain this usage
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
https://reviews.apache.org/r/2084/#comment4948
license
- Jonathan
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116588#comment-13116588 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/ --- (Updated 2011-09-28 16:35:57.691899) Review request for hbase. Changes --- Merge with latest trunk. Run unit tests couple more times. Summary --- 1. Collect client-side scan related metrics during scan operation. It is turned off by default. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. 3. Clean up some minor issues in tableInputFormat as well as test code. This addresses bug hbase-4145. https://issues.apache.org/jira/browse/hbase-4145 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScan.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 Diff: https://reviews.apache.org/r/1674/diff Testing --- 1. Verified on a small cluster. 2. Existing unit tests. 3. Added new tests. Thanks, Ming Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances. However, it might be good enough for now. With this approach, the metrics value will be available via MapReduce counter. a) Have ResultScanner return a new ResultScannerMetrics interface. b) TableInputFormat will access data from ResultScannerMetrics and populate MapReduce counters accordingly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more
[jira] [Commented] (HBASE-4506) [hbck] Allow HBaseFsck to be instantiated without connecting
[ https://issues.apache.org/jira/browse/HBASE-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116622#comment-13116622 ] jirapos...@reviews.apache.org commented on HBASE-4506: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2085/ --- Review request for hbase and Michael Stack. Summary --- commit d51a9fa5f3419114deca8ecd71f4f1ec4d2a6bc5 Author: Jonathan Hsieh j...@cloudera.com Date: Wed Sep 28 10:18:00 2011 -0700 HBASE-4506 [hbck] Allow HBaseFsck to be instantiated without connecting This is a semantics preserving patch that allows for offline meta rebuild (HBASE-4377) to reuse code in the existing hbck code when hbase is down. This addresses bug HBASE-4506. https://issues.apache.org/jira/browse/HBASE-4506 Diffs - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 Diff: https://reviews.apache.org/r/2085/diff Testing --- TestHBaseFsck passes. Thanks, jmhsieh [hbck] Allow HBaseFsck to be instantiated without connecting Key: HBASE-4506 URL: https://issues.apache.org/jira/browse/HBASE-4506 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4506-hbck-Allow-HBaseFsck-to-be-instantiated-w.patch This is a semantics preserving patch that allows for offline meta rebuild (HBASE-4377) to reuse code in the existing hbck code when hbase is down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116628#comment-13116628
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2137
---
This is an important feature.
Since the boolean parameter, forward, correlates so closely with reseek, can we
give it a better name ?
I was thinking about either reseek or forwardOnly.
- Ted
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116720#comment-13116720 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/ --- Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. Summary --- Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) This addresses bug HBASE-4422. https://issues.apache.org/jira/browse/HBASE-4422 Diffs - Diff: https://reviews.apache.org/r/2089/diff Testing --- Still working through some tests that aren't passing. Thanks, Jonathan Move block cache parameters and references into single CacheConf class -- Key: HBASE-4422 URL: https://issues.apache.org/jira/browse/HBASE-4422 Project: HBase Issue Type: Improvement Components: io Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.92.0 From StoreFile down to HFile, we currently use a boolean argument for each of the various block cache configuration parameters that exist. The number of parameters is going to continue to increase as we look at compressed cache, delta encoding, and more specific L1/L2 configuration. Every new config currently requires changing many constructors because it introduces a new boolean. We should move everything into a single class so that modifications are much less disruptive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116742#comment-13116742 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/ --- (Updated 2011-09-28 19:56:14.698336) Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. Changes --- Diff attached now. Summary --- Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) This addresses bug HBASE-4422. https://issues.apache.org/jira/browse/HBASE-4422 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java 1177030 /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 1177030 Diff: https://reviews.apache.org/r/2089/diff Testing --- Still working through some tests that aren't passing. Thanks, Jonathan Move block cache parameters and references into single CacheConf class -- Key: HBASE-4422 URL: https://issues.apache.org/jira/browse/HBASE-4422 Project: HBase Issue Type: Improvement Components: io Reporter: Jonathan Gray Assignee: Jonathan Gray Fix For: 0.92.0 From StoreFile down
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116884#comment-13116884 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/ --- (Updated 2011-09-28 23:03:54.523337) Review request for hbase. Changes --- Thanks, Todd. Rename counter name from COUNT_OF_RPC_CALLS to RPC_CALLS, etc. Summary --- 1. Collect client-side scan related metrics during scan operation. It is turned off by default. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. 3. Clean up some minor issues in tableInputFormat as well as test code. This addresses bug hbase-4145. https://issues.apache.org/jira/browse/hbase-4145 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 Diff: https://reviews.apache.org/r/1674/diff Testing --- 1. Verified on a small cluster. 2. Existing unit tests. 3. Added new tests. Thanks, Ming Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances. However, it might be good enough for now. With this approach, the metrics value will be available via MapReduce counter. a) Have ResultScanner return a new ResultScannerMetrics interface. b) TableInputFormat will access data from ResultScannerMetrics and populate MapReduce counters accordingly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116924#comment-13116924 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2145 --- RB couldn't handle the diffs to StoreFile, something weird there. I'll look at this again when the tests are working. /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java https://reviews.apache.org/r/2089/#comment4979 CacheConfig could extend Configuration? It's only extra constants and constructors, really. /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java https://reviews.apache.org/r/2089/#comment4980 See above - Andrew On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1177030 bq.
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116935#comment-13116935 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2147 --- Cool... This makes it much simpler in the future. Are you still planning to fold HBASE-4496 into this? /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java https://reviews.apache.org/r/2089/#comment4981 Hmm... The new code does not honor the cacheBlock flag that was passed into the method. /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java https://reviews.apache.org/r/2089/#comment4982 Same here. blockCache flag is not honored. - Lars On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1177030 bq.
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116985#comment-13116985 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2148 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4983 Should read 'can be easily' - Ted On 2011-09-28 23:03:54, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-28 23:03:54) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances.
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13116989#comment-13116989 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2149 --- Nice work. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4986 Should be declared as implementing VersionedWritable. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4984 It is a bit hard to read this counter. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4985 This is count of regions scanned, right ? If so, please name it that way. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4987 mb should be included in the exception. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4988 Why do we need this check again ? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4989 Value of version should be included here. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4990 I think we should have else block where the unsupported mb is logged. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java https://reviews.apache.org/r/1674/#comment4991 This name doesn't really match the constant above. I think HBase mapreduce Counters would be better. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java https://reviews.apache.org/r/1674/#comment4992 This should not be a tongue twister. How about naming it retrieveGetCounterWithStrings ? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java https://reviews.apache.org/r/1674/#comment4993 Shall we create the Object array outside the for loop and only fill in Metric name here ? - Ted On 2011-09-28 23:03:54, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-28 23:03:54) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942
[jira] [Commented] (HBASE-4510) HDFS-1620 related changes downstream (For compiling with HDFS 0.23+)
[ https://issues.apache.org/jira/browse/HBASE-4510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117022#comment-13117022 ] jirapos...@reviews.apache.org commented on HBASE-4510: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2108/ --- Review request for hbase. Summary --- HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: 1. We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) 2. HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. Way (2) seems more viable. I've attached an initial patch that doesn't fix all trouble yet, but introduces the first way of changes. The remaining issue lies in the use of DistributedFileSystem's safemode methods (which are private audience as well) inside of HBase for safemode waits and checks (via HBases' FSUtils class). Since this uses an enum, it is more difficult to handle without upstream interventions - thoughts? This addresses bug HBASE-4510. https://issues.apache.org/jira/browse/HBASE-4510 Diffs - src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 Diff: https://reviews.apache.org/r/2108/diff Testing --- Thanks, Harsh HDFS-1620 related changes downstream (For compiling with HDFS 0.23+) Key: HBASE-4510 URL: https://issues.apache.org/jira/browse/HBASE-4510 Project: HBase Issue Type: Task Affects Versions: 0.94.0 Reporter: Harsh J Assignee: Harsh J Priority: Blocker HBase isn't seemingly compiling anymore on 0.23 after the HDFS-1620 naming refactorings were carried out. Two solutions: * We use new classnames. This breaks HBase's backward compatibility with older Hadoop releases (is that a concern with future releases?) * HBase gets its own sets of constants as the upstream one is not marked for public usage. This needs a little more maintenance on HBases' side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4509) [hbck] Improve region map output
[ https://issues.apache.org/jira/browse/HBASE-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117093#comment-13117093 ] jirapos...@reviews.apache.org commented on HBASE-4509: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2111/ --- Review request for hbase, Todd Lipcon and Michael Stack. Summary --- This is on top of HBASE-4506 but doesn't require it. commit 17d3b8856a8e7c803463fb37b1bd4691d0a02ba9 Author: Jonathan Hsieh j...@cloudera.com Date: Wed Sep 28 20:32:02 2011 -0700 HBASE-4509 [hbck] Improve region map output Details: HBASE-4375 added a region coverage visualization to hbck in details mode. When users have binary row keys the output is difficult to parse (awk/sed) or pull into programs (numeric, excel) capable of handling tsv formatted data. This patch: * improves output by using Bytes.toStringBinary (which escapes binary) instead of Bytes.toString when printing keys, * suggests some repair actions, and * collects overlap group that groups regions that are overlapping. This addresses bug HBASE-4509. https://issues.apache.org/jira/browse/HBASE-4509 Diffs - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 Diff: https://reviews.apache.org/r/2111/diff Testing --- Updated unit test passes. Used on a live cluster to get better information when dianosing problem. Thanks, jmhsieh [hbck] Improve region map output Key: HBASE-4509 URL: https://issues.apache.org/jira/browse/HBASE-4509 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0, 0.94.0, 0.90.5 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4509-hbck-Improve-region-map-output.patch HBASE-4375 added a region coverage visualization to hbck in details mode. When users have binary row keys the output is difficult to parse (awk/sed) or pull into programs (numeric, excel) capable of handling tsv formatted data. This patch * improves output by using Bytes.toStringBinary (which escapes binary) instead of Bytes.toString when printing keys, * suggests some repair actions, and * collects problem group that groups regions that are overlapping. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117295#comment-13117295 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2150 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment4994 I think we need not give copyright information. - ramkrishna On 2011-09-28 23:03:54, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-28 23:03:54) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117456#comment-13117456 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-09-29 00:50:58, Andrew Purtell wrote: bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java, line 112 bq. https://reviews.apache.org/r/2089/diff/1/?file=46305#file46305line112 bq. bq. CacheConfig could extend Configuration? It's only extra constants and constructors, really. I considered doing that. But they seem orthogonal in most of their usage and I think it'd be weird to pass our Configuration all the way down into HFile reading and such. Changing config values dynamically (like disabling caching for a specific read) would then require changing the base Configuration or would the local booleans in the CacheConfig not match what's in the underlying Configuration? - Jonathan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2145 --- On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 bq.
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117458#comment-13117458 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-09-29 01:10:28, Lars Hofhansl wrote: bq. Cool... This makes it much simpler in the future. bq. Are you still planning to fold HBASE-4496 into this? Yes, I will look at that today. I'm thinking about a new test for all this stuff since this diff is riddled with bugs and it's a bit random which tests fail :) bq. On 2011-09-29 01:10:28, Lars Hofhansl wrote: bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java, line 244 bq. https://reviews.apache.org/r/2089/diff/1/?file=46310#file46310line244 bq. bq. Hmm... The new code does not honor the cacheBlock flag that was passed into the method. Yeah, this is the stuff I need to go back and review. Thanks for pointing me to the right place :) Will have a new diff up in a bit. - Jonathan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2147 --- On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177030 bq.
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117517#comment-13117517
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2161
---
src/main/java/org/apache/hadoop/hbase/KeyValue.java
https://reviews.apache.org/r/2084/#comment5035
I was implying that this is also a method argument when I wrote this
comment. I will edit this to make it clearer.
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
https://reviews.apache.org/r/2084/#comment5036
Yes, I will modify the javadoc of this method.
- Mikhail
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117523#comment-13117523
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
bq. On 2011-09-28 17:42:46, Ted Yu wrote:
bq. This is an important feature.
bq.
bq. Since the boolean parameter, forward, correlates so closely with reseek,
can we give it a better name ?
bq. I was thinking about either reseek or forwardOnly.
We have a few diffs in the pipeline that depend on this one. Can we rename the
boolean flag after we commit those diffs?
- Mikhail
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2137
---
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117524#comment-13117524
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
bq. On 2011-09-28 17:42:46, Ted Yu wrote:
bq. This is an important feature.
bq.
bq. Since the boolean parameter, forward, correlates so closely with reseek,
can we give it a better name ?
bq. I was thinking about either reseek or forwardOnly.
bq.
bq. Mikhail Bautin wrote:
bq. We have a few diffs in the pipeline that depend on this one. Can we
rename the boolean flag after we commit those diffs?
I am fine with the current name of forward.
- Ted
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2137
---
On 2011-09-28 16:03:52, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-28 16:03:52)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117546#comment-13117546 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- bq. On 2011-09-29 00:50:58, Andrew Purtell wrote: bq. /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java, line 112 bq. https://reviews.apache.org/r/2089/diff/1/?file=46305#file46305line112 bq. bq. CacheConfig could extend Configuration? It's only extra constants and constructors, really. bq. bq. Jonathan Gray wrote: bq. I considered doing that. But they seem orthogonal in most of their usage and I think it'd be weird to pass our Configuration all the way down into HFile reading and such. Changing config values dynamically (like disabling caching for a specific read) would then require changing the base Configuration or would the local booleans in the CacheConfig not match what's in the underlying Configuration? Fair enough, but it's a common pattern to mutate a Configuration object and then pass it along. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/#review2145 --- On 2011-09-28 19:56:14, Jonathan Gray wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2089/ bq. --- bq. bq. (Updated 2011-09-28 19:56:14) bq. bq. bq. Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. bq. bq. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) bq. bq. bq. This addresses bug HBASE-4422. bq. https://issues.apache.org/jira/browse/HBASE-4422 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177030 bq. /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177030 bq./src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177030 bq. /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177030 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177030 bq.
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117556#comment-13117556 ] jirapos...@reviews.apache.org commented on HBASE-4487: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2116/ --- Review request for hbase and Ted Yu. Summary --- The increment operation releases the rowlock before doing the sync to the HLog. This improves performance of increments on hot rows. Introduced method HLog.appendNoSync() that returns a txid. The increment method then release the rowlock and invokes HLog.sync(txid). The HLog.sync(txid) returns only if all the transactions upto the one identified by that txid has been successfully sycned to HDFS. There is a single test TestIncrement that creates a 100 threads and ensures that all the concurrent increments match the final expected result. There is a benchmark TestHLogBench that measures the performance of the appendNoSync call. This addresses bug HBASE-4487. https://issues.apache.org/jira/browse/HBASE-4487 Diffs - /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177401 /src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1177401 /src/test/java/org/apache/hadoop/hbase/regionserver/TestIncrement.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java 1177401 /src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 1177401 Diff: https://reviews.apache.org/r/2116/diff Testing --- All unit tests pass now (expect TestDistributedLogSplitting, TestRollingRestart, TestHTablePool), but I am seeing the same test to fail on trunk, so these failures do not seem to be related to this patch. Thanks, Dhruba The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117583#comment-13117583 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-29 20:15:12.832998) Review request for hbase and Mingjie Lai. Changes --- show both tests. Summary --- Proposed fix for HBASE-4070. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/main/ruby/hbase/admin.rb b244ffe src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c Diff: https://reviews.apache.org/r/2029/diff Testing (updated) --- Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117581#comment-13117581 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-29 20:13:32.064908) Review request for hbase and Mingjie Lai. Changes --- Remove relies on HBASE-4014: this is no longer true. Summary (updated) --- Proposed fix for HBASE-4070. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/main/ruby/hbase/admin.rb b244ffe src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c Diff: https://reviews.apache.org/r/2029/diff Testing --- One new test : testServerManagerCoprocessorReport, added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117574#comment-13117574 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-29 19:56:54.298361) Review request for hbase and Mingjie Lai. Changes --- Updated patch that shows both master as well as regionserver loaded coprocessors. Master UI shows both master and regionserver coprocessor information, while regionserver UI (/rs-status) shows regionserver coprocessor information. See JIRA for screenshots. Summary --- Proposed fix for HBASE-4070. Relies on functionality provided by HBASE-4014. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs (updated) - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/main/ruby/hbase/admin.rb b244ffe src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c Diff: https://reviews.apache.org/r/2029/diff Testing --- One new test : testServerManagerCoprocessorReport, added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117608#comment-13117608 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- bq. On 2011-09-29 13:33:05, ramkrishna vasudevan wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java, line 2 bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line2 bq. bq. I think we need not give copyright information. Fixed. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2150 --- On 2011-09-29 21:00:18, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:00:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117611#comment-13117611 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- bq. On 2011-09-29 03:48:24, Ted Yu wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java, line 43 bq. https://reviews.apache.org/r/1674/diff/3/?file=46268#file46268line43 bq. bq. Should read 'can be easily' Fixed. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2148 --- On 2011-09-29 21:00:18, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:00:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[
https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117609#comment-13117609
]
jirapos...@reviews.apache.org commented on HBASE-4145:
--
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 48
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line48
bq.
bq. Should be declared as implementing VersionedWritable.
The issue with VersionedWritable is it throws VersionMismatchException if the
version doesn't match.
public void readFields(DataInput in) throws IOException {
byte version = in.readByte(); // read version
if (version != getVersion())
throw new VersionMismatchException(getVersion(), version);
}
I want to make it backward compatible to support version = getVersion(). The
program could catch VersionMismatchException, however, there is no way to find
out the expectedVersion and foundVersion, given they are private members.
public class VersionMismatchException extends IOException {
private byte expectedVersion;
private byte foundVersion;
...
}
Any other suggestions, or Is it something that need to be fixed in
VersionedWritable, VersionMismatchException?
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 76
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line76
bq.
bq. It is a bit hard to read this counter.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 94
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line94
bq.
bq. This is count of regions scanned, right ?
bq. If so, please name it that way.
Todd suggested to rename it from COUNT_OF_REGIONS to REGIONS, given the fact
that it is a counter is implicit in mapreduce framework.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 127
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line127
bq.
bq. mb should be included in the exception.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 133
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line133
bq.
bq. Why do we need this check again ?
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 143
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line143
bq.
bq. Value of version should be included here.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java,
line 151
bq. https://reviews.apache.org/r/1674/diff/4/?file=46377#file46377line151
bq.
bq. I think we should have else block where the unsupported mb is logged.
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java,
line 52
bq. https://reviews.apache.org/r/1674/diff/4/?file=46380#file46380line52
bq.
bq. This name doesn't really match the constant above. I think HBase
mapreduce Counters would be better.
The name should show up in mapreduce UI and report. Other group names don't
have mapreduce. So keep it as HBase Counters and rename the variable.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java,
line 83
bq. https://reviews.apache.org/r/1674/diff/4/?file=46380#file46380line83
bq.
bq. This should not be a tongue twister.
bq. How about naming it retrieveGetCounterWithStrings ?
Fixed.
bq. On 2011-09-29 04:18:54, Ted Yu wrote:
bq.
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java,
line 232
bq. https://reviews.apache.org/r/1674/diff/4/?file=46380#file46380line232
bq.
bq. Shall we create the Object array outside the for loop and only fill
in Metric name here ?
Fixed. Don't create Object at all, pass the parameters directly.
- Ming
---
This is an automatically generated e-mail. To reply, visit:
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117612#comment-13117612 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/ --- (Updated 2011-09-29 21:00:18.525989) Review request for hbase. Changes --- Thanks for the review, Ted, Ram. Most are fixed. Please find comments inline. Summary --- 1. Collect client-side scan related metrics during scan operation. It is turned off by default. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. 3. Clean up some minor issues in tableInputFormat as well as test code. This addresses bug hbase-4145. https://issues.apache.org/jira/browse/hbase-4145 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 Diff: https://reviews.apache.org/r/1674/diff Testing --- 1. Verified on a small cluster. 2. Existing unit tests. 3. Added new tests. Thanks, Ming Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances. However, it might be good enough for now. With this approach, the metrics value will be available via MapReduce counter. a) Have ResultScanner return a new ResultScannerMetrics interface. b) TableInputFormat will access data from ResultScannerMetrics and populate MapReduce counters accordingly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) ROWCOL bloom filter not used if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117615#comment-13117615
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/
---
(Updated 2011-09-29 21:05:20.334849)
Review request for hbase.
Changes
---
Addressing Jonathan's comments.
Summary
---
Previously we only used row-column Bloom filters for scans that only requested
one column. We have seen production queries that request up to 200 columns, and
with say ~6 store files per store (region / column family combination) this
might have resulted in 1200 block read operations in the worst case. With this
diff we will be avoiding seeks on store files that we know don't contain the
row/column of interest when using an ExplicitColumnTracker. The performance
should remain the same for column range queries.
This addresses bug HBASE-2794.
https://issues.apache.org/jira/browse/HBASE-2794
Diffs (updated)
-
src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java 9d9895c
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java ac2348e
src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
Diff: https://reviews.apache.org/r/2084/diff
Testing
---
Existing unit tests. A new unit test (TestScanWithBloomError). Load testing
using HBaseTest.
Thanks,
Mikhail
ROWCOL bloom filter not used if multiple columns within same family are
requested in a Get
--
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1, then we currently don't take advantage of the bloom
filter. We should optimize this to check bloom for each of columns and if
none of the columns are present in the bloom avoid opening the file.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4497) If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE
[ https://issues.apache.org/jira/browse/HBASE-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117617#comment-13117617 ] jirapos...@reviews.apache.org commented on HBASE-4497: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/ --- Review request for hbase. Summary --- Adds a checkAndPut that takes a timestamp This addresses bug hbase-4497. https://issues.apache.org/jira/browse/hbase-4497 Diffs - src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc Diff: https://reviews.apache.org/r/2118/diff Testing --- Thanks, Michael If region opening fails after updating META HBCK reports it as inconsistent and scanning the region throws NSRE --- Key: HBASE-4497 URL: https://issues.apache.org/jira/browse/HBASE-4497 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Priority: Critical As per the discussion in the mail chain HBCK reporting of possible mismatch in RS assignment this JIRA is created. Consider two RS- RS1 and RS2. A region tries to open in RS1. But it takes a while. The RS1 has still not updated meta and transitioned the node from OPENING to OPENED So timeout assigns the region to RS2. RS2 successfully updates the META and opens the region. Now RS1 tries to act on the region by first updating the META and then transiting the node to OPENING to OPENED. RS1 transiting the node to OPENING to OPENED will fail. But the META entry will have RS1 as the latest. Now HBCK reports this as an inconsistency and if we try to scan the Region we get NotServingRegionException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117618#comment-13117618 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/ --- (Updated 2011-09-29 21:10:29.425308) Review request for hbase. Changes --- Fixed Summary. Summary (updated) --- Adds a checkAndPut that takes a timestamp This addresses bug hbase-4507. https://issues.apache.org/jira/browse/hbase-4507 Diffs - src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc Diff: https://reviews.apache.org/r/2118/diff Testing --- Thanks, Michael Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117621#comment-13117621 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2175 --- Only one minor comment left, see below. If test suite passes, I would vote +1 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment5068 VersionedWritable from hadoop left something to be desired. Such discussion came up during HBASE-2195. We can address this elsewhere. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java https://reviews.apache.org/r/1674/#comment5069 I meant that this counter can be named REGIONS_SCANNED because its value may be lower than the total number of regions in the table(s). - Ted On 2011-09-29 21:00:18, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:00:18) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117627#comment-13117627 ] jirapos...@reviews.apache.org commented on HBASE-4503: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2119/ --- Review request for hbase. Summary --- Replace deprecated HBaseClusterTestCase and subclasses. This addresses bug hbase-4503. https://issues.apache.org/jira/browse/hbase-4503 Diffs - Diff: https://reviews.apache.org/r/2119/diff Testing --- Unit tests pass Thanks, Michael Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4145) Provide metrics for hbase client
[ https://issues.apache.org/jira/browse/HBASE-4145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117634#comment-13117634 ] jirapos...@reviews.apache.org commented on HBASE-4145: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1674/#review2178 --- Ship it! - Ted On 2011-09-29 21:36:19, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1674/ bq. --- bq. bq. (Updated 2011-09-29 21:36:19) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. 1. Collect client-side scan related metrics during scan operation. It is turned off by default. bq. 2. TableInputFormat enables metrics collection on scan and pass the data to mapreduce framework. It only works with new mapreduce APIs that allow TableInputFormat to get access to mapreduce Counter. bq. 3. Clean up some minor issues in tableInputFormat as well as test code. bq. bq. bq. This addresses bug hbase-4145. bq. https://issues.apache.org/jira/browse/hbase-4145 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/metrics/ScanMetrics.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReader.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1176942 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableInputFormat.java 1176942 bq. bq. Diff: https://reviews.apache.org/r/1674/diff bq. bq. bq. Testing bq. --- bq. bq. 1. Verified on a small cluster. bq. 2. Existing unit tests. bq. 3. Added new tests. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Provide metrics for hbase client Key: HBASE-4145 URL: https://issues.apache.org/jira/browse/HBASE-4145 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.94.0 Attachments: HBaseClientSideMetrics.jpg Sometimes it is useful to get some metrics from hbase client point of view. This will help understand the metrics for scan/TableInputFormat map job scenario. What to capture, for example, for each ResultScanner object, 1. The number of RPC calls to RSs. 2. The delta time between consecutive RPC calls in the current serialized scan implementation. 3. The number of RPC retry to RSs. 4. The number of NotServingRegionException got. 5. The number of remote RPC calls. This excludes those call that hbase client calls the RS on the same machine. 6. The number of regions accessed. How to capture 1. Metrics framework works for a fixed number of metrics. It doesn't fit this scenario. 2. Use some TBD solution in HBase to capture such dynamic metrics. If we assume there is a solution in HBase that HBase client can use to log such kind of metrics, TableInputFormat can pass in mapreduce task ID as application scan ID to HBase client as small addition to existing scan API; and HBase client can log metrics accordingly with such ID. That will allow query, analysis later on the metrics data for specific map reduce job. 3. Expose via MapReduce counter. It lacks certain features, for example, there is no good way to access the metrics on per map instance; the MapReduce framework only performs sum on the counter values so it is tricky to find the max of certain metrics in all mapper instances. However, it might be good enough for now. With this approach, the metrics value will be available via MapReduce counter. a) Have ResultScanner return a new
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117646#comment-13117646 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2179 --- Ship it! - Andrew On 2011-09-29 20:15:12, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-29 20:15:12) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/main/ruby/hbase/admin.rb b244ffe bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117647#comment-13117647 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2176 --- src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java https://reviews.apache.org/r/2118/#comment5070 expected is not needed here. src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java https://reviews.apache.org/r/2118/#comment5071 Should be existence. src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java https://reviews.apache.org/r/2118/#comment5072 Should be executed. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5073 I don't see this parameter below. I think these four lines can be moved into the body of the method. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5076 Writable parameter is missing here. It would be nice to add it now. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5078 Should be checkAndPut src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5079 How about naming this method doTimestampsClash ? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5081 I wish there is a better way to handle this situation :-( src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5080 'that of the' can be omitted. - Ted On 2011-09-29 21:10:29, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 21:10:29) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117653#comment-13117653 ] jirapos...@reviews.apache.org commented on HBASE-4503: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2119/ --- (Updated 2011-09-29 22:01:14.094149) Review request for hbase. Changes --- Include actual diff Summary --- Replace deprecated HBaseClusterTestCase and subclasses. This addresses bug hbase-4503. https://issues.apache.org/jira/browse/hbase-4503 Diffs (updated) - src/test/java/org/apache/hadoop/hbase/HBaseClusterTestCase.java c18cddb src/test/java/org/apache/hadoop/hbase/MultiRegionTable.java a8fd4e5 src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java 555174a src/test/java/org/apache/hadoop/hbase/TestInfoServers.java 638df08 src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 9023af8 src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java d7baba4 src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 221352c src/test/java/org/apache/hadoop/hbase/client/TestGetRowVersions.java 27842ed src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java 9204539 src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java 5a5c3c6 src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java 0b8ff30 src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java a772360 src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java a4cd9a3 src/test/java/org/apache/hadoop/hbase/regionserver/DisabledTestRegionServerExit.java 5b8b464 src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java 5802686 Diff: https://reviews.apache.org/r/2119/diff Testing --- Unit tests pass Thanks, Michael Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117655#comment-13117655 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- bq. On 2011-09-29 21:52:40, Andrew Purtell wrote: bq. Actually, I do have a comment. Do we need a method in HMasterInterface for getting the coprocessor set? Can't it be extracted from HSL? - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2179 --- On 2011-09-29 20:15:12, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-29 20:15:12) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/main/ruby/hbase/admin.rb b244ffe bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117656#comment-13117656 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- bq. On 2011-09-29 21:52:40, Andrew Purtell wrote: bq. bq. bq. Andrew Purtell wrote: bq. Actually, I do have a comment. Do we need a method in HMasterInterface for getting the coprocessor set? Can't it be extracted from HSL? I meant via ClusterStatus, therefore via HBaseAdmin.getClusterStatus(). - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2179 --- On 2011-09-29 20:15:12, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-29 20:15:12) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/main/ruby/hbase/admin.rb b244ffe bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117664#comment-13117664 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2185 --- src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/2118/#comment5089 Since expectedTimestampOfLatestVersion would only be used for use case from HBASE04497, I think we just need a reliable way of enforcing this 1 ms advance. Maybe spinlocking till EnvironmentEdgeManager.currentTimeMillis() returns now+1 ? - Ted On 2011-09-29 21:10:29, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 21:10:29) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[
https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117673#comment-13117673
]
jirapos...@reviews.apache.org commented on HBASE-4503:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2119/#review2184
---
A couple of slight cleanup things, but overall +1.
src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
https://reviews.apache.org/r/2119/#comment5084
Does this file really need to be here? Yeah, its a little whitespace
cleanup, but I don't think its *that* necessary
src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
https://reviews.apache.org/r/2119/#comment5086
why not just assertEquals(${desc}, expected, content) here?
src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
https://reviews.apache.org/r/2119/#comment5088
Couldn't this just go into @BeforeClass?
src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
https://reviews.apache.org/r/2119/#comment5090
nitpick: just use iterable properties here?
Also, maybe abstract this into a single checking method? Not gaining a
whole lot, but saves you some code copy.
src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
https://reviews.apache.org/r/2119/#comment5093
nitpick: why drop the import?
src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java
https://reviews.apache.org/r/2119/#comment5094
nitpick: (Again) import Assert.*?
src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
https://reviews.apache.org/r/2119/#comment5095
This doesn't seem to fail into the same review need as the rest of the
patch. I'm ok with the changes, but it would be better if they were for the
same reason.
+/-0
- Jesse
On 2011-09-29 22:01:14, Michael Stack wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2119/
bq. ---
bq.
bq. (Updated 2011-09-29 22:01:14)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Replace deprecated HBaseClusterTestCase and subclasses.
bq.
bq.
bq. This addresses bug hbase-4503.
bq. https://issues.apache.org/jira/browse/hbase-4503
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/test/java/org/apache/hadoop/hbase/HBaseClusterTestCase.java c18cddb
bq.src/test/java/org/apache/hadoop/hbase/MultiRegionTable.java a8fd4e5
bq.src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
555174a
bq.src/test/java/org/apache/hadoop/hbase/TestInfoServers.java 638df08
bq.src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 9023af8
bq.src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java
d7baba4
bq.src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 221352c
bq.src/test/java/org/apache/hadoop/hbase/client/TestGetRowVersions.java
27842ed
bq.src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java 9204539
bq.src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
5a5c3c6
bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
0b8ff30
bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java
a772360
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
a4cd9a3
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/DisabledTestRegionServerExit.java
5b8b464
bq.src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
5802686
bq.
bq. Diff: https://reviews.apache.org/r/2119/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Unit tests pass
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
Purge deprecated HBaseClusterTestCase
-
Key: HBASE-4503
URL: https://issues.apache.org/jira/browse/HBASE-4503
Project: HBase
Issue Type: Improvement
Reporter: stack
Assignee: stack
Attachments: 4503-v2.txt, 4503.txt
It could gain us a few minutes on overall test run in the cases where we
don't spin up a cluster for each test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117677#comment-13117677 ] jirapos...@reviews.apache.org commented on HBASE-4487: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2116/#review2186 --- Looks good. I like the bench marking tool. This is like a group commit in HLog. We have a group committing going on inside in sync down in dfsclient. Does this group commit not 'group' enough. With this change, its no longer possible to sync each increment. We ok w/ that? We should at least release note this difference in increment behaviors and erring on the conservative side, I'd mark this an 'incompatible change' rather than an 'improvement' just so its more likely folks will know of this changed increment behavior. - Michael On 2011-09-29 19:27:36, Dhruba Borthakur wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2116/ bq. --- bq. bq. (Updated 2011-09-29 19:27:36) bq. bq. bq. Review request for hbase and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. The increment operation releases the rowlock before doing the sync to the HLog. This improves performance of increments on hot rows. Introduced method HLog.appendNoSync() that returns a txid. The increment method then release the rowlock and invokes HLog.sync(txid). The HLog.sync(txid) returns only if all the transactions upto the one identified by that txid has been successfully sycned to HDFS. bq. bq. There is a single test TestIncrement that creates a 100 threads and ensures that all the concurrent increments match the final expected result. There is a benchmark TestHLogBench that measures the performance of the appendNoSync call. bq. bq. bq. This addresses bug HBASE-4487. bq. https://issues.apache.org/jira/browse/HBASE-4487 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177401 bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1177401 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestIncrement.java PRE-CREATION bq. /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java PRE-CREATION bq. /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java 1177401 bq. /src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 1177401 bq. bq. Diff: https://reviews.apache.org/r/2116/diff bq. bq. bq. Testing bq. --- bq. bq. All unit tests pass now (expect TestDistributedLogSplitting, TestRollingRestart, TestHTablePool), but I am seeing the same test to fail on trunk, so these failures do not seem to be related to this patch. bq. bq. bq. Thanks, bq. bq. Dhruba bq. bq. The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117694#comment-13117694 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2183 --- src/main/java/org/apache/hadoop/hbase/HServerLoad.java https://reviews.apache.org/r/2029/#comment5082 an extra indent? src/main/java/org/apache/hadoop/hbase/HServerLoad.java https://reviews.apache.org/r/2029/#comment5083 Just a naming suggestion for walCoprocessors=coprocessors. Although right now WALObserver is the only RS level CP, but it'd better not to indicate here. src/main/java/org/apache/hadoop/hbase/HServerLoad.java https://reviews.apache.org/r/2029/#comment5085 getSimpleName() only returns the class name w/o package info. I think we should use the whole class name for the comparator. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/2029/#comment5092 do we really need to provide it thru HTAdmin? if we do, the returned value is not correct in case of exception. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java https://reviews.apache.org/r/2029/#comment5087 The new method is a little odd. how about just getCoprocessors(). src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/2029/#comment5091 still the method name? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/2029/#comment5096 same as HMasterInterface. can we just generateCoprocessorString() = String getCoprocessors()? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/2029/#comment5097 is it okay just reuse the existing serverLoad, without rebuilding the whole thing? this.serverInfo.getLoad() but it may not be a big issue. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java https://reviews.apache.org/r/2029/#comment5098 The file TestCoprocessorEndpoint is particularly for testing cp endpoint. It's odd to put the new test cases to this file. Maybe TestClassLoading is a better place. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java https://reviews.apache.org/r/2029/#comment5100 according the previous override comparator in HRL, the cps should be in alphabet order. so you don't need to compare this 2 strings. - Mingjie On 2011-09-29 20:15:12, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-29 20:15:12) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java a55a4b1 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java a069400 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/main/ruby/hbase/admin.rb b244ffe bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java a8f2a9c bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() added to src/test/java/o.a.h.h/coprocessor/TestCoprocessorEndpoint.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments:
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[
https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117705#comment-13117705
]
jirapos...@reviews.apache.org commented on HBASE-4503:
--
bq. On 2011-09-29 22:26:55, Jesse Yates wrote:
bq. src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java, line
150
bq. https://reviews.apache.org/r/2119/diff/1/?file=46504#file46504line150
bq.
bq. Does this file really need to be here? Yeah, its a little whitespace
cleanup, but I don't think its *that* necessary
Agreed. Will purge it from the diff.
bq. On 2011-09-29 22:26:55, Jesse Yates wrote:
bq. src/test/java/org/apache/hadoop/hbase/TestInfoServers.java, line 105
bq. https://reviews.apache.org/r/2119/diff/1/?file=46505#file46505line105
bq.
bq. why not just assertEquals(${desc}, expected, content) here?
Will do.
bq. On 2011-09-29 22:26:55, Jesse Yates wrote:
bq. src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java, line 72
bq. https://reviews.apache.org/r/2119/diff/1/?file=46506#file46506line72
bq.
bq. Couldn't this just go into @BeforeClass?
That is static but this.admin is not. It could but saw no harm doing it this
way.
bq. On 2011-09-29 22:26:55, Jesse Yates wrote:
bq. src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java, line 240
bq. https://reviews.apache.org/r/2119/diff/1/?file=46506#file46506line240
bq.
bq. nitpick: just use iterable properties here?
bq.
bq. Also, maybe abstract this into a single checking method? Not gaining
a whole lot, but saves you some code copy.
The tests here were copied from elsewhere. I tried not to touch what they did,
just convert the chassis they used. I can change this np.
bq. On 2011-09-29 22:26:55, Jesse Yates wrote:
bq. src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java,
line 191
bq. https://reviews.apache.org/r/2119/diff/1/?file=46513#file46513line191
bq.
bq. nitpick: why drop the import?
Will fix.
bq. On 2011-09-29 22:26:55, Jesse Yates wrote:
bq.
src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java, line
217
bq. https://reviews.apache.org/r/2119/diff/1/?file=46514#file46514line217
bq.
bq. nitpick: (Again) import Assert.*?
Will fix.
- Michael
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2119/#review2184
---
On 2011-09-29 22:01:14, Michael Stack wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2119/
bq. ---
bq.
bq. (Updated 2011-09-29 22:01:14)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Replace deprecated HBaseClusterTestCase and subclasses.
bq.
bq.
bq. This addresses bug hbase-4503.
bq. https://issues.apache.org/jira/browse/hbase-4503
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/test/java/org/apache/hadoop/hbase/HBaseClusterTestCase.java c18cddb
bq.src/test/java/org/apache/hadoop/hbase/MultiRegionTable.java a8fd4e5
bq.src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
555174a
bq.src/test/java/org/apache/hadoop/hbase/TestInfoServers.java 638df08
bq.src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 9023af8
bq.src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java
d7baba4
bq.src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 221352c
bq.src/test/java/org/apache/hadoop/hbase/client/TestGetRowVersions.java
27842ed
bq.src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java 9204539
bq.src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
5a5c3c6
bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
0b8ff30
bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java
a772360
bq.
src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java
a4cd9a3
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/DisabledTestRegionServerExit.java
5b8b464
bq.src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
5802686
bq.
bq. Diff: https://reviews.apache.org/r/2119/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Unit tests pass
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
Purge deprecated HBaseClusterTestCase
-
Key: HBASE-4503
URL: https://issues.apache.org/jira/browse/HBASE-4503
Project:
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117704#comment-13117704 ] jirapos...@reviews.apache.org commented on HBASE-4503: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2119/#review2189 --- src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java https://reviews.apache.org/r/2119/#comment5109 You are right - Michael On 2011-09-29 22:01:14, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2119/ bq. --- bq. bq. (Updated 2011-09-29 22:01:14) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Replace deprecated HBaseClusterTestCase and subclasses. bq. bq. bq. This addresses bug hbase-4503. bq. https://issues.apache.org/jira/browse/hbase-4503 bq. bq. bq. Diffs bq. - bq. bq.src/test/java/org/apache/hadoop/hbase/HBaseClusterTestCase.java c18cddb bq.src/test/java/org/apache/hadoop/hbase/MultiRegionTable.java a8fd4e5 bq.src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java 555174a bq.src/test/java/org/apache/hadoop/hbase/TestInfoServers.java 638df08 bq.src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 9023af8 bq.src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java d7baba4 bq.src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 221352c bq.src/test/java/org/apache/hadoop/hbase/client/TestGetRowVersions.java 27842ed bq.src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java 9204539 bq.src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java 5a5c3c6 bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java 0b8ff30 bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java a772360 bq. src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java a4cd9a3 bq. src/test/java/org/apache/hadoop/hbase/regionserver/DisabledTestRegionServerExit.java 5b8b464 bq.src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java 5802686 bq. bq. Diff: https://reviews.apache.org/r/2119/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests pass bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4515) User.getCurrent() can cause NPE.
[
https://issues.apache.org/jira/browse/HBASE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117723#comment-13117723
]
jirapos...@reviews.apache.org commented on HBASE-4515:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2123/#review2190
---
IMO, it would be best to avoid letting either a null or NPE escape.
- Andrew
On 2011-09-29 21:41:46, jmhsieh wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2123/
bq. ---
bq.
bq. (Updated 2011-09-29 21:41:46)
bq.
bq.
bq. Review request for hbase and Gary Helmling.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. commit aa319da9896588f97aeeb50a39bc5641ae58d537
bq. Author: Jonathan Hsieh j...@cloudera.com
bq. Date: Thu Sep 29 13:27:22 2011 -0700
bq.
bq. HBASE-4515 User.getCurrent() can cause NPE
bq.
bq.
bq. This addresses bug HBASE-4515.
bq. https://issues.apache.org/jira/browse/HBASE-4515
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7
bq.src/test/java/org/apache/hadoop/hbase/security/TestUser.java 6e497da
bq.
bq. Diff: https://reviews.apache.org/r/2123/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. New unit tests passes although the behavior it displays is a bit suspect.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
User.getCurrent() can cause NPE.
Key: HBASE-4515
URL: https://issues.apache.org/jira/browse/HBASE-4515
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments: 0001-HBASE-4515-User.getCurrent-can-cause-NPE.patch,
HBASE-4515_trunk.patch
When testing with miniclusters that shutdown and are restarted, sometimes a
call to User.getCurrent().getName() NPEs when attempting to restart hbase.
Oddly this happens consistently on particular branches and not on others. I
don't know or understand why this happens but it has something to do with the
getCurrentUGI call in o.a.h.h.security.User.HadoopUser sometimes returning
null and sometimes returning data.
{code}
private HadoopUser() {
try {
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
if (ugi == null) {
LOG.warn(Although successfully retrieved UserGroupInformation
+ it was null!);
}
} catch (RuntimeException re) {
{code}
This patch essentially is a workaround -- it propagates the null so that
clients can check and avoid the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4515) User.getCurrent() can cause NPE.
[
https://issues.apache.org/jira/browse/HBASE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117740#comment-13117740
]
jirapos...@reviews.apache.org commented on HBASE-4515:
--
bq. On 2011-09-29 23:08:13, Andrew Purtell wrote:
bq. IMO, it would be best to avoid letting either a null or NPE escape.
I agree, I just didn't figure out how the value got populated. Gary's posted a
patch that avoids the npe/null by logging in and that code worked for me.
- jmhsieh
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2123/#review2190
---
On 2011-09-29 21:41:46, jmhsieh wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2123/
bq. ---
bq.
bq. (Updated 2011-09-29 21:41:46)
bq.
bq.
bq. Review request for hbase and Gary Helmling.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. commit aa319da9896588f97aeeb50a39bc5641ae58d537
bq. Author: Jonathan Hsieh j...@cloudera.com
bq. Date: Thu Sep 29 13:27:22 2011 -0700
bq.
bq. HBASE-4515 User.getCurrent() can cause NPE
bq.
bq.
bq. This addresses bug HBASE-4515.
bq. https://issues.apache.org/jira/browse/HBASE-4515
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7
bq.src/test/java/org/apache/hadoop/hbase/security/TestUser.java 6e497da
bq.
bq. Diff: https://reviews.apache.org/r/2123/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. New unit tests passes although the behavior it displays is a bit suspect.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
User.getCurrent() can cause NPE.
Key: HBASE-4515
URL: https://issues.apache.org/jira/browse/HBASE-4515
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Gary Helmling
Attachments: 0001-HBASE-4515-User.getCurrent-can-cause-NPE.patch,
HBASE-4515_trunk.patch
When testing with miniclusters that shutdown and are restarted, sometimes a
call to User.getCurrent().getName() NPEs when attempting to restart hbase.
Oddly this happens consistently on particular branches and not on others. I
don't know or understand why this happens but it has something to do with the
getCurrentUGI call in o.a.h.h.security.User.HadoopUser sometimes returning
null and sometimes returning data.
{code}
private HadoopUser() {
try {
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
if (ugi == null) {
LOG.warn(Although successfully retrieved UserGroupInformation
+ it was null!);
}
} catch (RuntimeException re) {
{code}
This patch essentially is a workaround -- it propagates the null so that
clients can check and avoid the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4476) Compactions must fail if column tracker gets columns out of order
[
https://issues.apache.org/jira/browse/HBASE-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117749#comment-13117749
]
jirapos...@reviews.apache.org commented on HBASE-4476:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2050/
---
(Updated 2011-09-29 23:45:46.477105)
Review request for hbase and Kannan Muthukkaruppan.
Changes
---
Linking to HBASE-4476.
Summary
---
We found this in ScanWildcardColumnTracker:
// new col oldcol
// if (cmp 0) {
// WARNING: This means that very likely an edit for some other family
// was incorrectly stored into the store for this one. Continue, but
// complain.
LOG.error(ScanWildcardColumnTracker.checkColumn ran +
into a column actually smaller than the previous column: +
This went under the radar in our dark launch cluster when a column family name
was misspelled first, but then was renamed by renaming directories in the
HBase storage directory tree. We ended up with inconsistent data, but
compactions still succeeded most of the time, likely discarding part of input
data.
This addresses bug HBASE-4476.
https://issues.apache.org/jira/browse/HBASE-4476
Diffs
-
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java
1176613
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
1176613
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
1176613
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
1176613
http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
1176613
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java
1176613
http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
1176613
Diff: https://reviews.apache.org/r/2050/diff
Testing
---
Run unit tests.
Will also push to a 5-node dev cluster.
Thanks,
Mikhail
Compactions must fail if column tracker gets columns out of order
-
Key: HBASE-4476
URL: https://issues.apache.org/jira/browse/HBASE-4476
Project: HBase
Issue Type: Bug
Reporter: Mikhail Bautin
Assignee: Mikhail Bautin
Fix For: 0.92.0, 0.94.0
We found this in ScanWildcardColumnTracker:
// new col oldcol
// if (cmp 0) {
// WARNING: This means that very likely an edit for some other family
// was incorrectly stored into the store for this one. Continue, but
// complain.
LOG.error(ScanWildcardColumnTracker.checkColumn ran +
into a column actually smaller than the previous column: +
This went under the radar in our dark launch cluster when a column family
name was misspelled first, but then was renamed by renaming directories in
the HBase storage directory tree. We ended up with inconsistent data, but
compactions still succeeded most of the time, likely discarding part of input
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117753#comment-13117753 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- bq. On 2011-09-29 21:54:42, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 1942 bq. https://reviews.apache.org/r/2118/diff/1/?file=46471#file46471line1942 bq. bq. How about naming this method doTimestampsClash ? Because I wanted to follow beans convention for naming methods that have boolean return (its not the best english but tells you more about what to expect of the method than a 'do'). bq. On 2011-09-29 21:54:42, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 1948 bq. https://reviews.apache.org/r/2118/diff/1/?file=46471#file46471line1948 bq. bq. I wish there is a better way to handle this situation :-( This situation is going to be extremely rare. If it happens at all, we do this +1 on the ts. I prefer this to a spin-wait holding up the update. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2176 --- On 2011-09-29 21:10:29, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 21:10:29) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0c06f4f bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117756#comment-13117756 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/ --- (Updated 2011-09-29 23:52:43.167788) Review request for hbase. Changes --- Address Ted's feedback. Made all changes but suggestion to do a spin-wait. Summary --- Adds a checkAndPut that takes a timestamp This addresses bug hbase-4507. https://issues.apache.org/jira/browse/hbase-4507 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc Diff: https://reviews.apache.org/r/2118/diff Testing --- Thanks, Michael Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4516) HFile-level load tester with compaction and random-read workloads
[ https://issues.apache.org/jira/browse/HBASE-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117757#comment-13117757 ] jirapos...@reviews.apache.org commented on HBASE-4516: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2122/ --- Review request for hbase, Michael Stack and Jonathan Gray. Summary --- This is a load testing tool for HFile implementations, which supports two workloads: Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. Example output for the random read workload: Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 Also refactoring and clarifying the confusing situation when a StoreFile happens to have a different Bloom filter type than what is configured for the column family. This addresses bug HBASE-4516. https://issues.apache.org/jira/browse/HBASE-4516 Diffs - src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b429819 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java PRE-CREATION Diff: https://reviews.apache.org/r/2122/diff Testing --- Running unit tests and the load tester tool itself. Thanks, Mikhail HFile-level load tester with compaction and random-read workloads - Key: HBASE-4516 URL: https://issues.apache.org/jira/browse/HBASE-4516 Project: HBase Issue Type: Test Reporter: Mikhail Bautin Priority: Minor Fix For: 0.94.0 This is a load testing tool for HFile implementations, which supports two workloads: - Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. - Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. Example output for the random read workload: Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4516) HFile-level load tester with compaction and random-read workloads
[ https://issues.apache.org/jira/browse/HBASE-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117761#comment-13117761 ] jirapos...@reviews.apache.org commented on HBASE-4516: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2122/ --- (Updated 2011-09-29 23:54:05.869273) Review request for hbase, Michael Stack and Jonathan Gray. Changes --- Updating summary (sorry for spam). Summary (updated) --- This is a load testing tool for HFile implementations, which supports two workloads: Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. Example output for the random read workload: Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 Also refactoring and clarifying the confusing situation when a StoreFile happens to have a different Bloom filter type than what is configured for the column family. This addresses bug HBASE-4516. https://issues.apache.org/jira/browse/HBASE-4516 Diffs - src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b429819 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java PRE-CREATION Diff: https://reviews.apache.org/r/2122/diff Testing --- Running unit tests and the load tester tool itself. Thanks, Mikhail HFile-level load tester with compaction and random-read workloads - Key: HBASE-4516 URL: https://issues.apache.org/jira/browse/HBASE-4516 Project: HBase Issue Type: Test Reporter: Mikhail Bautin Priority: Minor Fix For: 0.94.0 This is a load testing tool for HFile implementations, which supports two workloads: - Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. - Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. Example output for the random read workload: Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.
[ https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117762#comment-13117762 ] jirapos...@reviews.apache.org commented on HBASE-4377: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2126/ --- Review request for hbase, Michael Stack and Andrew Purtell. Summary --- commit fbf82c17be6b3ecca5a981f5270cf93aac26e479 Author: Jonathan Hsieh j...@cloudera.com Date: Wed Sep 28 10:18:11 2011 -0700 HBASE-4377 [hbck] Offline rebuild .META. from fs data only This patch rebuilds a new .META. table by reading all the .regioninfo files in the hbase main directory. It depends on the yet to be committed HBASE-4515 (either my verison or Gary's version), HBASE-4509, and HBASE-4506. Some follow on work includes backporting to 0.90, auto-patching true holes, and adding documentation. This addresses bug HBASE-4377. https://issues.apache.org/jira/browse/HBASE-4377 Diffs - src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java b9c850d src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 8465724 src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java fae0881 src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuild.java PRE-CREATION Diff: https://reviews.apache.org/r/2126/diff Testing --- An earlier version of this code (backported to 0.90) was used to diagnose and repair a cluster that had 2700 inconsistencies due to failed splits (the cluster was underprovisioned memory-wise, and on restart, the some regions would start splitting and then die due to oome's). This was not actually used on a live cluster -- it was used to reconstruct a .META. from .regioninfo's laid out in hbase's directory structure. Note also that this is not an automatic fix -- whenever any problems are found, this bails out but dumps info on holes, suggests some fixes, and displays sets of overlapping regions. It is up to the user to merge regions, to create .regioninfo files to plug hole, and to do any potential data loosing operations. The tests demonstrate current expected behavior -- rebuild meta if things line up, and fail without making modifications if holes or overlaps exist. Thanks, jmhsieh [hbck] Offline rebuild .META. from fs data only. Key: HBASE-4377 URL: https://issues.apache.org/jira/browse/HBASE-4377 Project: HBase Issue Type: New Feature Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch In a worst case situation, it may be helpful to have an offline .META. rebuilder that just looks at the file system's .regioninfos and rebuilds meta from scratch. Users could move bad regions out until there is a clean rebuild. It would likely fill in region split holes. Follow on work could given options to merge or select regions that overlap, or do online rebuilds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4516) HFile-level load tester with compaction and random-read workloads
[ https://issues.apache.org/jira/browse/HBASE-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117766#comment-13117766 ] jirapos...@reviews.apache.org commented on HBASE-4516: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2122/#review2194 --- src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java https://reviews.apache.org/r/2122/#comment5113 I don't think you meant to include this file in your loading tool patch? Just let me know and I can exclude on commit (maybe its part of another JIRA)? src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java https://reviews.apache.org/r/2122/#comment5112 Did you intend to include this 'fix' in your loading tool patch? src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java https://reviews.apache.org/r/2122/#comment5114 FYI, going forward, this copyright line is no longer needed. - Michael On 2011-09-29 23:54:05, Mikhail Bautin wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2122/ bq. --- bq. bq. (Updated 2011-09-29 23:54:05) bq. bq. bq. Review request for hbase, Michael Stack and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. This is a load testing tool for HFile implementations, which supports two workloads: bq. bq. Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. bq. Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. bq. bq. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. bq. bq. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. bq. bq. Example output for the random read workload: bq. Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 bq. bq. Also refactoring and clarifying the confusing situation when a StoreFile happens to have a different Bloom filter type than what is configured for the column family. bq. bq. bq. This addresses bug HBASE-4516. bq. https://issues.apache.org/jira/browse/HBASE-4516 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b429819 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef bq. src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2122/diff bq. bq. bq. Testing bq. --- bq. bq. Running unit tests and the load tester tool itself. bq. bq. bq. Thanks, bq. bq. Mikhail bq. bq. HFile-level load tester with compaction and random-read workloads - Key: HBASE-4516 URL: https://issues.apache.org/jira/browse/HBASE-4516 Project: HBase Issue Type: Test Reporter: Mikhail Bautin Priority: Minor Fix For: 0.94.0 This is a load testing tool for HFile implementations, which supports two workloads: - Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. - Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. Example output for the random read workload: Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys:
[jira] [Commented] (HBASE-4516) HFile-level load tester with compaction and random-read workloads
[ https://issues.apache.org/jira/browse/HBASE-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117767#comment-13117767 ] jirapos...@reviews.apache.org commented on HBASE-4516: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2122/#review2195 --- Ship it! Answer my question above M and I'll exclude on commit. Will add a little note to the 'book' too that this tool now exists. Nice. - Michael On 2011-09-29 23:54:05, Mikhail Bautin wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2122/ bq. --- bq. bq. (Updated 2011-09-29 23:54:05) bq. bq. bq. Review request for hbase, Michael Stack and Jonathan Gray. bq. bq. bq. Summary bq. --- bq. bq. This is a load testing tool for HFile implementations, which supports two workloads: bq. bq. Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. bq. Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. bq. bq. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. bq. bq. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. bq. bq. Example output for the random read workload: bq. Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 bq. bq. Also refactoring and clarifying the confusing situation when a StoreFile happens to have a different Bloom filter type than what is configured for the column family. bq. bq. bq. This addresses bug HBASE-4516. bq. https://issues.apache.org/jira/browse/HBASE-4516 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java b429819 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java fd9e7ef bq. src/test/java/org/apache/hadoop/hbase/regionserver/HFileReadWriteTest.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2122/diff bq. bq. bq. Testing bq. --- bq. bq. Running unit tests and the load tester tool itself. bq. bq. bq. Thanks, bq. bq. Mikhail bq. bq. HFile-level load tester with compaction and random-read workloads - Key: HBASE-4516 URL: https://issues.apache.org/jira/browse/HBASE-4516 Project: HBase Issue Type: Test Reporter: Mikhail Bautin Priority: Minor Fix For: 0.94.0 This is a load testing tool for HFile implementations, which supports two workloads: - Compactions (merge the input HFiles). A special case of this is only one input, which allows to do HFile format conversions. - Random reads. Launches the specified number of threads that do seeks and short scans on randomly generated keys. The original purpose of this tool was to ensure that HFile format v2 did not introduce performance regressions. Keys for the read workload are generated randomly between the first and the last key of the HFile. At each position, instead of precisely calculating the correct probability for every byte value b, we select a uniformly random byte between in the allowed [low, high] range. In addition, there is a heuristic that determines the positions at which the key has hex characters, and the random key contains hex characters at those positions as well. Example output for the random read workload: Time: 120 sec, seek/sec: 8290, kv/sec: 30351, kv bytes/sec: 91868121, blk/sec: 10147, unique keys: 232779 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4503) Purge deprecated HBaseClusterTestCase
[ https://issues.apache.org/jira/browse/HBASE-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117768#comment-13117768 ] jirapos...@reviews.apache.org commented on HBASE-4503: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2119/#review2196 --- Ship it! - Jesse On 2011-09-29 23:26:34, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2119/ bq. --- bq. bq. (Updated 2011-09-29 23:26:34) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Replace deprecated HBaseClusterTestCase and subclasses. bq. bq. bq. This addresses bug hbase-4503. bq. https://issues.apache.org/jira/browse/hbase-4503 bq. bq. bq. Diffs bq. - bq. bq.src/test/java/org/apache/hadoop/hbase/HBaseClusterTestCase.java c18cddb bq.src/test/java/org/apache/hadoop/hbase/MultiRegionTable.java a8fd4e5 bq.src/test/java/org/apache/hadoop/hbase/TestInfoServers.java 638df08 bq.src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java 9023af8 bq.src/test/java/org/apache/hadoop/hbase/TestScanMultipleVersions.java d7baba4 bq.src/test/java/org/apache/hadoop/hbase/TimestampTestBase.java 221352c bq.src/test/java/org/apache/hadoop/hbase/client/TestGetRowVersions.java 27842ed bq.src/test/java/org/apache/hadoop/hbase/client/TestTimestamp.java 9204539 bq.src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java 5a5c3c6 bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java 0b8ff30 bq.src/test/java/org/apache/hadoop/hbase/mapreduce/TestTimeRangeMapRed.java a772360 bq. src/test/java/org/apache/hadoop/hbase/master/TestDefaultLoadBalancer.java a4cd9a3 bq. src/test/java/org/apache/hadoop/hbase/regionserver/DisabledTestRegionServerExit.java 5b8b464 bq.src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java 5802686 bq. bq. Diff: https://reviews.apache.org/r/2119/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests pass bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Purge deprecated HBaseClusterTestCase - Key: HBASE-4503 URL: https://issues.apache.org/jira/browse/HBASE-4503 Project: HBase Issue Type: Improvement Reporter: stack Assignee: stack Attachments: 4503-v2.txt, 4503.txt It could gain us a few minutes on overall test run in the cases where we don't spin up a cluster for each test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4515) User.getCurrent() can cause NPE.
[
https://issues.apache.org/jira/browse/HBASE-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117771#comment-13117771
]
jirapos...@reviews.apache.org commented on HBASE-4515:
--
bq. On 2011-09-29 23:08:13, Andrew Purtell wrote:
bq. IMO, it would be best to avoid letting either a null or NPE escape.
bq.
bq. jmhsieh wrote:
bq. I agree, I just didn't figure out how the value got populated. Gary's
posted a patch that avoids the npe/null by logging in and that code worked for
me.
bq.
bq. Michael Stack wrote:
bq. Should we cancel this review then?
done.
- jmhsieh
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2123/#review2190
---
On 2011-09-29 21:41:46, jmhsieh wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2123/
bq. ---
bq.
bq. (Updated 2011-09-29 21:41:46)
bq.
bq.
bq. Review request for hbase and Gary Helmling.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. commit aa319da9896588f97aeeb50a39bc5641ae58d537
bq. Author: Jonathan Hsieh j...@cloudera.com
bq. Date: Thu Sep 29 13:27:22 2011 -0700
bq.
bq. HBASE-4515 User.getCurrent() can cause NPE
bq.
bq.
bq. This addresses bug HBASE-4515.
bq. https://issues.apache.org/jira/browse/HBASE-4515
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/security/User.java d90f2c7
bq.src/test/java/org/apache/hadoop/hbase/security/TestUser.java 6e497da
bq.
bq. Diff: https://reviews.apache.org/r/2123/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. New unit tests passes although the behavior it displays is a bit suspect.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
User.getCurrent() can cause NPE.
Key: HBASE-4515
URL: https://issues.apache.org/jira/browse/HBASE-4515
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.0, 0.94.0
Reporter: Jonathan Hsieh
Assignee: Gary Helmling
Attachments: 0001-HBASE-4515-User.getCurrent-can-cause-NPE.patch,
HBASE-4515_trunk.patch
When testing with miniclusters that shutdown and are restarted, sometimes a
call to User.getCurrent().getName() NPEs when attempting to restart hbase.
Oddly this happens consistently on particular branches and not on others. I
don't know or understand why this happens but it has something to do with the
getCurrentUGI call in o.a.h.h.security.User.HadoopUser sometimes returning
null and sometimes returning data.
{code}
private HadoopUser() {
try {
ugi = (UserGroupInformation) callStatic(getCurrentUGI);
if (ugi == null) {
LOG.warn(Although successfully retrieved UserGroupInformation
+ it was null!);
}
} catch (RuntimeException re) {
{code}
This patch essentially is a workaround -- it propagates the null so that
clients can check and avoid the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[
https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117780#comment-13117780
]
jirapos...@reviews.apache.org commented on HBASE-4487:
--
bq. On 2011-09-29 22:31:37, Michael Stack wrote:
bq. Looks good. I like the bench marking tool.
bq.
bq. This is like a group commit in HLog. We have a group committing going
on inside in sync down in dfsclient. Does this group commit not 'group' enough.
bq.
bq. With this change, its no longer possible to sync each increment. We ok
w/ that? We should at least release note this difference in increment
behaviors and erring on the conservative side, I'd mark this an 'incompatible
change' rather than an 'improvement' just so its more likely folks will know of
this changed increment behavior.
HRegion has:
{code}
final Configuration conf;
{code}
Can we pass new config parameter through conf which enables this new feature ?
- Ted
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2116/#review2186
---
On 2011-09-29 19:27:36, Dhruba Borthakur wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2116/
bq. ---
bq.
bq. (Updated 2011-09-29 19:27:36)
bq.
bq.
bq. Review request for hbase and Ted Yu.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. The increment operation releases the rowlock before doing the sync to the
HLog. This improves performance of increments on hot rows. Introduced method
HLog.appendNoSync() that returns a txid. The increment method then release the
rowlock and invokes HLog.sync(txid). The HLog.sync(txid) returns only if all
the transactions upto the one identified by that txid has been successfully
sycned to HDFS.
bq.
bq. There is a single test TestIncrement that creates a 100 threads and
ensures that all the concurrent increments match the final expected result.
There is a benchmark TestHLogBench that measures the performance of the
appendNoSync call.
bq.
bq.
bq. This addresses bug HBASE-4487.
bq. https://issues.apache.org/jira/browse/HBASE-4487
bq.
bq.
bq. Diffs
bq. -
bq.
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177401
bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
1177401
bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestIncrement.java
PRE-CREATION
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java
PRE-CREATION
bq.
/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java
1177401
bq.
/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
1177401
bq.
bq. Diff: https://reviews.apache.org/r/2116/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. All unit tests pass now (expect TestDistributedLogSplitting,
TestRollingRestart, TestHTablePool), but I am seeing the same test to fail on
trunk, so these failures do not seem to be related to this patch.
bq.
bq.
bq. Thanks,
bq.
bq. Dhruba
bq.
bq.
The increment operation can release the rowlock before sync-ing the Hlog
Key: HBASE-4487
URL: https://issues.apache.org/jira/browse/HBASE-4487
Project: HBase
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt
This allows for better throughput when there are hot rows.I have seen this
change make a single row update improve from 400 increments/sec/server to
4000 increments/sec/server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117801#comment-13117801 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/ --- (Updated 2011-09-30 01:05:40.429350) Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. Changes --- Fixed bugs mentioned by Lars and a few others. Lots of tests are passing (TestCacheOnWrite for example). Running test suite now. Also added a small test in TestBlocksRead that I was attempting to test for HBASE-4496 but it's passing so still need to dig on what exactly is wrong there. Summary --- Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) This addresses bug HBASE-4422. https://issues.apache.org/jira/browse/HBASE-4422 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177471 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177471 /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177471 /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177471 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177471 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177471 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177471 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177471 /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177471 /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177471 /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177471 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177471 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177471 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java 1177471 /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 1177471 Diff: https://reviews.apache.org/r/2089/diff Testing --- Still working through some tests that aren't passing. Thanks, Jonathan Move block cache parameters and references into single CacheConf class -- Key: HBASE-4422
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117808#comment-13117808 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- bq. On 2011-09-29 22:18:31, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 1948 bq. https://reviews.apache.org/r/2118/diff/1/?file=46471#file46471line1948 bq. bq. Since expectedTimestampOfLatestVersion would only be used for use case from HBASE04497, I think we just need a reliable way of enforcing this 1 ms advance. bq. Maybe spinlocking till EnvironmentEdgeManager.currentTimeMillis() returns now+1 ? Why do we need to wait until currentTimeMillis() rolls? This is pretty ugly but I think we'll need to do something for HBASE-4497 and this will work (as I'm understanding it). But if we didn't do the +1, wouldn't the only time this would be an issue be if RS1 was hosting R1, it then got unassigned and reassigned BACK to RS1 again, and RS1 updates META with the new location of R1 in the same millisecond that it did the original update to META the last time it opened R1. That seems overwhelmingly impossible :) I'd be okay without the +1 business, but at the least, let's make sure it's documented in the top-level API javadoc (i think it is) - Jonathan --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2185 --- On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117807#comment-13117807 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2202 --- Ship it! Good stuff. I'm not crazy about the +1 ms but I think it's harmless (we'll literally never run into this for the META use case, as I've understood it) - Jonathan On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117811#comment-13117811 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2203 --- Nice stuff, Stack. Some questions: 1. The approach of isTimestampClash and the nowBytes = Bytes.toBytes(now + 1). It seems there is still a rare case, where EnvironmentEdgeManager.currentTimeMillis() returns expectedTimestampOfLatestVersion - 1. Then later on in put, updateKVTimestamps set it to the new now value, which happens to be expectedTimestampOfLatestVersion. Perhaps we can use now = expectedTimestampOfLatestVersion condition instead of now == expectedTimestampOfLatestVersion in isTimestampClash. Set the new time value as nowBytes = Bytes.toBytes(expectedTimestampOfLatestVersion + 1). 2. Do we need to modify coprocessor interfaces preCheckAndPut, postCheckAndPut, etc.? Perhaps we don't have any scenario for this yet. 3. Do we need the same thing for checkAndDelete? Perhaps we don't have any scenario for this yet. - Ming On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117849#comment-13117849 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-30 03:29:57.261186) Review request for hbase and Mingjie Lai. Changes --- Address reviews by Andrew Purtell and Mingjie Lai. Summary --- Proposed fix for HBASE-4070. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/HMaster.java 270f3f3 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 Diff: https://reviews.apache.org/r/2029/diff Testing (updated) --- Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() included in a new source file src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117884#comment-13117884 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/ --- (Updated 2011-09-30 04:58:20.313786) Review request for hbase and Mingjie Lai. Changes --- serialize/deserialize masterCoprocessors String value in ClusterStatus. Needed by HBase clients (e.g. for HBASE-3512). Summary --- Proposed fix for HBASE-4070. This addresses bug HBASE-4070. https://issues.apache.org/jira/browse/HBASE-4070 Diffs (updated) - src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java PRE-CREATION Diff: https://reviews.apache.org/r/2029/diff Testing --- Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() included in a new source file src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java. Thanks, Eugene [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4487) The increment operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117885#comment-13117885 ] jirapos...@reviews.apache.org commented on HBASE-4487: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2116/#review2206 --- /src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java https://reviews.apache.org/r/2116/#comment5140 Am not sure. Just to clarify Is this check not needed here then? - ramkrishna On 2011-09-29 19:27:36, Dhruba Borthakur wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2116/ bq. --- bq. bq. (Updated 2011-09-29 19:27:36) bq. bq. bq. Review request for hbase and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. The increment operation releases the rowlock before doing the sync to the HLog. This improves performance of increments on hot rows. Introduced method HLog.appendNoSync() that returns a txid. The increment method then release the rowlock and invokes HLog.sync(txid). The HLog.sync(txid) returns only if all the transactions upto the one identified by that txid has been successfully sycned to HDFS. bq. bq. There is a single test TestIncrement that creates a 100 threads and ensures that all the concurrent increments match the final expected result. There is a benchmark TestHLogBench that measures the performance of the appendNoSync call. bq. bq. bq. This addresses bug HBASE-4487. bq. https://issues.apache.org/jira/browse/HBASE-4487 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177401 bq./src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 1177401 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestIncrement.java PRE-CREATION bq. /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogBench.java PRE-CREATION bq. /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java 1177401 bq. /src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 1177401 bq. bq. Diff: https://reviews.apache.org/r/2116/diff bq. bq. bq. Testing bq. --- bq. bq. All unit tests pass now (expect TestDistributedLogSplitting, TestRollingRestart, TestHTablePool), but I am seeing the same test to fail on trunk, so these failures do not seem to be related to this patch. bq. bq. bq. Thanks, bq. bq. Dhruba bq. bq. The increment operation can release the rowlock before sync-ing the Hlog Key: HBASE-4487 URL: https://issues.apache.org/jira/browse/HBASE-4487 Project: HBase Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: appendNoSync4.txt, appendNoSync5.txt, appendNoSync6.txt This allows for better throughput when there are hot rows.I have seen this change make a single row update improve from 400 increments/sec/server to 4000 increments/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117904#comment-13117904 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- bq. On 2011-09-29 22:18:31, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 1948 bq. https://reviews.apache.org/r/2118/diff/1/?file=46471#file46471line1948 bq. bq. Since expectedTimestampOfLatestVersion would only be used for use case from HBASE04497, I think we just need a reliable way of enforcing this 1 ms advance. bq. Maybe spinlocking till EnvironmentEdgeManager.currentTimeMillis() returns now+1 ? bq. bq. Jonathan Gray wrote: bq. Why do we need to wait until currentTimeMillis() rolls? bq. bq. This is pretty ugly but I think we'll need to do something for HBASE-4497 and this will work (as I'm understanding it). bq. bq. But if we didn't do the +1, wouldn't the only time this would be an issue be if RS1 was hosting R1, it then got unassigned and reassigned BACK to RS1 again, and RS1 updates META with the new location of R1 in the same millisecond that it did the original update to META the last time it opened R1. That seems overwhelmingly impossible :) I'd be okay without the +1 business, but at the least, let's make sure it's documented in the top-level API javadoc (i think it is) I'm thinking that we should never need it but in case we do.. Two regionservers would have to be trying to update meta at the same time AND somehow the host that was carrying .META. had its clock go backwards -- e.g. .META. moved to a server whose clock was behind -- and then the update of location by one of the regionservers would have to arrive at EXACTLY the millisecond the original edit had AND the update that is going in would have to be the same value as is already there. I think its impossible but Ming postulated it so for peace of mind, we'll make sure every edit IF the value being updated has same coordinates for sure has its on timestamp... we'll +1 it if they happen to land in same millisecond (Yes its up in the javadoc that each edit has for sure unique timestamp). Ted, you +1 or you have a better idea for the +1'ing (The +1'ing has a unit test!) - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2185 --- On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117903#comment-13117903 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- bq. On 2011-09-30 01:19:20, Jonathan Gray wrote: bq. Good stuff. I'm not crazy about the +1 ms but I think it's harmless (we'll literally never run into this for the META use case, as I've understood it) Thanks boss. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2202 --- On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117905#comment-13117905 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- bq. On 2011-09-30 01:26:24, Ming Ma wrote: bq. Nice stuff, Stack. bq. bq. Some questions: bq. bq. 1. The approach of isTimestampClash and the nowBytes = Bytes.toBytes(now + 1). It seems there is still a rare case, where EnvironmentEdgeManager.currentTimeMillis() returns expectedTimestampOfLatestVersion - 1. Then later on in put, updateKVTimestamps set it to the new now value, which happens to be expectedTimestampOfLatestVersion. Perhaps we can use now = expectedTimestampOfLatestVersion condition instead of now == expectedTimestampOfLatestVersion in isTimestampClash. Set the new time value as nowBytes = Bytes.toBytes(expectedTimestampOfLatestVersion + 1). bq. 2. Do we need to modify coprocessor interfaces preCheckAndPut, postCheckAndPut, etc.? Perhaps we don't have any scenario for this yet. bq. 3. Do we need the same thing for checkAndDelete? Perhaps we don't have any scenario for this yet. 1. Doesn't updateKVTimestamps only mess with timestamps if the timestamp is set to HConstants.LATEST_TIMESTAMP? My setting the timestamp here before we go down into internalPut means the later updateKVTimestamps will not change the timestamps I've set? Let me check for sure. 2. Let me look into this. 3. I considered it but stopped myself. Figured I'd let someone ask for it before I went and implemented new functionality. Thanks Ming for review. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2203 --- On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13117967#comment-13117967 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2215 --- Ship it! Please replace checkAndSet with checkAndPut in javadoc - Ted On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118209#comment-13118209 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review --- src/main/java/org/apache/hadoop/hbase/HServerLoad.java https://reviews.apache.org/r/2029/#comment5157 Should return String[]? Trivial to do here given the data is in a Set. Then callers can enumerate. src/main/java/org/apache/hadoop/hbase/HServerLoad.java https://reviews.apache.org/r/2029/#comment5156 Make a list of unique names for HSL instead of a string? src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/2029/#comment5155 Should return String[]? Trivial to do here given the data is in a Set. Then callers can enumerate. Easy enough to display a list of strings in the jamon templates. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/2029/#comment5154 Should return String[]? Then callers can enumerate. Easy enough to display a list of strings in the jamon templates. - Andrew On 2011-09-30 04:58:20, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-30 04:58:20) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() included in a new source file src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118217#comment-13118217 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 3245 bq. https://reviews.apache.org/r/2029/diff/2/?file=46422#file46422line3245 bq. bq. is it okay just reuse the existing serverLoad, without rebuilding the whole thing? bq. bq. this.serverInfo.getLoad() bq. bq. but it may not be a big issue. sorry, I don't see a serverInfo member in HRegionServer..? bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 190 bq. https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line190 bq. bq. an extra indent? fixed; thanks. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 465 bq. https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line465 bq. bq. Just a naming suggestion for walCoprocessors=coprocessors. Although right now WALObserver is the only RS level CP, but it'd better not to indicate here. I changed the parameter name to rsCoprocessors and added a javadoc @param note; what do you think? Could also just call it coprocessors as you suggested. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 505 bq. https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line505 bq. bq. getSimpleName() only returns the class name w/o package info. I think we should use the whole class name for the comparator. bq. bq. Andrew Purtell wrote: bq. Full classnames are canonical and avoid accidental conflict, but are unwieldy and not so nice to look at. Judgement call, I'd say. I think it unlikely that for any given deployment two coprocessors will have the same class name but be in a different package. I would agree with Andrew and use getSimpleName(). bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java, line 261 bq. https://reviews.apache.org/r/2029/diff/2/?file=46420#file46420line261 bq. bq. The new method is a little odd. how about just getCoprocessors(). Removed HMasterInterface.java changes at Andy's recommendation: now no difference with trunk. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java, line 1514 bq. https://reviews.apache.org/r/2029/diff/2/?file=46421#file46421line1514 bq. bq. still the method name? changed to getCoprocessors() at Andy's recommendation. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 3244 bq. https://reviews.apache.org/r/2029/diff/2/?file=46422#file46422line3244 bq. bq. same as HMasterInterface. can we just generateCoprocessorString() = String getCoprocessors()? changed; thanks. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 1652 bq. https://reviews.apache.org/r/2029/diff/2/?file=46418#file46418line1652 bq. bq. do we really need to provide it thru HTAdmin? if we do, the returned value is not correct in case of exception. removed differences from trunk; you are right: we don't need it here. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java, line 24 bq. https://reviews.apache.org/r/2029/diff/2/?file=46424#file46424line24 bq. bq. The file TestCoprocessorEndpoint is particularly for testing cp endpoint. It's odd to put the new test cases to this file. Maybe TestClassLoading is a better place. Good idea; moved to a new source file: TestCoprocessorReporting.java. bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java, line 201 bq. https://reviews.apache.org/r/2029/diff/2/?file=46424#file46424line201 bq. bq. according the previous override comparator in HRL, the cps should be in alphabet order. so you don't need to compare this 2 strings. Thanks; removed extra string and added comment with your point. - Eugene --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2183 --- On 2011-09-30 04:58:20, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq.
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118218#comment-13118218 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- bq. On 2011-09-29 21:52:40, Andrew Purtell wrote: bq. bq. bq. Andrew Purtell wrote: bq. Actually, I do have a comment. Do we need a method in HMasterInterface for getting the coprocessor set? Can't it be extracted from HSL? bq. bq. Andrew Purtell wrote: bq. I meant via ClusterStatus, therefore via HBaseAdmin.getClusterStatus(). Removed HMasterInterface change since we don't need it. ClusterStatus constructor has an additional param that is the string representation of the Master's loaded coprocessor set. HBaseAdmin can now use getClusterStatus() to get Master's coprocessor set. HBASE-3512's patch will be updated to use this to show master coprocessors in the shell. - Eugene --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2179 --- On 2011-09-30 04:58:20, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-30 04:58:20) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() included in a new source file src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118220#comment-13118220 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- bq. On 2011-09-29 21:52:40, Andrew Purtell wrote: bq. bq. bq. Andrew Purtell wrote: bq. Actually, I do have a comment. Do we need a method in HMasterInterface for getting the coprocessor set? Can't it be extracted from HSL? bq. bq. Andrew Purtell wrote: bq. I meant via ClusterStatus, therefore via HBaseAdmin.getClusterStatus(). bq. bq. Eugene Koontz wrote: bq. Removed HMasterInterface change since we don't need it. bq. bq. ClusterStatus constructor has an additional param that is the string representation of the Master's loaded coprocessor set. bq. bq. HBaseAdmin can now use getClusterStatus() to get Master's coprocessor set. HBASE-3512's patch will be updated to use this to show master coprocessors in the shell. Cool, thanks Eugene - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2179 --- On 2011-09-30 04:58:20, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-30 04:58:20) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() included in a new source file src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4070) [Coprocessors] Improve region server metrics to report loaded coprocessors to master
[ https://issues.apache.org/jira/browse/HBASE-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118221#comment-13118221 ] jirapos...@reviews.apache.org commented on HBASE-4070: -- bq. On 2011-09-29 22:42:26, Mingjie Lai wrote: bq. src/main/java/org/apache/hadoop/hbase/HServerLoad.java, line 465 bq. https://reviews.apache.org/r/2029/diff/2/?file=46417#file46417line465 bq. bq. Just a naming suggestion for walCoprocessors=coprocessors. Although right now WALObserver is the only RS level CP, but it'd better not to indicate here. bq. bq. Eugene Koontz wrote: bq. I changed the parameter name to rsCoprocessors and added a javadoc @param note; what do you think? Could also just call it coprocessors as you suggested. If the coprocessor list is within a regionserver load report, it's implicitly about the regionserver so no rs prefix is needed on the member, no? Seems cleaner not to use one. - Andrew --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2029/#review2183 --- On 2011-09-30 04:58:20, Eugene Koontz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2029/ bq. --- bq. bq. (Updated 2011-09-30 04:58:20) bq. bq. bq. Review request for hbase and Mingjie Lai. bq. bq. bq. Summary bq. --- bq. bq. Proposed fix for HBASE-4070. bq. bq. bq. This addresses bug HBASE-4070. bq. https://issues.apache.org/jira/browse/HBASE-4070 bq. bq. bq. Diffs bq. - bq. bq.src/main/jamon/org/apache/hbase/tmpl/master/MasterStatusTmpl.jamon abeb850 bq.src/main/jamon/org/apache/hbase/tmpl/regionserver/RSStatusTmpl.jamon be6fceb bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 01bc1dd bq.src/main/java/org/apache/hadoop/hbase/HServerLoad.java 0c680e4 bq.src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java dbae4fd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java f80d232 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq. src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorReporting.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2029/diff bq. bq. bq. Testing bq. --- bq. bq. Two new tests : testRegionServerCoprocessorReported() and testMasterServerCoprocessorsReported() included in a new source file src/test/java/o.a.h.h/coprocessor/TestCoprocessorReporting.java. bq. bq. bq. Thanks, bq. bq. Eugene bq. bq. [Coprocessors] Improve region server metrics to report loaded coprocessors to master Key: HBASE-4070 URL: https://issues.apache.org/jira/browse/HBASE-4070 Project: HBase Issue Type: Improvement Affects Versions: 0.90.3 Reporter: Mingjie Lai Assignee: Eugene Koontz Attachments: HBASE-4070.patch, HBASE-4070.patch, master-web-ui.jpg, rs-status-web-ui.jpg HBASE-3512 is about listing loaded cp classes at shell. To make it more generic, we need a way to report this piece of information from region to master (or just at region server level). So later on, we can display the loaded class names at shell as well as web console. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2794) Utilize ROWCOL bloom filter if multiple columns within same family are requested in a Get
[
https://issues.apache.org/jira/browse/HBASE-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118294#comment-13118294
]
jirapos...@reviews.apache.org commented on HBASE-2794:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2084/#review2226
---
Ship it!
I'm +0 on commmitting this. I tried reviewing it but I don't know this code
well. The added unit test is nicely intrusive and the asserts look right.
What about Nicolas's performance concerns. How are they addressed by this
patch? I'm running a build of the patch and if that passes I'm +1 on commit.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/2084/#comment5175
Interesting method name. We should use this pattern everywhere we have to
do this.
src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
https://reviews.apache.org/r/2084/#comment5176
Should we get rid of this javadoc if an override? (Let us know can do on
commit)
- Michael
On 2011-09-29 21:05:20, Mikhail Bautin wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2084/
bq. ---
bq.
bq. (Updated 2011-09-29 21:05:20)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Previously we only used row-column Bloom filters for scans that only
requested one column. We have seen production queries that request up to 200
columns, and with say ~6 store files per store (region / column family
combination) this might have resulted in 1200 block read operations in the
worst case. With this diff we will be avoiding seeks on store files that we
know don't contain the row/column of interest when using an
ExplicitColumnTracker. The performance should remain the same for column range
queries.
bq.
bq.
bq. This addresses bug HBASE-2794.
bq. https://issues.apache.org/jira/browse/HBASE-2794
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 585c4a8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java
f5173c4
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java a3d778e
bq.
src/main/java/org/apache/hadoop/hbase/regionserver/AbstractKeyValueScanner.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 7cbdb98
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
9d9895c
bq.src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
6cdada7
bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 4aa72de
bq.src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
68cdac5
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
fd9e7ef
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
08d3ba4
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
ac2348e
bq.src/main/java/org/apache/hadoop/hbase/util/CollectionBackedScanner.java
32f88fb
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestKeyValueHeap.java
a5d13f7
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestMultiColumnScanner.java
baee696
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWithBloomError.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2084/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Existing unit tests. A new unit test (TestScanWithBloomError). Load
testing using HBaseTest.
bq.
bq.
bq. Thanks,
bq.
bq. Mikhail
bq.
bq.
Utilize ROWCOL bloom filter if multiple columns within same family are
requested in a Get
-
Key: HBASE-2794
URL: https://issues.apache.org/jira/browse/HBASE-2794
Project: HBase
Issue Type: Improvement
Components: performance
Reporter: Kannan Muthukkaruppan
Assignee: Mikhail Bautin
Fix For: 0.92.0
Noticed the following snippet in StoreFile.java:Scanner:shouldSeek():
{code}
switch(bloomFilterType) {
case ROW:
key = row;
break;
case ROWCOL:
if (columns.size() == 1) {
byte[] col = columns.first();
key = Bytes.add(row, col);
break;
}
//$FALL-THROUGH$
default:
return true;
}
{code}
If columns.size 1,
[jira] [Commented] (HBASE-4422) Move block cache parameters and references into single CacheConf class
[ https://issues.apache.org/jira/browse/HBASE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118323#comment-13118323 ] jirapos...@reviews.apache.org commented on HBASE-4422: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2089/ --- (Updated 2011-09-30 19:36:04.480088) Review request for hbase, Dhruba Borthakur, Michael Stack, and Li Pi. Changes --- More bug fixes and removed some debug output. Unit tests are passing for me when I run them in eclipse. For some reason, lots of stuff is frozen when I run it on my remote dev box but having problems with and without this patch. Appreciate reviews and unit test runs! Thanks Summary --- Creates a new CacheConfig class and moves almost everything block cache related into this single class. Adding new configuration params and booleans and such should be much better. All tests are NOT passing yet, still working on it, but wanted to have something up today. Basically code complete but broken :) This addresses bug HBASE-4422. https://issues.apache.org/jira/browse/HBASE-4422 Diffs (updated) - /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java 1177760 /src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java 1177760 /src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1177760 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java 1177760 /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 1177760 /src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java 1177760 /src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java 1177760 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java 1177760 /src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileReaderV1.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java 1177760 /src/test/java/org/apache/hadoop/hbase/HFilePerformanceEvaluation.java 1177760 /src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java 1177760 /src/main/java/org/apache/hadoop/hbase/util/CompressionTest.java 1177760 /src/main/java/org/apache/hadoop/hbase/util/BloomFilterFactory.java 1177760 /src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java 1177760 /src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1177760 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1177760 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1177760 /src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1177760 /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java 1177760 /src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1177760 Diff: https://reviews.apache.org/r/2089/diff Testing --- Still working through some tests that aren't passing. Thanks, Jonathan Move block cache parameters and references into single CacheConf class -- Key: HBASE-4422 URL: https://issues.apache.org/jira/browse/HBASE-4422 Project: HBase
[jira] [Commented] (HBASE-4507) Create checkAndPut variant that exposes timestamp / UUID
[ https://issues.apache.org/jira/browse/HBASE-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118383#comment-13118383 ] jirapos...@reviews.apache.org commented on HBASE-4507: -- bq. On 2011-09-30 01:26:24, Ming Ma wrote: bq. Nice stuff, Stack. bq. bq. Some questions: bq. bq. 1. The approach of isTimestampClash and the nowBytes = Bytes.toBytes(now + 1). It seems there is still a rare case, where EnvironmentEdgeManager.currentTimeMillis() returns expectedTimestampOfLatestVersion - 1. Then later on in put, updateKVTimestamps set it to the new now value, which happens to be expectedTimestampOfLatestVersion. Perhaps we can use now = expectedTimestampOfLatestVersion condition instead of now == expectedTimestampOfLatestVersion in isTimestampClash. Set the new time value as nowBytes = Bytes.toBytes(expectedTimestampOfLatestVersion + 1). bq. 2. Do we need to modify coprocessor interfaces preCheckAndPut, postCheckAndPut, etc.? Perhaps we don't have any scenario for this yet. bq. 3. Do we need the same thing for checkAndDelete? Perhaps we don't have any scenario for this yet. bq. bq. Michael Stack wrote: bq. 1. Doesn't updateKVTimestamps only mess with timestamps if the timestamp is set to HConstants.LATEST_TIMESTAMP? My setting the timestamp here before we go down into internalPut means the later updateKVTimestamps will not change the timestamps I've set? Let me check for sure. bq. 2. Let me look into this. bq. 3. I considered it but stopped myself. Figured I'd let someone ask for it before I went and implemented new functionality. bq. bq. Thanks Ming for review. bq. I took another look and now I see what you are saying Ming. New patch coming. For 1., for the case where we are all up in the same millisecond, then I'll pass the 'now' to the innerPut so that when it does its updateKVTimestamps, its operating with the same 'now'. I fixed 2. adding this new method to RegionObserver and handling the ripple through the code base (Man, its harder now adding new methods to a regionserver!) I'm not going to do 3. in scope of this issue. @Ted New patch will include your suggestion too. Thanks for review lads. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2118/#review2203 --- On 2011-09-29 23:52:43, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2118/ bq. --- bq. bq. (Updated 2011-09-29 23:52:43) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds a checkAndPut that takes a timestamp bq. bq. bq. This addresses bug hbase-4507. bq. https://issues.apache.org/jira/browse/hbase-4507 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 3679c02 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 6ec857c bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 96b763b bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java 99b34cc bq. bq. Diff: https://reviews.apache.org/r/2118/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Create checkAndPut variant that exposes timestamp / UUID Key: HBASE-4507 URL: https://issues.apache.org/jira/browse/HBASE-4507 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: stack Michael checked the checkAndPut which doesn't expose timestamp. A variant of checkAndPut should be created to expose timestamp which is written into a column specified by additional parameters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4522) Make hbase-site-custom.xml override the hbase-site.xml
[ https://issues.apache.org/jira/browse/HBASE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118401#comment-13118401 ] jirapos...@reviews.apache.org commented on HBASE-4522: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2135/ --- Review request for Paul Lindner. Summary --- The motivation for diff is that we want to override some config change for any specific cluster easily by just adding the config entries in the hbase-site-custom.xml for that cluster. This change adds the hbase-site-custom.xml configuration file into HBaseConfiguration. This addresses bug HBASE-4522. https://issues.apache.org/jira/browse/HBASE-4522 Diffs - src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java 06fd29c Diff: https://reviews.apache.org/r/2135/diff Testing --- This has been in production on multiple clusters for a few weeks. Thanks, Mikhail Make hbase-site-custom.xml override the hbase-site.xml -- Key: HBASE-4522 URL: https://issues.apache.org/jira/browse/HBASE-4522 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Liyin Tang Priority: Minor Fix For: 0.94.0 The motivation for diff is that we want to override some config change for any specific cluster easily by just adding the config entries in the hbase-site-custom.xml for that cluster. This change adds the hbase-site-custom.xml configuration file into HBaseConfiguration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
