[jira] [Created] (HBASE-4680) FSUtils.isInSafeMode() requires superuser privileges when HDFS permissions checking enabled
FSUtils.isInSafeMode() requires superuser privileges when HDFS permissions
checking enabled
---
Key: HBASE-4680
URL: https://issues.apache.org/jira/browse/HBASE-4680
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.92.0
Reporter: Gary Helmling
Priority: Critical
Fix For: 0.92.0
The HDFS safe mode check workaround introduced by HBASE-4510 performs a
{{FileSystem.setPermission()}} operation on the root directory (/) when
attempting to trigger a {{SafeModeException}}. As a result, it requires
superuser privileges when running with DFS permission checking enabled.
Changing the operations to act on the HBase root directory should be safe,
since the master process must have write access to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4681) StringIndexOutOfBoundsException parsing Hostname
[
https://issues.apache.org/jira/browse/HBASE-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135739#comment-13135739
]
stack commented on HBASE-4681:
--
The second start worked.
StringIndexOutOfBoundsException parsing Hostname
Key: HBASE-4681
URL: https://issues.apache.org/jira/browse/HBASE-4681
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.92.0
Starting a 0.92 on 0.90 data with 0.90 zk ensemble I got this:
{code}
2011-10-26 06:13:53,920 ERROR
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
already exists and this is not a retry
2011-10-26 06:13:53,927 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1937)
at
org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81)
at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63)
at
org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148)
at
org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:346)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
at java.lang.Thread.run(Thread.java:662)
2011-10-26 06:13:53,929 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2011-10-26 06:13:53,929 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service thre
{code}
I thought this had been fixed. Dig in .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4388) Second start after migration from 90 to trunk crashes
[ https://issues.apache.org/jira/browse/HBASE-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135743#comment-13135743 ] stack commented on HBASE-4388: -- I just started 0.92 over a crashed out 0.90 cluster that had 700 regions in it. Migration looks like it went off fine (Excepting HBASE-4681). I'm going to commit this in next day or so so reviews appreciated (I'll address Ted's last comment on commit). Second start after migration from 90 to trunk crashes - Key: HBASE-4388 URL: https://issues.apache.org/jira/browse/HBASE-4388 Project: HBase Issue Type: Bug Components: master, migration Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Blocker Fix For: 0.92.0 Attachments: 4388-v2.txt, 4388-v3.txt, 4388-v4.txt, 4388.txt, hbase-4388-root.dir.tgz, hbase-master-nase.log, meta.tgz I started a trunk cluster to upgrade from 90, inserted a ton of data, then did a clean shutdown. When I started again, I got the following exception: 11/09/13 12:29:09 INFO master.HMaster: Meta has HRI with HTDs. Updating meta now. 11/09/13 12:29:09 FATAL master.HMaster: Unhandled exception. Starting shutdown. java.lang.NegativeArraySizeException: -102 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:147) at org.apache.hadoop.hbase.HTableDescriptor.readFields(HTableDescriptor.java:606) at org.apache.hadoop.hbase.migration.HRegionInfo090x.readFields(HRegionInfo090x.java:641) at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:133) at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:103) at org.apache.hadoop.hbase.util.Writables.getHRegionInfoForMigration(Writables.java:228) at org.apache.hadoop.hbase.catalog.MetaEditor.getHRegionInfoForMigration(MetaEditor.java:350) at org.apache.hadoop.hbase.catalog.MetaEditor$1.visit(MetaEditor.java:273) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:633) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:255) at org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:235) at org.apache.hadoop.hbase.catalog.MetaEditor.updateMetaWithNewRegionInfo(MetaEditor.java:284) at org.apache.hadoop.hbase.catalog.MetaEditor.migrateRootAndMeta(MetaEditor.java:298) at org.apache.hadoop.hbase.master.HMaster.updateMetaWithNewHRI(HMaster.java:529) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:472) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4669) Add an option of using round-robin assignment for enabling table
[
https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jieshan Bean updated HBASE-4669:
Attachment: HBASE-4669-Trunk-V2.patch
HBASE-4669-90.patch
{noformat}
Do we have to introduce assignUserRegionsToOnlineServers() ?
{noformat}
I think it's necessary. assignUserRegionsToOnlineServers is a polymorphic
method of assignUserRegions. But it can get the online servers first.
{noformat}
I don't think IOE should be swallowed:
{noformat}
Yes, I agree with it. I've changed it in v2.
Add an option of using round-robin assignment for enabling table
Key: HBASE-4669
URL: https://issues.apache.org/jira/browse/HBASE-4669
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 0.90.4, 0.94.0
Reporter: Jieshan Bean
Priority: Minor
Fix For: 0.94.0, 0.90.5
Attachments: HBASE-4669-90.patch, HBASE-4669-Trunk-V2.patch,
HBASE-4669-Trunk.patch
Under some scenarios, we use the function of disable/enable HTable. But
currently, enable HTable uses the random-assignment. We hope all the regions
show a better distribution, no matter how many regions and how many
regionservers.
So I suggest to add an option of using round-robin assignment on
enable-table.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4669) Add an option of using round-robin assignment for enabling table
[ https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135754#comment-13135754 ] jirapos...@reviews.apache.org commented on HBASE-4669: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2568/ --- Review request for Ted Yu, Michael Stack and ramkrishna vasudevan. Summary --- Under some scenarios, we use the function of disable/enable HTable. But currently, enable HTable uses the random-assignment. We hope all the regions show a better distribution, no matter how many regions and how many regionservers. So I suggest to add an option of using round-robin assignment on enable-table. This addresses bug HBASE-4669. https://issues.apache.org/jira/browse/HBASE-4669 Diffs - /src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1188986 /src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java 1188986 /src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java 1188986 /src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java 1188986 /src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java 1188986 Diff: https://reviews.apache.org/r/2568/diff Testing --- Thanks, Jieshan Add an option of using round-robin assignment for enabling table Key: HBASE-4669 URL: https://issues.apache.org/jira/browse/HBASE-4669 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4, 0.94.0 Reporter: Jieshan Bean Priority: Minor Fix For: 0.94.0, 0.90.5 Attachments: HBASE-4669-90.patch, HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch Under some scenarios, we use the function of disable/enable HTable. But currently, enable HTable uses the random-assignment. We hope all the regions show a better distribution, no matter how many regions and how many regionservers. So I suggest to add an option of using round-robin assignment on enable-table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.
[ https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Bauer updated HBASE-4377: --- Attachment: hbase-4377.trunk.v4.txt i think hbase-4377.trunk.v3.txt cannot comile because of removing doFsck(boolean) function and now we have doFsck(Configuration, boolean), so this patch correct this to. Apply clear to trunk and 0.92 [hbck] Offline rebuild .META. from fs data only. Key: HBASE-4377 URL: https://issues.apache.org/jira/browse/HBASE-4377 Project: HBase Issue Type: New Feature Affects Versions: 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90-v4.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90.v3.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.trunk.v3.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v1.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v2.patch, hbase-4377-trunk.v2.patch, hbase-4377.trunk.v3.txt, hbase-4377.trunk.v4.txt In a worst case situation, it may be helpful to have an offline .META. rebuilder that just looks at the file system's .regioninfos and rebuilds meta from scratch. Users could move bad regions out until there is a clean rebuild. It would likely fill in region split holes. Follow on work could given options to merge or select regions that overlap, or do online rebuilds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.
[ https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135758#comment-13135758 ] Hadoop QA commented on HBASE-4377: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500832/EXT_ATU_05f84d32cbc0bdabf00e00bc2f3570f0.regioninfo against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/69//console This message is automatically generated. [hbck] Offline rebuild .META. from fs data only. Key: HBASE-4377 URL: https://issues.apache.org/jira/browse/HBASE-4377 Project: HBase Issue Type: New Feature Affects Versions: 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90-v4.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90.v3.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.trunk.v3.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v1.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v2.patch, EXT_AC.regioninfo, EXT_ATU_05f84d32cbc0bdabf00e00bc2f3570f0.regioninfo, hbase-4377-trunk.v2.patch, hbase-4377.trunk.v3.txt, hbase-4377.trunk.v4.txt In a worst case situation, it may be helpful to have an offline .META. rebuilder that just looks at the file system's .regioninfos and rebuilds meta from scratch. Users could move bad regions out until there is a clean rebuild. It would likely fill in region split holes. Follow on work could given options to merge or select regions that overlap, or do online rebuilds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4377) [hbck] Offline rebuild .META. from fs data only.
[ https://issues.apache.org/jira/browse/HBASE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Bauer updated HBASE-4377: --- Attachment: EXT_ATU_05f84d32cbc0bdabf00e00bc2f3570f0.regioninfo EXT_AC.regioninfo If i remember both regioninfo cause problem with empty HRegionInfo.tableName. My test instance almost all time is running trunk, so many thinks could be broken :) [hbck] Offline rebuild .META. from fs data only. Key: HBASE-4377 URL: https://issues.apache.org/jira/browse/HBASE-4377 Project: HBase Issue Type: New Feature Affects Versions: 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90-v4.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.0.90.v3.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data-.trunk.v3.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v1.patch, 0001-HBASE-4377-hbck-Offline-rebuild-.META.-from-fs-data.0.92.v2.patch, EXT_AC.regioninfo, EXT_ATU_05f84d32cbc0bdabf00e00bc2f3570f0.regioninfo, hbase-4377-trunk.v2.patch, hbase-4377.trunk.v3.txt, hbase-4377.trunk.v4.txt In a worst case situation, it may be helpful to have an offline .META. rebuilder that just looks at the file system's .regioninfos and rebuilds meta from scratch. Users could move bad regions out until there is a clean rebuild. It would likely fill in region split holes. Follow on work could given options to merge or select regions that overlap, or do online rebuilds. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4669) Add an option of using round-robin assignment for enabling table
[ https://issues.apache.org/jira/browse/HBASE-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135810#comment-13135810 ] jirapos...@reviews.apache.org commented on HBASE-4669: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2568/#review2852 --- /src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java https://reviews.apache.org/r/2568/#comment6386 Please add enclosing curly braces for line 179. /src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java https://reviews.apache.org/r/2568/#comment6387 I think setting hbase.master.enabletable.roundrobin should be done in this method. - Ted On 2011-10-26 06:51:56, Jieshan Bean wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2568/ bq. --- bq. bq. (Updated 2011-10-26 06:51:56) bq. bq. bq. Review request for Ted Yu, Michael Stack and ramkrishna vasudevan. bq. bq. bq. Summary bq. --- bq. bq. Under some scenarios, we use the function of disable/enable HTable. But currently, enable HTable uses the random-assignment. We hope all the regions show a better distribution, no matter how many regions and how many regionservers. bq. bq. So I suggest to add an option of using round-robin assignment on enable-table. bq. bq. bq. This addresses bug HBASE-4669. bq. https://issues.apache.org/jira/browse/HBASE-4669 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1188986 bq./src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java 1188986 bq./src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java 1188986 bq. /src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java 1188986 bq./src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java 1188986 bq. bq. Diff: https://reviews.apache.org/r/2568/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Jieshan bq. bq. Add an option of using round-robin assignment for enabling table Key: HBASE-4669 URL: https://issues.apache.org/jira/browse/HBASE-4669 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4, 0.94.0 Reporter: Jieshan Bean Priority: Minor Fix For: 0.94.0, 0.90.5 Attachments: HBASE-4669-90.patch, HBASE-4669-Trunk-V2.patch, HBASE-4669-Trunk.patch Under some scenarios, we use the function of disable/enable HTable. But currently, enable HTable uses the random-assignment. We hope all the regions show a better distribution, no matter how many regions and how many regionservers. So I suggest to add an option of using round-robin assignment on enable-table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13135894#comment-13135894 ] jirapos...@reviews.apache.org commented on HBASE-4120: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1421/ --- (Updated 2011-10-26 12:24:20.090230) Review request for hbase. Changes --- Move most of the priority function to PriorityFunction class. Make the PriorityHBaseServer much more slim.Fix all review comments. Summary --- Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like get,scan,put and delete can have priorities. This addresses bug HBase-4120. https://issues.apache.org/jira/browse/HBase-4120 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1155226 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1155226 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1155226 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION Diff: https://reviews.apache.org/r/1421/diff Testing --- Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client. Thanks, Jia isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136004#comment-13136004 ] jirapos...@reviews.apache.org commented on HBASE-4120: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1421/ --- (Updated 2011-10-26 14:13:10.677021) Review request for hbase. Changes --- Fix bugs Summary --- Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like get,scan,put and delete can have priorities. This addresses bug HBase-4120. https://issues.apache.org/jira/browse/HBase-4120 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION Diff: https://reviews.apache.org/r/1421/diff Testing --- Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client. Thanks, Jia isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4508) Backport HBASE-3777 to 0.90 branch
[ https://issues.apache.org/jira/browse/HBASE-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4508: -- Resolution: Fixed Release Note: A new config parameter, hbase.connection.per.config, has been added. If set to true, there is no connection sharing. This is default setting. If set to false, connection sharing would be enabled so that fewer connections to zookeeper are used. was: A new config parameter, hbase.connection.per.config, has been added. If set to true, there is no connection sharing. If set to false, connection sharing would be enabled so that fewer connections to zookeeper are used. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Backport HBASE-3777 to 0.90 branch -- Key: HBASE-4508 URL: https://issues.apache.org/jira/browse/HBASE-4508 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Bright Fulton Fix For: 0.90.5 Attachments: HBASE-4508.v1.patch, HBASE-4508.v2.patch, HBASE-4508.v3.patch, HBASE-4508.v4.git.patch, HBASE-4508.v4.patch, HBASE-4508.v5.patch See discussion here: http://search-hadoop.com/m/MJBId1aazTR1/backporting+HBASE-3777+to+0.90subj=backporting+HBASE+3777+to+0+90 Rocketfuel has been running 0.90.3 with HBASE-3777 since its resolution. They have 10 RS nodes , 1 Master and 1 Zookeeper Live writes and reads but super heavy on reads. Cache hit is pretty high. The qps on one of their data centers is 50K. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option
[ https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136040#comment-13136040 ] Akash Ashok commented on HBASE-4224: Thanks Ted. I shall make those changes. I was testing a few more things and have a few queries As per the Javadoc the flush() method in HBaseAdmin is supposed to be asynchronous but its actually not. 1. It might make sense to make this async for table flushes, where the regions on different Region Servers can be flushed simultaneously instead of waiting for each flush to complete. 2. Same holds good for log rolling Also it might even make sense to have a method to flush a list of regions belonging to a particular region server to support table flushes, instead of making round trip to and from the server for every region. Does this sound reasonable ? Need a flush by regionserver rather than by table option Key: HBASE-4224 URL: https://issues.apache.org/jira/browse/HBASE-4224 Project: HBase Issue Type: Bug Components: shell Reporter: stack Assignee: Akash Ashok Attachments: HBase-4224.patch This evening needed to clean out logs on the cluster. logs are by regionserver. to let go of logs, we need to have all edits emptied from memory. only flush is by table or region. We need to be able to flush the regionserver. Need to add this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4679) Thrift null mutation error
[ https://issues.apache.org/jira/browse/HBASE-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136049#comment-13136049 ] Nicolas Spiegelberg commented on HBASE-4679: does Hadoop QA not understand files made by GIT? This applied cleanly to 92 trunk as of rev 1188410. Thrift null mutation error -- Key: HBASE-4679 URL: https://issues.apache.org/jira/browse/HBASE-4679 Project: HBase Issue Type: Bug Affects Versions: 0.92.0, 0.94.0 Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Attachments: HBASE-4679.patch When using null as a value for a mutation, HBasse thrift client failed and threw an error. We should instad check for a null byte buffer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option
[ https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136059#comment-13136059 ] Ted Yu commented on HBASE-4224: --- For #1 above, I think we should keep the current behavior. People may have become dependent on the current behavior. For #2, we can introduce async log rolling. For #3, supporting list of regions is nice to have. All new commands and parameters should be documented in respective .rb files. Thanks Need a flush by regionserver rather than by table option Key: HBASE-4224 URL: https://issues.apache.org/jira/browse/HBASE-4224 Project: HBase Issue Type: Bug Components: shell Reporter: stack Assignee: Akash Ashok Attachments: HBase-4224.patch This evening needed to clean out logs on the cluster. logs are by regionserver. to let go of logs, we need to have all edits emptied from memory. only flush is by table or region. We need to be able to flush the regionserver. Need to add this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option
[ https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136066#comment-13136066 ] stack commented on HBASE-4224: -- On #1, sounds great. Add a new API? On #2, ditto. On #3, that'd be useful, for sure. Need a flush by regionserver rather than by table option Key: HBASE-4224 URL: https://issues.apache.org/jira/browse/HBASE-4224 Project: HBase Issue Type: Bug Components: shell Reporter: stack Assignee: Akash Ashok Attachments: HBase-4224.patch This evening needed to clean out logs on the cluster. logs are by regionserver. to let go of logs, we need to have all edits emptied from memory. only flush is by table or region. We need to be able to flush the regionserver. Need to add this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4649) Add atomic bulk load function to region server
[ https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4649: -- Attachment: hbase-4649.v2.patch Updated to address comments. hbase-4649.v2.patch Add atomic bulk load function to region server -- Key: HBASE-4649 URL: https://issues.apache.org/jira/browse/HBASE-4649 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.90.4, 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch, hbase-4649.v2.patch Add a method that atomically bulk load multiple hfiles. Row atomicity guarantees for multi-column family rows require this functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
[ https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4650: -- Attachment: hbase-4650.v2.patch Updated patch to address comments and add error handling tests. hbase-4650.v2.patch Update LoadIncrementalHFiles to use atomic bulk load RS mechanism - Key: HBASE-4650 URL: https://issues.apache.org/jira/browse/HBASE-4650 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch, 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch, hbase-4650.v2.patch MR jobs and command line bulk load execution runs use the LoadIncrementalHFile.doBulkLoad. This needs to be updated to group HFiles by row/region so that rows can be atomically loaded multiple column families. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4677) Remove old single bulkLoadHFile method
[ https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4677: -- Attachment: hbase-4677.patch Removes methods (for 0.92/trunk branch) Remove old single bulkLoadHFile method -- Key: HBASE-4677 URL: https://issues.apache.org/jira/browse/HBASE-4677 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4677.patch In review for HBASE-4649, there is some debate as whether to remove, deprecate, or leave the HRegionServer.bulkLoadHFile method. https://reviews.apache.org/r/2545/ . This jira will take care of that for the 0.92 and trunk releases, and allow the same patch to remain for an optional 0.90.x patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4677) Remove old single bulkLoadHFile method
[ https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4677: -- Status: Patch Available (was: Open) Remove old single bulkLoadHFile method -- Key: HBASE-4677 URL: https://issues.apache.org/jira/browse/HBASE-4677 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4677.patch In review for HBASE-4649, there is some debate as whether to remove, deprecate, or leave the HRegionServer.bulkLoadHFile method. https://reviews.apache.org/r/2545/ . This jira will take care of that for the 0.92 and trunk releases, and allow the same patch to remain for an optional 0.90.x patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option
[ https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136093#comment-13136093 ] Akash Ashok commented on HBASE-4224: For #1: I was thinking I'd be better to incorporate withing the same flush() function instead of a new API. Considering Ted's comment, It actually makes sense to retain the behavior, thus we could still have the same behavior for the flush() function but make sure it only exit after all the required regions have been flushed, by having waits within the function. That way we don't have to add a new API. The behavior would remain the same. Just that thing would be a little faster as flushes happen in parallel. Need a flush by regionserver rather than by table option Key: HBASE-4224 URL: https://issues.apache.org/jira/browse/HBASE-4224 Project: HBase Issue Type: Bug Components: shell Reporter: stack Assignee: Akash Ashok Attachments: HBase-4224.patch This evening needed to clean out logs on the cluster. logs are by regionserver. to let go of logs, we need to have all edits emptied from memory. only flush is by table or region. We need to be able to flush the regionserver. Need to add this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4650) Update LoadIncrementalHFiles to use atomic bulk load RS mechanism
[ https://issues.apache.org/jira/browse/HBASE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136094#comment-13136094 ] Hadoop QA commented on HBASE-4650: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500899/hbase-4650.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/71//console This message is automatically generated. Update LoadIncrementalHFiles to use atomic bulk load RS mechanism - Key: HBASE-4650 URL: https://issues.apache.org/jira/browse/HBASE-4650 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.patch, 0001-HBASE-4650-Update-LoadIncrementalHFiles-to-use-atomi.prelim.patch, hbase-4650.v2.patch MR jobs and command line bulk load execution runs use the LoadIncrementalHFile.doBulkLoad. This needs to be updated to group HFiles by row/region so that rows can be atomically loaded multiple column families. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4649) Add atomic bulk load function to region server
[ https://issues.apache.org/jira/browse/HBASE-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136097#comment-13136097 ] Hadoop QA commented on HBASE-4649: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500898/hbase-4649.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/72//console This message is automatically generated. Add atomic bulk load function to region server -- Key: HBASE-4649 URL: https://issues.apache.org/jira/browse/HBASE-4649 Project: HBase Issue Type: Sub-task Components: regionserver Affects Versions: 0.90.4, 0.92.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4649-Add-atomic-bulk-load-function-to-region-s.patch, hbase-4649.v2.patch Add a method that atomically bulk load multiple hfiles. Row atomicity guarantees for multi-column family rows require this functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method
[ https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136095#comment-13136095 ] Hadoop QA commented on HBASE-4677: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500902/hbase-4677.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/70//console This message is automatically generated. Remove old single bulkLoadHFile method -- Key: HBASE-4677 URL: https://issues.apache.org/jira/browse/HBASE-4677 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4677.patch In review for HBASE-4649, there is some debate as whether to remove, deprecate, or leave the HRegionServer.bulkLoadHFile method. https://reviews.apache.org/r/2545/ . This jira will take care of that for the 0.92 and trunk releases, and allow the same patch to remain for an optional 0.90.x patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4677) Remove old single bulkLoadHFile method
[ https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4677: -- Status: Open (was: Patch Available) Forgot to update RPC version. Remove old single bulkLoadHFile method -- Key: HBASE-4677 URL: https://issues.apache.org/jira/browse/HBASE-4677 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4677.patch In review for HBASE-4649, there is some debate as whether to remove, deprecate, or leave the HRegionServer.bulkLoadHFile method. https://reviews.apache.org/r/2545/ . This jira will take care of that for the 0.92 and trunk releases, and allow the same patch to remain for an optional 0.90.x patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136107#comment-13136107
]
Jean-Daniel Cryans commented on HBASE-4470:
---
I'd say yes.
ServerNotRunningException coming out of assignRootAndMeta kills the Master
--
Key: HBASE-4470
URL: https://issues.apache.org/jira/browse/HBASE-4470
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.90.5
I'm surprised we still have issues like that and I didn't get a hit while
googling so forgive me if there's already a jira about it.
When the master starts it verifies the locations of root and meta before
assigning them, if the server is started but not running you'll get this:
{quote}
2011-09-23 04:47:44,859 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
RemoteException connecting to RS
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
{quote}
I hit that 3-4 times this week while debugging something else. The worst is
that when you restart the master it sees that as a failover, but none of the
regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option
[ https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136128#comment-13136128 ] Ted Yu commented on HBASE-4224: --- That sounds good, Akash. Need a flush by regionserver rather than by table option Key: HBASE-4224 URL: https://issues.apache.org/jira/browse/HBASE-4224 Project: HBase Issue Type: Bug Components: shell Reporter: stack Assignee: Akash Ashok Attachments: HBase-4224.patch This evening needed to clean out logs on the cluster. logs are by regionserver. to let go of logs, we need to have all edits emptied from memory. only flush is by table or region. We need to be able to flush the regionserver. Need to add this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset
[ https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4648: - Hadoop Flags: Incompatible change,Reviewed Bytes.toBigDecimal() doesn't use offset --- Key: HBASE-4648 URL: https://issues.apache.org/jira/browse/HBASE-4648 Project: HBase Issue Type: Bug Components: util Affects Versions: 0.90.4 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6 Reporter: Bryan Keller Attachments: bigdec.patch, bigdec2.patch The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, thus you will get an incorrect result for the BigDecimal unless the BigDecimal's bytes are at the beginning of the byte array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4300: - Status: In Progress (was: Patch Available) Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4300: - Status: Patch Available (was: In Progress) Resubmit patch to trigger build over in https://builds.apache.org/view/G-L/view/HBase/job/PreCommit-HBASE-Build/ Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4645) Edits Log recovery losing data across column families
[ https://issues.apache.org/jira/browse/HBASE-4645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136175#comment-13136175 ] Hudson commented on HBASE-4645: --- Integrated in HBase-TRUNK #2370 (See [https://builds.apache.org/job/HBase-TRUNK/2370/]) HBASE-4645 Edits Log recovery losing data across column families nspiegelberg : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java Edits Log recovery losing data across column families - Key: HBASE-4645 URL: https://issues.apache.org/jira/browse/HBASE-4645 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100924, 0.92.0 Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Attachments: 4645-v9.diff There is a data loss happening (for some of the column families) when we do the replay logs. The bug seems to be from the fact that during replay-logs we only choose to replay the logs from the maximumSequenceID across *ALL* the stores. This is wrong. If a column family is ahead of others (because the crash happened before all the column families were flushed), then we lose data for the column families that have not yet caught up. The correct logic for replay should begin the replay from the minimum across the maximum in each store. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset
[ https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-4648. -- Resolution: Fixed Fix Version/s: 0.94.0 0.92.0 Committed to 0.92 and trunk. Since this is a (slightly) incompatible change I did not commit it to 0.90... Open again if you feel otherwise. Bytes.toBigDecimal() doesn't use offset --- Key: HBASE-4648 URL: https://issues.apache.org/jira/browse/HBASE-4648 Project: HBase Issue Type: Bug Components: util Affects Versions: 0.90.4 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6 Reporter: Bryan Keller Fix For: 0.92.0, 0.94.0 Attachments: bigdec.patch, bigdec2.patch The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, thus you will get an incorrect result for the BigDecimal unless the BigDecimal's bytes are at the beginning of the byte array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset
[ https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136176#comment-13136176 ] Lars Hofhansl commented on HBASE-4648: -- Thanks for the patch Bryan. (and I just realized I misspelled your name in the commit log) Bytes.toBigDecimal() doesn't use offset --- Key: HBASE-4648 URL: https://issues.apache.org/jira/browse/HBASE-4648 Project: HBase Issue Type: Bug Components: util Affects Versions: 0.90.4 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6 Reporter: Bryan Keller Fix For: 0.92.0, 0.94.0 Attachments: bigdec.patch, bigdec2.patch The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, thus you will get an incorrect result for the BigDecimal unless the BigDecimal's bytes are at the beginning of the byte array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4673: - Attachment: 4673.txt One liner... NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Priority: Minor Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HBASE-4300: -- Status: Open (was: Patch Available) Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HBASE-4300: -- Status: Patch Available (was: Open) Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136309#comment-13136309 ] jirapos...@reviews.apache.org commented on HBASE-2742: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1991/ --- (Updated 2011-10-26 20:23:19.711393) Review request for hbase. Changes --- This updated patch just changes the POM security profile from the previous version: * Renames the profile from hadoop-0.20S to security * Since the default hadoop profile now uses 0.20.205.0 as the build version, the hadoop version is no longer needed in the security profile. The security profile can now be used in combination with any of the Hadoop profiles (though I've only tested with 0.20.205.0, aka profile hadoop-0.20). * The security profile now overrides the hbase-site.xml used for testing with one that enables SecureRpcEngine. This should make testing with the RPC changes easier, though the pom.xml changes to accommodate it are a little clunky. To run tests using SecureRpcEngine, just do: mvn clean test -P security Summary --- This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication of clients, and a token-based authentication mechanism for mapreduce jobs. Primary components of the patch are: - a new maven profile for secure Hadoop/HBase: hadoop-0.20S - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory. These source and test directories are only including if building the secure Hadoop profile - Currently the security classes get packaged with the regular HBase build artifacts. We need a way to at least override project.version, so we can append something like a -security suffix indicating the additional security components. - The pseudo-module here is really a half-step forward. It enables the security code to be optionally included in the build for now, and sets up the structure for a security module. But we still will want to pursue full modularization (see HBASE-4336), which will allow packing the security code in a separate build artifact. - a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/ - The implementation classes extend the existing HBaseClient and HBaseServer to share as much of the RPC code as possible. The main override is of the connection classes to allow control over the SASL negotiation of secure connections - existing RPC changes - The existing HBaseClient and HBaseServer have been modified to make subclassing possible - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User to insulate from future dependencies on specific Hadoop versions - a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting classes for token generation and synchronization (incorporating HBASE-3615) - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/ - Secret keys for token generation and verification are synchronized throughout the cluster in zookeeper, under /hbase/tokenauth/keys To enable secure RPC, add the following configuration to hbase-site.xml: property namehadoop.security.authorization/name valuetrue/value /property property namehadoop.security.authentication/name valuekerberos/value /property property namehbase.rpc.engine/name valueorg.apache.hadoop.hbase.ipc.SecureRpcEngine/value /property property namehbase.coprocessor.region.classes/name valueorg.apache.hadoop.hbase.security.token.TokenProvider/value /property In addition, the master and regionserver processes must be configured for kerberos authentication using the properties: * hbase.(master|regionserver).keytab.file * hbase.(master|regionserver).kerberos.principal * hbase.(master|regionserver).kerberos.https.principal This addresses bug HBASE-2742. https://issues.apache.org/jira/browse/HBASE-2742 Diffs (updated) - conf/hbase-policy.xml PRE-CREATION pom.xml 9d42e2b security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
[jira] [Commented] (HBASE-4648) Bytes.toBigDecimal() doesn't use offset
[ https://issues.apache.org/jira/browse/HBASE-4648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136315#comment-13136315 ] Hudson commented on HBASE-4648: --- Integrated in HBase-0.92 #82 (See [https://builds.apache.org/job/HBase-0.92/82/]) HBASE-4648 Bytes.toBigDecimal() doesn't use offset larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java Bytes.toBigDecimal() doesn't use offset --- Key: HBASE-4648 URL: https://issues.apache.org/jira/browse/HBASE-4648 Project: HBase Issue Type: Bug Components: util Affects Versions: 0.90.4 Environment: Java 1.6.0_26, Mac OS X 10.7 and CentOS 6 Reporter: Bryan Keller Fix For: 0.92.0, 0.94.0 Attachments: bigdec.patch, bigdec2.patch The Bytes.toBigDecimal(byte[], offset, len) method does not use the offset, thus you will get an incorrect result for the BigDecimal unless the BigDecimal's bytes are at the beginning of the byte array. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136320#comment-13136320 ] stack commented on HBASE-4673: -- +1 NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Priority: Minor Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136346#comment-13136346 ] Hadoop QA commented on HBASE-4300: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494701/4300-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/73//console This message is automatically generated. Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136348#comment-13136348 ] jirapos...@reviews.apache.org commented on HBASE-4120: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1421/#review2867 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java https://reviews.apache.org/r/1421/#comment6408 No year please. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java https://reviews.apache.org/r/1421/#comment6409 Remove extra 'the' http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java https://reviews.apache.org/r/1421/#comment6410 Should read 'the priority map, cache the region, table and' http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6399 Year should be removed. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6400 PriorityHBaseServer is not abstract, right ? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6402 Unless the line gets too wide, explanation would be on the same line as name of parameter. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6403 What does metaHandlerCount mean ? Please add javadoc above. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6404 Should default value be related to the handler count ? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6406 Write javadoc to describe algorithm. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java https://reviews.apache.org/r/1421/#comment6405 Since 10 has special meaning, we should create constant for it. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java https://reviews.apache.org/r/1421/#comment6407 Year isn't needed. - Ted On 2011-10-26 14:13:10, Jia Liu wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1421/ bq. --- bq. bq. (Updated 2011-10-26 14:13:10) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like get,scan,put and delete can have priorities. bq. bq. bq. This addresses bug HBase-4120. bq. https://issues.apache.org/jira/browse/HBase-4120 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1189169 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1189169 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1189169 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1421/diff bq. bq. bq. Testing bq. --- bq. bq. Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch bq. please apply the patch of HBASE-4181
[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer
[ https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136383#comment-13136383 ] Jonathan Gray commented on HBASE-4658: -- How does this relate to HBASE-1744? That's slated for 0.94, should we just put this in 0.92? And I guess we should ensure that attributes are supported over there. I'm +1 on putting this in 0.92 since it makes it possible to add whatever we want without changing the API in 92 minor releases. Put attributes are not exposed via the ThriftServer --- Key: HBASE-4658 URL: https://issues.apache.org/jira/browse/HBASE-4658 Project: HBase Issue Type: Bug Components: thrift Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: ThriftPutAttributes1.txt The Put api also takes in a bunch of arbitrary attributes that an application can use to associate metadata with each put operation. This is not exposed via Thrift. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Gray updated HBASE-4528: - Attachment: HBASE-4528-Trunk-FINAL.patch Patch being committed. This is dhruba's appendNoSyncPut8.txt rebased on trunk. One small change was needed in TestParallelPut in order to compile. The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136404#comment-13136404 ] Ted Yu commented on HBASE-4528: --- What about TimeRangeTracker ? The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-4673: Assignee: Lars Hofhansl NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136412#comment-13136412 ] Jonathan Gray commented on HBASE-4528: -- Sorry Ted, I'm not clear on what exactly you're pointing out. Is something broken there? The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-4673. -- Resolution: Fixed Fix Version/s: 0.94.0 Hadoop Flags: Reviewed NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.94.0 Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136421#comment-13136421 ] Ted Yu commented on HBASE-4528: --- I think TimeRangeTracker should be updated during rollback. See my comment @ 23/Oct/11 10:45 The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136444#comment-13136444 ] Todd Lipcon commented on HBASE-4673: This should go in 0.92 also, right? (HFv2 is in 92) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.94.0 Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4682) Support deleted rows using Import/Export
Support deleted rows using Import/Export Key: HBASE-4682 URL: https://issues.apache.org/jira/browse/HBASE-4682 Project: HBase Issue Type: Sub-task Components: mapreduce Reporter: Lars Hofhansl Parent allow keeping deleted rows around. Would be nice if those could be exported and imported as well. All the building blocks are there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136469#comment-13136469 ] stack commented on HBASE-4300: -- Whho! Patch build is working You the man Giri! (Stack does repeat of the little dance he did yesterday for Giri). Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reopened HBASE-4673: -- You are right. Didn't realized that jgray had checked the cache config code in 0.92 as well. NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.94.0 Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-4673. -- Resolution: Fixed Fix Version/s: 0.92.0 Done NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4300: - Status: Open (was: Patch Available) Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136498#comment-13136498
]
Ted Yu commented on HBASE-4605:
---
@Jesse:
Can you publish your patch on review board ?
I see some nits which can be better expressed there.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4300: - Attachment: 4300-v3.txt Updated patch. Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300-v3.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4300) Start of new-version master fails if old master's znode is hanging around
[ https://issues.apache.org/jira/browse/HBASE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4300: - Status: Patch Available (was: Open) Start of new-version master fails if old master's znode is hanging around - Key: HBASE-4300 URL: https://issues.apache.org/jira/browse/HBASE-4300 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: stack Priority: Critical Fix For: 0.92.0 Attachments: 4300-v2.txt, 4300-v3.txt, 4300.txt I shut down an 0.90 cluster, and had to do so uncleanly. I then started a trunk (0.92) cluster before the old master znode had expired. This cased: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1937) at org.apache.hadoop.hbase.ServerName.parseHostname(ServerName.java:81) at org.apache.hadoop.hbase.ServerName.init(ServerName.java:63) at org.apache.hadoop.hbase.master.ActiveMasterManager.blockUntilBecomingActiveMaster(ActiveMasterManager.java:148) at org.apache.hadoop.hbase.master.HMaster.becomeActiveMaster(HMaster.java:342) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:297) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136531#comment-13136531
]
jirapos...@reviews.apache.org commented on HBASE-4605:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2579/
---
Review request for hbase.
Summary
---
Most of the implementation for adding constraints as a coprocessor.
Looking for general comments on style/structure, though nitpicks are ok too.
Currently missing implementation for disableConstraints() since that will
require adding removeCoprocessor() to HTD (also comments on if this is worth it
would be good).
This addresses bug HBASE-4605.
https://issues.apache.org/jira/browse/HBASE-4605
Diffs
-
src/main/java/org/apache/hadoop/hbase/constraint/BaseConstraint.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/constraint/IntegrationTestConstraint.java
PRE-CREATION
Diff: https://reviews.apache.org/r/2579/diff
Testing
---
Adding IntegrationTestConstraint and unit tests for Constraints and
IntegerConstraint. All of those pass.
Thanks,
Jesse
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136563#comment-13136563
]
jirapos...@reviews.apache.org commented on HBASE-4605:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2579/#review2870
---
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
https://reviews.apache.org/r/2579/#comment6424
This condition can be expressed as !constraints.isEmpty()
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
https://reviews.apache.org/r/2579/#comment6423
Please use curly braces for line 60.
This log can be combined with log on line 63.
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
https://reviews.apache.org/r/2579/#comment6425
This has been done, right ?
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
https://reviews.apache.org/r/2579/#comment6426
Contents of put should be included.
src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java
https://reviews.apache.org/r/2579/#comment6427
We can either add removeCoprocessor() to HTD or, persist a flag to tell
ConstraintProcessor that it is disabled.
I think the second approach may be better in our case.
src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java
https://reviews.apache.org/r/2579/#comment6428
Should be 'desc HTableDescriptor to read from'
- Ted
On 2011-10-26 22:34:35, Jesse Yates wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2579/
bq. ---
bq.
bq. (Updated 2011-10-26 22:34:35)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Most of the implementation for adding constraints as a coprocessor.
bq.
bq. Looking for general comments on style/structure, though nitpicks are ok
too.
bq.
bq. Currently missing implementation for disableConstraints() since that will
require adding removeCoprocessor() to HTD (also comments on if this is worth it
would be good).
bq.
bq.
bq. This addresses bug HBASE-4605.
bq. https://issues.apache.org/jira/browse/HBASE-4605
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/constraint/BaseConstraint.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/constraint/IntegrationTestConstraint.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2579/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Adding IntegrationTestConstraint and unit tests for Constraints and
IntegerConstraint. All of those pass.
bq.
bq.
bq. Thanks,
bq.
bq. Jesse
bq.
bq.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136593#comment-13136593 ] Hadoop QA commented on HBASE-4528: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500956/HBASE-4528-Trunk-FINAL.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 13 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -167 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/74//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/74//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/74//console This message is automatically generated. The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[
https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136598#comment-13136598
]
stack commented on HBASE-4528:
--
Do these tests fail w/o this patch?
{code}
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
org.apache.hadoop.hbase.coprocessor.TestMasterObserver
org.apache.hadoop.hbase.master.TestMasterFailover
{code}
I see that TestMasterObserver has been failing recently but
TestMasterFailover and TestDistributedLogSplitting?
Im looking at TestMasterObserver here. When I run it singly, it passes -- the
f'er
The put operation can release the rowlock before sync-ing the Hlog
--
Key: HBASE-4528
URL: https://issues.apache.org/jira/browse/HBASE-4528
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Fix For: 0.94.0
Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt,
appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt,
appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt,
appendNoSyncPut7.txt, appendNoSyncPut8.txt
This allows for better throughput when there are hot rows. A single row
update improves from 100 puts/sec/server to 5000 puts/sec/server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4683) Create config option to only cache index blocks
Create config option to only cache index blocks --- Key: HBASE-4683 URL: https://issues.apache.org/jira/browse/HBASE-4683 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.0 This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the cache. This is the equivalent of setting all cacheBlocks to false on all scans (including scans on behalf of gets). I would like to general feeling about what folks think about this. The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Create config option to only cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136609#comment-13136609 ] Jean-Daniel Cryans commented on HBASE-4683: --- Sounds sophisticated! +1 Create config option to only cache index blocks --- Key: HBASE-4683 URL: https://issues.apache.org/jira/browse/HBASE-4683 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.0 This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to general feeling about what folks think about this. The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4683) Create config option to only cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4683: - Description: This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to general feeling about what folks think about this. The change itself would be simple. was: This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the cache. This is the equivalent of setting all cacheBlocks to false on all scans (including scans on behalf of gets). I would like to general feeling about what folks think about this. The change itself would be simple. Create config option to only cache index blocks --- Key: HBASE-4683 URL: https://issues.apache.org/jira/browse/HBASE-4683 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.0 This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to general feeling about what folks think about this. The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer
[ https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136610#comment-13136610 ] stack commented on HBASE-4658: -- Is this patch missing the generated code? I'm good w/ it in 0.92. Put attributes are not exposed via the ThriftServer --- Key: HBASE-4658 URL: https://issues.apache.org/jira/browse/HBASE-4658 Project: HBase Issue Type: Bug Components: thrift Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: ThriftPutAttributes1.txt The Put api also takes in a bunch of arbitrary attributes that an application can use to associate metadata with each put operation. This is not exposed via Thrift. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1744) Thrift server to match the new java api.
[
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-1744:
-
Status: Open (was: Patch Available)
Thrift server to match the new java api.
Key: HBASE-1744
URL: https://issues.apache.org/jira/browse/HBASE-1744
Project: HBase
Issue Type: Improvement
Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
Fix For: 0.94.0
Attachments:
0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch,
HBASE-1744.2.patch, HBASE-1744.3.patch, HBASE-1744.4.patch,
HBASE-1744.5.patch, HBASE-1744.6.patch, HBASE-1744.7.patch,
HBASE-1744.8.patch, HBASE-1744.9.patch, HBASE-1744.preview.1.patch,
thriftexperiment.patch
This mutateRows, etc.. is a little confusing compared to the new cleaner java
client.
Thinking of ways to make a thrift client that is just as elegant. something
like:
void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
with:
struct TColumn {
1:Bytes family,
2:Bytes qualifier,
3:i64 timestamp
}
struct TPut {
1:Bytes row,
2:mapTColumn, Bytes values
}
This creates more verbose rpc than if the columns in TPut were just
mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and
still be intuitive from say python.
Presumably the goal of a thrift gateway is to be easy first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1744) Thrift server to match the new java api.
[
https://issues.apache.org/jira/browse/HBASE-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-1744:
-
Status: Patch Available (was: Open)
Submitting patch to see if it breaks any unit tests.
Thrift server to match the new java api.
Key: HBASE-1744
URL: https://issues.apache.org/jira/browse/HBASE-1744
Project: HBase
Issue Type: Improvement
Components: thrift
Reporter: Tim Sell
Assignee: Tim Sell
Priority: Critical
Fix For: 0.94.0
Attachments:
0001-thrift2-enable-usage-of-.deleteColumns-for-thrift.patch,
HBASE-1744.2.patch, HBASE-1744.3.patch, HBASE-1744.4.patch,
HBASE-1744.5.patch, HBASE-1744.6.patch, HBASE-1744.7.patch,
HBASE-1744.8.patch, HBASE-1744.9.patch, HBASE-1744.preview.1.patch,
thriftexperiment.patch
This mutateRows, etc.. is a little confusing compared to the new cleaner java
client.
Thinking of ways to make a thrift client that is just as elegant. something
like:
void put(1:Bytes table, 2:TPut put) throws (1:IOError io)
with:
struct TColumn {
1:Bytes family,
2:Bytes qualifier,
3:i64 timestamp
}
struct TPut {
1:Bytes row,
2:mapTColumn, Bytes values
}
This creates more verbose rpc than if the columns in TPut were just
mapBytes, mapBytes, Bytes, but that is harder to fit timestamps into and
still be intuitive from say python.
Presumably the goal of a thrift gateway is to be easy first.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136616#comment-13136616
]
stack commented on HBASE-4470:
--
The fix in 0.92 was radical putting HTable in place everywhere we previously
used bare HConnection to get retries. That ain't going to happen for 0.90.5.
ServerNotRunningException coming out of assignRootAndMeta kills the Master
--
Key: HBASE-4470
URL: https://issues.apache.org/jira/browse/HBASE-4470
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.90.5
I'm surprised we still have issues like that and I didn't get a hit while
googling so forgive me if there's already a jira about it.
When the master starts it verifies the locations of root and meta before
assigning them, if the server is started but not running you'll get this:
{quote}
2011-09-23 04:47:44,859 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
RemoteException connecting to RS
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
{quote}
I hit that 3-4 times this week while debugging something else. The worst is
that when you restart the master it sees that as a failover, but none of the
regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4684) REST server is leaking ZK connections in 0.90
REST server is leaking ZK connections in 0.90 - Key: HBASE-4684 URL: https://issues.apache.org/jira/browse/HBASE-4684 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.90.5 As reported a month ago, http://search-hadoop.com/m/FD6gmKzrxY1, the REST server is leak ZK connections. Upon investigation I see that TableResource.scanTransformAttrs creates a new HBA per minute per table (when the server is getting requests) but never deletes the connection created in there. There are a bunch of other places where HBAs are created but not cleaned after like SchemaResource, StorageClusterStatusResource, StorageClusterVersionResource, ExistsResource, etc. Those places shouldn't be as leaky under normal circumstances tho. Thanks to Jack Levin for bringing up this issue again when he tried to upgrade. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4470) ServerNotRunningException coming out of assignRootAndMeta kills the Master
[
https://issues.apache.org/jira/browse/HBASE-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136623#comment-13136623
]
Jean-Daniel Cryans commented on HBASE-4470:
---
We could just add retries for this case like we did for others.
ServerNotRunningException coming out of assignRootAndMeta kills the Master
--
Key: HBASE-4470
URL: https://issues.apache.org/jira/browse/HBASE-4470
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Priority: Blocker
Fix For: 0.90.5
I'm surprised we still have issues like that and I didn't get a hit while
googling so forgive me if there's already a jira about it.
When the master starts it verifies the locations of root and meta before
assigning them, if the server is started but not running you'll get this:
{quote}
2011-09-23 04:47:44,859 WARN
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
RemoteException connecting to RS
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hbase.ipc.ServerNotRunningException: Server is not running
yet
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1038)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy6.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:969)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getCachedConnection(CatalogTracker.java:388)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:287)
at
org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:484)
at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:441)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:388)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:282)
{quote}
I hit that 3-4 times this week while debugging something else. The worst is
that when you restart the master it sees that as a failover, but none of the
regions are assigned so it takes an eternity to get back fully online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) test.build.data property overused leading to write data at the wrong place
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136626#comment-13136626 ] Hudson commented on HBASE-4634: --- Integrated in HBase-0.92 #83 (See [https://builds.apache.org/job/HBase-0.92/83/]) HBASE-4634 'test.build.data' property overused leading to write data at the wrong place stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java *
[jira] [Commented] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136625#comment-13136625 ] Hudson commented on HBASE-4673: --- Integrated in HBase-0.92 #83 (See [https://builds.apache.org/job/HBase-0.92/83/]) HBASE-4673 NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 larsh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4658) Put attributes are not exposed via the ThriftServer
[ https://issues.apache.org/jira/browse/HBASE-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136683#comment-13136683 ] dhruba borthakur commented on HBASE-4658: - This is complimentary to the things occuring in HBASE-1744. This just adds the Put attributes to the Put/Get/Scan call. There is no risk in this piece of code, so I am ok if you folks decide to put this into 0.92. Stack: can the committer pl generate the generated code and commit it (the only generated file that changes is Hbase.java)? Alternatively, if somebody can tell me which version of thrift to use (and where to pick up that version of the thrift binary), I can upload the generated file as part of my patch too. Put attributes are not exposed via the ThriftServer --- Key: HBASE-4658 URL: https://issues.apache.org/jira/browse/HBASE-4658 Project: HBase Issue Type: Bug Components: thrift Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: ThriftPutAttributes1.txt The Put api also takes in a bunch of arbitrary attributes that an application can use to associate metadata with each put operation. This is not exposed via Thrift. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4683) Create config option to only cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4683: - Description: This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the (aggregate) cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to get a general feeling about what folks think about this. The change itself would be simple. was: This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to general feeling about what folks think about this. The change itself would be simple. Create config option to only cache index blocks --- Key: HBASE-4683 URL: https://issues.apache.org/jira/browse/HBASE-4683 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.0 This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the (aggregate) cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to get a general feeling about what folks think about this. The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136695#comment-13136695 ] jirapos...@reviews.apache.org commented on HBASE-4120: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1421/ --- (Updated 2011-10-27 01:57:57.358569) Review request for hbase. Changes --- fixed Summary --- Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like get,scan,put and delete can have priorities. This addresses bug HBase-4120. https://issues.apache.org/jira/browse/HBase-4120 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION Diff: https://reviews.apache.org/r/1421/diff Testing --- Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client. Thanks, Jia isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4304) requestsPerSecond counter stuck at 0
[ https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Pi updated HBASE-4304: - Attachment: 4304v1.txt requestsPerSecond counter stuck at 0 Key: HBASE-4304 URL: https://issues.apache.org/jira/browse/HBASE-4304 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: 4304v1.txt Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 both in the master UI and in the RS UI. The writeRequestsCount metric is properly updating in the RS UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4304) requestsPerSecond counter stuck at 0
[ https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Pi updated HBASE-4304: - Status: Patch Available (was: Open) fixed. requestsPerSecond counter stuck at 0 Key: HBASE-4304 URL: https://issues.apache.org/jira/browse/HBASE-4304 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: 4304v1.txt Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 both in the master UI and in the RS UI. The writeRequestsCount metric is properly updating in the RS UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4634) test.build.data property overused leading to write data at the wrong place
[ https://issues.apache.org/jira/browse/HBASE-4634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136701#comment-13136701 ] Hudson commented on HBASE-4634: --- Integrated in HBase-TRUNK #2372 (See [https://builds.apache.org/job/HBase-TRUNK/2372/]) HBASE-4634 'test.build.data' property overused leading to write data at the wrong place stack : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditor.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestHTablePool.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestColumnPrefixFilter.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestDependentColumnFilter.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/filter/TestMultipleColumnPrefixFilter.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/TestHalfStoreFileReader.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestFixedFileTrailer.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFilePerformance.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestReseekTo.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFiles.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterRestartAfterDisablingTable.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestOpenedRegionHandler.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestRestartCluster.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestZKBasedOpenCloseRegion.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestColumnSeeking.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundBloomFilter.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java *
[jira] [Commented] (HBASE-4673) NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0
[ https://issues.apache.org/jira/browse/HBASE-4673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136700#comment-13136700 ] Hudson commented on HBASE-4673: --- Integrated in HBase-TRUNK #2372 (See [https://builds.apache.org/job/HBase-TRUNK/2372/]) HBASE-4673 NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 larsh : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java NPE in HFileReaderV2.close during major compaction when hfile.block.cache.size is set to 0 --- Key: HBASE-4673 URL: https://issues.apache.org/jira/browse/HBASE-4673 Project: HBase Issue Type: Bug Affects Versions: 0.94.0 Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.92.0, 0.94.0 Attachments: 4673.txt On a test system got this exception when hfile.block.cache.size is set to 0: java.lang.NullPointerException at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.close(HFileReaderV2.java:321) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.close(StoreFile.java:1065) at org.apache.hadoop.hbase.regionserver.StoreFile.closeReader(StoreFile.java:539) at org.apache.hadoop.hbase.regionserver.StoreFile.deleteReader(StoreFile.java:549) at org.apache.hadoop.hbase.regionserver.Store.completeCompaction(Store.java:1314) at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:686) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1016) at org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(CompactionRequest.java:178) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Minor issue as nobody in their right mind with have hfile.block.cache.size=0 Looks like this is due to HBASE-4422 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4528) The put operation can release the rowlock before sync-ing the Hlog
[ https://issues.apache.org/jira/browse/HBASE-4528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136714#comment-13136714 ] Ted Yu commented on HBASE-4528: --- I am fine with the current form of this feature. Depending on how often memstoreRollback is executed in production, we can formulate the next step with another JIRA. Thanks for the nice job done, Dhruba. The put operation can release the rowlock before sync-ing the Hlog -- Key: HBASE-4528 URL: https://issues.apache.org/jira/browse/HBASE-4528 Project: HBase Issue Type: Improvement Components: regionserver Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.94.0 Attachments: HBASE-4528-Trunk-FINAL.patch, appendNoSync5.txt, appendNoSyncPut1.txt, appendNoSyncPut2.txt, appendNoSyncPut3.txt, appendNoSyncPut4.txt, appendNoSyncPut5.txt, appendNoSyncPut6.txt, appendNoSyncPut7.txt, appendNoSyncPut8.txt This allows for better throughput when there are hot rows. A single row update improves from 100 puts/sec/server to 5000 puts/sec/server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4304) requestsPerSecond counter stuck at 0
[ https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136715#comment-13136715 ] Hadoop QA commented on HBASE-4304: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501015/4304v1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/76//console This message is automatically generated. requestsPerSecond counter stuck at 0 Key: HBASE-4304 URL: https://issues.apache.org/jira/browse/HBASE-4304 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: 4304v1.txt Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 both in the master UI and in the RS UI. The writeRequestsCount metric is properly updating in the RS UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4304) requestsPerSecond counter stuck at 0
[ https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136726#comment-13136726 ] Li Pi commented on HBASE-4304: -- Do these patches need to svn patches? requestsPerSecond counter stuck at 0 Key: HBASE-4304 URL: https://issues.apache.org/jira/browse/HBASE-4304 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: 4304v1.txt Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 both in the master UI and in the RS UI. The writeRequestsCount metric is properly updating in the RS UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136729#comment-13136729
]
Jesse Yates commented on HBASE-4605:
Also, this patch (clearly) does not include changes to the shell or book
documentation. Any pointers on where to look for the former would be really
helpful, as I haven't looked into the shell much before (and don't have a ton
of ruby experience). However, going to be looking into that next week.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4304) requestsPerSecond counter stuck at 0
[ https://issues.apache.org/jira/browse/HBASE-4304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136740#comment-13136740 ] Todd Lipcon commented on HBASE-4304: Try git show --no-prefix or git diff --no-prefix instead. That way it will apply with patch -p0 and should work in test-patch (if it's like Hadoop's old test-patch script) requestsPerSecond counter stuck at 0 Key: HBASE-4304 URL: https://issues.apache.org/jira/browse/HBASE-4304 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Li Pi Priority: Critical Fix For: 0.92.0 Attachments: 4304v1.txt Running trunk @ r1163343, all of the requestsPerSecond counters are showing 0 both in the master UI and in the RS UI. The writeRequestsCount metric is properly updating in the RS UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4683) Create config option to only cache index blocks
[ https://issues.apache.org/jira/browse/HBASE-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136766#comment-13136766 ] Lars Hofhansl commented on HBASE-4683: -- As a separate issue... index blocks could be treated with MEMORY priority in the LRU cache. Create config option to only cache index blocks --- Key: HBASE-4683 URL: https://issues.apache.org/jira/browse/HBASE-4683 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Fix For: 0.94.0 This would add a new boolean config option: hfile.block.cache.datablocks Default would be true. Setting this to false allows HBase in a mode where only index blocks are cached, which is useful for analytical scenarios where a useful working set of the data cannot be expected to fit into the (aggregate) cache. This is the equivalent of setting cacheBlocks to false on all scans (including scans on behalf of gets). I would like to get a general feeling about what folks think about this. The change itself would be simple. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13136793#comment-13136793 ] jirapos...@reviews.apache.org commented on HBASE-2856: -- bq. On 2011-10-17 06:06:23, Kannan Muthukkaruppan wrote: bq. Amit: Did you rebase before uploading the new patch. That, unfortunately, is making it hard to isolate the changes between r6 and r7. Will review tomorrow morning. bq. bq. But I did read your description about the issues you mentioned. bq. bq. Regarding (b)-- we had already discussed in person. That makes sense. bq. bq. And really nice catch on (a) too!! That is indeed subtle and tricky. Super!!! bq. bq. bq. Amitanand Aiyer wrote: bq. Looks like a lot has changed since the original revision that I based my first patch off. bq. bq. Please disregard v7. bq. bq. Let me submit these modifications as a separate diff. I have a sub-jira created for each part. bq. bq. Seems like we are all basically +1 on this patch upto to some point - could we commit till there? We can add the other changes as separate tasks under the same umbrella task. The other diffs are based on this diff... and they are separate issues from what this is addressing (persisting memstoreTS). Its getting confusing if we keep adding to this diff. - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2224/#review2614 --- On 2011-10-15 04:08:41, Amitanand Aiyer wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2224/ bq. --- bq. bq. (Updated 2011-10-15 04:08:41) bq. bq. bq. Review request for Ted Yu, Michael Stack, Kannan Muthukkaruppan, and Karthik Ranganathan. bq. bq. bq. Summary bq. --- bq. bq. address the 2856 issues by writing the memstoreTS to the disk. bq. bq. version v11 of the patch. bq. bq. uploading it here for easier review process. bq. bq. bq. This addresses bug HBASE-2856. bq. https://issues.apache.org/jira/browse/HBASE-2856 bq. bq. bq. Diffs bq. - bq. bq./pom.xml 1183581 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileReader.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/ColumnCount.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/ColumnTracker.java 1183581 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ExplicitColumnTracker.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/MemStore.java 1183581 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ReadWriteConsistencyControl.java 1183581 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java 1183581 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1183581 bq. /src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java 1183581 bq./src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java 1183581 bq./src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 1183581 bq./src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java 1183581 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java 1183581 bq./src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java 1183581 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/TestExplicitColumnTracker.java 1183581 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java 1183581 bq. /src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java 1183581 bq./src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 1183581 bq. bq. Diff: https://reviews.apache.org/r/2224/diff bq. bq. bq. Testing bq. --- bq. bq. mvn test bq. bq. bq. Thanks, bq. bq. Amitanand bq. bq. TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856
