[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095793#comment-13095793 ] ramkrishna.s.vasudevan commented on HBASE-4308: --- I too have noticed this. but it doesnt cause any issues as such. Race between RegionOpenedHandler and AssignmentManager -- Key: HBASE-4308 URL: https://issues.apache.org/jira/browse/HBASE-4308 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 When the master is processing a ZK event for REGION_OPENED, it calls delete() on the znode before it removes the node from RegionsInTransition. If the notification of that delete comes back into AssignmentManager before the region is removed from RIT, you see an error like: 2011-08-30 17:43:29,537 WARN [main-EventThread] master.AssignmentManager(861): Node deleted but still in RIT: .META.,,1.1028785192 state=OPEN, ts=1314751409532, server=todd-w510,55655,1314751396840 Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4265) zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled
[ https://issues.apache.org/jira/browse/HBASE-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095794#comment-13095794 ] jirapos...@reviews.apache.org commented on HBASE-4265: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1685/#review1732 --- Patch looks fine to me. - ramkrishna On 2011-08-31 01:37:46, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1685/ bq. --- bq. bq. (Updated 2011-08-31 01:37:46) bq. bq. bq. Review request for hbase, Ted Yu and ramkrishna vasudevan. bq. bq. bq. Summary bq. --- bq. bq. The issue is disableTable tries to work on those regions in transition. disableTable already has code to bypass those regions in transition. The issue is recoverTableInDisablingState is called before processRegionsInTransition(which updates regions-in-transition list) is called at startup. Thus the regions-in-transition list hasn't been updated when recoverTableInDisablingState is called. bq. bq. The fix is to postpone recoverTableInDisablingState, after processRegionsInTransition is called. bq. bq. bq. This addresses bug hbase-4265. bq. https://issues.apache.org/jira/browse/hbase-4265 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1163346 bq. bq. Diff: https://reviews.apache.org/r/1685/diff bq. bq. bq. Testing bq. --- bq. bq. On a small cluster, stop HMaster when disableTable is in progress. Make sure there are some regions-in-transition in zk when the HMaster shudown occurs. Without the fix, we get such exception. With the fix, HMaster can continue disabling process after restart and table can get to disabled state. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled --- Key: HBASE-4265 URL: https://issues.apache.org/jira/browse/HBASE-4265 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.92.0 There seems to be more than just one issue regarding the following scenario. I would provide a fix later just for this exception. 1. A table is being disabled. 2. HMaster restarted. 3. At HMaster startup, it tries to transition from disabling to disabled state. It got the following exception. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /hbase/unassigned/419b902243c836c285108ba555b712fa at org.apache.zookeeper.KeeperException.create(KeeperException.java:110) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:475) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:742) at org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:461) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1440) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1406) at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:141) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) This issue is this specific region is in a special state before HMaster restarts; it has been closed by RS properly thus the zk state is RS_ZK_REGION_CLOSED. However, HMaster hasn't got a chance to process ClosedRegionHandler yet and thus the node remains at zk. After RS restarts, this node is added to online region list first in AssignmentManager.rebuildUserRegions and tries to unassign it later. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-4268: - Assignee: ramkrishna.s.vasudevan Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: ramkrishna.s.vasudevan In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095802#comment-13095802 ] Todd Lipcon commented on HBASE-4268: I actually started this, this afternoon, and forgot to assign to myself. Ram, unless you have some code, mind assigning it to me? Or I can upload what I have if you want to finish it off. Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: ramkrishna.s.vasudevan In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095804#comment-13095804 ] Sebastian Bauer commented on HBASE-4301: I had try this patch on broken instance, with trunk running on it before and added some logs and it doesn't help. In myFSUtils.rootTableInfoExists(fs, rd) return true, so FSUtils.createTableDescriptor isn't run on my instance and scanning META still raise TableNotFoundException. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-4268: - Assignee: Todd Lipcon (was: ramkrishna.s.vasudevan) Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4268) Add utility to entirely clear out ZK
[ https://issues.apache.org/jira/browse/HBASE-4268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095817#comment-13095817 ] ramkrishna.s.vasudevan commented on HBASE-4268: --- I have reassigned to you Todd :) Add utility to entirely clear out ZK Key: HBASE-4268 URL: https://issues.apache.org/jira/browse/HBASE-4268 Project: HBase Issue Type: New Feature Components: scripts Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon In disaster scenarios, sometimes some cruft is left over in ZK, when it would be better to do a truely clean startup. We should add a script which allows the admin to clear out ZK while the cluster is down. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095822#comment-13095822 ] Sebastian Bauer commented on HBASE-4301: Even checking ROOT and META tableInfoExists(as in 4301_v2 patch) says that my instance have it META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095832#comment-13095832
]
Ted Yu commented on HBASE-4301:
---
@Sebastian:
Thanks for your investigation.
You may have noticed the following method in FSUtils:
{code}
public static HTableDescriptor getTableDescriptor(FileSystem fs, Path
tableDir)
{code}
which calls hTableDescriptor.readFields().
You can write a short standalone program utilizing FSUtils.getTableDescriptor()
to analyze what is inside .tableinfo for .META. table.
If you don't time for the above action, please attach log snippet showing more
details around TableNotFoundException. The log Todd posted doesn't give me much
clue.
META migration from 0.90 to trunk fails
---
Key: HBASE-4301
URL: https://issues.apache.org/jira/browse/HBASE-4301
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Ted Yu
Priority: Blocker
Fix For: 0.92.0
Attachments: 4301.txt, 4301_v2.txt, master-log.txt
I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table
is screwed. I can't scan it, etc, and other operations fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4301) META migration from 0.90 to trunk fails
[
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095838#comment-13095838
]
Ted Yu edited comment on HBASE-4301 at 9/2/11 7:27 AM:
---
First step should be analyzing contents of {code}-ROOT-{code} table.
was (Author: yuzhih...@gmail.com):
First step should be analyzing contents of -ROOT- table.
META migration from 0.90 to trunk fails
---
Key: HBASE-4301
URL: https://issues.apache.org/jira/browse/HBASE-4301
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Ted Yu
Priority: Blocker
Fix For: 0.92.0
Attachments: 4301.txt, 4301_v2.txt, master-log.txt
I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table
is screwed. I can't scan it, etc, and other operations fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095838#comment-13095838 ] Ted Yu commented on HBASE-4301: --- First step should be analyzing contents of -ROOT- table. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095864#comment-13095864
]
Sebastian Bauer commented on HBASE-4301:
This is what i get from HTableDescriptor.toString():
{NAME = '-ROOT-', IS_ROOT = 'true', IS_META = 'true', FAMILIES = [{NAME =
'info', BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0', VERSIONS = '10',
COMPRESSION = 'NONE', MIN_VERSIONS = '0', TTL = '2147483647', BLOCKSIZE =
'8192', IN_MEMORY = 'true', BLOCKCACHE = 'true'}]}
{NAME = '.META.', IS_META = 'true', FAMILIES = [{NAME = 'info', BLOOMFILTER
= 'NONE', REPLICATION_SCOPE = '0', VERSIONS = '10', COMPRESSION = 'NONE',
MIN_VERSIONS = '0', TTL = '2147483647', BLOCKSIZE = '8192', IN_MEMORY =
'true', BLOCKCACHE = 'true'}]}
META migration from 0.90 to trunk fails
---
Key: HBASE-4301
URL: https://issues.apache.org/jira/browse/HBASE-4301
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Ted Yu
Priority: Blocker
Fix For: 0.92.0
Attachments: 4301.txt, 4301_v2.txt, master-log.txt
I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table
is screwed. I can't scan it, etc, and other operations fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13095992#comment-13095992 ] Ted Yu commented on HBASE-4301: --- @Sebastian: Can you post log showing more details for TableNotFoundException ? Your case seems to be different from Todd's. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096008#comment-13096008 ] Sebastian Bauer commented on HBASE-4301: Todd's TableNotFoundException was when master starting, mine is when i try to do any operation on tables: Exception in thread main org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server null for region , row '', but failed after 10 attempts. Exceptions: org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. org.apache.hadoop.hbase.TableNotFoundException: Table '.META.' was not found. at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1245) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:194) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:52) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:127) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:340) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:127) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:103) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:81) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:65) at org.apache.hadoop.hbase.client.MetaScanner.listAllRegions(MetaScanner.java:269) at org.apache.hadoop.hbase.client.MetaScanner.listAllRegions(MetaScanner.java:236) at gl.tools.TableDescriptor.doMain(TableDescriptor.java:24) META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-4321) Add more comprehensive region split calculator
[ https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-4321 started by Jonathan Hsieh. Add more comprehensive region split calculator -- Key: HBASE-4321 URL: https://issues.apache.org/jira/browse/HBASE-4321 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Hbck currently scans through meta one entry at a time, only keeping a reference to the previous meta entry. This is insufficient for capturing all the possible problems in meta and needs something more to properly identify holes, overlaps, duplicate start keys, and otherwise invalid meta entries. Ideally, this calculator could also be used online interrogating an existing meta (HBASE-4058), and also used to generate a completely new meta offline just from regioninfo and in hdfs (HBASE-3505). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4321) Add more comprehensive region split calculator
[ https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4321: -- Attachment: 0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch This is just the calculator algorithm, updates to hbck to follow. Add more comprehensive region split calculator -- Key: HBASE-4321 URL: https://issues.apache.org/jira/browse/HBASE-4321 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch Hbck currently scans through meta one entry at a time, only keeping a reference to the previous meta entry. This is insufficient for capturing all the possible problems in meta and needs something more to properly identify holes, overlaps, duplicate start keys, and otherwise invalid meta entries. Ideally, this calculator could also be used online interrogating an existing meta (HBASE-4058), and also used to generate a completely new meta offline just from regioninfo and in hdfs (HBASE-3505). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4321) Add more comprehensive region split calculator
[ https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096048#comment-13096048 ] jirapos...@reviews.apache.org commented on HBASE-4321: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1703/ --- Review request for hbase, Todd Lipcon and Michael Stack. Summary --- This is the core region split calculator. It provides a sorted set of region split points, and a coverage multimap. This is enough information to properly determine dupe start keys, all the different kinds of overlap, as well as holes. This addresses bug hbase-4321. https://issues.apache.org/jira/browse/hbase-4321 Diffs - src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java PRE-CREATION Diff: https://reviews.apache.org/r/1703/diff Testing --- Unit tests just checks algorithm, no dependencies on HBase yet. Thanks, jmhsieh Add more comprehensive region split calculator -- Key: HBASE-4321 URL: https://issues.apache.org/jira/browse/HBASE-4321 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch Hbck currently scans through meta one entry at a time, only keeping a reference to the previous meta entry. This is insufficient for capturing all the possible problems in meta and needs something more to properly identify holes, overlaps, duplicate start keys, and otherwise invalid meta entries. Ideally, this calculator could also be used online interrogating an existing meta (HBASE-4058), and also used to generate a completely new meta offline just from regioninfo and in hdfs (HBASE-3505). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4321) Add more comprehensive region split calculator
[ https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4321: -- Affects Version/s: 0.90.4 Status: Patch Available (was: In Progress) Add more comprehensive region split calculator -- Key: HBASE-4321 URL: https://issues.apache.org/jira/browse/HBASE-4321 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch Hbck currently scans through meta one entry at a time, only keeping a reference to the previous meta entry. This is insufficient for capturing all the possible problems in meta and needs something more to properly identify holes, overlaps, duplicate start keys, and otherwise invalid meta entries. Ideally, this calculator could also be used online interrogating an existing meta (HBASE-4058), and also used to generate a completely new meta offline just from regioninfo and in hdfs (HBASE-3505). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4321) Add more comprehensive region split calculator
[ https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096051#comment-13096051 ] stack commented on HBASE-4321: -- So far, excellent. Its a bit odd methinks that the public signature of the RegionSplitCalculator class takes extensions of a generic that is an abstract interior class. Do you think the interior class has enough heft to live outside of RegionSplitCalculator? What are the start/end keys passed here? Are they .META. rows or the start/end rows from regions? If the former, the comparator as is won't work -- Bytes only (see the meta comparators in KeyValue) -- else what if the problem in .META. spans tables? (It doesn't look easy getting other comparators into the code as written) The dump function in the test looks great. Good stuff Jon. Add more comprehensive region split calculator -- Key: HBASE-4321 URL: https://issues.apache.org/jira/browse/HBASE-4321 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch Hbck currently scans through meta one entry at a time, only keeping a reference to the previous meta entry. This is insufficient for capturing all the possible problems in meta and needs something more to properly identify holes, overlaps, duplicate start keys, and otherwise invalid meta entries. Ideally, this calculator could also be used online interrogating an existing meta (HBASE-4058), and also used to generate a completely new meta offline just from regioninfo and in hdfs (HBASE-3505). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4321) Add more comprehensive region split calculator
[ https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096084#comment-13096084 ] Jonathan Hsieh commented on HBASE-4321: --- I think the main concern is that this is a public static abstract. I can pull the Range out and make it a separate file. Would it makes sense to pull this out in to a subpackage? o.a.h.h.util.splits? o.a.h.h.util.hbck? I'm not clear if the generic a concern. Is it? The start keys and end keys can be whatever, I wrote it generically to make testing easy and straight forward. In a follow on patch, the start/end keys are the actual start and end keys of a region -- not the .META. rows. Basically, I subclass and wrap hbckinfos and use hbckinfo.metaData.getStartKey() and hbckinfo.metaData.getEndKey() as the range's getStartKey/getEndKeys. I know Todd wants to do an offline meta table rebuild that just looks at .regioninfo's on HDFS. So for that case, the Range would subclass wrap an HRegionInfo and use hri.getStartKey(), and hri.getEndKey(). Add more comprehensive region split calculator -- Key: HBASE-4321 URL: https://issues.apache.org/jira/browse/HBASE-4321 Project: HBase Issue Type: Improvement Affects Versions: 0.90.4 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: 0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch Hbck currently scans through meta one entry at a time, only keeping a reference to the previous meta entry. This is insufficient for capturing all the possible problems in meta and needs something more to properly identify holes, overlaps, duplicate start keys, and otherwise invalid meta entries. Ideally, this calculator could also be used online interrogating an existing meta (HBASE-4058), and also used to generate a completely new meta offline just from regioninfo and in hdfs (HBASE-3505). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4015) Refactor the TimeoutMonitor to make it less racy
[
https://issues.apache.org/jira/browse/HBASE-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096126#comment-13096126
]
ramkrishna.s.vasudevan commented on HBASE-4015:
---
The following tests were performed
The cluster has 3 nodes.
Created some 4000+ regions. And i followed the startegy like
Initially create regions
Balancer will balance the regions across all the three.
Using a script kill the RS in these 3 nodes at fixed time.(at different time
intervals).
Then start it after sometime.
Sometimes once all the 3 RS are up invoke the balancer command manually.
While balancing kill one of the RS which was the destination during balancer
calculation.
Again bring it up.
The number of regions remained constant and every time the Timeout detected it
was able to overcome it with no issues at all.
I also calculated some metrics
There were instance were 144, 249,1334, 30 regions were timed out.
For any timeout region to be monitored and completed it took at the max 1.5 sec
minimum being 0.35 secs.
The following logs shows the scenario where the timeout monitor tries to assign
a node to OFFLINE but by the time the node is changed to OPENED by RS.
{code}
2011-09-02 18:15:55,621 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Region has been PENDING_OPEN for too long, reassigning
region=testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.
2011-09-02 18:15:55,709 INFO org.mortbay.log: Timeout detected for the region
testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88. at Fri Sep 02
18:15:55 IST 2011
2011-09-02 18:15:55,942 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x13229c29f830010 Creating (or updating) unassigned node for
b7a45cf31651d5cbba682ae24b83bf88 with OFFLINE state
2011-09-02 18:15:55,967 INFO org.apache.hadoop.hbase.zookeeper.ZKAssign:
Current data in znode
.region=testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.,
origin=HOST-10-18-52-198,60020,1314967317396, state=RS_ZK_REGION_OPENED
2011-09-02 18:15:55,996 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Attempted to create/force node into OFFLINE state before completing assignment
but failed to do so for
testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.
state=PENDING_OPEN, ts=1314967555942, server=null
2011-09-02 18:15:55,999 INFO org.mortbay.log: Time taken to complete the action
on timeout for the reigon
testram2,9970,1314962425951.b7a45cf31651d5cbba682ae24b83bf88.291
2011-09-02 18:16:24,115 DEBUG org.apache.hadoop.hbase.master.AssignmentManager:
Handling transition=RS_ZK_REGION_OPENED,
server=HOST-10-18-52-198,60020,1314967317396,
region=b7a45cf31651d5cbba682ae24b83bf88, which is more than 15 seconds late
2011-09-02 18:16:24,122 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x13229c29f830010 Successfully deleted unassigned node for region
b7a45cf31651d5cbba682ae24b83bf88 in expected state RS_ZK_REGION_OPENED
{code}
Refactor the TimeoutMonitor to make it less racy
Key: HBASE-4015
URL: https://issues.apache.org/jira/browse/HBASE-4015
Project: HBase
Issue Type: Sub-task
Affects Versions: 0.90.3
Reporter: Jean-Daniel Cryans
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.92.0
Attachments: HBASE-4015_1_trunk.patch, HBASE-4015_2_trunk.patch,
HBASE-4015_reprepared_trunk_2.patch, Timeoutmonitor with state diagrams.pdf
The current implementation of the TimeoutMonitor acts like a race condition
generator, mostly making things worse rather than better. It does it's own
thing for a while without caring for what's happening in the rest of the
master.
The first thing that needs to happen is that the regions should not be
processed in one big batch, because that sometimes can take minutes to
process (meanwhile a region that timed out opening might have opened, then
what happens is it will be reassigned by the TimeoutMonitor generating the
never ending PENDING_OPEN situation).
Those operations should also be done more atomically, although I'm not sure
how to do it in a scalable way in this case.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4310) SlabCache metrics bugfix.
[ https://issues.apache.org/jira/browse/HBASE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096140#comment-13096140 ] Hudson commented on HBASE-4310: --- Integrated in HBase-TRUNK #2171 (See [https://builds.apache.org/job/HBase-TRUNK/2171/]) HBASE-4310 Fix check for lower and upper bounds tedyu : Files : * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java SlabCache metrics bugfix. - Key: HBASE-4310 URL: https://issues.apache.org/jira/browse/HBASE-4310 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor Fix For: 0.92.0 Attachments: metrics.txt, metrics.txt, metrics.txt, metricsv2.txt, metricsv2.txt, metricsv3.txt math error in metrics makes it display incorrect metrics. also no longer logs metrics of size 0 to save space. Also added second log for those things that are successfully cached. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4257) Limit the number of regions in transitions displayed on master webpage.
[ https://issues.apache.org/jira/browse/HBASE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HBASE-4257: -- Assignee: Todd Lipcon Limit the number of regions in transitions displayed on master webpage. --- Key: HBASE-4257 URL: https://issues.apache.org/jira/browse/HBASE-4257 Project: HBase Issue Type: Improvement Components: master Reporter: Jonathan Hsieh Assignee: Todd Lipcon HBASE-3837 added a table to the master web page that displays regions in transitions. There is a possibility that a massive number of RIT's could be present which would make displaying this page slow. This should be limited to a reasonable number (100?) on the main page and provide a link to another page that displays all information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096169#comment-13096169 ] Ted Yu commented on HBASE-4301: --- Summarizing Sebastian's scenario: First attempt on that cluster was made about 2011-08-19. Then patched TRUNK was loaded. .tableinfo for ROOT and .META. have timestamp of 2011-09-02 08:19 The patch is supposed to be applied before the first attempt of migration. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4257) Limit the number of regions in transitions displayed on master webpage.
[ https://issues.apache.org/jira/browse/HBASE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-4257: --- Attachment: hbase-4257.txt accidental other change slipped into previous patch. Limit the number of regions in transitions displayed on master webpage. --- Key: HBASE-4257 URL: https://issues.apache.org/jira/browse/HBASE-4257 Project: HBase Issue Type: Improvement Components: master Reporter: Jonathan Hsieh Assignee: Todd Lipcon Attachments: hbase-4257.txt, hbase-4257.txt HBASE-3837 added a table to the master web page that displays regions in transitions. There is a possibility that a massive number of RIT's could be present which would make displaying this page slow. This should be limited to a reasonable number (100?) on the main page and provide a link to another page that displays all information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4257) Limit the number of regions in transitions displayed on master webpage.
[ https://issues.apache.org/jira/browse/HBASE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-4257: --- Attachment: hbase-4257.txt Here's a simple patch that limits it to 100 RITs on the main page, and indicates if there are more that are not shown. It will also always show META or ROOT if they're in transition. I was too lazy to add pagination at this point. You can see the full list in the new status dump servlet at /dump. Limit the number of regions in transitions displayed on master webpage. --- Key: HBASE-4257 URL: https://issues.apache.org/jira/browse/HBASE-4257 Project: HBase Issue Type: Improvement Components: master Reporter: Jonathan Hsieh Assignee: Todd Lipcon Attachments: hbase-4257.txt, hbase-4257.txt HBASE-3837 added a table to the master web page that displays regions in transitions. There is a possibility that a massive number of RIT's could be present which would make displaying this page slow. This should be limited to a reasonable number (100?) on the main page and provide a link to another page that displays all information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096176#comment-13096176 ] Sebastian Bauer commented on HBASE-4301: I have hadoop directory with backup before upgrading to TRUNK so i need about half an hour to unpack it and test with patched TRUNK ;) META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4257) Limit the number of regions in transitions displayed on master webpage.
[ https://issues.apache.org/jira/browse/HBASE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096178#comment-13096178 ] stack commented on HBASE-4257: -- +1 Limit the number of regions in transitions displayed on master webpage. --- Key: HBASE-4257 URL: https://issues.apache.org/jira/browse/HBASE-4257 Project: HBase Issue Type: Improvement Components: master Reporter: Jonathan Hsieh Assignee: Todd Lipcon Attachments: hbase-4257.txt, hbase-4257.txt HBASE-3837 added a table to the master web page that displays regions in transitions. There is a possibility that a massive number of RIT's could be present which would make displaying this page slow. This should be limited to a reasonable number (100?) on the main page and provide a link to another page that displays all information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096185#comment-13096185 ] stack commented on HBASE-4308: -- Any harm swapping the order in which we do the delete and RIT remove? Race between RegionOpenedHandler and AssignmentManager -- Key: HBASE-4308 URL: https://issues.apache.org/jira/browse/HBASE-4308 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 When the master is processing a ZK event for REGION_OPENED, it calls delete() on the znode before it removes the node from RegionsInTransition. If the notification of that delete comes back into AssignmentManager before the region is removed from RIT, you see an error like: 2011-08-30 17:43:29,537 WARN [main-EventThread] master.AssignmentManager(861): Node deleted but still in RIT: .META.,,1.1028785192 state=OPEN, ts=1314751409532, server=todd-w510,55655,1314751396840 Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4321) Add more comprehensive region split calculator
[
https://issues.apache.org/jira/browse/HBASE-4321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096189#comment-13096189
]
jirapos...@reviews.apache.org commented on HBASE-4321:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1703/#review1735
---
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3957
Instead of this, why not make:
interface KeyRange {
byte[] getStartKey();
byte[] getEndKey();
}
and then a ComparatorKeyRange? Then HRegionInfo could implement KeyRange
with no wrappers.
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3949
check out ComparisonChain from guava. Something like:
ComparisonChain.start()
.compare(getStartKey(), r2.getStartKey())
.compare(getEndKey(), r2.getEndKey())
.result()
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3950
typo is contains
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3951
what's n?
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3952
remove empty javadoc
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3953
rename obj to range or something
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3954
maybe log a warning? or at least a debug.
Should this be = 0? I don't know if other stuff would fall apart if he had
a degenerate/empty region
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3955
if I understand this correctly, it might be easier written as something
like:
for (byte[] coveredSplit : splits.subSet(r.getStartKey(), r.endKey()) {
regions.put(coveredSplit, r);
}
src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java
https://reviews.apache.org/r/1703/#comment3956
I think it would be nice if you actually had this return sb.toString(), and
then for each of the cases, actually asserted equality against something you've
hardcoded. eg:
assertEquals(
A:t[A,B]\n +
B:\t[B,C]\t[B,D]\n +
D:\t\n +
E:\t[E,F]\n,
dump(foo));
- Todd
On 2011-09-02 15:28:35, jmhsieh wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1703/
bq. ---
bq.
bq. (Updated 2011-09-02 15:28:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon and Michael Stack.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This is the core region split calculator. It provides a sorted set of
region split points, and a coverage multimap. This is enough information to
properly determine dupe start keys, all the different kinds of overlap, as well
as holes.
bq.
bq.
bq. This addresses bug hbase-4321.
bq. https://issues.apache.org/jira/browse/hbase-4321
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1703/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Unit tests just checks algorithm, no dependencies on HBase yet.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
Add more comprehensive region split calculator
--
Key: HBASE-4321
URL: https://issues.apache.org/jira/browse/HBASE-4321
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Attachments:
0001-HBASE-4321-Add-more-comprehensive-region-split-calcu.patch
Hbck currently scans through meta one entry at a time, only keeping a
reference to the previous meta entry. This is insufficient for capturing all
the possible problems in meta and needs something more to properly identify
holes, overlaps, duplicate start keys, and otherwise invalid meta entries.
Ideally, this calculator could also be used online interrogating an existing
meta (HBASE-4058), and also used to generate a completely new meta offline
just from regioninfo and in
[jira] [Commented] (HBASE-4308) Race between RegionOpenedHandler and AssignmentManager
[ https://issues.apache.org/jira/browse/HBASE-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096191#comment-13096191 ] Jean-Daniel Cryans commented on HBASE-4308: --- I was also mentioning that we have to keep in mind that ZK notifications can be delayed, sometimes by tens of seconds when there's a huge backup. Race between RegionOpenedHandler and AssignmentManager -- Key: HBASE-4308 URL: https://issues.apache.org/jira/browse/HBASE-4308 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 When the master is processing a ZK event for REGION_OPENED, it calls delete() on the znode before it removes the node from RegionsInTransition. If the notification of that delete comes back into AssignmentManager before the region is removed from RIT, you see an error like: 2011-08-30 17:43:29,537 WARN [main-EventThread] master.AssignmentManager(861): Node deleted but still in RIT: .META.,,1.1028785192 state=OPEN, ts=1314751409532, server=todd-w510,55655,1314751396840 Not certain if it causes issues, but it's a concerning log message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2195) Support cyclic replication
[ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096195#comment-13096195 ] stack commented on HBASE-2195: -- So, we'd go with using the attribute and the attribute would have the uuid for the cluster. It'd be set on every edit rather than per every HLogEdit (which could have one or a batch of edits). Chatting w/ J-D, he reminds me that there is not yet a means of correlating the byte-level cluster identifier with the uuid cluster id we have for CPs (there is no cluster registry where the byte gets mapped to uuid). Downsides: + Going this route would make the byte id redundant; might as well clear it out if we are not going to use it + Fat UUID repeated per individual edit (in a Map so fatter still, rather than as data member bits in byte array) Upsides: + 'safe' -- no need of cluster registry + Minimal changes required to core types getting cyclic replication working Whats your thought Lars... get this attribute based version working then optimize space-used, etc., later? Support cyclic replication -- Key: HBASE-2195 URL: https://issues.apache.org/jira/browse/HBASE-2195 Project: HBase Issue Type: Sub-task Components: replication Reporter: Jean-Daniel Cryans Attachments: 2195.txt We need to support cyclic replication by using the cluster id of each HlogKey and stop replicating when it goes back to the original cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4328) make HBaseAdmin.modifyTable sync
make HBaseAdmin.modifyTable sync Key: HBASE-4328 URL: https://issues.apache.org/jira/browse/HBASE-4328 Project: HBase Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma HBaseAdmin.modifyTable calls HMasterInterface.modifyTable and returns right away. HMasterInterface.modifyTable is async. Thus HBaseAdmin.modifyTable is async. It will be nice to make HBaseAdmin.modifyTable sync given the rest of table operation APIs on HBaseAdmin are sync. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096209#comment-13096209 ] Michael Weng commented on HBASE-4327: - I added a separate 0.22 profile without annotations. Build is fine. And no issue found so far while running hbase trunk on hadoop 0.22. Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096211#comment-13096211 ] stack commented on HBASE-4327: -- Good on you Michael. Post a patch? Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096212#comment-13096212 ] Joep Rottinghuis commented on HBASE-4327: - Testing a patch locally, will post soon. Initially I thought that HBase depended on hadoop-annotations, but that is a problem with 0.23 (probably a missed dependency in those POMs). Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096213#comment-13096213
]
Sebastian Bauer commented on HBASE-4301:
unfortunately after clean upgrade from backup running before branch-0.90 to
patched TRUNK i still cannot even scan .META. with ./hbase shell
with v2 patch i get this:
2011-09-02 20:49:30,473 INFO org.apache.hadoop.hbase.master.SplitLogManager:
found 0 orphan tasks and 0 rescan nodes
2011-09-02 20:49:30,787 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
Creating HTableDescriptor for ROOT tables
2011-09-02 20:49:30,788 INFO org.apache.hadoop.hbase.util.FSUtils: Current
tableInfoPath = hdfs://lala.golden:50001/hbase/-ROOT-/.tableinfo
2011-09-02 20:49:30,790 INFO org.apache.hadoop.hbase.util.FSUtils:
TableInfoPath = hdfs://lala.golden:50001/hbase/-ROOT-/.tableinfo tmpPath =
hdfs://lala.golden:50001/hbase/-ROOT-/.tmp/.tableinfo
2011-09-02 20:49:30,862 INFO org.apache.hadoop.hbase.util.FSUtils:
TableDescriptor stored. TableInfoPath =
hdfs://lala.golden:50001/hbase/-ROOT-/.tableinfo
2011-09-02 20:49:30,864 INFO org.apache.hadoop.hbase.master.MasterFileSystem:
Creating HTableDescriptor for .META. tables
2011-09-02 20:49:30,865 INFO org.apache.hadoop.hbase.util.FSUtils: Current
tableInfoPath = hdfs://lala.golden:50001/hbase/.META./.tableinfo
2011-09-02 20:49:30,866 INFO org.apache.hadoop.hbase.util.FSUtils:
TableInfoPath = hdfs://lala.golden:50001/hbase/.META./.tableinfo tmpPath =
hdfs://lala.golden:50001/hbase/.META./.tmp/.tableinfo
2011-09-02 20:49:30,877 INFO org.apache.hadoop.hbase.util.FSUtils:
TableDescriptor stored. TableInfoPath =
hdfs://lala.golden:50001/hbase/.META./.tableinfo
and later:
2011-09-02 20:50:40,374 INFO org.apache.hadoop.hbase.catalog.CatalogTracker:
Passed metaserver is null
2011-09-02 20:50:40,405 INFO org.apache.hadoop.hbase.master.AssignmentManager:
No previous transition plan was found (or we are ignoring an existing plan) for
.META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=,
dest=lala.golden,60020,1314989371098; 1 (online=1, exclude=null) available serve
rs
2011-09-02 20:50:40,405 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Assigning region .META.,,1.1028785192 to lala.golden,60020,1314989371098
2011-09-02 20:50:40,636 INFO
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED
event for 1028785192; deleting unassigned node
2011-09-02 20:50:40,638 WARN org.apache.hadoop.hbase.master.AssignmentManager:
Node deleted but still in RIT: .META.,,1.1028785192 state=OPEN,
ts=1314989440635, server=lala.golden,60020,1314989371098
2011-09-02 20:50:40,638 INFO
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region
.META.,,1.1028785192 on lala.golden,60020,1314989371098
2011-09-02 20:50:40,639 INFO org.apache.hadoop.hbase.zookeeper.MetaNodeTracker:
Detected completed assignment of META, notifying catalog tracker
2011-09-02 20:50:40,667 INFO org.apache.hadoop.hbase.master.HMaster: .META.
assigned=2, rit=false, location=lala.golden,60020,1314989371098
2011-09-02 20:50:40,671 INFO org.apache.hadoop.hbase.master.HMaster:
metaUpdated = NULL.
2011-09-02 20:50:40,671 INFO org.apache.hadoop.hbase.master.HMaster: Meta
updated status = false
2011-09-02 20:50:40,672 INFO org.apache.hadoop.hbase.master.HMaster: Meta has
HRI with HTDs. Updating meta now.
2011-09-02 20:50:40,720 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Current INFO from scan results = REGION = {NAME =
'EXT_AC,,1295533530463.4e36610aa00b7e8743c1816b12c2e698.', STARTKEY = '',
ENDKEY = '', ENCODED = 4e36610aa00b7e8743c1816b12c2e698, TABLE = {{NAME =
'EXT_AC', FAMILIES = [{NAME = 'c', BLOO
MFILTER = 'ROW', REPLICATION_SCOPE = '0', VERSIONS = '1', COMPRESSION =
'LZO', TTL = '-1', BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE =
'true'}]}}
META migration from 0.90 to trunk fails
---
Key: HBASE-4301
URL: https://issues.apache.org/jira/browse/HBASE-4301
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Ted Yu
Priority: Blocker
Fix For: 0.92.0
Attachments: 4301.txt, 4301_v2.txt, master-log.txt
I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table
is screwed. I can't scan it, etc, and other operations fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096221#comment-13096221 ] Joep Rottinghuis commented on HBASE-4327: - Still seeing compilation error: [ERROR] Failed to execute goal on project hbase: Could not resolve dependencies for project org.apache.hbase:hbase:jar:0.91.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hadoop:hadoop-test:jar:0.22.0-SNAPSHOT, org.apache.hadoop:guava:jar:r09: Could not find artifact org.apache.hadoop:hadoop-test:jar:0.22.0-SNAPSHOT in apache release (https://repository.apache.org/content/repositories/releases/) - [Help 1] The guava one is related to HDFS-2189 (and HDFS-2214), which for some reason keeps rearing it's ugly head. Just wiped out ~/.m2/repository. Will download the offending POM manually to double-check. Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4320) Off Heap Cache never creates Slabs
[ https://issues.apache.org/jira/browse/HBASE-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4320: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to TRUNK. Thanks for patch Li and review Ted. Off Heap Cache never creates Slabs -- Key: HBASE-4320 URL: https://issues.apache.org/jira/browse/HBASE-4320 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Fix For: 0.92.0 Attachments: confnotloading.txt On testing, the configuration file is never loaded by the off heap cache. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4319) site.xml
[ https://issues.apache.org/jira/browse/HBASE-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096223#comment-13096223 ] stack commented on HBASE-4319: -- We should have added a redirect? site.xml Key: HBASE-4319 URL: https://issues.apache.org/jira/browse/HBASE-4319 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: site_HBASE_4319.xml.patch site.xml Removing the link to the not so quick guide. That section doesn't exist anymore since the Configuration chapter was overhauled. Unfortunately, that book section is still on the public website, which another cleanup item. Historical note: Not So Quick and Config used to be two different chapters, separated by the Upgrade chapter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4319) site.xml
[ https://issues.apache.org/jira/browse/HBASE-4319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096226#comment-13096226 ] Doug Meil commented on HBASE-4319: -- I think that should also happen. This site.xml change I think needed to happen regardless, but redirecting the quickstart page on the website to the book would be a good idea. site.xml Key: HBASE-4319 URL: https://issues.apache.org/jira/browse/HBASE-4319 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: site_HBASE_4319.xml.patch site.xml Removing the link to the not so quick guide. That section doesn't exist anymore since the Configuration chapter was overhauled. Unfortunately, that book section is still on the public website, which another cleanup item. Historical note: Not So Quick and Config used to be two different chapters, separated by the Upgrade chapter. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4206) jenkins hash implementation uses longs unnecessarily
[ https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096227#comment-13096227 ] stack commented on HBASE-4206: -- Patch looks good to me. Lots of changes though. Before integrating, how do we know your changes have not changed how the hash engine works? Thanks Ron. jenkins hash implementation uses longs unnecessarily Key: HBASE-4206 URL: https://issues.apache.org/jira/browse/HBASE-4206 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.90.4 Reporter: Ron Yang Priority: Minor Fix For: 0.92.0, 0.90.5 Attachments: ryang.patch I don't believe you need to use long for a,b,c and as a result no longer need to against INT_MASK. At a minimum the private static longs should be made final, and the main method should not print the absolute value of the hash but instead use something like Integer.toHexString -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096241#comment-13096241 ] Joep Rottinghuis commented on HBASE-4327: - Problem still there on the Hadoop side: https://repository.apache.org/content/groups/snapshots/org/apache/hadoop/hadoop-hdfs/0.22.0-SNAPSHOT/hadoop-hdfs-0.22.0-20110726.103826-380.pom The POM still points to org.apache.hadoop#guava and it should be com.google.guava. The source code in hdfs is correct, but the last published build is stale. Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Bauer updated HBASE-4301: --- Attachment: logs.tar.gz logs from upgrading hbase from branch-0.90 to TRUNK META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[
https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096277#comment-13096277
]
Ted Yu commented on HBASE-4301:
---
The only exception I saw was:
{code}
2011-09-02 20:53:45,587 WARN org.apache.hadoop.hbase.master.HMaster: Failed
getting all descriptors
java.io.FileNotFoundException: No status for hdfs://x:50001/hbase/me/.tableinfo
at
org.apache.hadoop.hbase.util.FSUtils.getTableInfoModtime(FSUtils.java:876)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:127)
at
org.apache.hadoop.hbase.util.FSTableDescriptors.getAll(FSTableDescriptors.java:154)
at
org.apache.hadoop.hbase.master.HMaster.getHTableDescriptors(HMaster.java:1375)
{code}
Startup seems to be normal:
{code}
2011-09-02 20:55:08,198 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Bulk assigning done
2011-09-02 20:55:08,201 INFO org.apache.hadoop.hbase.master.HMaster: Master has
completed initialization
{code}
Trace of HBASE-4308 is clearly visible from the log.
META migration from 0.90 to trunk fails
---
Key: HBASE-4301
URL: https://issues.apache.org/jira/browse/HBASE-4301
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Ted Yu
Priority: Blocker
Fix For: 0.92.0
Attachments: 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt
I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table
is screwed. I can't scan it, etc, and other operations fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4257) Limit the number of regions in transitions displayed on master webpage.
[ https://issues.apache.org/jira/browse/HBASE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HBASE-4257. Resolution: Fixed Fix Version/s: 0.92.0 Hadoop Flags: [Reviewed] Committed to trunk, thanks for review. Limit the number of regions in transitions displayed on master webpage. --- Key: HBASE-4257 URL: https://issues.apache.org/jira/browse/HBASE-4257 Project: HBase Issue Type: Improvement Components: master Reporter: Jonathan Hsieh Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-4257.txt, hbase-4257.txt HBASE-3837 added a table to the master web page that displays regions in transitions. There is a possibility that a massive number of RIT's could be present which would make displaying this page slow. This should be limited to a reasonable number (100?) on the main page and provide a link to another page that displays all information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4301) META migration from 0.90 to trunk fails
[ https://issues.apache.org/jira/browse/HBASE-4301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096281#comment-13096281 ] Sebastian Bauer commented on HBASE-4301: This exception is from list command from shell and doesn't matter in this case because i have directory me in hbase rootdir. Problem is that i can't do anything with normal tables, thrift and hbase shell can't find .META. table and this problem doesn't log anything. Maybe HBASE-4308 is the root problem for this, when there will be some patch i'll try it. META migration from 0.90 to trunk fails --- Key: HBASE-4301 URL: https://issues.apache.org/jira/browse/HBASE-4301 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Ted Yu Priority: Blocker Fix For: 0.92.0 Attachments: 4301.txt, 4301_v2.txt, logs.tar.gz, master-log.txt I started a trunk cluster as an upgrade from 0.90.4ish, and now my META table is screwed. I can't scan it, etc, and other operations fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joep Rottinghuis updated HBASE-4327: Attachment: HBASE-4327.patch Preliminary patch. Still need to resolve downstream hdfs dependency issue first. Asked Konstantin to initiate integration build on hadoop-0.22 in order to publish fixed jar+pom. Once in I need to check where the hadoop-test error comes from. Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Attachments: HBASE-4327.patch Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4206) jenkins hash implementation uses longs unnecessarily
[ https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096302#comment-13096302 ] Ron Yang commented on HBASE-4206: - I wrote a data generator and did before/after hash comparisons. I also wrote a test to verify that signed int arithmetic behaves the same as long arithmetic truncated to int. The new code has been running in production here for a few weeks and behaving satisfactorily, although in a non-mission-critical capacity. jenkins hash implementation uses longs unnecessarily Key: HBASE-4206 URL: https://issues.apache.org/jira/browse/HBASE-4206 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.90.4 Reporter: Ron Yang Priority: Minor Fix For: 0.92.0, 0.90.5 Attachments: ryang.patch I don't believe you need to use long for a,b,c and as a result no longer need to against INT_MASK. At a minimum the private static longs should be made final, and the main method should not print the absolute value of the hash but instead use something like Integer.toHexString -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2195) Support cyclic replication
[ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096314#comment-13096314 ] Lars Hofhansl commented on HBASE-2195: -- I have been thinking about this too. Actually I found that the cluster id in ReplicationSource is always 0 (i.e. it is never set for the Master - and my tests just worked by accident :( ). In fact I came to the same conclusion... I think the main reason that it is nice that the replication members do not need to be labeled. Knowing their zookeeper ensemble allows us to get the UUID. I have a patch that uses the UUID about 70% done. One of the problems is that now one cannot enable replication without the other side running as it needs to connect to the other zookeeper to get the cluster id (but I can probably work around that by retrieving the UUID later). I think getting the attribute based version running is a good start. Then we can decide whether this is safe enough for 0.92 or we'll do a space optimized version for 0.94. Support cyclic replication -- Key: HBASE-2195 URL: https://issues.apache.org/jira/browse/HBASE-2195 Project: HBase Issue Type: Sub-task Components: replication Reporter: Jean-Daniel Cryans Attachments: 2195.txt We need to support cyclic replication by using the cluster id of each HlogKey and stop replicating when it goes back to the original cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2195) Support cyclic replication
[ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096326#comment-13096326 ] stack commented on HBASE-2195: -- +1 on the above. Support cyclic replication -- Key: HBASE-2195 URL: https://issues.apache.org/jira/browse/HBASE-2195 Project: HBase Issue Type: Sub-task Components: replication Reporter: Jean-Daniel Cryans Attachments: 2195.txt We need to support cyclic replication by using the cluster id of each HlogKey and stop replicating when it goes back to the original cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4206) jenkins hash implementation uses longs unnecessarily
[ https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096332#comment-13096332 ] Ted Yu commented on HBASE-4206: --- +1 on patch. It would be nice if the test for verifying signed int arithmetic can be published as well. jenkins hash implementation uses longs unnecessarily Key: HBASE-4206 URL: https://issues.apache.org/jira/browse/HBASE-4206 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.90.4 Reporter: Ron Yang Priority: Minor Fix For: 0.92.0, 0.90.5 Attachments: ryang.patch I don't believe you need to use long for a,b,c and as a result no longer need to against INT_MASK. At a minimum the private static longs should be made final, and the main method should not print the absolute value of the hash but instead use something like Integer.toHexString -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4206) jenkins hash implementation uses longs unnecessarily
[ https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096330#comment-13096330 ] stack commented on HBASE-4206: -- @Ron Can you attach your data generator code in case someone later has a similar issue and want to check their work or if an issue w/ this patch and they need to see how it was that the data generator found no issue w/ your patch above? If you can't the above testimony works for me and I'll apply your patch... jenkins hash implementation uses longs unnecessarily Key: HBASE-4206 URL: https://issues.apache.org/jira/browse/HBASE-4206 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.90.4 Reporter: Ron Yang Priority: Minor Fix For: 0.92.0, 0.90.5 Attachments: ryang.patch I don't believe you need to use long for a,b,c and as a result no longer need to against INT_MASK. At a minimum the private static longs should be made final, and the main method should not print the absolute value of the hash but instead use something like Integer.toHexString -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4206) jenkins hash implementation uses longs unnecessarily
[ https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-4206: - Assignee: Ron Yang jenkins hash implementation uses longs unnecessarily Key: HBASE-4206 URL: https://issues.apache.org/jira/browse/HBASE-4206 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.90.4 Reporter: Ron Yang Assignee: Ron Yang Priority: Minor Fix For: 0.92.0, 0.90.5 Attachments: ryang.patch I don't believe you need to use long for a,b,c and as a result no longer need to against INT_MASK. At a minimum the private static longs should be made final, and the main method should not print the absolute value of the hash but instead use something like Integer.toHexString -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4329) Use NextGen Hadoop to deploy HBase
Use NextGen Hadoop to deploy HBase -- Key: HBASE-4329 URL: https://issues.apache.org/jira/browse/HBASE-4329 Project: HBase Issue Type: Brainstorming Reporter: Arun C Murthy Currently (circa 2011), with due respect, it's not practical to run shared, multi-tenant HBase clusters on the largest Hadoop installs (of 4000+ nodes). As an interim, I'd like to brainstorm using NextGen Hadoop (MAPREDUCE-279) to deploy HBase for focussed sets of applications/users/organizations. Thus, one could deploy a smaller instance of HBase (100s of nodes) in a large Hadoop cluster and use it for a set of applications. The other advantage is that the resource usage of HBase (master, region-server etc.) is accounted for in the overall utilization of the cluster and, conceivably, aid in resource tracking, capacity planning etc. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4329) Use NextGen Hadoop to deploy HBase
[ https://issues.apache.org/jira/browse/HBASE-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096348#comment-13096348 ] Arun C Murthy commented on HBASE-4329: -- Potentially this is related to Andrew's ideas in HBASE-4047 for using NextGen Hadoop to run generic co-processors. Use NextGen Hadoop to deploy HBase -- Key: HBASE-4329 URL: https://issues.apache.org/jira/browse/HBASE-4329 Project: HBase Issue Type: Brainstorming Reporter: Arun C Murthy Currently (circa 2011), with due respect, it's not practical to run shared, multi-tenant HBase clusters on the largest Hadoop installs (of 4000+ nodes). As an interim, I'd like to brainstorm using NextGen Hadoop (MAPREDUCE-279) to deploy HBase for focussed sets of applications/users/organizations. Thus, one could deploy a smaller instance of HBase (100s of nodes) in a large Hadoop cluster and use it for a set of applications. The other advantage is that the resource usage of HBase (master, region-server etc.) is accounted for in the overall utilization of the cluster and, conceivably, aid in resource tracking, capacity planning etc. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096353#comment-13096353
]
Hudson commented on HBASE-4027:
---
Integrated in HBase-TRUNK #2172 (See
[https://builds.apache.org/job/HBase-TRUNK/2172/])
HBASE-4027 Off Heap Cache never creates Slabs
stack :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2195) Support cyclic replication
[ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096354#comment-13096354 ] Lars Hofhansl commented on HBASE-2195: -- Now I have to wrap my head around the naming convention in Zookeeper. Currently this is all labeled with the 1 digit cluster id. '-' is used as separator in case of ZK failures. So the ClusterUUID needs to be encoded (maybe as simple as replacing - with x). Base64 encoding is no good because it contains / . Or maybe it is simpler to use ; as separator now. I am not aware of any length limits for ZNode names, is that correct? Is the description here http://hbase.apache.org/replication.html, still completely up-to-date? Support cyclic replication -- Key: HBASE-2195 URL: https://issues.apache.org/jira/browse/HBASE-2195 Project: HBase Issue Type: Sub-task Components: replication Reporter: Jean-Daniel Cryans Attachments: 2195.txt We need to support cyclic replication by using the cluster id of each HlogKey and stop replicating when it goes back to the original cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4302) don't run SNAPPY compression test
[ https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096363#comment-13096363 ] Todd Lipcon commented on HBASE-4302: Does SnappyCodec not provide any test/boolean to check whether the native libraries are available? don't run SNAPPY compression test - Key: HBASE-4302 URL: https://issues.apache.org/jira/browse/HBASE-4302 Project: HBase Issue Type: Test Components: test Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HBASE-4302v1.patch The presence of the Snappy CODEC it does not mean that the Snappy JNI-bindings and native library are available. Because of this it is not possible to assert SNAPPY compression -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4299) Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes
[
https://issues.apache.org/jira/browse/HBASE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096397#comment-13096397
]
Todd Lipcon commented on HBASE-4299:
The tree in {{target/generated-sources}} is a little strange after this patch:
{noformat}
todd@todd-w510:~/svn/hbase-trunk$ ls target/generated-sources/
avro org
todd@todd-w510:~/svn/hbase-trunk$ ls target/generated-sources/avro/
org
{noformat}
Is this right? Why do we have one generated src dir directly in
generated-sources but the avro stuff inside an avro subdir?
Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes
--
Key: HBASE-4299
URL: https://issues.apache.org/jira/browse/HBASE-4299
Project: HBase
Issue Type: Improvement
Components: build
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HBASE-4299v1.patch
Besides updating to the latest Avro, the generated Avro files should be
generated by the build (using the Avro Maven plugin)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2195) Support cyclic replication
[ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096440#comment-13096440 ] Lars Hofhansl commented on HBASE-2195: -- Ah one more. Can I just how change HLogKey is read or written? It seems that this was done before (when byte cluster id was added), but I don't see how that would just work - it only guards again EOFException in readFields. Support cyclic replication -- Key: HBASE-2195 URL: https://issues.apache.org/jira/browse/HBASE-2195 Project: HBase Issue Type: Sub-task Components: replication Reporter: Jean-Daniel Cryans Attachments: 2195.txt We need to support cyclic replication by using the cluster id of each HlogKey and stop replicating when it goes back to the original cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4265) zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled
[ https://issues.apache.org/jira/browse/HBASE-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096441#comment-13096441 ] Ted Yu commented on HBASE-4265: --- +1 on patch. zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled --- Key: HBASE-4265 URL: https://issues.apache.org/jira/browse/HBASE-4265 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.92.0 There seems to be more than just one issue regarding the following scenario. I would provide a fix later just for this exception. 1. A table is being disabled. 2. HMaster restarted. 3. At HMaster startup, it tries to transition from disabling to disabled state. It got the following exception. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /hbase/unassigned/419b902243c836c285108ba555b712fa at org.apache.zookeeper.KeeperException.create(KeeperException.java:110) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:475) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:742) at org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:461) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1440) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1406) at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:141) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) This issue is this specific region is in a special state before HMaster restarts; it has been closed by RS properly thus the zk state is RS_ZK_REGION_CLOSED. However, HMaster hasn't got a chance to process ClosedRegionHandler yet and thus the node remains at zk. After RS restarts, this node is added to online region list first in AssignmentManager.rebuildUserRegions and tries to unassign it later. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4327) Compile HBase against hadoop 0.22
[ https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Weng updated HBASE-4327: Attachment: HBASE-4327-Michael.patch Here is my patch. Compile HBase against hadoop 0.22 - Key: HBASE-4327 URL: https://issues.apache.org/jira/browse/HBASE-4327 Project: HBase Issue Type: Bug Components: build Affects Versions: 0.92.0 Reporter: Joep Rottinghuis Fix For: 0.92.0 Attachments: HBASE-4327-Michael.patch, HBASE-4327.patch Pom contains a profile for hadoop-0.20 and one for hadoop-0.23, but not one for hadoop-0.22. When overriding hadoop.version to 0.22, then the (compile-time) dependency on hadoop-annotations cannot be met. That exists on 0.23 and 0.24/trunk, but not on 0.22. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4271) Clean up coprocessor's handlings of table operations
[ https://issues.apache.org/jira/browse/HBASE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096452#comment-13096452 ] jirapos...@reviews.apache.org commented on HBASE-4271: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1691/#review1729 --- Looks really good, Ming. See just my comments on the preCreateTable and postCreateTable hooks. Nice work beefing up javadoc and clearly indicating where bypass is not supported as well. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java https://reviews.apache.org/r/1691/#comment3963 Move above @param tags? http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java https://reviews.apache.org/r/1691/#comment3944 I think we still need to pass HTableDescriptor here. It's part of what's being created. And I actually have a case right now where I need to be able to tweak the table settings. I also think we should change the return type on this to HRegionInfo[]. This would allow MasterObserver implementations to modify the initial table regions, if desired. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java https://reviews.apache.org/r/1691/#comment3945 Given the need to pass HTableDescriptor to the preCreateTable method, I suppose we should also pass it here. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java https://reviews.apache.org/r/1691/#comment3964 Nice additional tests. I like the checks for bypass behavior. - Gary On 2011-09-01 07:19:49, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1691/ bq. --- bq. bq. (Updated 2011-09-01 07:19:49) bq. bq. bq. Review request for hbase, Gary Helmling and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. Along with coprocessor work, also fix HBaseAdmin.deleteTable so that it waits until the table is removed from HMaster's HTableDescriptor list. bq. bq. 1. Fix preCreateTable, postCreateTable, preAssign, postAssign, preUnassign APIs. bq. 2. Make coprocessor honor bypass request from preMove, preAddColumn, preModifyColumn, preDeleteColumn. bq. bq. bq. This addresses bug HBASE-4271. bq. https://issues.apache.org/jira/browse/HBASE-4271 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 1163917 bq. bq. Diff: https://reviews.apache.org/r/1691/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Clean up coprocessor's handlings of table operations Key: HBASE-4271 URL: https://issues.apache.org/jira/browse/HBASE-4271 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Couple fixes we can do w.r.t coprocessor's handlings of table operations. 1. Honor MasterObserver's requests to bypass default action. 2. Fix up the function signatures for preCreateTable to use HRegionInfo as parameter instead. 3. Invoke postEnableTable, etc. methods after the operations are done. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4299) Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes
[ https://issues.apache.org/jira/browse/HBASE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HBASE-4299: -- Attachment: HBASE-4299v2.patch v2 patch Normalized location of generated sources to *target/generated-sources/java*. Using a single dir to simplify things for eclipse users. The eclipse maven plugin does not recognize the generated sources directories, because of this the developer must add it by hand. Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes -- Key: HBASE-4299 URL: https://issues.apache.org/jira/browse/HBASE-4299 Project: HBase Issue Type: Improvement Components: build Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HBASE-4299v1.patch, HBASE-4299v2.patch Besides updating to the latest Avro, the generated Avro files should be generated by the build (using the Avro Maven plugin) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4299) Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes
[ https://issues.apache.org/jira/browse/HBASE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-4299: --- Resolution: Fixed Fix Version/s: 0.92.0 Release Note: Upgraded the Avro dependency to 1.5.3. Those using both Avro and HBase may want to double check their dependencies are aligned. Hadoop Flags: [Incompatible change, Reviewed] Status: Resolved (was: Patch Available) +1. Committed to trunk. I listed this as an incompatible change since those relying on hbase's libraries and using avro themselves may have a version conflict to resolve with this change. Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes -- Key: HBASE-4299 URL: https://issues.apache.org/jira/browse/HBASE-4299 Project: HBase Issue Type: Improvement Components: build Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.92.0 Attachments: HBASE-4299v1.patch, HBASE-4299v2.patch Besides updating to the latest Avro, the generated Avro files should be generated by the build (using the Avro Maven plugin) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4257) Limit the number of regions in transitions displayed on master webpage.
[ https://issues.apache.org/jira/browse/HBASE-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096477#comment-13096477 ] Hudson commented on HBASE-4257: --- Integrated in HBase-TRUNK #2173 (See [https://builds.apache.org/job/HBase-TRUNK/2173/]) HBASE-4257 Limit the number of regions in transitions displayed on master webpage. todd : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/jamon/org/apache/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/master/TestMasterStatusServlet.java Limit the number of regions in transitions displayed on master webpage. --- Key: HBASE-4257 URL: https://issues.apache.org/jira/browse/HBASE-4257 Project: HBase Issue Type: Improvement Components: master Reporter: Jonathan Hsieh Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-4257.txt, hbase-4257.txt HBASE-3837 added a table to the master web page that displays regions in transitions. There is a possibility that a massive number of RIT's could be present which would make displaying this page slow. This should be limited to a reasonable number (100?) on the main page and provide a link to another page that displays all information. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4271) Clean up coprocessor's handlings of table operations
[ https://issues.apache.org/jira/browse/HBASE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096503#comment-13096503 ] jirapos...@reviews.apache.org commented on HBASE-4271: -- bq. On 2011-09-02 22:58:46, Gary Helmling wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java, line 40 bq. https://reviews.apache.org/r/1691/diff/2/?file=37257#file37257line40 bq. bq. I think we still need to pass HTableDescriptor here. It's part of what's being created. And I actually have a case right now where I need to be able to tweak the table settings. bq. bq. I also think we should change the return type on this to HRegionInfo[]. This would allow MasterObserver implementations to modify the initial table regions, if desired. Thanks, Gray for the review. I will fix that soon. One thing to note is HBaseAdmin verifies the number of requested regions matches the number of regions created. Thus if preCreateTable modifies the size of HRegionInfo[], it will cause issue with HBaseAdmin. If that happens, We could use the default HRegionInfo[] and document this behavior for preCreateTable. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1691/#review1729 --- On 2011-09-01 07:19:49, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1691/ bq. --- bq. bq. (Updated 2011-09-01 07:19:49) bq. bq. bq. Review request for hbase, Gary Helmling and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. Along with coprocessor work, also fix HBaseAdmin.deleteTable so that it waits until the table is removed from HMaster's HTableDescriptor list. bq. bq. 1. Fix preCreateTable, postCreateTable, preAssign, postAssign, preUnassign APIs. bq. 2. Make coprocessor honor bypass request from preMove, preAddColumn, preModifyColumn, preDeleteColumn. bq. bq. bq. This addresses bug HBASE-4271. bq. https://issues.apache.org/jira/browse/HBASE-4271 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 1163917 bq. bq. Diff: https://reviews.apache.org/r/1691/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Clean up coprocessor's handlings of table operations Key: HBASE-4271 URL: https://issues.apache.org/jira/browse/HBASE-4271 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Couple fixes we can do w.r.t coprocessor's handlings of table operations. 1. Honor MasterObserver's requests to bypass default action. 2. Fix up the function signatures for preCreateTable to use HRegionInfo as parameter instead. 3. Invoke postEnableTable, etc. methods after the operations are done. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4206) jenkins hash implementation uses longs unnecessarily
[ https://issues.apache.org/jira/browse/HBASE-4206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Yang updated HBASE-4206: Attachment: JenkinsHashTest.tgz eclipse project with my tests that try to verify the changes didn't regress JenkinsHash jenkins hash implementation uses longs unnecessarily Key: HBASE-4206 URL: https://issues.apache.org/jira/browse/HBASE-4206 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.90.4 Reporter: Ron Yang Assignee: Ron Yang Priority: Minor Fix For: 0.92.0, 0.90.5 Attachments: JenkinsHashTest.tgz, ryang.patch I don't believe you need to use long for a,b,c and as a result no longer need to against INT_MASK. At a minimum the private static longs should be made final, and the main method should not print the absolute value of the hash but instead use something like Integer.toHexString -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4302) don't run SNAPPY compression test
[ https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096515#comment-13096515 ] Alejandro Abdelnur commented on HBASE-4302: --- Actually you are right, it does, but it is wrong, HADOOP-7608. Still, this will not fix the issue here in Hbase because we cannot use the SnappyCodec class directly, because it is not avail in currently released versions of Hadoop. Thoughts? don't run SNAPPY compression test - Key: HBASE-4302 URL: https://issues.apache.org/jira/browse/HBASE-4302 Project: HBase Issue Type: Test Components: test Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HBASE-4302v1.patch The presence of the Snappy CODEC it does not mean that the Snappy JNI-bindings and native library are available. Because of this it is not possible to assert SNAPPY compression -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4265) zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled
[ https://issues.apache.org/jira/browse/HBASE-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096520#comment-13096520 ] Ted Yu commented on HBASE-4265: --- All tests passed. zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled --- Key: HBASE-4265 URL: https://issues.apache.org/jira/browse/HBASE-4265 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.92.0 There seems to be more than just one issue regarding the following scenario. I would provide a fix later just for this exception. 1. A table is being disabled. 2. HMaster restarted. 3. At HMaster startup, it tries to transition from disabling to disabled state. It got the following exception. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /hbase/unassigned/419b902243c836c285108ba555b712fa at org.apache.zookeeper.KeeperException.create(KeeperException.java:110) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:475) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:742) at org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:461) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1440) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1406) at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:141) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) This issue is this specific region is in a special state before HMaster restarts; it has been closed by RS properly thus the zk state is RS_ZK_REGION_CLOSED. However, HMaster hasn't got a chance to process ClosedRegionHandler yet and thus the node remains at zk. After RS restarts, this node is added to online region list first in AssignmentManager.rebuildUserRegions and tries to unassign it later. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4302) don't run SNAPPY compression test
[ https://issues.apache.org/jira/browse/HBASE-4302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096527#comment-13096527 ] Alejandro Abdelnur commented on HBASE-4302: --- After chatting with Todd, we may be able to get the test to work by trapping JNI related exceptions. don't run SNAPPY compression test - Key: HBASE-4302 URL: https://issues.apache.org/jira/browse/HBASE-4302 Project: HBase Issue Type: Test Components: test Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HBASE-4302v1.patch The presence of the Snappy CODEC it does not mean that the Snappy JNI-bindings and native library are available. Because of this it is not possible to assert SNAPPY compression -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4271) Clean up coprocessor's handlings of table operations
[ https://issues.apache.org/jira/browse/HBASE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096530#comment-13096530 ] jirapos...@reviews.apache.org commented on HBASE-4271: -- bq. On 2011-09-02 22:58:46, Gary Helmling wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java, line 40 bq. https://reviews.apache.org/r/1691/diff/2/?file=37257#file37257line40 bq. bq. I think we still need to pass HTableDescriptor here. It's part of what's being created. And I actually have a case right now where I need to be able to tweak the table settings. bq. bq. I also think we should change the return type on this to HRegionInfo[]. This would allow MasterObserver implementations to modify the initial table regions, if desired. bq. bq. Ming Ma wrote: bq. Thanks, Gray for the review. I will fix that soon. One thing to note is HBaseAdmin verifies the number of requested regions matches the number of regions created. Thus if preCreateTable modifies the size of HRegionInfo[], it will cause issue with HBaseAdmin. If that happens, We could use the default HRegionInfo[] and document this behavior for preCreateTable. Hmm, we may be starting to wade into deeper water here. We could change HMasterInterface#createTable to return an int, indicating the number of regions that are actually being created. Then HBaseAdmin could use that value when checking for region counts. Or even have createTable return HRegionInfo[]? Either would make this an incompatible change at the RPC level, so we would need to flag it in JIRA. Or we could drop my suggestion to have preCreateTable return HRegionInfo[]. It was just a thought and I don't have any solid need for it. What do you think? - Gary --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1691/#review1729 --- On 2011-09-01 07:19:49, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1691/ bq. --- bq. bq. (Updated 2011-09-01 07:19:49) bq. bq. bq. Review request for hbase, Gary Helmling and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. Along with coprocessor work, also fix HBaseAdmin.deleteTable so that it waits until the table is removed from HMaster's HTableDescriptor list. bq. bq. 1. Fix preCreateTable, postCreateTable, preAssign, postAssign, preUnassign APIs. bq. 2. Make coprocessor honor bypass request from preMove, preAddColumn, preModifyColumn, preDeleteColumn. bq. bq. bq. This addresses bug HBASE-4271. bq. https://issues.apache.org/jira/browse/HBASE-4271 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 1163917 bq. bq. Diff: https://reviews.apache.org/r/1691/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Clean up coprocessor's handlings of table operations Key: HBASE-4271 URL: https://issues.apache.org/jira/browse/HBASE-4271 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Couple fixes we can do w.r.t coprocessor's handlings of table operations. 1. Honor MasterObserver's requests to bypass default action. 2. Fix up the function signatures for preCreateTable to use HRegionInfo as parameter instead. 3. Invoke postEnableTable, etc. methods after the operations are done. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4330) Fix races in slab cache
Fix races in slab cache --- Key: HBASE-4330 URL: https://issues.apache.org/jira/browse/HBASE-4330 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.92.0 A few races are still lingering in the slab cache. Here are some tests and proposed fixes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4271) Clean up coprocessor's handlings of table operations
[ https://issues.apache.org/jira/browse/HBASE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096543#comment-13096543 ] jirapos...@reviews.apache.org commented on HBASE-4271: -- bq. On 2011-09-02 22:58:46, Gary Helmling wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java, line 40 bq. https://reviews.apache.org/r/1691/diff/2/?file=37257#file37257line40 bq. bq. I think we still need to pass HTableDescriptor here. It's part of what's being created. And I actually have a case right now where I need to be able to tweak the table settings. bq. bq. I also think we should change the return type on this to HRegionInfo[]. This would allow MasterObserver implementations to modify the initial table regions, if desired. bq. bq. Ming Ma wrote: bq. Thanks, Gray for the review. I will fix that soon. One thing to note is HBaseAdmin verifies the number of requested regions matches the number of regions created. Thus if preCreateTable modifies the size of HRegionInfo[], it will cause issue with HBaseAdmin. If that happens, We could use the default HRegionInfo[] and document this behavior for preCreateTable. bq. bq. Gary Helmling wrote: bq. Hmm, we may be starting to wade into deeper water here. We could change HMasterInterface#createTable to return an int, indicating the number of regions that are actually being created. Then HBaseAdmin could use that value when checking for region counts. Or even have createTable return HRegionInfo[]? Either would make this an incompatible change at the RPC level, so we would need to flag it in JIRA. bq. bq. Or we could drop my suggestion to have preCreateTable return HRegionInfo[]. It was just a thought and I don't have any solid need for it. What do you think? I will vote for not making such change for now to allow preCreateTable return HRegionInfo[], given we don't have a use case for it. That is also the reason preCreateTable can't bypass default action. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1691/#review1729 --- On 2011-09-01 07:19:49, Ming Ma wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1691/ bq. --- bq. bq. (Updated 2011-09-01 07:19:49) bq. bq. bq. Review request for hbase, Gary Helmling and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. Along with coprocessor work, also fix HBaseAdmin.deleteTable so that it waits until the table is removed from HMaster's HTableDescriptor list. bq. bq. 1. Fix preCreateTable, postCreateTable, preAssign, postAssign, preUnassign APIs. bq. 2. Make coprocessor honor bypass request from preMove, preAddColumn, preModifyColumn, preDeleteColumn. bq. bq. bq. This addresses bug HBASE-4271. bq. https://issues.apache.org/jira/browse/HBASE-4271 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 1163917 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 1163917 bq. bq. Diff: https://reviews.apache.org/r/1691/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests bq. bq. bq. Thanks, bq. bq. Ming bq. bq. Clean up coprocessor's handlings of table operations Key: HBASE-4271 URL: https://issues.apache.org/jira/browse/HBASE-4271 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Couple fixes we can do w.r.t coprocessor's handlings of table operations. 1. Honor MasterObserver's requests to bypass default action. 2. Fix up the function signatures for preCreateTable to use HRegionInfo as parameter instead. 3. Invoke
[jira] [Updated] (HBASE-4330) Fix races in slab cache
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-4330: --- Attachment: hbase-4330.txt Some new tests and some fixes, let me know what you think Fix races in slab cache --- Key: HBASE-4330 URL: https://issues.apache.org/jira/browse/HBASE-4330 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-4330.txt A few races are still lingering in the slab cache. Here are some tests and proposed fixes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4271) Clean up coprocessor's handlings of table operations
[ https://issues.apache.org/jira/browse/HBASE-4271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096552#comment-13096552 ] jirapos...@reviews.apache.org commented on HBASE-4271: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1691/ --- (Updated 2011-09-03 01:47:00.947743) Review request for hbase, Gary Helmling and Ted Yu. Changes --- Here is the update. Summary --- Along with coprocessor work, also fix HBaseAdmin.deleteTable so that it waits until the table is removed from HMaster's HTableDescriptor list. 1. Fix preCreateTable, postCreateTable, preAssign, postAssign, preUnassign APIs. 2. Make coprocessor honor bypass request from preMove, preAddColumn, preModifyColumn, preDeleteColumn. This addresses bug HBASE-4271. https://issues.apache.org/jira/browse/HBASE-4271 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java 1164783 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java 1164783 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1164783 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1164783 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1164783 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseMasterObserver.java 1164783 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/coprocessor/MasterObserver.java 1164783 Diff: https://reviews.apache.org/r/1691/diff Testing --- Unit tests Thanks, Ming Clean up coprocessor's handlings of table operations Key: HBASE-4271 URL: https://issues.apache.org/jira/browse/HBASE-4271 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Couple fixes we can do w.r.t coprocessor's handlings of table operations. 1. Honor MasterObserver's requests to bypass default action. 2. Fix up the function signatures for preCreateTable to use HRegionInfo as parameter instead. 3. Invoke postEnableTable, etc. methods after the operations are done. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Spiegelberg reassigned HBASE-1730:
--
Assignee: Nicolas Spiegelberg (was: stack)
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Nicolas Spiegelberg
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Spiegelberg updated HBASE-1730:
---
Attachment: HBASE-1730-2.patch
Nileema sent me this patch to address the peer feedback (note, she volunteered
this after her internship, +1). Made minor modifications. Passes TestAdmin
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Nicolas Spiegelberg
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730-2.patch, HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4299) Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes
[ https://issues.apache.org/jira/browse/HBASE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096569#comment-13096569 ] Hudson commented on HBASE-4299: --- Integrated in HBase-TRUNK #2174 (See [https://builds.apache.org/job/HBase-TRUNK/2174/]) HBASE-4299 Update to Avro 1.5.3 and use Avro Maven plugin to generate Avro classes. todd : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/pom.xml * /hbase/trunk/src/main/avro * /hbase/trunk/src/main/avro/hbase.avpr * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/AvroServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AAlreadyExists.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AClusterStatus.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AColumn.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AColumnFamilyDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AColumnValue.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/ACompressionAlgorithm.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/ADelete.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AFamilyDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AGet.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AIOError.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AIllegalArgument.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AMasterNotRunning.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/APut.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/ARegionLoad.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AResult.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AResultEntry.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AScan.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AServerAddress.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AServerInfo.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/AServerLoad.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/ATableDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/ATableExists.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/ATimeRange.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/HBase.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/IOError.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/generated/TCell.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/avro/hbase.avpr Upgrade to Avro 1.5.3 use Avro Maven plugin to generate Avro classes -- Key: HBASE-4299 URL: https://issues.apache.org/jira/browse/HBASE-4299 Project: HBase Issue Type: Improvement Components: build Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.92.0 Attachments: HBASE-4299v1.patch, HBASE-4299v2.patch Besides updating to the latest Avro, the generated Avro files should be generated by the build (using the Avro Maven plugin) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096573#comment-13096573 ] Ted Yu commented on HBASE-4330: --- When I ran TestSingleSizeCache, it seemed to hang. Here is part of jstack: http://pastebin.com/juL5ezzt Fix races in slab cache --- Key: HBASE-4330 URL: https://issues.apache.org/jira/browse/HBASE-4330 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-4330.txt A few races are still lingering in the slab cache. Here are some tests and proposed fixes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4310) SlabCache metrics bugfix.
[ https://issues.apache.org/jira/browse/HBASE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Pi updated HBASE-4310: - Resolution: Fixed Status: Resolved (was: Patch Available) SlabCache metrics bugfix. - Key: HBASE-4310 URL: https://issues.apache.org/jira/browse/HBASE-4310 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Li Pi Priority: Minor Fix For: 0.92.0 Attachments: metrics.txt, metrics.txt, metrics.txt, metricsv2.txt, metricsv2.txt, metricsv3.txt math error in metrics makes it display incorrect metrics. also no longer logs metrics of size 0 to save space. Also added second log for those things that are successfully cached. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096577#comment-13096577 ] Li Pi commented on HBASE-4330: -- Do we really need: contentBlock.serializedData.rewind();? As we always do a ByteBuffer.duplicate later. Fix races in slab cache --- Key: HBASE-4330 URL: https://issues.apache.org/jira/browse/HBASE-4330 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-4330.txt A few races are still lingering in the slab cache. Here are some tests and proposed fixes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2195) Support cyclic replication
[ https://issues.apache.org/jira/browse/HBASE-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096578#comment-13096578 ] Lars Hofhansl commented on HBASE-2195: -- Amending my solution a bit. The Edits will be written with the full id as described above. In the near term I think it is better keep the 1 character ID for local identification. The reason is: 1. It is nice to be able to identify every slave by a number chosen by the master. (addPeer, deletePeer, listPeers, etc) - once we support multiple slaves that is. 2. A user at the master does not have to know about cluster UUID. 3. The ZNode handling would not change. These id is never shared to the slave/sink. Under the hood the full 16byte cluster id is used in the Edits to avoid replication cycles. I assume it would be worth going through the trouble of not storing those 16 bytes of UUID when replication is not enabled(?) Support cyclic replication -- Key: HBASE-2195 URL: https://issues.apache.org/jira/browse/HBASE-2195 Project: HBase Issue Type: Sub-task Components: replication Reporter: Jean-Daniel Cryans Attachments: 2195.txt We need to support cyclic replication by using the cluster id of each HlogKey and stop replicating when it goes back to the original cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4330) Fix races in slab cache
[ https://issues.apache.org/jira/browse/HBASE-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096579#comment-13096579 ] Li Pi commented on HBASE-4330: -- Also, whats the advantage of using a synchronize block and checking whether eviction has been completed yet vs using a RWL? Fix races in slab cache --- Key: HBASE-4330 URL: https://issues.apache.org/jira/browse/HBASE-4330 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-4330.txt A few races are still lingering in the slab cache. Here are some tests and proposed fixes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4265) zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled
[ https://issues.apache.org/jira/browse/HBASE-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096581#comment-13096581 ] Ted Yu commented on HBASE-4265: --- Integrated to TRUNK. Thanks for the patch Ming. Thanks for the review Ramkrishna. zookeeper.KeeperException$NodeExistsException if HMaster restarts while table is being disabled --- Key: HBASE-4265 URL: https://issues.apache.org/jira/browse/HBASE-4265 Project: HBase Issue Type: Bug Reporter: Ming Ma Assignee: Ming Ma Fix For: 0.92.0 There seems to be more than just one issue regarding the following scenario. I would provide a fix later just for this exception. 1. A table is being disabled. 2. HMaster restarted. 3. At HMaster startup, it tries to transition from disabling to disabled state. It got the following exception. org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /hbase/unassigned/419b902243c836c285108ba555b712fa at org.apache.zookeeper.KeeperException.create(KeeperException.java:110) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:475) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:457) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndWatch(ZKUtil.java:742) at org.apache.hadoop.hbase.zookeeper.ZKAssign.createNodeClosing(ZKAssign.java:461) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1440) at org.apache.hadoop.hbase.master.AssignmentManager.unassign(AssignmentManager.java:1406) at org.apache.hadoop.hbase.master.handler.DisableTableHandler$BulkDisabler$1.run(DisableTableHandler.java:141) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) This issue is this specific region is in a special state before HMaster restarts; it has been closed by RS properly thus the zk state is RS_ZK_REGION_CLOSED. However, HMaster hasn't got a chance to process ClosedRegionHandler yet and thus the node remains at zk. After RS restarts, this node is added to online region list first in AssignmentManager.rebuildUserRegions and tries to unassign it later. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096582#comment-13096582
]
Hudson commented on HBASE-1730:
---
Integrated in HBase-TRUNK #2175 (See
[https://builds.apache.org/job/HBase-TRUNK/2175/])
HBASE-1730 Online schema changes for HBase
nspiegelberg :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/BulkReOpen.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ClosedRegionHandler.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java
* /hbase/trunk/src/main/ruby/hbase/admin.rb
* /hbase/trunk/src/main/ruby/shell.rb
* /hbase/trunk/src/main/ruby/shell/commands/alter.rb
* /hbase/trunk/src/main/ruby/shell/commands/alter_async.rb
* /hbase/trunk/src/main/ruby/shell/commands/alter_status.rb
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Nicolas Spiegelberg
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730-2.patch, HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1730) Near-instantaneous online schema and table state updates
[
https://issues.apache.org/jira/browse/HBASE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-1730:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Latest patch has been integrated by Nicolas.
Near-instantaneous online schema and table state updates
Key: HBASE-1730
URL: https://issues.apache.org/jira/browse/HBASE-1730
Project: HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Nicolas Spiegelberg
Priority: Critical
Fix For: 0.92.0
Attachments: 1730-v2.patch, 1730-v3.patch, 1730.patch,
HBASE-1730-2.patch, HBASE-1730.patch
We should not need to take a table offline to update HCD or HTD.
One option for that is putting HTDs and HCDs up into ZK, with mirror on disk
catalog tables to be used only for cold init scenarios, as discussed on IRC.
In this scheme, regionservers hosting regions of a table would watch
permanent nodes in ZK associated with that table for schema updates and take
appropriate actions out of the watcher. In effect, schema updates become
another item in the ToDo list.
{{/hbase/tables/table-name/schema}}
Must be associated with a write locking scheme also handled with ZK
primitives to avoid situations where one concurrent update clobbers another.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4224) Need a flush by regionserver rather than by table option
[ https://issues.apache.org/jira/browse/HBASE-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096584#comment-13096584 ] Akash Ashok commented on HBASE-4224: Hey Stack, I would actually like to work on this. I don't know If I am able to grasp the true gravity and understand how big a problem this is or even If I am equipped enough to handle this. But with a little help I could work on this. Please let me know. Need a flush by regionserver rather than by table option Key: HBASE-4224 URL: https://issues.apache.org/jira/browse/HBASE-4224 Project: HBase Issue Type: Bug Components: shell Reporter: stack This evening needed to clean out logs on the cluster. logs are by regionserver. to let go of logs, we need to have all edits emptied from memory. only flush is by table or region. We need to be able to flush the regionserver. Need to add this. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
