[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508573#comment-13508573 ] Ted Yu commented on HBASE-7259: --- Can you provide patch for trunk so that Hadoop QA can run through test suite ? Thanks Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: ZookeeperNodeTracker.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above:
[jira] [Commented] (HBASE-1648) heap usage and limit reporting doesn't work as expected when using G1 GC
[
https://issues.apache.org/jira/browse/HBASE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508579#comment-13508579
]
liang xie commented on HBASE-1648:
--
[~saint@gmail.com] yes, it should be safe. we should not use G1 before
JDK7u4
And also, i tried in a hbase0.94.0 + jdk6u22 + G1GC test env(not a good choice,
aha):
hbase(main):003:0 status 'simple'
2 live servers
ha3:60020 1354522267193
requestsPerSecond=0, numberOfOnlineRegions=1, usedHeapMB=38,
maxHeapMB=200
ha2:60020 1354551060862
requestsPerSecond=0, numberOfOnlineRegions=6, usedHeapMB=88,
maxHeapMB=200
0 dead servers
Aggregate load: 0, regions: 7
the output is fine
And, i tried with jdk7u9 + G1GC as well:
hbase(main):002:0 status 'simple'
2 live servers
ha2:60020 1354552532489
requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=46,
maxHeapMB=200
ha3:60020 1354523738930
requestsPerSecond=0, numberOfOnlineRegions=2, usedHeapMB=44,
maxHeapMB=200
0 dead servers
Aggregate load: 0, regions: 7
still good
heap usage and limit reporting doesn't work as expected when using G1 GC
Key: HBASE-1648
URL: https://issues.apache.org/jira/browse/HBASE-1648
Project: HBase
Issue Type: Bug
Environment: Linux Centos x86_64 5.3, Sun JDK 1.6.0_14 (HotSpot
64-Bit Server VM build 14.0-b16)
Reporter: Andrew Purtell
Priority: Minor
Getting some early experience with the G1 GC.
Running with -Xmx1000m and HBASE_OPTS set to -XX:+UnlockExperimentalVMOpts
-XX:+UseG1GC. Regionserver heap use reports are not of much use:
{noformat}
hbase(main):001:0 status 'simple'
3 live servers
test3:60020 1247389283042
requests=0, regions=1, usedHeap=0, maxHeap=41
test2:60020 1247389219994
requests=0, regions=1, usedHeap=0, maxHeap=41
test4:60020 1247389324563
requests=0, regions=2, usedHeap=0, maxHeap=41
0 dead servers
{noformat}
top is about right:
{noformat}
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
23988 hadoop24 0 1492m 98m 10m S 0.0 2.5 0:02.65 java
{noformat}
Incidentally, don't try -XX:+UseG1GC and -XX:+DoEscapeAnalysis together or
the JVM will rapidly segfault.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508580#comment-13508580 ] Enis Soztutar commented on HBASE-6367: -- bq. Leaving the backup master header even though there is no backup masters is more clear to users espeically when a user migrates to new hbase version from an old one +1. There might be cases where the backup master is dead, but the user was expecting it. Showing the section with total 0 can be useful. List backup masters in ui. -- Key: HBASE-6367 URL: https://issues.apache.org/jira/browse/HBASE-6367 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Jeffrey Zhong Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: Has BackupMasters.png, hbase-6367.patch, No BackupMasters.png Right now only the active master shows any information on the web ui. It would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei updated HBASE-7259: - Attachment: (was: ZookeeperNodeTracker.patch) Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: === catalina-exec-800: at
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei updated HBASE-7259: - Description: HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: === catalina-exec-800: at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943) - waiting to lock 0x000731902200 (a java.lang.Object) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei updated HBASE-7259: - Attachment: HConnectionManager.patch Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: === catalina-exec-800: at
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei updated HBASE-7259: - Attachment: (was: HConnectionManager.patch) Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: === catalina-exec-800: at
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liwei updated HBASE-7259: - Attachment: HConnectionManager.patch Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: === catalina-exec-800: at
[jira] [Created] (HBASE-7260) Upgrade hadoop 1 dependency to hadoop 1.1.1
Ted Yu created HBASE-7260: - Summary: Upgrade hadoop 1 dependency to hadoop 1.1.1 Key: HBASE-7260 URL: https://issues.apache.org/jira/browse/HBASE-7260 Project: HBase Issue Type: Bug Reporter: Ted Yu hadoop 1.1.1 has been released with 20 bug fixes and improvements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7260) Upgrade hadoop 1 dependency to hadoop 1.1.1
[ https://issues.apache.org/jira/browse/HBASE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7260: -- Attachment: 7260.txt Upgrade hadoop 1 dependency to hadoop 1.1.1 --- Key: HBASE-7260 URL: https://issues.apache.org/jira/browse/HBASE-7260 Project: HBase Issue Type: Bug Reporter: Ted Yu Attachments: 7260.txt hadoop 1.1.1 has been released with 20 bug fixes and improvements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7260) Upgrade hadoop 1 dependency to hadoop 1.1.1
[ https://issues.apache.org/jira/browse/HBASE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-7260: - Assignee: Ted Yu Upgrade hadoop 1 dependency to hadoop 1.1.1 --- Key: HBASE-7260 URL: https://issues.apache.org/jira/browse/HBASE-7260 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Attachments: 7260.txt hadoop 1.1.1 has been released with 20 bug fixes and improvements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7260) Upgrade hadoop 1 dependency to hadoop 1.1.1
[ https://issues.apache.org/jira/browse/HBASE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7260: -- Fix Version/s: 0.96.0 Status: Patch Available (was: Open) Upgrade hadoop 1 dependency to hadoop 1.1.1 --- Key: HBASE-7260 URL: https://issues.apache.org/jira/browse/HBASE-7260 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 7260.txt hadoop 1.1.1 has been released with 20 bug fixes and improvements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508779#comment-13508779 ] Ted Yu commented on HBASE-7259: --- The latest patch was generated from 0.94 branch. Please check out trunk repo: http://svn.apache.org/repos/asf/hbase/trunk And attach patch based on trunk. Normally the patch filename contains the JIRA number (and optionally short description). Thanks Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
[jira] [Updated] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Meil updated HBASE-7221:
-
Attachment: hbase-common_hbase_7221_v3.patch
Ok, I think this one's a winner. :-)
There is a RowKeySchema and a RowKey. This creates fixed-length keys without
delimiters (generally considered to be a best practice), and enforces the
defined lengths when the elements are set. It's also bi-directional, so that
you can pass in a byte-array (i.e., rowkey) from a table and then read the key
elements back.
Creation example...
{code}
int elements[] = {RowKeySchema.SIZEOF_MD5_HASH, RowKeySchema.SIZEOF_INT,
RowKeySchema.SIZEOF_LONG};
RowKeySchema schema = new RowKeySchema(elements);
RowKey rowkey = schema.createRowKey();
rowkey.setHash(0, hashVal);
rowkey.setInt(1, intVal);
rowkey.setLong(2, longVal);
byte bytes[] = rowkey.getBytes();
{code}
RowKey utility class for rowkey construction
Key: HBASE-7221
URL: https://issues.apache.org/jira/browse/HBASE-7221
Project: HBase
Issue Type: Improvement
Reporter: Doug Meil
Assignee: Doug Meil
Priority: Minor
Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch,
hbase-common_hbase_7221_v3.patch
A common question in the dist-lists is how to construct rowkeys, particularly
composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
you to sensibly populate that byte-array, and that's where things tend to go
off the rails.
The intent of this RowKey utility class isn't meant to add functionality into
Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
Example:
{code}
RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
key.addHash(a);
key.add(b);
byte bytes[] = key.getBytes();
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7260) Upgrade hadoop 1 dependency to hadoop 1.1.1
[
https://issues.apache.org/jira/browse/HBASE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508816#comment-13508816
]
Hadoop QA commented on HBASE-7260:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555759/7260.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 26 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3435//console
This message is automatically generated.
Upgrade hadoop 1 dependency to hadoop 1.1.1
---
Key: HBASE-7260
URL: https://issues.apache.org/jira/browse/HBASE-7260
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.96.0
Attachments: 7260.txt
hadoop 1.1.1 has been released with 20 bug fixes and improvements
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7245) Recovery on failed restore.
[ https://issues.apache.org/jira/browse/HBASE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508857#comment-13508857 ] Matteo Bertozzi commented on HBASE-7245: The set of operations that have this kind of problem are: * create table: remove the table if failed (rollback) the user already received the failure * delete table: finish removing the table (rollforward) restoring the table is impossible * clone table: remove the table if failed (rollback) same as create table * restore table: finish restoring the table (rollforward) finish the restore * snapshot: removing the tmp folder (rollback) One simple solution is to drop a operation lock file in the table folder, and on master startup, if the file is present look at the operation enum serialized and execute the rollback/rollforward. (Note that if the master is not down, you can do the recovery catching the exception) Recovery on failed restore. --- Key: HBASE-7245 URL: https://issues.apache.org/jira/browse/HBASE-7245 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Restore will do updates to the file system and to meta. it seems that an inopportune failure before meta is completely updated could result in an inconsistent state that would require hbck to fix. We should define what the semantics are for recovering from this. Some suggestions: 1) Fail Forward (see some log saying restore's meta edits not completed, then gather information necessary to build it all from fs, and complete meta edits.). 2) Fail backwards (see some log saying restore's meta edits not completed, delete incomplete snapshot region entries from meta.) I think I prefer 1 -- if two processes end somehow updating (somehow the original master didn't die, and a new one started up) they would be idempotent. If we used 2, we could still have a race and still be in a bad place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508863#comment-13508863 ] liwei commented on HBASE-7259: -- I don't have privilege to commit. How to do it? Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by catalina-exec-710 Java stack information for the threads listed above: ===
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508871#comment-13508871 ] Ted Yu commented on HBASE-7259: --- I was talking about how to create the patch. 1. svn co http://svn.apache.org/repos/asf/hbase/trunk 2. cd trunk 3. modify HConnectionManager.java 4. at the root of trunk (your workspace) issue: svn diff 7259-remove-deadlock-in-case-keeper-exception.txt Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
[jira] [Updated] (HBASE-7245) Recovery on failed snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7245: -- Summary: Recovery on failed snapshot restore (was: Recovery on failed restore.) The subject of this JIRA is w.r.t. snapshot restore. So the first two scenarios above can be handled in separate JIRA. The operation directive file can be created in zookeeper. Recovery on failed snapshot restore --- Key: HBASE-7245 URL: https://issues.apache.org/jira/browse/HBASE-7245 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Restore will do updates to the file system and to meta. it seems that an inopportune failure before meta is completely updated could result in an inconsistent state that would require hbck to fix. We should define what the semantics are for recovering from this. Some suggestions: 1) Fail Forward (see some log saying restore's meta edits not completed, then gather information necessary to build it all from fs, and complete meta edits.). 2) Fail backwards (see some log saying restore's meta edits not completed, delete incomplete snapshot region entries from meta.) I think I prefer 1 -- if two processes end somehow updating (somehow the original master didn't die, and a new one started up) they would be idempotent. If we used 2, we could still have a race and still be in a bad place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7245) Recovery on failed snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508872#comment-13508872 ] Ted Yu edited comment on HBASE-7245 at 12/3/12 5:20 PM: The subject of this JIRA is w.r.t. snapshot restore. So the first two scenarios above can be handled in separate JIRA. The operation directive can be created in zookeeper. was (Author: yuzhih...@gmail.com): The subject of this JIRA is w.r.t. snapshot restore. So the first two scenarios above can be handled in separate JIRA. The operation directive file can be created in zookeeper. Recovery on failed snapshot restore --- Key: HBASE-7245 URL: https://issues.apache.org/jira/browse/HBASE-7245 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Restore will do updates to the file system and to meta. it seems that an inopportune failure before meta is completely updated could result in an inconsistent state that would require hbck to fix. We should define what the semantics are for recovering from this. Some suggestions: 1) Fail Forward (see some log saying restore's meta edits not completed, then gather information necessary to build it all from fs, and complete meta edits.). 2) Fail backwards (see some log saying restore's meta edits not completed, delete incomplete snapshot region entries from meta.) I think I prefer 1 -- if two processes end somehow updating (somehow the original master didn't die, and a new one started up) they would be idempotent. If we used 2, we could still have a race and still be in a bad place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[ https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508873#comment-13508873 ] Lars Hofhansl commented on HBASE-7259: -- Things are much different in trunk as far as the client is concerned. A trunk patch for this neither realistic nor useful. Thanks for the analysis [~boneylw] Deadlock in HBaseClient when KeeperException occured Key: HBASE-7259 URL: https://issues.apache.org/jira/browse/HBASE-7259 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.0, 0.94.1, 0.94.2 Reporter: liwei Priority: Critical Attachments: HConnectionManager.patch HBaseClient was running after a period of time, all of get operation became too slow. From the client logs I could see the following: 1. Unable to get data of znode /hbase/root-region-server java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725) at org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82) at org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162) at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685) at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366) 2. Catalina.out found one Java-level deadlock: = catalina-exec-800: waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a java.lang.Object), which is held by catalina-exec-710 catalina-exec-710: waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation), which is held by catalina-exec-29-EventThread catalina-exec-29-EventThread: waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a org.apache.hadoop.hbase.zookeeper.RootRegionTracker), which is held by
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-7259:
--
Description:
HBaseClient was running after a period of time, all of get operation became too
slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by catalina-exec-710
Java stack information for the threads listed above:
===
catalina-exec-800:
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:943)
- waiting to lock 0x000731902200 (a java.lang.Object)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.relocateRegion(HConnectionManager.java:807)
at
[jira] [Commented] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508884#comment-13508884
]
Lars Hofhansl commented on HBASE-7259:
--
Patch looks good.
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Attachments: HConnectionManager.patch
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by catalina-exec-710
Java stack information for the threads listed above:
===
[jira] [Commented] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508896#comment-13508896 ] Sergey Shelukhin commented on HBASE-1299: - +1 on patch JSPs don't HTML escape literals (ie: table names, region names, start end keys) - Key: HBASE-1299 URL: https://issues.apache.org/jira/browse/HBASE-1299 Project: HBase Issue Type: Bug Affects Versions: 0.19.0, 0.19.1 Reporter: Hoss Man Assignee: Nick Dimiduk Attachments: 1299.patch similar to HBASE-1298, the various JSPs included with HBase for monitoring the system don't seem to do any HTML escaping when displaying user entered data which may contain special characters: table names, region names, start Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508900#comment-13508900 ] Sergey Shelukhin commented on HBASE-7055: - bq. The family settings are pulled in from the CF descriptor, there isn't any per-CF configuration in XML, though, right? Yes. port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically) - Key: HBASE-7055 URL: https://issues.apache.org/jira/browse/HBASE-7055 Project: HBase Issue Type: Task Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.96.0 Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, HBASE-6371-v3-refactor-only-squashed.patch, HBASE-6371-v4-refactor-only-squashed.patch, HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, HBASE-7055-v1.patch There's divergence in the code :( See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508916#comment-13508916 ] nkeywal commented on HBASE-7247: An possible solution could be: - use a watcher to see if someone else is updating the znode - use an asynchronous write to update the znode when we want to say to the master we're still there. It mostly as of today, except that we suppressed the synchronous write on the regionserver path. Pros: should be a small enough change. Cons: we're still loading ZK (and the master as a side effect). What do you think? Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening Key: HBASE-7247 URL: https://issues.apache.org/jira/browse/HBASE-7247 Project: HBase Issue Type: Improvement Components: master, Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Critical The regionserver.OpenRegionHandler#tickleOpening updates the region znode as Do this so master doesn't timeout this region-in-transition.. However, on the usual test, this makes the assignment time of 1500 regions goes from 70s to 100s, that is, we're 50% slower because of this. More generally, ZooKeper commits to disk all the data update, and this takes time. Using it to provide a keep alive seems overkill. At the very list, it could be made asynchronous. I'm not sure how necessary these updates are required (I need to go deeper in the internal, feedback welcome), but it seems very important to optimize this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7219) Make it so can connect remotely to a standalone hbase
[ https://issues.apache.org/jira/browse/HBASE-7219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508922#comment-13508922 ] nkeywal commented on HBASE-7219: I'm putting this on my low priority todo list. I will assign it to me when I start working on it. If someone wants to pick it up, that's fine with me as well. Make it so can connect remotely to a standalone hbase - Key: HBASE-7219 URL: https://issues.apache.org/jira/browse/HBASE-7219 Project: HBase Issue Type: Bug Reporter: stack Should be able to connect from a remote client to a standalone instance. HBase has 'localhost' in regionservers file and will write 'localhost' to znode for master location which remote client can't use. Fix. This comes up on mailing list w/ some frequency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7187: - Attachment: HBASE-7187-3.patch Rebased patch on trunk Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7261) batchMutate/put release the same lock twice
Jimmy Xiang created HBASE-7261: -- Summary: batchMutate/put release the same lock twice Key: HBASE-7261 URL: https://issues.apache.org/jira/browse/HBASE-7261 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Put uses batchMutate to do the actual work. Put calls startRegionOperation/closeRegionOperation. BatchMutate does the same. If the same thread already holds the lock, lock it again doesn't increase the lock count. However, releasing lock is a little different. If the lock is already released, IllegalMonitorStateException will throw if it is released again. There could be other calls. I will look into it more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Attachment: trunk-6423_v3.4.patch 0.94-6423.patch Writes should not block reads on blocking updates to memstores -- Key: HBASE-6423 URL: https://issues.apache.org/jira/browse/HBASE-6423 Project: HBase Issue Type: Bug Reporter: Karthik Ranganathan Assignee: Jimmy Xiang Attachments: 0.94-6423.patch, trunk-6423.patch, trunk-6423_v2.1.patch, trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch, trunk-6423_v3.4.patch We have a big data use case where we turn off WAL and have a ton of reads and writes. We found that: 1. flushing a memstore takes a while (GZIP compression) 2. incoming writes cause the new memstore to grow in an unbounded fashion 3. this triggers blocking memstore updates 4. in turn, this causes all the RPC handler threads to block on writes to that memstore 5. we are not able to read during this time as RPC handlers are blocked At a higher level, we should not hold up the RPC threads while blocking updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508935#comment-13508935 ] Jimmy Xiang commented on HBASE-6423: Thanks for the review. Added some minor change. Also uploaded a patch for 0.94. I will commit it tonight if no objection. Writes should not block reads on blocking updates to memstores -- Key: HBASE-6423 URL: https://issues.apache.org/jira/browse/HBASE-6423 Project: HBase Issue Type: Bug Reporter: Karthik Ranganathan Assignee: Jimmy Xiang Attachments: 0.94-6423.patch, trunk-6423.patch, trunk-6423_v2.1.patch, trunk-6423_v2.patch, trunk-6423_v3.2.patch, trunk-6423_v3.3.patch, trunk-6423_v3.4.patch We have a big data use case where we turn off WAL and have a ton of reads and writes. We found that: 1. flushing a memstore takes a while (GZIP compression) 2. incoming writes cause the new memstore to grow in an unbounded fashion 3. this triggers blocking memstore updates 4. in turn, this causes all the RPC handler threads to block on writes to that memstore 5. we are not able to read during this time as RPC handlers are blocked At a higher level, we should not hold up the RPC threads while blocking updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7236: Attachment: HBASE-7236-PROTOTYPE-v1.patch add per-table/per-cf compaction configuration via metadata -- Key: HBASE-7236 URL: https://issues.apache.org/jira/browse/HBASE-7236 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE-v1.patch Regardless of the compaction policy, it makes sense to have separate configuration for compactions for different tables and column families, as their access patterns and workloads can be different. In particular, for tiered compactions that are being ported from 0.89-fb branch it is necessary to have, to use it properly. We might want to add support for compaction configuration via metadata on table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508937#comment-13508937 ] Sergey Shelukhin commented on HBASE-7236: - bq. Regards descriptor attributes, a configuration override is just another attribute. Does it make sense to go in the other direction and fix where descriptors have metadata which are configuration overrides with custom names, meaning: rename them to the convention for Configuration? Otherwise now we have not only attributes, some of which override settings in the XML configuration, but now also configuration overrides that also do so? Do you want to store them in the attributes dictionary though? Some attributes are not config overrides (e.g. IS_ROOT/IS_META, in-memory, etc.), there are also user attributes; above problems with having the same dictionary will remain. I can move the existing attributes into config overrides instead; some backward compat might be necessary. bq. Regards CompoundConfiguration, as an API user why should I care about tagging if something I add to CompoundConfiguration is an 'override' or not. Seems any .add() should simply override values added to the configuration by a previous .add() ? Or are some overrides special that will continue to override values even if they are provided in a subsequent .add(), so some of those values in the .add() will override previous values from an earlier .add() as I would expect but there are these other values changed with an .addOverride() that I don't know about? Will an second addOverride override the previous addOverride overrides? Confusing – See? The problem here is that you cannot have .add(ListA) and .add(ListB) due to type erasure on generics. I will rename both methods to more elaborate, semantically similar names. add per-table/per-cf compaction configuration via metadata -- Key: HBASE-7236 URL: https://issues.apache.org/jira/browse/HBASE-7236 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE-v1.patch Regardless of the compaction policy, it makes sense to have separate configuration for compactions for different tables and column families, as their access patterns and workloads can be different. In particular, for tiered compactions that are being ported from 0.89-fb branch it is necessary to have, to use it properly. We might want to add support for compaction configuration via metadata on table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[ https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508939#comment-13508939 ] stack commented on HBASE-7247: -- bq. use a watcher to see if someone else is updating the znode Do we not have this currently? If someone else changes the znode, we don't notice? We only notice when we go to update the znode? Should we list how many znode updates happen for a region open? Async'ing the OPENINGs so master gets a tickle that the OPEN is progressing sounds fine... We don't actually move any files around on OPEN so its not a 'problem' if not the owner, really... Its only later after compaction, etc., that the damage is done... that is what we should prevent happening for sure. Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening Key: HBASE-7247 URL: https://issues.apache.org/jira/browse/HBASE-7247 Project: HBase Issue Type: Improvement Components: master, Region Assignment, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Critical The regionserver.OpenRegionHandler#tickleOpening updates the region znode as Do this so master doesn't timeout this region-in-transition.. However, on the usual test, this makes the assignment time of 1500 regions goes from 70s to 100s, that is, we're 50% slower because of this. More generally, ZooKeper commits to disk all the data update, and this takes time. Using it to provide a keep alive seems overkill. At the very list, it could be made asynchronous. I'm not sure how necessary these updates are required (I need to go deeper in the internal, feedback welcome), but it seems very important to optimize this... The trival fix would be to make this optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5686) Should add backup masters to AvroUtil.csToACS()
[ https://issues.apache.org/jira/browse/HBASE-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-5686: - Affects Version/s: (was: 0.96.0) Should add backup masters to AvroUtil.csToACS() --- Key: HBASE-5686 URL: https://issues.apache.org/jira/browse/HBASE-5686 Project: HBase Issue Type: Improvement Components: avro Affects Versions: 0.92.1, 0.94.0 Reporter: David S. Wang Changes similar to HBASE-5209/HBASE-5596 need to be added for Avro. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7262) Move HBaseRPC metrics to metrics2
Elliott Clark created HBASE-7262: Summary: Move HBaseRPC metrics to metrics2 Key: HBASE-7262 URL: https://issues.apache.org/jira/browse/HBASE-7262 Project: HBase Issue Type: Sub-task Components: metrics Affects Versions: 0.96.0 Reporter: Elliott Clark Priority: Blocker HBase RPC is the last thing still publishing through metrics1. We should move this into metrics2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6055) Snapshots in HBase 0.96
[ https://issues.apache.org/jira/browse/HBASE-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6055: -- Priority: Blocker (was: Major) Snapshots in HBase 0.96 --- Key: HBASE-6055 URL: https://issues.apache.org/jira/browse/HBASE-6055 Project: HBase Issue Type: New Feature Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jesse Yates Assignee: Jesse Yates Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: Snapshots in HBase.docx Continuation of HBASE-50 for the current trunk. Since the implementation has drastically changed, opening as a new ticket. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7261) batchMutate/put release the same lock twice
[ https://issues.apache.org/jira/browse/HBASE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508959#comment-13508959 ] Jimmy Xiang commented on HBASE-7261: This happens in 0.94 too. In 0.94, put and internalObtainRowLock call startRegionOperation/closeRegionOperation twice in the same thread. Something surprising me is that the IllegalMonitorStateException doesn't happen all the time. Did I miss anything? batchMutate/put release the same lock twice --- Key: HBASE-7261 URL: https://issues.apache.org/jira/browse/HBASE-7261 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Put uses batchMutate to do the actual work. Put calls startRegionOperation/closeRegionOperation. BatchMutate does the same. If the same thread already holds the lock, lock it again doesn't increase the lock count. However, releasing lock is a little different. If the lock is already released, IllegalMonitorStateException will throw if it is released again. There could be other calls. I will look into it more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6058) Use ZK 3.4 API 'multi' in bulk assignment
[ https://issues.apache.org/jira/browse/HBASE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508963#comment-13508963 ] nkeywal commented on HBASE-6058: I've tested ZK#multi on assignment in master: no real improvement actually because we’re actually spending our time in the region server. We lower the load on ZK, but it would be visible only on a large cluster. As using multi would require to use ZK 3.4, it’s not compelling enough to do the move. Note that's because: - the master part I've changed is doing asynchronous writes, faster than synchronous writes - ZooKeeper does nothing else. On a large cluster, it would be more interesting. - there is no real bulk assign in the region server (i.e. a regionserver receives 20 regions simultaneously). So we don't need multi there today. Use ZK 3.4 API 'multi' in bulk assignment - Key: HBASE-6058 URL: https://issues.apache.org/jira/browse/HBASE-6058 Project: HBase Issue Type: Improvement Components: master, Zookeeper Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We use async API today. This is already much much faster than the sync API. Still, it makes sense to use the 'multi' function: this will decrease the network zookeeper load at startup/rolling restart. On a 500 nodes cluster, we see 3 that 3 seconds are spent on updating ZK per bulk assignment. This should cut it in half (+ the benefits on the network/zk load). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6367) List backup masters in ui.
[ https://issues.apache.org/jira/browse/HBASE-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508964#comment-13508964 ] Elliott Clark commented on HBASE-6367: -- bq. I can move the backup master section under region servers. Where do you think it's more appropriate(at the very bottom)? Either at the very bottom or after region servers list. My vote would be after the region server's list. My thinking is this. Most people will be combing to this page in order to see how their cluster is doing. This data is best represented by the regionservers list. Thoughts ? bq.There might be cases where the backup master is dead, but the user was expecting it. That's fair. My thought was just that users might be confused by empty headers' for things that are optional. All users need region servers but not all users will have a backup master. But discoverability could win out. List backup masters in ui. -- Key: HBASE-6367 URL: https://issues.apache.org/jira/browse/HBASE-6367 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Jeffrey Zhong Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: Has BackupMasters.png, hbase-6367.patch, No BackupMasters.png Right now only the active master shows any information on the web ui. It would be nice to see that there are backup masters waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7263) Investigate more fine grain locking for checkAndPut/append/increment
Gregory Chanan created HBASE-7263:
-
Summary: Investigate more fine grain locking for
checkAndPut/append/increment
Key: HBASE-7263
URL: https://issues.apache.org/jira/browse/HBASE-7263
Project: HBase
Issue Type: Improvement
Components: Transactions/MVCC
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
HBASE-7051 lists 3 options for fixing an ACID-violation wrt checkAndPut:
{quote}
1) Waiting for the MVCC to advance for read/updates: the downside is that you
have to wait for updates on other rows.
2) Have an MVCC per-row (table configuration): this avoids the unnecessary
contention of 1)
3) Transform the read/updates to write-only with rollup on read.. E.g. an
increment would just have the number of values to increment.
{quote}
HBASE-7051 and HBASE-4583 implement option #1. The downside, as mentioned, is
that you have to wait for updates on other rows, since MVCC is per-row.
Another option occurred to me that I think is worth investigating: rely on a
row-level read/write lock rather than MVCC.
Here is pseudo-code for what exists today for read/updates like checkAndPut
{code}
(1) Acquire RowLock
(1a) BeginMVCC + Finish MVCC
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
{code}
Write-only operations (e.g. puts) are the same, just without step 1a.
Now, consider the following instead:
{code}
(1) Acquire RowLock
(1a) Grab+Release RowWriteLock (instead of BeginMVCC + Finish MVCC)
(1b) Grab RowReadLock (new step!)
(2) Begin MVCC
(3) Do work
(4) Release RowLock
(5) Append to WAL
(6) Finish MVCC
(7) Release RowReadLock (new step!)
{code}
As before, write-only operations are the same, just without step 1a.
The difference here is that writes grab a row-level read lock and hold it until
the MVCC is completed. The nice property that this gives you is that
read/updates can tell when the MVCC is done on a per-row basis, because they
can just try to acquire the write-lock which will block until the MVCC is
competed for that row in step 7.
There is overhead for acquiring the read lock that I need to measure, but it
should be small, since there will never be any blocking on acquiring the
row-level read lock. This is because the read lock can only block if someone
else holds the write lock, but both the write and read lock are only acquired
under the row lock.
I ran a quick test of this approach over a region (this directly interacts with
HRegion, so no client effects):
- 30 threads
- 5000 increments per thread
- 30 columns per increment
- Each increment uniformly distributed over 500,000 rows
- 5 trials
Better-Than-Theoretical-Max: (No locking or MVCC on step 1a): 10362.2 ms
Today: 13950 ms
The locking approach: 10877 ms
So it looks like an improvement, at least wrt increment. As mentioned, I need
to measure the overhead of acquiring the read lock for puts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508966#comment-13508966 ] Elliott Clark commented on HBASE-1299: -- +1 JSPs don't HTML escape literals (ie: table names, region names, start end keys) - Key: HBASE-1299 URL: https://issues.apache.org/jira/browse/HBASE-1299 Project: HBase Issue Type: Bug Affects Versions: 0.19.0, 0.19.1 Reporter: Hoss Man Assignee: Nick Dimiduk Attachments: 1299.patch similar to HBASE-1298, the various JSPs included with HBase for monitoring the system don't seem to do any HTML escaping when displaying user entered data which may contain special characters: table names, region names, start Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6055) Offline Snapshots in HBase 0.96
[ https://issues.apache.org/jira/browse/HBASE-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-6055: --- Summary: Offline Snapshots in HBase 0.96 (was: Snapshots in HBase 0.96) Offline Snapshots in HBase 0.96 --- Key: HBASE-6055 URL: https://issues.apache.org/jira/browse/HBASE-6055 Project: HBase Issue Type: New Feature Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jesse Yates Assignee: Jesse Yates Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: Snapshots in HBase.docx Continuation of HBASE-50 for the current trunk. Since the implementation has drastically changed, opening as a new ticket. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-1299: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk (tried to commit to 0.94 but failed... so passing for now). Thanks for the patch Nick. JSPs don't HTML escape literals (ie: table names, region names, start end keys) - Key: HBASE-1299 URL: https://issues.apache.org/jira/browse/HBASE-1299 Project: HBase Issue Type: Bug Affects Versions: 0.19.0, 0.19.1 Reporter: Hoss Man Assignee: Nick Dimiduk Fix For: 0.96.0 Attachments: 1299.patch similar to HBASE-1298, the various JSPs included with HBase for monitoring the system don't seem to do any HTML escaping when displaying user entered data which may contain special characters: table names, region names, start Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6055) Offline Snapshots in HBase 0.96
[ https://issues.apache.org/jira/browse/HBASE-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-6055: --- Attachment: Offline Snapshots.docx Attaching doc that covers a range of information on how offline snapshots and recovery work. It starts out talking a bit about the high-level of each feature, and then does a walk-through of offline snapshots, restore, clone and export. This doc describes out general FS layouts, ordering of events, process ownership, and pending concerns. Offline Snapshots in HBase 0.96 --- Key: HBASE-6055 URL: https://issues.apache.org/jira/browse/HBASE-6055 Project: HBase Issue Type: New Feature Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jesse Yates Assignee: Jesse Yates Priority: Blocker Fix For: hbase-6055, 0.96.0 Attachments: Offline Snapshots.docx, Snapshots in HBase.docx Continuation of HBASE-50 for the current trunk. Since the implementation has drastically changed, opening as a new ticket. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Remove Writable Interface from KeyValue
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508973#comment-13508973 ] Matt Corgan commented on HBASE-7233: Not sure I follow everything so far, but I'm wondering if KeyValue should just keep the Writable interface since KeyValue is the unit of input/output in certain map-reduce jobs. The Cell interface improves on KeyValue when you are passing around blobs of many Cells (since they can share common row-prefixes, etc), but for map-reduce we are passing around individual Cells, so might as well just keep using KeyValue. The Cells need to be standalone, so KeyValue may be required. Are there benefits to removing Writable for this particular class beyond cleaning up the code? Maybe saving 4-8 bytes memory per KV in the memstore. Remove Writable Interface from KeyValue --- Key: HBASE-7233 URL: https://issues.apache.org/jira/browse/HBASE-7233 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Priority: Blocker Attachments: 7233.txt, 7233-v2.txt Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508974#comment-13508974 ] Shawn Quinn commented on HBASE-3996: So, this is something we'd really start liking to use here as well, as we're trying to stay within the released HBase APIs (so, we're currently using a custom TableInputFormatBase extension which hasn't been ideal.) Based on the comments here and the references to this ticket in the mailing list, it appears there's a good amount of interest in this enhancement. I've monkeyed with a few things within the HBase code locally here, but haven't yet tried to submit a patch. Lars/Stack, if you let me know you wouldn't mind another person's contribution being added to the mix here, I'd be glad to give this one a go and submit an updated patch. I don't want cause you guys any headaches if adding another person into the mix is just going to complicate or slow this one down though. Support multiple tables and scanners as input to the mapper in map/reduce jobs -- Key: HBASE-3996 URL: https://issues.apache.org/jira/browse/HBASE-3996 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Eran Kutner Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.4 Attachments: 3996-v2.txt, 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, HBase-3996.patch It seems that in many cases feeding data from multiple tables or multiple scanners on a single table can save a lot of time when running map/reduce jobs. I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7264) Improve Snappy installation documentation
Jean-Marc Spaggiari created HBASE-7264: -- Summary: Improve Snappy installation documentation Key: HBASE-7264 URL: https://issues.apache.org/jira/browse/HBASE-7264 Project: HBase Issue Type: Bug Components: documentation Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-7264.patch Snappy installation process is lacking some details. I tried to give some. There is also some mistakes it's vs its. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7264) Improve Snappy installation documentation
[ https://issues.apache.org/jira/browse/HBASE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7264: --- Status: Patch Available (was: Open) Adding more details in the documentation Fixing it's vs its mistakes. Improve Snappy installation documentation - Key: HBASE-7264 URL: https://issues.apache.org/jira/browse/HBASE-7264 Project: HBase Issue Type: Bug Components: documentation Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Labels: documentation Attachments: HBASE-7264.patch Original Estimate: 1h Remaining Estimate: 1h Snappy installation process is lacking some details. I tried to give some. There is also some mistakes it's vs its. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7264) Improve Snappy installation documentation
[ https://issues.apache.org/jira/browse/HBASE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7264: --- Attachment: HBASE-7264.patch Improve Snappy installation documentation - Key: HBASE-7264 URL: https://issues.apache.org/jira/browse/HBASE-7264 Project: HBase Issue Type: Bug Components: documentation Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Labels: documentation Attachments: HBASE-7264.patch Original Estimate: 1h Remaining Estimate: 1h Snappy installation process is lacking some details. I tried to give some. There is also some mistakes it's vs its. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7262) Move HBaseRPC metrics to metrics2
[ https://issues.apache.org/jira/browse/HBASE-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HBASE-7262: Assignee: Elliott Clark Move HBaseRPC metrics to metrics2 - Key: HBASE-7262 URL: https://issues.apache.org/jira/browse/HBASE-7262 Project: HBase Issue Type: Sub-task Components: metrics Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Blocker Fix For: 0.96.0 HBase RPC is the last thing still publishing through metrics1. We should move this into metrics2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508990#comment-13508990 ] Nick Dimiduk commented on HBASE-1299: - Fails 0.94 because these files moved on trunk. `git checkout 0.94 cd src/main/resources/hbase-webapps/master patch` 1299.patch may work. JSPs don't HTML escape literals (ie: table names, region names, start end keys) - Key: HBASE-1299 URL: https://issues.apache.org/jira/browse/HBASE-1299 Project: HBase Issue Type: Bug Affects Versions: 0.19.0, 0.19.1 Reporter: Hoss Man Assignee: Nick Dimiduk Fix For: 0.96.0 Attachments: 1299.patch similar to HBASE-1298, the various JSPs included with HBase for monitoring the system don't seem to do any HTML escaping when displaying user entered data which may contain special characters: table names, region names, start Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6302) Document how to run integration tests
[ https://issues.apache.org/jira/browse/HBASE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-6302: - Resolution: Fixed Status: Resolved (was: Patch Available) Resolving the issue. Stack, I could not find info about how do we update the site. Document how to run integration tests - Key: HBASE-6302 URL: https://issues.apache.org/jira/browse/HBASE-6302 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: Enis Soztutar Priority: Blocker Fix For: 0.96.0 Attachments: 6302v3.txt, HBASE-6302_v1.patch, hbase-6302_v2.patch HBASE-6203 has attached the old IT doc with some mods. When we figure how ITs are to be run, update it and apply the documentation under this issue. Making a blocker against 0.96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[
https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508995#comment-13508995
]
Hadoop QA commented on HBASE-7187:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555798/HBASE-7187-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 26 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.master.TestMasterFailover
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3436//console
This message is automatically generated.
Create empty hbase-client module
Key: HBASE-7187
URL: https://issues.apache.org/jira/browse/HBASE-7187
Project: HBase
Issue Type: Sub-task
Components: Client
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 0.96.0
Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch,
HBASE-7187-2.patch, HBASE-7187-3.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7250) create integration test for balancing regions and killing region servers - 2
[ https://issues.apache.org/jira/browse/HBASE-7250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508996#comment-13508996 ] Sergey Shelukhin commented on HBASE-7250: - renamed the action; wrt SecureRandom - are you sure it's necessary? Nothing particularly security-sensitive about ChaosMonkey. Or is it a general guideline to use it? On minicluster, the test fails, I only ran it on full cluster before... I will see why it fails, looks like clients aren't syncing meta properly. create integration test for balancing regions and killing region servers - 2 Key: HBASE-7250 URL: https://issues.apache.org/jira/browse/HBASE-7250 Project: HBase Issue Type: Improvement Components: test Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Attachments: HBASE-7250-v0.patch The original test is too general; need another one that would be more targeted and would test master logic in particular (e.g. not kill master). I re-discovered HBASE-6060 using it on the first run :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508999#comment-13508999 ] Enis Soztutar commented on HBASE-7213: -- bq. and in case folks feel that ROOT should be handled as well, it probably can be done in a follow up I think JD has updated the patch for removing ROOT in favor of ZK. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7245) Recovery on failed snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509005#comment-13509005 ] Matteo Bertozzi commented on HBASE-7245: If we use ZooKeeper what happens when the user erase the zk content before starting the master? we still have the bad state on disk and meta but no hint on the unfinished operation. If I understood correctly we are using zookeeper as ephimeral only (except for replication) do we want to move to something that rely fully on zookeeper? Recovery on failed snapshot restore --- Key: HBASE-7245 URL: https://issues.apache.org/jira/browse/HBASE-7245 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Restore will do updates to the file system and to meta. it seems that an inopportune failure before meta is completely updated could result in an inconsistent state that would require hbck to fix. We should define what the semantics are for recovering from this. Some suggestions: 1) Fail Forward (see some log saying restore's meta edits not completed, then gather information necessary to build it all from fs, and complete meta edits.). 2) Fail backwards (see some log saying restore's meta edits not completed, delete incomplete snapshot region entries from meta.) I think I prefer 1 -- if two processes end somehow updating (somehow the original master didn't die, and a new one started up) they would be idempotent. If we used 2, we could still have a race and still be in a bad place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7253: --- Attachment: HBASE-7253-v1.patch v1 patch with fixes to missing options explanation. Compaction Tool --- Key: HBASE-7253 URL: https://issues.apache.org/jira/browse/HBASE-7253 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch In HBASE-5616, as part of the compaction code refactor, a CompactionTool was added. but there are some issues: * The tool is under test/ * mockito is required, so the test scope should be removed from the pom.xml, otherwise the tool doesn't start * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some code (Store) uses HRegion.regionInfo directly HStore.java#L2021, HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. * The Mocked Store uses a dummy family and the compacted files doesn't get the same family properties specified (compression, encoding, ...) * at the end of compaction CompactionTool.java#L155, on by default, the compaction file is removed (note that the compacted one are already removed inside the store.compact()... and you end up with an empty dir, if you compact everything. I've fixed some stuff and added support to: * Run the compaction as a MR Job * Specify a Table (compact each region/family) * Specify a Region (compact each family) * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509010#comment-13509010 ] stack commented on HBASE-7187: -- htrace is hard dependency of hbase-client? I suppose it makes sense. We skip client tests? The removals from hbase-common look like duh! ones... thanks for taking them out. +1 on commit Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7245) Recovery on failed snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509013#comment-13509013 ] Ted Yu commented on HBASE-7245: --- The discussion from HBASE-6721 is related in this regard. Francis started with storing group information on hdfs. Later he switched to storage in table. Whether storing in zookeeper is under review. I am fine with storing operation directive on hdfs. Recovery on failed snapshot restore --- Key: HBASE-7245 URL: https://issues.apache.org/jira/browse/HBASE-7245 Project: HBase Issue Type: Sub-task Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jonathan Hsieh Assignee: Matteo Bertozzi Fix For: hbase-6055, 0.96.0 Restore will do updates to the file system and to meta. it seems that an inopportune failure before meta is completely updated could result in an inconsistent state that would require hbck to fix. We should define what the semantics are for recovering from this. Some suggestions: 1) Fail Forward (see some log saying restore's meta edits not completed, then gather information necessary to build it all from fs, and complete meta edits.). 2) Fail backwards (see some log saying restore's meta edits not completed, delete incomplete snapshot region entries from meta.) I think I prefer 1 -- if two processes end somehow updating (somehow the original master didn't die, and a new one started up) they would be idempotent. If we used 2, we could still have a race and still be in a bad place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7253: - Resolution: Fixed Release Note: Tool to run compactions external to hbase: Usage: java + this.getClass().getName() + [-compactOnce] [-mapred] [-Dproperty=value]* files... Options: mapred Use MapReduce to run compaction. compactOnceExecute just one compaction step. (default: while needed) Note: -D properties will be applied to the conf used. For example: To preserve input files, pass -D+CONF_COMPLETE_COMPACTION+=false To stop delete of compacted file, pass -D+CONF_DELETE_COMPACTED+=false To set tmp dir, pass -D+CONF_TMP_DIR+=ALTERNATE_DIR Examples: To compact the full 'TestTable' using MapReduce: $ bin/hbase + this.getClass().getName() + -mapred hdfs:///hbase/TestTable To compact column family 'x' of the table 'TestTable' region 'abc': $ bin/hbase + this.getClass().getName() + hdfs:///hbase/TestTable/abc/x Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Compaction Tool --- Key: HBASE-7253 URL: https://issues.apache.org/jira/browse/HBASE-7253 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch In HBASE-5616, as part of the compaction code refactor, a CompactionTool was added. but there are some issues: * The tool is under test/ * mockito is required, so the test scope should be removed from the pom.xml, otherwise the tool doesn't start * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some code (Store) uses HRegion.regionInfo directly HStore.java#L2021, HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. * The Mocked Store uses a dummy family and the compacted files doesn't get the same family properties specified (compression, encoding, ...) * at the end of compaction CompactionTool.java#L155, on by default, the compaction file is removed (note that the compacted one are already removed inside the store.compact()... and you end up with an empty dir, if you compact everything. I've fixed some stuff and added support to: * Run the compaction as a MR Job * Specify a Table (compact each region/family) * Specify a Region (compact each family) * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509024#comment-13509024 ] stack commented on HBASE-7253: -- Committed to trunk. Thanks for the patch Matteo. Make new issue for 0.94 since it'll be a little different. Good stuff. Compaction Tool --- Key: HBASE-7253 URL: https://issues.apache.org/jira/browse/HBASE-7253 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch In HBASE-5616, as part of the compaction code refactor, a CompactionTool was added. but there are some issues: * The tool is under test/ * mockito is required, so the test scope should be removed from the pom.xml, otherwise the tool doesn't start * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some code (Store) uses HRegion.regionInfo directly HStore.java#L2021, HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. * The Mocked Store uses a dummy family and the compacted files doesn't get the same family properties specified (compression, encoding, ...) * at the end of compaction CompactionTool.java#L155, on by default, the compaction file is removed (note that the compacted one are already removed inside the store.compact()... and you end up with an empty dir, if you compact everything. I've fixed some stuff and added support to: * Run the compaction as a MR Job * Specify a Table (compact each region/family) * Specify a Region (compact each family) * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[
https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509023#comment-13509023
]
Elliott Clark commented on HBASE-7187:
--
There are two skip things in the pom:
The first is that maven doesn't run any second part tests. That's because all
second part tests need a server and the client module doesn't depend upon the
server.
The second is so that a developer can run something like:
{code}mvn clean test -Dskip-client-tests{code}
and that will run test but won't test the client. This is useful while you
know a test is failing or you just want to skip to the module's tests that you
are debugging.
Create empty hbase-client module
Key: HBASE-7187
URL: https://issues.apache.org/jira/browse/HBASE-7187
Project: HBase
Issue Type: Sub-task
Components: Client
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 0.96.0
Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch,
HBASE-7187-2.patch, HBASE-7187-3.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7253: --- Release Note: The CompactionTool works at file-system level, so the table should be disabled. The compaction process uses the same hbase-site.xml configuration property used by the server, like hbase.hstore.compactionThreshold co. You can compact the whole table or just a single region or family, and the input of the CompactionTool is a fs path. You can run the compaction as a MapReduce Job, or as a local process. Each family can be compacted in parallel if you use the -mapreduce option. To compact TestTable family cf1 of region e450da04b1a10099b618bec031e0f951 bin/hbase org.apache.hadoop.hbase.regionserver.CompactionTool hdfs:///hbase/TestTable/e450da04b1a10099b618bec031e0f951/cf1 To compact all the families of region e450da04b1a10099b618bec031e0f951: bin/hbase org.apache.hadoop.hbase.regionserver.CompactionTool hdfs:///hbase/TestTable/e450da04b1a10099b618bec031e0f951 To compact all regions and family of the Table: bin/hbase org.apache.hadoop.hbase.regionserver.CompactionTool -mapred hdfs:///hbase/TestTable was: Tool to run compactions external to hbase: Usage: java + this.getClass().getName() + [-compactOnce] [-mapred] [-Dproperty=value]* files... Options: mapred Use MapReduce to run compaction. compactOnceExecute just one compaction step. (default: while needed) Note: -D properties will be applied to the conf used. For example: To preserve input files, pass -D+CONF_COMPLETE_COMPACTION+=false To stop delete of compacted file, pass -D+CONF_DELETE_COMPACTED+=false To set tmp dir, pass -D+CONF_TMP_DIR+=ALTERNATE_DIR Examples: To compact the full 'TestTable' using MapReduce: $ bin/hbase + this.getClass().getName() + -mapred hdfs:///hbase/TestTable To compact column family 'x' of the table 'TestTable' region 'abc': $ bin/hbase + this.getClass().getName() + hdfs:///hbase/TestTable/abc/x Hadoop Flags: (was: Reviewed) Compaction Tool --- Key: HBASE-7253 URL: https://issues.apache.org/jira/browse/HBASE-7253 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch In HBASE-5616, as part of the compaction code refactor, a CompactionTool was added. but there are some issues: * The tool is under test/ * mockito is required, so the test scope should be removed from the pom.xml, otherwise the tool doesn't start * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some code (Store) uses HRegion.regionInfo directly HStore.java#L2021, HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. * The Mocked Store uses a dummy family and the compacted files doesn't get the same family properties specified (compression, encoding, ...) * at the end of compaction CompactionTool.java#L155, on by default, the compaction file is removed (note that the compacted one are already removed inside the store.compact()... and you end up with an empty dir, if you compact everything. I've fixed some stuff and added support to: * Run the compaction as a MR Job * Specify a Table (compact each region/family) * Specify a Region (compact each family) * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1299) JSPs don't HTML escape literals (ie: table names, region names, start end keys)
[ https://issues.apache.org/jira/browse/HBASE-1299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509025#comment-13509025 ] Hudson commented on HBASE-1299: --- Integrated in HBase-TRUNK #3587 (See [https://builds.apache.org/job/HBase-TRUNK/3587/]) HBASE-1299 JSPs don't HTML escape literals (ie: table names, region names, start end keys) (Revision 1416645) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/master/table.jsp * /hbase/trunk/hbase-server/src/main/resources/hbase-webapps/master/tablesDetailed.jsp JSPs don't HTML escape literals (ie: table names, region names, start end keys) - Key: HBASE-1299 URL: https://issues.apache.org/jira/browse/HBASE-1299 Project: HBase Issue Type: Bug Affects Versions: 0.19.0, 0.19.1 Reporter: Hoss Man Assignee: Nick Dimiduk Fix For: 0.96.0 Attachments: 1299.patch similar to HBASE-1298, the various JSPs included with HBase for monitoring the system don't seem to do any HTML escaping when displaying user entered data which may contain special characters: table names, region names, start Keys, or end Keys -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4791) Allow Secure Zookeeper JAAS configuration to be programmatically set (rather than only by reading JAAS configuration file)
[ https://issues.apache.org/jira/browse/HBASE-4791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-4791: --- Attachment: HBASE-4791-v3.patch Now that ZOOKEEPER-1437 I think that we're ready to go with this one. I've rebased the patch and fixed some comments. Allow Secure Zookeeper JAAS configuration to be programmatically set (rather than only by reading JAAS configuration file) -- Key: HBASE-4791 URL: https://issues.apache.org/jira/browse/HBASE-4791 Project: HBase Issue Type: Improvement Components: security, Zookeeper Reporter: Eugene Koontz Assignee: Matteo Bertozzi Labels: security, zookeeper Attachments: DemoConfig.java, HBASE-4791-v1.patch, HBASE-4791-v2.patch, HBASE-4791-v3.patch In the currently proposed fix for HBASE-2418, there must be a JAAS file specified in System.setProperty(java.security.auth.login.config). However, it might be preferable to construct a JAAS configuration programmatically, as is done with secure Hadoop (see https://github.com/apache/hadoop-common/blob/a48eceb62c9b5c1a5d71ee2945d9eea2ed62527b/src/java/org/apache/hadoop/security/UserGroupInformation.java#L175). This would have the benefit of avoiding a usage of a system property setting, and allow instead an HBase-local configuration setting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7264) Improve Snappy installation documentation
[
https://issues.apache.org/jira/browse/HBASE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509034#comment-13509034
]
Hadoop QA commented on HBASE-7264:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12555814/HBASE-7264.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 26 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestDrainingServer
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3437//console
This message is automatically generated.
Improve Snappy installation documentation
-
Key: HBASE-7264
URL: https://issues.apache.org/jira/browse/HBASE-7264
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Jean-Marc Spaggiari
Assignee: Jean-Marc Spaggiari
Priority: Minor
Labels: documentation
Attachments: HBASE-7264.patch
Original Estimate: 1h
Remaining Estimate: 1h
Snappy installation process is lacking some details. I tried to give some.
There is also some mistakes it's vs its.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-7213: --- Attachment: 7213-in-progress.2.2.patch Addresses most of the comments.. One thing I didn't change is bq. The check for ROOT region above is for assigning ROOT .. In HBASE-3171, there is work underway to remove -ROOT- .. So not much gain here to address things to do with how the root region is handled. I have left it as such (with a known and a very corner case that if the server holding the root's region crashes before the region's edits are persisted, the assignment of the region might be an issue). Manual tests seemed to indicate things are working well. I'll start the full set of unit tests now. Have HLog files for .META. edits only - Key: HBASE-7213 URL: https://issues.apache.org/jira/browse/HBASE-7213 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Attachments: 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch Over on HBASE-6774, there is a discussion on separating out the edits for .META. regions from the other regions' edits w.r.t where the edits are written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509049#comment-13509049 ] Hudson commented on HBASE-7187: --- Integrated in HBase-TRUNK #3588 (See [https://builds.apache.org/job/HBase-TRUNK/3588/]) HBASE-7187 Create empty hbase-client module (Revision 1416664) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterListTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/trunk/pom.xml Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509050#comment-13509050 ] Hudson commented on HBASE-7253: --- Integrated in HBase-TRUNK #3588 (See [https://builds.apache.org/job/HBase-TRUNK/3588/]) HBASE-7253 CompactionTool (Revision 1416657) Result = FAILURE stack : Files : * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/mapreduce * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/mapreduce/JobUtil.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/mapreduce * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/mapreduce/JobUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java Compaction Tool --- Key: HBASE-7253 URL: https://issues.apache.org/jira/browse/HBASE-7253 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch In HBASE-5616, as part of the compaction code refactor, a CompactionTool was added. but there are some issues: * The tool is under test/ * mockito is required, so the test scope should be removed from the pom.xml, otherwise the tool doesn't start * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some code (Store) uses HRegion.regionInfo directly HStore.java#L2021, HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. * The Mocked Store uses a dummy family and the compacted files doesn't get the same family properties specified (compression, encoding, ...) * at the end of compaction CompactionTool.java#L155, on by default, the compaction file is removed (note that the compacted one are already removed inside the store.compact()... and you end up with an empty dir, if you compact everything. I've fixed some stuff and added support to: * Run the compaction as a MR Job * Specify a Table (compact each region/family) * Specify a Region (compact each family) * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7264) Improve Snappy installation documentation
[ https://issues.apache.org/jira/browse/HBASE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7264: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch J-M. Applied. Improve Snappy installation documentation - Key: HBASE-7264 URL: https://issues.apache.org/jira/browse/HBASE-7264 Project: HBase Issue Type: Bug Components: documentation Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Labels: documentation Fix For: 0.96.0 Attachments: HBASE-7264.patch Original Estimate: 1h Remaining Estimate: 1h Snappy installation process is lacking some details. I tried to give some. There is also some mistakes it's vs its. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509056#comment-13509056 ] stack commented on HBASE-7253: -- [~mbertozzi] Thanks for fixing up the release note. Compaction Tool --- Key: HBASE-7253 URL: https://issues.apache.org/jira/browse/HBASE-7253 Project: HBase Issue Type: New Feature Components: Compaction Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch In HBASE-5616, as part of the compaction code refactor, a CompactionTool was added. but there are some issues: * The tool is under test/ * mockito is required, so the test scope should be removed from the pom.xml, otherwise the tool doesn't start * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some code (Store) uses HRegion.regionInfo directly HStore.java#L2021, HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. * The Mocked Store uses a dummy family and the compacted files doesn't get the same family properties specified (compression, encoding, ...) * at the end of compaction CompactionTool.java#L155, on by default, the compaction file is removed (note that the compacted one are already removed inside the store.compact()... and you end up with an empty dir, if you compact everything. I've fixed some stuff and added support to: * Run the compaction as a MR Job * Specify a Table (compact each region/family) * Specify a Region (compact each family) * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509059#comment-13509059 ] stack commented on HBASE-7187: -- It should be -DskipClientTests rather than skip-client-tests? We have -DskipTests and -DskipJavadocs currently. If you want to be like the others Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6302) Document how to run integration tests
[ https://issues.apache.org/jira/browse/HBASE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509065#comment-13509065 ] stack commented on HBASE-6302: -- [~enis] See http://hbase.apache.org/book.html#hbase.org Let me know if doesn't make sense. I just pushed out doc so your fancy integration test stuff should show on the site soon. Document how to run integration tests - Key: HBASE-6302 URL: https://issues.apache.org/jira/browse/HBASE-6302 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: Enis Soztutar Priority: Blocker Fix For: 0.96.0 Attachments: 6302v3.txt, HBASE-6302_v1.patch, hbase-6302_v2.patch HBASE-6203 has attached the old IT doc with some mods. When we figure how ITs are to be run, update it and apply the documentation under this issue. Making a blocker against 0.96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509067#comment-13509067 ] Elliott Clark commented on HBASE-7187: -- The rest of the modules are -Dskip-module name-tests. We can change them though if you want. Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6302) Document how to run integration tests
[ https://issues.apache.org/jira/browse/HBASE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509069#comment-13509069 ] Enis Soztutar commented on HBASE-6302: -- Cool. Thanks Stack. I had checked the book, but somehow I missed it apparently. My b. Document how to run integration tests - Key: HBASE-6302 URL: https://issues.apache.org/jira/browse/HBASE-6302 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: Enis Soztutar Priority: Blocker Fix For: 0.96.0 Attachments: 6302v3.txt, HBASE-6302_v1.patch, hbase-6302_v2.patch HBASE-6203 has attached the old IT doc with some mods. When we figure how ITs are to be run, update it and apply the documentation under this issue. Making a blocker against 0.96. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509070#comment-13509070 ] stack commented on HBASE-7187: -- If not hard, would say lets go for consistency.. skipCommonTests, skipServerTests Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7256) Quick Start Guide shows stable version as 0.95, in the stable folder it is 0.94
[ https://issues.apache.org/jira/browse/HBASE-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-7256. -- Resolution: Fixed Fix Version/s: 0.96.0 Assignee: stack I changed the doc. Thanks for reporting the issue Sumod. Quick Start Guide shows stable version as 0.95, in the stable folder it is 0.94 --- Key: HBASE-7256 URL: https://issues.apache.org/jira/browse/HBASE-7256 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.94.2 Environment: HBase Repositories Reporter: Sumod Pawgi Assignee: stack Priority: Minor Fix For: 0.96.0 In the Quick Start Guide for HBase - http://hbase.apache.org/book/quickstart.html The stable version is mentioned as - 0.95 in the line - Choose a download site from this list of Apache Download Mirrors. Click on the suggested top link. This will take you to a mirror of HBase Releases. Click on the folder named stable and then download the file that ends in .tar.gz to your local filesystem; e.g. hbase-0.95-SNAPSHOT.tar.gz. But in the download folder at - http://apache.techartifact.com/mirror/hbase/stable/ The version that can be found is - hbase-0.94.2-security.tar.gz i.e. 0.94 So either the documentation or the download folder needs to be updated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7265) Make Maven skip module test properties consistent
Elliott Clark created HBASE-7265: Summary: Make Maven skip module test properties consistent Key: HBASE-7265 URL: https://issues.apache.org/jira/browse/HBASE-7265 Project: HBase Issue Type: Bug Components: build Reporter: Elliott Clark Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509077#comment-13509077 ] Elliott Clark commented on HBASE-7187: -- Filed HBASE-7265. Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509078#comment-13509078 ] Enis Soztutar commented on HBASE-7249: -- Patch looks good. Two minors: - It's better to use -test rather than -tests, to be consistent with maven test -Dtest=pattern - Can you add a 1 sentence doc about -test to docs add test name filter to IntegrationTestsDriver -- Key: HBASE-7249 URL: https://issues.apache.org/jira/browse/HBASE-7249 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3, 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Fix For: 0.94.3, 0.96.0 Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7251) Avoid flood logs during client disconnect during batch get operation
[
https://issues.apache.org/jira/browse/HBASE-7251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7251:
-
Fix Version/s: (was: 0.94.3)
0.94.4
Avoid flood logs during client disconnect during batch get operation
Key: HBASE-7251
URL: https://issues.apache.org/jira/browse/HBASE-7251
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.2
Reporter: Fengdong Yu
Labels: patch
Fix For: 0.94.4
Attachments: HBASE-7251.patch
Background:
A smart guy in the company want to read data from the HBASE in batch, the
code like the following:(just demonstrate, not runnable):
ListGet gets = new ArrayListGet();
for(int i =0; i n; ++i){
gets.add(some row key here);
if (i % 1 == 0){
Results[] results = htable.get(gets);
//process results here. so delete some code
}
}
Yes, you know that, this guy forgot gets.clear() after each htable.get()
operation in his code.
One region server becomes very slow, and crashed after 30mins becauseof OOM,
but we got 15GB log file.
there are flood logs as following:
ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
multi(org.apache.hadoop.hbase.client.MultiAction@49540d8d), rpc version=1,
client version=29, methodsFingerPrint=-56040613 from 10.1.1.1:57933 after
3980 ms, since caller disconnected
at
org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3468)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3425)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3449)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4198)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4171)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1993)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3410)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1409)
Fix:
Server is stop get but cannot exit from the for loop, so write flood logs
here.
My patch just log one exception log instead of flood logs.
Importantly, server stop processing immediately if client timeout or
disconnect.
Test:
I used this guy's wrong code read data( NO gets.clear() ) from the HBASE,
when it becomes very slow to get results, I pressed ctrl+C, then there is
only ONE CallerDisconnectedException exception log and the server stop
reading immediately, LOG generate the last log entry:
WARN org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020 caught
a ClosedChannelException, this means that the server was processing a request
but the client went away. The error message was: null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7259) Deadlock in HBaseClient when KeeperException occured
[
https://issues.apache.org/jira/browse/HBASE-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-7259:
-
Fix Version/s: 0.94.4
Deadlock in HBaseClient when KeeperException occured
Key: HBASE-7259
URL: https://issues.apache.org/jira/browse/HBASE-7259
Project: HBase
Issue Type: Bug
Components: Zookeeper
Affects Versions: 0.94.0, 0.94.1, 0.94.2
Reporter: liwei
Priority: Critical
Fix For: 0.94.4
Attachments: HConnectionManager.patch
HBaseClient was running after a period of time, all of get operation became
too slow.
From the client logs I could see the following:
1. Unable to get data of znode /hbase/root-region-server
{code}
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156)
at
org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:174)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:48)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:126)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:359)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:123)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:99)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.prefetchRegionCache(HConnectionManager.java:894)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:948)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionLocation(HConnectionManager.java:725)
at
org.apache.hadoop.hbase.client.ServerCallable.connect(ServerCallable.java:82)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:162)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:685)
at
org.apache.hadoop.hbase.client.HTablePool$PooledHTable.get(HTablePool.java:366)
{code}
2. Catalina.out found one Java-level deadlock:
{code}
=
catalina-exec-800:
waiting to lock monitor 0x5f1f6530 (object 0x000731902200, a
java.lang.Object),
which is held by catalina-exec-710
catalina-exec-710:
waiting to lock monitor 0x2aaab9a05bd0 (object 0x0007321f8708, a
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation),
which is held by catalina-exec-29-EventThread
catalina-exec-29-EventThread:
waiting to lock monitor 0x5f9f0af0 (object 0x000732a9c7e0, a
org.apache.hadoop.hbase.zookeeper.RootRegionTracker),
which is held by catalina-exec-710
Java stack information for the threads listed above:
===
catalina-exec-800:
[jira] [Updated] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7249: - Fix Version/s: (was: 0.94.3) 0.94.4 add test name filter to IntegrationTestsDriver -- Key: HBASE-7249 URL: https://issues.apache.org/jira/browse/HBASE-7249 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.3, 0.96.0 Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor Fix For: 0.96.0, 0.94.4 Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7204: - Fix Version/s: (was: 0.94.3) 0.94.4 Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.96.0, 0.94.4 Attachments: 0.94-7204.patch, trunk-7204.addendum, trunk-7204.patch, trunk-7204_v2.1.patch, trunk-7204_v2.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7190) Add an option to hbck to check only meta and assignment
[ https://issues.apache.org/jira/browse/HBASE-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7190: - Fix Version/s: (was: 0.94.3) 0.94.4 Add an option to hbck to check only meta and assignment --- Key: HBASE-7190 URL: https://issues.apache.org/jira/browse/HBASE-7190 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0, 0.94.4 Attachments: trunk-7190.patch Currently, hbck loads region info from HDFS for each run. It may take some time if there are many regions. We need an option to not check HDFS, i.e. just checking meta and assignment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7008) Set scanner caching to a better default
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509090#comment-13509090 ] stack commented on HBASE-7008: -- [~lhofhansl] This is good to commit still? Has +1s and nice release note but not committed. Want me to do it? Set scanner caching to a better default --- Key: HBASE-7008 URL: https://issues.apache.org/jira/browse/HBASE-7008 Project: HBase Issue Type: Bug Components: Client Reporter: liang xie Assignee: liang xie Fix For: 0.96.0 Attachments: 7008-0.94.txt, 7008-0.94-v2.txt, 7008-0.94-v3.txt, 7008-trunk-v5.txt, 7008-trunk-v6.txt, 7008-v3.txt, 7008-v4.txt, HBASE-7008.patch, HBASE-7008-v2.patch per http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253Fsubj=Set+scanner+caching+to+a+better+default+ let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6944) Enhance test-patch.sh to run against both jdk 1.6 and jdk 1.7
[ https://issues.apache.org/jira/browse/HBASE-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509093#comment-13509093 ] stack commented on HBASE-6944: -- Should it just do 1.7 Ted? Seems a bit much doing 1.6 and 1.7 given little real diff between the two? Enhance test-patch.sh to run against both jdk 1.6 and jdk 1.7 - Key: HBASE-6944 URL: https://issues.apache.org/jira/browse/HBASE-6944 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6944.txt Currently test-patch.sh only runs against jdk 1.6 However trunk build is using jdk 1.7 test-patch.sh should be enhanced to run against both jdk versions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6944) Enhance test-patch.sh to run against both jdk 1.6 and jdk 1.7
[ https://issues.apache.org/jira/browse/HBASE-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509100#comment-13509100 ] Ted Yu commented on HBASE-6944: --- Running against 1.7 is fine. Enhance test-patch.sh to run against both jdk 1.6 and jdk 1.7 - Key: HBASE-6944 URL: https://issues.apache.org/jira/browse/HBASE-6944 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6944.txt Currently test-patch.sh only runs against jdk 1.6 However trunk build is using jdk 1.7 test-patch.sh should be enhanced to run against both jdk versions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7137) Improve Bytes to accept byte buffers which don't allow us to directly access their backing arrays
[ https://issues.apache.org/jira/browse/HBASE-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7137: - Resolution: Fixed Status: Resolved (was: Patch Available) Resolving. This patch was committed to trunk. Thanks for the patch Hiroshi. Improve Bytes to accept byte buffers which don't allow us to directly access their backing arrays - Key: HBASE-7137 URL: https://issues.apache.org/jira/browse/HBASE-7137 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Hiroshi Ikeda Assignee: Hiroshi Ikeda Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7137.patch, HBASE-7137-V2.patch, HBASE-7137-V3.patch Inside HBase, it seems that there is the implicit assumption that byte buffers have backed arrays and are not read-only, and we can freely call ByteBuffer.array() and arrayOffset() without runtime exceptions. But some classes, including Bytes, are supposed to be used by users from outside of HBase, and we should think the possibility that methods receive byte buffers which don't hold the assumption. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7264) Improve Snappy installation documentation
[ https://issues.apache.org/jira/browse/HBASE-7264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509105#comment-13509105 ] Hudson commented on HBASE-7264: --- Integrated in HBase-TRUNK #3589 (See [https://builds.apache.org/job/HBase-TRUNK/3589/]) HBASE-7264 Improve Snappy installation documentation (Revision 1416675) HBASE-7264 Improve Snappy installation documentation (Revision 1416667) Result = FAILURE stack : Files : * /hbase/trunk/src/docbkx/book.xml * /hbase/trunk/src/docbkx/developer.xml stack : Files : * /hbase/trunk/src/docbkx/book.xml Improve Snappy installation documentation - Key: HBASE-7264 URL: https://issues.apache.org/jira/browse/HBASE-7264 Project: HBase Issue Type: Bug Components: documentation Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Labels: documentation Fix For: 0.96.0 Attachments: HBASE-7264.patch Original Estimate: 1h Remaining Estimate: 1h Snappy installation process is lacking some details. I tried to give some. There is also some mistakes it's vs its. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509106#comment-13509106 ] Hudson commented on HBASE-7187: --- Integrated in HBase-TRUNK #3589 (See [https://builds.apache.org/job/HBase-TRUNK/3589/]) HBASE-7187 Create empty hbase-client module (Revision 1416676) HBASE-7187 Revert Dirty commit. (Revision 1416671) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-client * /hbase/trunk/hbase-client/pom.xml * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/pom.xml eclark : Files : * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-server/pom.xml * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/BackupMasterListTmpl.jamon * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon * /hbase/trunk/pom.xml Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7265) Make Maven skip module test properties consistent
[
https://issues.apache.org/jira/browse/HBASE-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-7265:
-
Attachment: HBASE-7265-0.patch
ran
{code}mvn clean package -DskipClientTests -DskipCommonTests -DskipExamplesTests
-DskipHadoopCompatTests -DskipHadoopTwoCompatTests -DskipHadoopOneCompatTests
-DskipIntegrationTests -DskipServerTests{code}
And no tests were run at all.
Make Maven skip module test properties consistent
-
Key: HBASE-7265
URL: https://issues.apache.org/jira/browse/HBASE-7265
Project: HBase
Issue Type: Bug
Components: build
Reporter: Elliott Clark
Priority: Minor
Fix For: 0.96.0
Attachments: HBASE-7265-0.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7265) Make Maven skip module test properties consistent
[ https://issues.apache.org/jira/browse/HBASE-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7265: - Fix Version/s: 0.96.0 Status: Patch Available (was: Open) Make Maven skip module test properties consistent - Key: HBASE-7265 URL: https://issues.apache.org/jira/browse/HBASE-7265 Project: HBase Issue Type: Bug Components: build Reporter: Elliott Clark Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7265-0.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7187) Create empty hbase-client module
[ https://issues.apache.org/jira/browse/HBASE-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-7187: - Resolution: Fixed Hadoop Flags: Incompatible change Status: Resolved (was: Patch Available) Create empty hbase-client module Key: HBASE-7187 URL: https://issues.apache.org/jira/browse/HBASE-7187 Project: HBase Issue Type: Sub-task Components: Client Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 0.96.0 Attachments: HBASE-7187-0.patch, HBASE-7187-1.patch, HBASE-7187-2.patch, HBASE-7187-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7265) Make Maven skip module test properties consistent
[ https://issues.apache.org/jira/browse/HBASE-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509130#comment-13509130 ] stack commented on HBASE-7265: -- +1 Make Maven skip module test properties consistent - Key: HBASE-7265 URL: https://issues.apache.org/jira/browse/HBASE-7265 Project: HBase Issue Type: Bug Components: build Reporter: Elliott Clark Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7265-0.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6944) Enhance test-patch.sh to run against both jdk 1.6 and jdk 1.7
[ https://issues.apache.org/jira/browse/HBASE-6944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6944. -- Resolution: Fixed Hadoop Flags: Incompatible change Changed https://builds.apache.org/view/G-L/view/HBase/job/PreCommit-HBASE-Build/configure to use 1.7 instead. Resolving. Enhance test-patch.sh to run against both jdk 1.6 and jdk 1.7 - Key: HBASE-6944 URL: https://issues.apache.org/jira/browse/HBASE-6944 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6944.txt Currently test-patch.sh only runs against jdk 1.6 However trunk build is using jdk 1.7 test-patch.sh should be enhanced to run against both jdk versions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7249) add test name filter to IntegrationTestsDriver
[
https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509137#comment-13509137
]
Hadoop QA commented on HBASE-7249:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1271/HBASE-7249-v0-094.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3439//console
This message is automatically generated.
add test name filter to IntegrationTestsDriver
--
Key: HBASE-7249
URL: https://issues.apache.org/jira/browse/HBASE-7249
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.94.3, 0.96.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Priority: Minor
Fix For: 0.96.0, 0.94.4
Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0.patch
As the number of tests grows, one might want to just run one, or a subset
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7256) Quick Start Guide shows stable version as 0.95, in the stable folder it is 0.94
[ https://issues.apache.org/jira/browse/HBASE-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509138#comment-13509138 ] Hudson commented on HBASE-7256: --- Integrated in HBase-TRUNK #3590 (See [https://builds.apache.org/job/HBase-TRUNK/3590/]) HBASE-7256 Quick Start Guide shows stable version as 0.95, in the stable folder it is 0.94 (Revision 1416679) Result = FAILURE stack : Files : * /hbase/trunk/src/docbkx/getting_started.xml Quick Start Guide shows stable version as 0.95, in the stable folder it is 0.94 --- Key: HBASE-7256 URL: https://issues.apache.org/jira/browse/HBASE-7256 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.94.2 Environment: HBase Repositories Reporter: Sumod Pawgi Assignee: stack Priority: Minor Fix For: 0.96.0 In the Quick Start Guide for HBase - http://hbase.apache.org/book/quickstart.html The stable version is mentioned as - 0.95 in the line - Choose a download site from this list of Apache Download Mirrors. Click on the suggested top link. This will take you to a mirror of HBase Releases. Click on the folder named stable and then download the file that ends in .tar.gz to your local filesystem; e.g. hbase-0.95-SNAPSHOT.tar.gz. But in the download folder at - http://apache.techartifact.com/mirror/hbase/stable/ The version that can be found is - hbase-0.94.2-security.tar.gz i.e. 0.94 So either the documentation or the download folder needs to be updated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7265) Make Maven skip module test properties consistent
[ https://issues.apache.org/jira/browse/HBASE-7265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark reassigned HBASE-7265: Assignee: Elliott Clark Make Maven skip module test properties consistent - Key: HBASE-7265 URL: https://issues.apache.org/jira/browse/HBASE-7265 Project: HBase Issue Type: Bug Components: build Reporter: Elliott Clark Assignee: Elliott Clark Priority: Minor Fix For: 0.96.0 Attachments: HBASE-7265-0.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
