[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153425#comment-13153425 ] Hudson commented on HBASE-4436: --- Integrated in HBase-0.92 #146 (See [https://builds.apache.org/job/HBase-0.92/146/]) HBASE-4436 Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * /hbase/branches/0.92/src/main/ruby/hbase/table.rb * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java Remove methods deprecated in 0.90 from TRUNK and 0.92 - Key: HBASE-4436 URL: https://issues.apache.org/jira/browse/HBASE-4436 Project: HBase Issue Type: Task Reporter: stack Assignee: Jonathan Hsieh Labels: noob Fix For: 0.92.0 Remove methods deprecated in 0.90 from codebase. i took a quick look. The messy bit is thrift referring to old stuff; will take a little work to do the convertion over to the new methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4256) Intra-row scanning (part deux)
[ https://issues.apache.org/jira/browse/HBASE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153438#comment-13153438 ] Hudson commented on HBASE-4256: --- Integrated in HBase-TRUNK #2460 (See [https://builds.apache.org/job/HBase-TRUNK/2460/]) HBASE-4256 Intra-row scanning (part deux) stack : Files : * /hbase/trunk/src/docbkx/book.xml Intra-row scanning (part deux) -- Key: HBASE-4256 URL: https://issues.apache.org/jira/browse/HBASE-4256 Project: HBase Issue Type: New Feature Affects Versions: 0.90.4 Reporter: Jean-Daniel Cryans Assignee: Dave Revell Priority: Critical Fix For: 0.94.0 Attachments: 4256.txt Dave Revell was asking on IRC today if there's a way to scan ranges of *qualifiers* within a row. That is, to be able to specify a *start qualifier* and an *end qualifier* so that the Get or Scan seeks directly to the first qualifier and stops at some point which can be predeterminate by a qualifier or simply a batch configuration (already exists). This is particularly useful for large rows with time-based qualifiers. Dave also mentioned that another popular database has such a feature that they call column slices. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4436) Remove methods deprecated in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153439#comment-13153439 ] Hudson commented on HBASE-4436: --- Integrated in HBase-TRUNK #2460 (See [https://builds.apache.org/job/HBase-TRUNK/2460/]) HBASE-4436 Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 stack : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Scan.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * /hbase/trunk/src/main/ruby/hbase/table.rb * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java Remove methods deprecated in 0.90 from TRUNK and 0.92 - Key: HBASE-4436 URL: https://issues.apache.org/jira/browse/HBASE-4436 Project: HBase Issue Type: Task Reporter: stack Assignee: Jonathan Hsieh Labels: noob Fix For: 0.92.0 Remove methods deprecated in 0.90 from codebase. i took a quick look. The messy bit is thrift referring to old stuff; will take a little work to do the convertion over to the new methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153440#comment-13153440 ] Hadoop QA commented on HBASE-4798: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504344/4798_trunk_all.v10.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 59 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.client.TestShell org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/309//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/309//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/309//console This message is automatically generated. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v10.patch, 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153451#comment-13153451 ] Ted Yu commented on HBASE-4213: --- TestInstantSchemaChange failures were due to 'Too many open files'. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk-v6.txt, 4213-trunk-v7.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch, schema-update.png This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153453#comment-13153453 ] Ted Yu commented on HBASE-4815: --- I think HadoopQA reported TestAdmin failure earlier which wasn't given enough attention. We need to set hbase.online.schema.update.enable to true for testDeleteEditUnknownColumnFamilyAndOrTable(). Otherwise the exception we got would be TableNotDisabledException, not InvalidFamilyOperationException. Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153454#comment-13153454 ] Ted Yu commented on HBASE-4583: --- bq. then could remove KVs never than that I guess the above should read 'then could remove KVs newer than that' Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4815: -- Attachment: 4815.addendum Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153460#comment-13153460 ] Ted Yu commented on HBASE-4815: --- Addendum integrated to TRUNK. Let's see if TestAdmin gets fixed. Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Open (was: Patch Available) Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v10.patch, 4798_trunk_all.v10.patch, 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Attachment: 4798_trunk_all.v10.patch Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v10.patch, 4798_trunk_all.v10.patch, 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Patch Available (was: Open) same patch try again Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v10.patch, 4798_trunk_all.v10.patch, 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153486#comment-13153486 ] Hudson commented on HBASE-4815: --- Integrated in HBase-TRUNK #2461 (See [https://builds.apache.org/job/HBase-TRUNK/2461/]) HBASE-4815 Addendum sets hbase.online.schema.update.enable to true tedyu : Files : * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153487#comment-13153487 ] Hadoop QA commented on HBASE-4798: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504358/4798_trunk_all.v10.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 59 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.regionserver.wal.TestLogRolling org.apache.hadoop.hbase.coprocessor.TestMasterObserver org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/310//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/310//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/310//console This message is automatically generated. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v10.patch, 4798_trunk_all.v10.patch, 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4781) Pom update to use the new versions of surefire junit.
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4781: --- Status: Open (was: Patch Available) Pom update to use the new versions of surefire junit. --- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4781) Pom update to use the new versions of surefire junit.
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4781: --- Attachment: 4781_trunk_pom.v6.patch Pom update to use the new versions of surefire junit. --- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4781) Pom update to use the new versions of surefire junit.
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4781: --- Status: Patch Available (was: Open) Pom update to use the new versions of surefire junit. --- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4763) Integrate surefire and junit for category management
[ https://issues.apache.org/jira/browse/HBASE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153494#comment-13153494 ] nkeywal commented on HBASE-4763: @stack: yes, but Gary has actually created a special repo for surefire and built it from this. Now we have: Junit: 4.10-HBASE-1, source code on https://github.com/nkeywal/junit, branch hbase Surefire: 2.11-TRUNK-HBASE-2, source code on https://github.com/ghelmling/maven-surefire branch 2.11-TRUNK-HBASE Both are in Gary's repo on http://people.apache.org/~garyh/mvn. In HBASE-4781, I am currently checking that it works well with Jenkins, if yes: - the patch on pom.xml in HBASE-4781 will be commited. - we will be able to close the two Jiras. Integrate surefire and junit for category management Key: HBASE-4763 URL: https://issues.apache.org/jira/browse/HBASE-4763 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: surefire_793_trunk.v3.patch, surefire_hbase.v2.patch, surefire_hbase.v3.patch As of today, Surefire integrates category on the trunk of 2.11 version: http://jira.codehaus.org/browse/SUREFIRE-329 . It may requires private patches as well. It may impact JUnit: https://github.com/KentBeck/junit/issues/359 This jira is about this integration. We will need a repo for this. For the naming of the versions to be created, I don't know if there is a convention. If not I would propose: 2.10-patched-HBASE Obviously, it's important to get our changes integrated in the main release: we're not forking surefire junit! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4781) Pom update to use the new versions of surefire junit.
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153515#comment-13153515 ] Hadoop QA commented on HBASE-4781: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504361/4781_trunk_pom.v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/311//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/311//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/311//console This message is automatically generated. Pom update to use the new versions of surefire junit. --- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4781) Pom update to use the new versions of surefire junit.
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153517#comment-13153517 ] nkeywal commented on HBASE-4781: org.apache.hadoop.hbase.client.TestAdmin: Caused by: java.io.IOException: Too many open files org.apache.hadoop.hbase.client.TestShell: failed on trunk as well Can be committed imho Pom update to use the new versions of surefire junit. --- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153518#comment-13153518 ] nkeywal commented on HBASE-4798: Intersection of the two tests and of trunk failures is: - org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testTableOperations - org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations - org.apache.hadoop.hbase.client.TestAdmin.testCheckHBaseAvailableClosesConnection (Caused by: java.io.IOException: Too many open files) So we're left with two new errors on coprocessor. Will have a look. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v10.patch, 4798_trunk_all.v10.patch, 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153519#comment-13153519 ] ramkrishna.s.vasudevan commented on HBASE-4815: --- Thanks Ted and Stack. I did not observe the failure as i had left the office by that time.. Regarding the config update in hbase-default.xml i was not very clear if it was needed. Thanks Stack for taking care of it. :) Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4781) Pom update to use the new versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153523#comment-13153523 ] Ted Yu commented on HBASE-4781: --- Integrated to TRUNK. Thanks for the patch N. Pom update to use the new versions of surefire junit -- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.94.0 Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4781) Pom update to use the new versions of surefire junit
[ https://issues.apache.org/jira/browse/HBASE-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4781: -- Fix Version/s: 0.94.0 Summary: Pom update to use the new versions of surefire junit (was: Pom update to use the new versions of surefire junit.) Pom update to use the new versions of surefire junit -- Key: HBASE-4781 URL: https://issues.apache.org/jira/browse/HBASE-4781 Project: HBase Issue Type: Improvement Components: build Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.94.0 Attachments: 4781_trunk_pom.patch, 4781_trunk_pom.v2.patch, 4781_trunk_pom.v6.patch Upgrading surefire junit requires some .pom modifications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153535#comment-13153535 ] Hudson commented on HBASE-4815: --- Integrated in HBase-0.92 #147 (See [https://builds.apache.org/job/HBase-0.92/147/]) HBASE-4815 Addendum sets hbase.online.schema.update.enable to true for TestAdmin tedyu : Files : * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
  -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings. This must be the new test, because the other changes are minimal.   -1 core tests. The patch failed these unit tests:            org.apache.hadoop.hbase.regionserver.wal.TestLogRolling          org.apache.hadoop.hbase.client.TestShell          org.apache.hadoop.hbase.client.TestAdmin These seem unrelated.   - Andy
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153538#comment-13153538
]
Andrew Purtell commented on HBASE-2418:
---
bq. Could the TestAdmin be because of this patch A?
I don't see it locally.
This change in the patch could be suspect but it's a shot in the dark:
{code}
--- src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
+++ src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
@@ -148,11 +156,14 @@ public class MiniZooKeeperCluster {
tickTimeToUse = TICK_TIME;
}
ZooKeeperServer server = new ZooKeeperServer(dir, dir, tickTimeToUse);
- NIOServerCnxn.Factory standaloneServerFactory;
+ NIOServerCnxnFactory standaloneServerFactory;
while (true) {
try {
- standaloneServerFactory = new NIOServerCnxn.Factory(
- new InetSocketAddress(tentativePort));
+ standaloneServerFactory = new NIOServerCnxnFactory();
+ standaloneServerFactory.configure(
+new InetSocketAddress(tentativePort),
+configuration.getInt(HConstants.ZOOKEEPER_MAX_CLIENT_CNXNS,
+ HConstants.DEFAULT_ZOOKEPER_MAX_CLIENT_CNXNS));
} catch (BindException e) {
LOG.debug(Failed binding ZK Server to client port: +
tentativePort);
{code}
I could change HConstants.DEFAULT_ZOOKEPER_MAX_CLIENT_CNXNS here to 1000 and
resubmit.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Attachments: HBASE-2418-5.patch, HBASE-2418-5.patch,
HBASE-2418-5.patch
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
security and helping to ensure that services don't interact negatively (touch
each other's data).
Today HBase does not have support for authentication or authorization. This
should be added to the HBase clients
that are accessing the ZK cluster. In general it means calling addAuthInfo
once after a session is established:
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String,
byte[])
with a user specific credential, often times this is a shared secret or
certificate. You may be able to statically configure this
in some cases (config string or file to read from), however in my case in
particular you may need to access it programmatically,
which adds complexity as the end user may need to load code into HBase for
accessing the credential.
Secondly you need to specify a non world ACL when interacting with znodes
(create primarily):
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html
Feel free to ping the ZooKeeper team if you have questions. It might also be
good to discuss with some
potential end users - in particular regarding how the end user can specify
the credential.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
From: Andrew Purtell apurt...@apache.org   -1 core tests. The patch failed these unit tests:            org.apache.hadoop.hbase.regionserver.wal.TestLogRolling          org.apache.hadoop.hbase.client.TestShell          org.apache.hadoop.hbase.client.TestAdmin These seem unrelated. Further discussion on the JIRA
[jira] [Created] (HBASE-4825) TestRegionServersMetrics and TestZKLeaderManager are not categorized (small/medium/large)
TestRegionServersMetrics and TestZKLeaderManager are not categorized (small/medium/large) - Key: HBASE-4825 URL: https://issues.apache.org/jira/browse/HBASE-4825 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal see title -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153539#comment-13153539 ] nkeywal commented on HBASE-4712: New proposal: 1) Running tests HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTests, MediumTests, LargeTests. - Small tests are executed in a shared JVM. We put in this category all the tests that can be executed quickly (the maximum execution time for a test is 15 seconds and they do not use a cluster) in a shared jvm. - Medium tests represents tests that must be executed before proposing a patch. They are designed to run in less than 30 minutes altogether, and are quite stable in their results. They're designed to less than 50 seconds. They can use a cluster, and each of them are executed in a separate JVM. - Large tests are everything else. They are typically integration tests, regression tests for specific bugs, timeout tests, performance tests. Some of them can be flaky. They are executed before a commit on the pre-integration machines. They can be run on the developper as well. Commands are: - mvn test - execute all tests in a separate JVM. All results are presented in a single report. - mvn -P runDevTests - execute small tests in a single JVM and medium tests in separates jvm. There are two reports. - mvn -P runAllTests - execute small tests in a single JVM and medium and large tests in separates jvm. There are two reports, one for small, one for medium and large. - mvn -P runSmallTests - execute small tests in a single JVM. It's as well possible to use the script 'hbasetests.sh'. This script runs the medium and large tests in parallel with two maven instances, and provide a single report. It must be executed from the directory which contains the pom.xml. Commands are: ./dev-support/hbasetests.sh - execute small and medium tests ./dev-support/hbasetests.sh runAllTests - execute all tests ./dev-support/hbasetests.sh replayFailed - rerun the failed tests a second time, in a separate jvm and without parallelisation. 2) Writing tests Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests must be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper functions to create a temp directory and do the cleanup, or to start a cluster. - Categories and executin time - All tests must be categorized, if not they could be skipped. - All tests should be written to be as fast as possible. - Small tests should last less than 15 seconds, and must not have any side effect. - Medium tests should last less than 45 seconds. - large tests should last less than 3 minutes, this ensure a good parallelisation for the ones using it, and ease the analysis when the test fails. - Sleeps: - Whenever possible, tests should not use sleep, but rather waiting for the real event. This is faster and clearer for the reader. - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local file system. - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class. - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later. @committers: Please tell me where I should put this, I will put it as a patch (with the new comments taken into account of course!) Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Status: Open (was: Patch Available) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-5.patch, HBASE-2418-5.patch, HBASE-2418-5.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Attachment: HBASE-2418-6.patch v6 patch with above described change. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Status: Patch Available (was: Open) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Attachment: (was: HBASE-2418-5.patch) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Attachment: (was: HBASE-2418-5.patch) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Attachment: (was: HBASE-2418-5.patch) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153546#comment-13153546 ] Hadoop QA commented on HBASE-2418: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504379/HBASE-2418-6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/312//console This message is automatically generated. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4826) Modify hbasetests.sh to take into account the new pom.xml with surefire
Modify hbasetests.sh to take into account the new pom.xml with surefire --- Key: HBASE-4826 URL: https://issues.apache.org/jira/browse/HBASE-4826 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal see title -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153551#comment-13153551 ] stack commented on HBASE-4815: -- Thanks Ted for coming along w/ the cleanup. Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Status: Open (was: Patch Available) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Attachment: HBASE-2418-6.patch Rebased patch on latest trunk. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153554#comment-13153554
]
stack commented on HBASE-2418:
--
Trunk is changing too fast on you Andrew!
{code}
patching file pom.xml
Hunk #1 FAILED at 276.
Hunk #2 succeeded at 861 (offset 41 lines).
Hunk #3 succeeded at 1404 with fuzz 2 (offset 3 lines).
1 out of 3 hunks FAILED -- saving rejects to file pom.xml.rej
patching file
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
patching file src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
patching file
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
patching file src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
patching file
src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PATCH APPLICATION FAILED
{code}
This is last 0.92 patch though Almost there.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
security and helping to ensure that services don't interact negatively (touch
each other's data).
Today HBase does not have support for authentication or authorization. This
should be added to the HBase clients
that are accessing the ZK cluster. In general it means calling addAuthInfo
once after a session is established:
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String,
byte[])
with a user specific credential, often times this is a shared secret or
certificate. You may be able to statically configure this
in some cases (config string or file to read from), however in my case in
particular you may need to access it programmatically,
which adds complexity as the end user may need to load code into HBase for
accessing the credential.
Secondly you need to specify a non world ACL when interacting with znodes
(create primarily):
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html
Feel free to ping the ZooKeeper team if you have questions. It might also be
good to discuss with some
potential end users - in particular regarding how the end user can specify
the credential.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Status: Patch Available (was: Open) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4789:
-
Resolution: Fixed
Assignee: stack
Status: Resolved (was: Patch Available)
Committed branch and trunk. Can open new issue if the logging in here turns up
more info on why this condition.
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789-v3.txt, 4789-v4.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4826) Modify hbasetests.sh to take into account the new pom.xml with surefire
[ https://issues.apache.org/jira/browse/HBASE-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4826: --- Attachment: 4826_trunk_hbasetests.patch Modify hbasetests.sh to take into account the new pom.xml with surefire --- Key: HBASE-4826 URL: https://issues.apache.org/jira/browse/HBASE-4826 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Attachments: 4826_trunk_hbasetests.patch see title -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4826) Modify hbasetests.sh to take into account the new pom.xml with surefire
[ https://issues.apache.org/jira/browse/HBASE-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153561#comment-13153561 ] nkeywal commented on HBASE-4826: local tests ok, can't be tested on pre-integration (it's a shell script). Modify hbasetests.sh to take into account the new pom.xml with surefire --- Key: HBASE-4826 URL: https://issues.apache.org/jira/browse/HBASE-4826 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Attachments: 4826_trunk_hbasetests.patch see title -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153578#comment-13153578 ] Hadoop QA commented on HBASE-2418: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504384/HBASE-2418-6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/313//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/313//console This message is automatically generated. add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
This seems to be the issue with the HBaseAdmin failure: Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO Caused by: java.io.IOException: Too many open files  I don't see this locally (obviously). I'd say the QA environment is insufficiently configured.   - Andy - Original Message - From: Hadoop QA (Commented) (JIRA) j...@apache.org To: issues@hbase.apache.org Cc: Sent: Saturday, November 19, 2011 12:28 PM Subject: [jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication   [ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153578#comment-13153578 ] Hadoop QA commented on HBASE-2418: -- -1 overall. Here are the results of testing the latest attachment  http://issues.apache.org/jira/secure/attachment/12504384/HBASE-2418-6.patch  against trunk revision .   +1 @author. The patch does not contain any @author tags.   +1 tests included. The patch appears to include 7 new or modified tests.   +1 javadoc. The javadoc tool did not generate any warning messages.   +1 javac. The applied patch does not increase the total number of javac compiler warnings.   -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings.   +1 release audit. The applied patch does not increase the total number of release audit warnings.   -1 core tests. The patch failed these unit tests:            org.apache.hadoop.hbase.client.TestAdmin          org.apache.hadoop.hbase.replication.TestReplication          org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/313//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/313//console This message is automatically generated.         add support for ZooKeeper authentication         Key: HBASE-2418         URL: https://issues.apache.org/jira/browse/HBASE-2418       Project: HBase      Issue Type: Improvement      Components: master, regionserver       Reporter: Patrick Hunt       Assignee: Eugene Koontz       Priority: Critical        Labels: security, zookeeper       Fix For: 0.92.0     Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153596#comment-13153596
]
Hudson commented on HBASE-4789:
---
Integrated in HBase-TRUNK #2463 (See
[https://builds.apache.org/job/HBase-TRUNK/2463/])
HBASE-4789 On split, parent region is sticking around in oldest sequenceid
to region map though not online; we don't cleanup WALs
stack :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789-v3.txt, 4789-v4.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4827) TestAdmin should clean up resources after tests
TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4827) TestAdmin should clean up resources after tests
[ https://issues.apache.org/jira/browse/HBASE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-4827: -- Attachment: HBASE-4827.patch TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-4827.patch TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4828) Fix failing TestShell, needs same addendum as HBASE-4815 got
[
https://issues.apache.org/jira/browse/HBASE-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4828:
-
Attachment: 4828.txt
Fix failing TestShell, needs same addendum as HBASE-4815 got
Key: HBASE-4828
URL: https://issues.apache.org/jira/browse/HBASE-4828
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 4828.txt
Do this:
{code}
Index: src/test/java/org/apache/hadoop/hbase/client/TestShell.java
===
--- src/test/java/org/apache/hadoop/hbase/client/TestShell.java (revision
1204082)
+++ src/test/java/org/apache/hadoop/hbase/client/TestShell.java (working copy)
@@ -45,6 +45,7 @@
@BeforeClass
public static void setUpBeforeClass() throws Exception {
// Start mini cluster
+
TEST_UTIL.getConfiguration().setBoolean(hbase.online.schema.update.enable,
true);
TEST_UTIL.getConfiguration().setInt(hbase.regionserver.msginterval,
100);
TEST_UTIL.getConfiguration().setInt(hbase.client.pause, 250);
TEST_UTIL.getConfiguration().setInt(hbase.client.retries.number, 6);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4829) Fix javadoc warnings in 0.92 branch
[ https://issues.apache.org/jira/browse/HBASE-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4829: - Attachment: javadoc.txt Fix javadoc warnings in 0.92 branch --- Key: HBASE-4829 URL: https://issues.apache.org/jira/browse/HBASE-4829 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.92.0 Attachments: javadoc.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4829) Fix javadoc warnings in 0.92 branch
[ https://issues.apache.org/jira/browse/HBASE-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-4829. -- Resolution: Fixed Fix Version/s: 0.92.0 Assignee: stack Committed branch and trunk Fix javadoc warnings in 0.92 branch --- Key: HBASE-4829 URL: https://issues.apache.org/jira/browse/HBASE-4829 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.0 Attachments: javadoc.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4829) Fix javadoc warnings in 0.92 branch
Fix javadoc warnings in 0.92 branch --- Key: HBASE-4829 URL: https://issues.apache.org/jira/browse/HBASE-4829 Project: HBase Issue Type: Bug Reporter: stack Attachments: javadoc.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4827) TestAdmin should clean up resources after tests
[ https://issues.apache.org/jira/browse/HBASE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153603#comment-13153603 ] stack commented on HBASE-4827: -- +1 Good one. TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell Attachments: HBASE-4827.patch TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4827) TestAdmin should clean up resources after tests
[ https://issues.apache.org/jira/browse/HBASE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-4827. --- Resolution: Fixed Fix Version/s: 0.90.5 0.94.0 0.92.0 Committed to trunk, 0.92, and 0.90 branches. TestAdmin passes locally in all three. TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: HBASE-4827.patch TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4827) TestAdmin should clean up resources after tests
[ https://issues.apache.org/jira/browse/HBASE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-4827: -- Attachment: HBASE-4827.patch Actual committed change. TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: HBASE-4827.patch, HBASE-4827.patch TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Status: Open (was: Patch Available) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2418) add support for ZooKeeper authentication
[ https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-2418: -- Status: Patch Available (was: Open) add support for ZooKeeper authentication Key: HBASE-2418 URL: https://issues.apache.org/jira/browse/HBASE-2418 Project: HBase Issue Type: Improvement Components: master, regionserver Reporter: Patrick Hunt Assignee: Eugene Koontz Priority: Critical Labels: security, zookeeper Fix For: 0.92.0 Attachments: HBASE-2418-6.patch, HBASE-2418-6.patch Some users may run a ZooKeeper cluster in multi tenant mode meaning that more than one client service would like to share a single ZooKeeper service instance (cluster). In this case the client services typically want to protect their data (ZK znodes) from access by other services (tenants) on the cluster. Say you are running HBase and Solr and Neo4j, or multiple HBase instances, etc... having authentication/authorization on the znodes is important for both security and helping to ensure that services don't interact negatively (touch each other's data). Today HBase does not have support for authentication or authorization. This should be added to the HBase clients that are accessing the ZK cluster. In general it means calling addAuthInfo once after a session is established: http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String, byte[]) with a user specific credential, often times this is a shared secret or certificate. You may be able to statically configure this in some cases (config string or file to read from), however in my case in particular you may need to access it programmatically, which adds complexity as the end user may need to load code into HBase for accessing the credential. Secondly you need to specify a non world ACL when interacting with znodes (create primarily): http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/data/ACL.html http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooDefs.html Feel free to ping the ZooKeeper team if you have questions. It might also be good to discuss with some potential end users - in particular regarding how the end user can specify the credential. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153609#comment-13153609 ] stack commented on HBASE-4712: -- This is excellent. Thanks for writing it up Nicolas. I believe it belongs in this chapter: http://hbase.apache.org/book.html#developer I can get it. Lets leave it hang out here a little while longer in case there are comments after your email today. Do you think the long tests play into HBASE-4821? Or at least, we should be able to run the long tests in whatever comes of hbase-4812. Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster. Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally. Small and medium tests must not need more than 30 minutes to run altogether. Small and medium tests should be executed by the developers before submitting a patch. Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests. Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - All tests should be written to be as fast as possible. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster. - Sleeps: - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local file system. - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class. - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4712) Document rules for writing tests
[ https://issues.apache.org/jira/browse/HBASE-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153612#comment-13153612 ] stack commented on HBASE-4712: -- On commit, add to the doc this note from mail on list: bq. By default there all executed in different JVM, and the second report is skipped. So you will see a new message at the end of the report: [INFO] Tests are skipped. It's harmless. Document rules for writing tests Key: HBASE-4712 URL: https://issues.apache.org/jira/browse/HBASE-4712 Project: HBase Issue Type: Task Components: test Affects Versions: 0.92.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor We saw that some tests could be improved. Documenting the general rules could help. Proposal: HBase tests are divided in three categories: small, medium and large, with corresponding JUnit categories: SmallTest, MediumTest, LargeTest Small tests are executed in parallel in a shared JVM. They must last less than 15 seconds. They must NOT use a cluster. Medium tests are executed in separate JVM. They must last less than 50 seconds. They can use a cluster. They must not fail occasionally. Small and medium tests must not need more than 30 minutes to run altogether. Small and medium tests should be executed by the developers before submitting a patch. Large tests are everything else. They are typically integration tests, non-regression tests for specific bugs, timeout tests, performance tests. Tests rules hints are: - As most as possible, tests should be written as small tests. - All tests should be written to support parallel execution on the same machine, hence should not use shared resources as fixed ports or fixed file names. - All tests should be written to be as fast as possible. - Tests should not overlog. More than 100 lines/second makes the logs complex to read and use i/o that are hence not available for the other tests. - Tests can be written with HBaseTestingUtility . This class offers helper function to create a temp directory and do the cleanup, or to start a cluster. - Sleeps: - Tests should not do a 'Thread.sleep' without testing an ending condition. This allows understanding what the test is waiting for. Moreover, the test will work whatever the machine performances. - Sleep should be minimal to be as fast as possible. Waiting for a variable should be done in a 40ms sleep loop. Waiting for a socket operation should be done in a 200 ms sleep loop. - Tests using cluster: - Tests using a HRegion do not have to start a cluster: A region can use the local file system. - Start/stopping a cluster cost around 10 seconds. They should not be started per test method but per class. - Started cluster must be shutdown using HBaseTestingUtility#shutdownMiniCluster, which cleans the directories. - As most as possible, tests should use the default settings for the cluster. When they don't, they should document it. This will allow to share the cluster later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4826) Modify hbasetests.sh to take into account the new pom.xml with surefire
[ https://issues.apache.org/jira/browse/HBASE-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-4826. -- Resolution: Fixed Fix Version/s: 0.94.0 Hadoop Flags: Reviewed Committed to trunk. Thanks for the patch Nicolas Modify hbasetests.sh to take into account the new pom.xml with surefire --- Key: HBASE-4826 URL: https://issues.apache.org/jira/browse/HBASE-4826 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Fix For: 0.94.0 Attachments: 4826_trunk_hbasetests.patch see title -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153616#comment-13153616
]
Hudson commented on HBASE-4789:
---
Integrated in HBase-0.92 #148 (See
[https://builds.apache.org/job/HBase-0.92/148/])
HBASE-4789 On split, parent region is sticking around in oldest sequenceid
to region map though not online; we don't cleanup WALs
stack :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
*
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789-v3.txt, 4789-v4.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see:
[jira] [Created] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800 nid=0x61b8
waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
[
https://issues.apache.org/jira/browse/HBASE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153622#comment-13153622
]
stack commented on HBASE-4830:
--
Hmmm... looks like I had an OOME first (Running w/ 1G/Default heap).
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800
nid=0x61b8 waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
[
https://issues.apache.org/jira/browse/HBASE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4830:
-
Attachment: hbase-stack-regionserver-sv4r9s38.out
Two thread dumps of hung regionserver
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: hbase-stack-regionserver-sv4r9s38.out
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800
nid=0x61b8 waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4826) Modify hbasetests.sh to take into account the new pom.xml with surefire
[ https://issues.apache.org/jira/browse/HBASE-4826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153632#comment-13153632 ] Hudson commented on HBASE-4826: --- Integrated in HBase-TRUNK #2464 (See [https://builds.apache.org/job/HBase-TRUNK/2464/]) HBASE-4826 Modify hbasetests.sh to take into account the new pom.xml with surefire stack : Files : * /hbase/trunk/dev-support/hbasetests.sh Modify hbasetests.sh to take into account the new pom.xml with surefire --- Key: HBASE-4826 URL: https://issues.apache.org/jira/browse/HBASE-4826 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Fix For: 0.94.0 Attachments: 4826_trunk_hbasetests.patch see title -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153634#comment-13153634 ] Hudson commented on HBASE-4815: --- Integrated in HBase-TRUNK #2464 (See [https://builds.apache.org/job/HBase-TRUNK/2464/]) HBASE-4828 Fix failing TestShell, needs same addendum as HBASE-4815 got stack : Files : * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestShell.java Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4829) Fix javadoc warnings in 0.92 branch
[ https://issues.apache.org/jira/browse/HBASE-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153636#comment-13153636 ] Hudson commented on HBASE-4829: --- Integrated in HBase-TRUNK #2464 (See [https://builds.apache.org/job/HBase-TRUNK/2464/]) HBASE-4829 Fix javadoc warnings in 0.92 branch stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/Result.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Fix javadoc warnings in 0.92 branch --- Key: HBASE-4829 URL: https://issues.apache.org/jira/browse/HBASE-4829 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.0 Attachments: javadoc.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4828) Fix failing TestShell, needs same addendum as HBASE-4815 got
[
https://issues.apache.org/jira/browse/HBASE-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153635#comment-13153635
]
Hudson commented on HBASE-4828:
---
Integrated in HBase-TRUNK #2464 (See
[https://builds.apache.org/job/HBase-TRUNK/2464/])
HBASE-4828 Fix failing TestShell, needs same addendum as HBASE-4815 got
stack :
Files :
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
Fix failing TestShell, needs same addendum as HBASE-4815 got
Key: HBASE-4828
URL: https://issues.apache.org/jira/browse/HBASE-4828
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 4828.txt
Do this:
{code}
Index: src/test/java/org/apache/hadoop/hbase/client/TestShell.java
===
--- src/test/java/org/apache/hadoop/hbase/client/TestShell.java (revision
1204082)
+++ src/test/java/org/apache/hadoop/hbase/client/TestShell.java (working copy)
@@ -45,6 +45,7 @@
@BeforeClass
public static void setUpBeforeClass() throws Exception {
// Start mini cluster
+
TEST_UTIL.getConfiguration().setBoolean(hbase.online.schema.update.enable,
true);
TEST_UTIL.getConfiguration().setInt(hbase.regionserver.msginterval,
100);
TEST_UTIL.getConfiguration().setInt(hbase.client.pause, 250);
TEST_UTIL.getConfiguration().setInt(hbase.client.retries.number, 6);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4827) TestAdmin should clean up resources after tests
[ https://issues.apache.org/jira/browse/HBASE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153633#comment-13153633 ] Hudson commented on HBASE-4827: --- Integrated in HBase-TRUNK #2464 (See [https://builds.apache.org/job/HBase-TRUNK/2464/]) HBASE-4827 TestAdmin should clean up resources after tests apurtell : Files : * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: HBASE-4827.patch, HBASE-4827.patch TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4830) Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno running 0.20.205.0+
[
https://issues.apache.org/jira/browse/HBASE-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153638#comment-13153638
]
Todd Lipcon commented on HBASE-4830:
Yea, I'm guessing this was a result of the OOME rather than any known HDFS bug.
Whatever that assertion failure was in DFSClient probably borked the internal
queues to some inconsistent state.
Regionserver BLOCKED on WAITING DFSClient$DFSOutputStream.waitForAckedSeqno
running 0.20.205.0+
---
Key: HBASE-4830
URL: https://issues.apache.org/jira/browse/HBASE-4830
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: hbase-stack-regionserver-sv4r9s38.out
Running 0.20.205.1 (I was not at tip of the branch) I ran into the following
hung regionserver:
{code}
regionserver7003.logRoller daemon prio=10 tid=0x7fd98028f800 nid=0x61af
in Object.wait() [0x7fd987bfa000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(DFSClient.java:3606)
- locked 0xf8656788 (a java.util.LinkedList)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(DFSClient.java:3595)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3687)
- locked 0xf8656458 (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3626)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:966)
- locked 0xf8655998 (a
org.apache.hadoop.io.SequenceFile$Writer)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.close(SequenceFileLogWriter.java:214)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.cleanupCurrentWriter(HLog.java:791)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:578)
- locked 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:94)
at java.lang.Thread.run(Thread.java:662)
{code}
Other threads are like this (here's a sample):
{code}
regionserver7003.logSyncer daemon prio=10 tid=0x7fd98025e000 nid=0x61ae
waiting for monitor entry [0x7fd987cfb000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1074)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1195)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1057)
at java.lang.Thread.run(Thread.java:662)
IPC Server handler 0 on 7003 daemon prio=10 tid=0x7fd98049b800
nid=0x61b8 waiting for monitor entry [0x7fd9872f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:1007)
- waiting to lock 0xc443deb0 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:1798)
at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1668)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:2980)
at sun.reflect.GeneratedMethodAccessor636.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
{code}
Looks like HDFS-1529? (Todd?)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153645#comment-13153645 ] Lars Hofhansl commented on HBASE-4583: -- Yep. Thinking about it, though, that wouldn't be very good, because any scanner scanning any row would prevent the older from being cleaned from the memstore. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4831) LRU stats thread should be a daemon thread
LRU stats thread should be a daemon thread -- Key: HBASE-4831 URL: https://issues.apache.org/jira/browse/HBASE-4831 Project: HBase Issue Type: Bug Reporter: Prakash Khemani Assignee: Prakash Khemani I have seen the hung processes where the following was the only non-daemon thread LRU Statistics #0 prio=10 tid=0x2ab0bc04f800 nid=0x11ac waiting on condition [0x42f57000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x2aaab9a1c000 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4823) long running scans lose benefit of bloomfilters and timerange hints
[
https://issues.apache.org/jira/browse/HBASE-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153655#comment-13153655
]
Prakash Khemani commented on HBASE-4823:
https://issues.apache.org/jira/browse/HBASE-3415 is also related
long running scans lose benefit of bloomfilters and timerange hints
---
Key: HBASE-4823
URL: https://issues.apache.org/jira/browse/HBASE-4823
Project: HBase
Issue Type: Bug
Reporter: Kannan Muthukkaruppan
Assignee: Kannan Muthukkaruppan
When you have a long running scan due to say an MR job, you can lose the
benefit of timerange hints bloom filters midway if your scanner gets reset.
[Note: The scanners can get reset say due to a flush or compaction].
In one of our workloads, we periodically want to do rollups on recent 15
minutes of data in a column family... but the timerange hint benefit is lost
midway when this resetScannerStack (shown below) happens. And end result-- we
end up reading all the old HFiles rather than just the recent HFiles.
{code}
private void resetScannerStack(KeyValue lastTopKey) throws IOException {
if (heap != null) {
throw new RuntimeException(StoreScanner.reseek run on an existing
heap!);
}
/* When we have the scan object, should we not pass it to getScanners()
* to get a limited set of scanners? We did so in the constructor and we
* could have done it now by storing the scan object from the constructor
*/
ListKeyValueScanner scanners = getScanners();
{code}
The comment in the code seems to be aware of this issue and even has the
suggested fix!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153678#comment-13153678 ] Lars Hofhansl commented on HBASE-2856: -- Since this mainly clashes with my change from HBASE-4536, I'm probably best qualified to adapt the patch to 0.92. I'm working on a 0.92 patch now. TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153679#comment-13153679 ] Lars Hofhansl commented on HBASE-2856: -- I extracted the trunk patch this way: svn diff -r r1203428:r1203468 TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153713#comment-13153713 ] Lars Hofhansl commented on HBASE-2856: -- Turns out this relies on API introduced in HBASE-4219 TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4827) TestAdmin should clean up resources after tests
[ https://issues.apache.org/jira/browse/HBASE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153720#comment-13153720 ] Hudson commented on HBASE-4827: --- Integrated in HBase-0.92 #149 (See [https://builds.apache.org/job/HBase-0.92/149/]) HBASE-4827 TestAdmin should clean up resources after tests apurtell : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java TestAdmin should clean up resources after tests --- Key: HBASE-4827 URL: https://issues.apache.org/jira/browse/HBASE-4827 Project: HBase Issue Type: Test Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.92.0, 0.94.0, 0.90.5 Attachments: HBASE-4827.patch, HBASE-4827.patch TestAdmin creates a new HBaseAdmin for each test case but does not clean up resources afterward. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4828) Fix failing TestShell, needs same addendum as HBASE-4815 got
[
https://issues.apache.org/jira/browse/HBASE-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153722#comment-13153722
]
Hudson commented on HBASE-4828:
---
Integrated in HBase-0.92 #149 (See
[https://builds.apache.org/job/HBase-0.92/149/])
HBASE-4828 Fix failing TestShell, needs same addendum as HBASE-4815 got
stack :
Files :
*
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestShell.java
Fix failing TestShell, needs same addendum as HBASE-4815 got
Key: HBASE-4828
URL: https://issues.apache.org/jira/browse/HBASE-4828
Project: HBase
Issue Type: Bug
Reporter: stack
Attachments: 4828.txt
Do this:
{code}
Index: src/test/java/org/apache/hadoop/hbase/client/TestShell.java
===
--- src/test/java/org/apache/hadoop/hbase/client/TestShell.java (revision
1204082)
+++ src/test/java/org/apache/hadoop/hbase/client/TestShell.java (working copy)
@@ -45,6 +45,7 @@
@BeforeClass
public static void setUpBeforeClass() throws Exception {
// Start mini cluster
+
TEST_UTIL.getConfiguration().setBoolean(hbase.online.schema.update.enable,
true);
TEST_UTIL.getConfiguration().setInt(hbase.regionserver.msginterval,
100);
TEST_UTIL.getConfiguration().setInt(hbase.client.pause, 250);
TEST_UTIL.getConfiguration().setInt(hbase.client.retries.number, 6);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153721#comment-13153721 ] Hudson commented on HBASE-4815: --- Integrated in HBase-0.92 #149 (See [https://builds.apache.org/job/HBase-0.92/149/]) HBASE-4828 Fix failing TestShell, needs same addendum as HBASE-4815 got stack : Files : * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestShell.java Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.addendum, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4829) Fix javadoc warnings in 0.92 branch
[ https://issues.apache.org/jira/browse/HBASE-4829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153723#comment-13153723 ] Hudson commented on HBASE-4829: --- Integrated in HBase-0.92 #149 (See [https://builds.apache.org/job/HBase-0.92/149/]) HBASE-4829 Fix javadoc warnings in 0.92 branch stack : Files : * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/Result.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormat.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Fix javadoc warnings in 0.92 branch --- Key: HBASE-4829 URL: https://issues.apache.org/jira/browse/HBASE-4829 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.0 Attachments: javadoc.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-2856: - Attachment: 2856-0.92.txt Here's a patch against 0.92. I pulled in the necessary API changes from HBASE-4219 (but not the rest of the functionality). I could use some help verifying the patch and testing this! TestAcidGuarantees passed (but only ran it once). TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-0.92.txt, 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
