[jira] [Updated] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-3025: - Status: Patch Available (was: Open) Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3615) Implement token based DIGEST-MD5 authentication for MapReduce tasks
[ https://issues.apache.org/jira/browse/HBASE-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling resolved HBASE-3615. -- Resolution: Fixed Fix Version/s: 0.92.0 Since tokens are one of the possible authentication methods during RPC, this was included as part of the change for HBASE-2742. Implement token based DIGEST-MD5 authentication for MapReduce tasks --- Key: HBASE-3615 URL: https://issues.apache.org/jira/browse/HBASE-3615 Project: HBase Issue Type: New Feature Components: ipc, security Reporter: Gary Helmling Assignee: Gary Helmling Fix For: 0.92.0 Attachments: HBASE-3615.patch HBase security currently supports Kerberos authentication for clients, but this isn't sufficient for map-reduce interoperability, where tasks execute without Kerberos credentials. In order to fully interoperate with map-reduce clients, we will need to provide our own token authentication mechanism, mirroring the Hadoop token authentication mechanisms. This will require obtaining an HBase authentication token for the user when the job is submitted, serializing it to a secure location, and then, at task execution, having the client or task code de-serialize the stored authentication token and use that in the HBase client authentication process. A detailed implementation proposal is sketched out on the wiki: http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4816) Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out
[ https://issues.apache.org/jira/browse/HBASE-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152719#comment-13152719 ] Hadoop QA commented on HBASE-4816: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504181/4816-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -163 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 56 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/290//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/290//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/290//console This message is automatically generated. Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out -- Key: HBASE-4816 URL: https://issues.apache.org/jira/browse/HBASE-4816 Project: HBase Issue Type: Bug Reporter: stack Fix For: 0.92.0 Attachments: 4816-v2.txt, 4816.txt A regionserver wouldn't go down because it was waiting on a user region to close only the user-space region had just been opened as part of a split transaction -- it was a new daughter -- just as we'd issued the bulk close to all user regions on receipt of a cluster shutdown call. We need to add a check for this condition -- user tables that did not get the close. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4818) HBase Shell - Add support for formatting row keys before output
[
https://issues.apache.org/jira/browse/HBASE-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152725#comment-13152725
]
Lars George commented on HBASE-4818:
I would also like to see this persisted then, i.e. a simply text property file,
or in the .irbrc where you can define this per table, so that these classes are
loaded implicitly.
HBase Shell - Add support for formatting row keys before output
---
Key: HBASE-4818
URL: https://issues.apache.org/jira/browse/HBASE-4818
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Eran Kampf
Priority: Trivial
Original Estimate: 24h
Remaining Estimate: 24h
As many HBase users use binary row keys rather than strings to optimize
memory consumption displaying an escaped string in the HBase shell isn't
useful (and takes a lot of screen space)
Allowing user to provide a row key formatter as part of the scan\get commands
would allow developers to display the row key in a way thats makes sense for
them.
Example:
scan 'stats', { ROWFORMATTER = MyRowFormatter.new }
The row formatter simply gets the bytes array key and formats it to a string.
Its an easy change tomake with simple monkey-patching of the shell commands
but I would be happy to see it as part of the shell itself.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152749#comment-13152749 ] Hadoop QA commented on HBASE-3025: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504183/HBASE-3025_5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/291//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/291//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/291//console This message is automatically generated. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152757#comment-13152757 ] Hudson commented on HBASE-2742: --- Integrated in HBase-0.92 #143 (See [https://builds.apache.org/job/HBase-0.92/143/]) HBASE-2742 Provide strong authentication with a secure RPC engine garyh : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/conf/hbase-policy.xml * /hbase/branches/0.92/pom.xml * /hbase/branches/0.92/security * /hbase/branches/0.92/security/src * /hbase/branches/0.92/security/src/main * /hbase/branches/0.92/security/src/main/java * /hbase/branches/0.92/security/src/main/java/org * /hbase/branches/0.92/security/src/main/java/org/apache * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java * /hbase/branches/0.92/security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java * /hbase/branches/0.92/security/src/test * /hbase/branches/0.92/security/src/test/java * /hbase/branches/0.92/security/src/test/java/org * /hbase/branches/0.92/security/src/test/java/org/apache * /hbase/branches/0.92/security/src/test/java/org/apache/hadoop * /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase * /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security * /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token * /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java * /hbase/branches/0.92/security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java * /hbase/branches/0.92/security/src/test/resources * /hbase/branches/0.92/security/src/test/resources/hbase-site.xml * /hbase/branches/0.92/src/assembly/all.xml * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/HServerAddress.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java *
[jira] [Commented] (HBASE-4818) HBase Shell - Add support for formatting row keys before output
[
https://issues.apache.org/jira/browse/HBASE-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152766#comment-13152766
]
Eran Kampf commented on HBASE-4818:
---
Thats a good idea!
A simple global hash that maps a table name to a row key formatter and then all
operations on the table use that formatter unless explicitly given one
HBase Shell - Add support for formatting row keys before output
---
Key: HBASE-4818
URL: https://issues.apache.org/jira/browse/HBASE-4818
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Eran Kampf
Priority: Trivial
Original Estimate: 24h
Remaining Estimate: 24h
As many HBase users use binary row keys rather than strings to optimize
memory consumption displaying an escaped string in the HBase shell isn't
useful (and takes a lot of screen space)
Allowing user to provide a row key formatter as part of the scan\get commands
would allow developers to display the row key in a way thats makes sense for
them.
Example:
scan 'stats', { ROWFORMATTER = MyRowFormatter.new }
The row formatter simply gets the bytes array key and formats it to a string.
Its an easy change tomake with simple monkey-patching of the shell commands
but I would be happy to see it as part of the shell itself.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152799#comment-13152799 ] Hudson commented on HBASE-2742: --- Integrated in HBase-TRUNK #2455 (See [https://builds.apache.org/job/HBase-TRUNK/2455/]) HBASE-2742 Provide strong authentication with a secure RPC engine garyh : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/conf/hbase-policy.xml * /hbase/trunk/pom.xml * /hbase/trunk/security * /hbase/trunk/security/src * /hbase/trunk/security/src/main * /hbase/trunk/security/src/main/java * /hbase/trunk/security/src/main/java/org * /hbase/trunk/security/src/main/java/org/apache * /hbase/trunk/security/src/main/java/org/apache/hadoop * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java * /hbase/trunk/security/src/test * /hbase/trunk/security/src/test/java * /hbase/trunk/security/src/test/java/org * /hbase/trunk/security/src/test/java/org/apache * /hbase/trunk/security/src/test/java/org/apache/hadoop * /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase * /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security * /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token * /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java * /hbase/trunk/security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java * /hbase/trunk/security/src/test/resources * /hbase/trunk/security/src/test/resources/hbase-site.xml * /hbase/trunk/src/assembly/all.xml * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HServerAddress.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java *
[jira] [Updated] (HBASE-4763) Integrate surefire and junit for category management
[ https://issues.apache.org/jira/browse/HBASE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4763: --- Attachment: surefire_793_trunk.v3.patch Integrate surefire and junit for category management Key: HBASE-4763 URL: https://issues.apache.org/jira/browse/HBASE-4763 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: surefire_793_trunk.v3.patch, surefire_hbase.v2.patch, surefire_hbase.v3.patch As of today, Surefire integrates category on the trunk of 2.11 version: http://jira.codehaus.org/browse/SUREFIRE-329 . It may requires private patches as well. It may impact JUnit: https://github.com/KentBeck/junit/issues/359 This jira is about this integration. We will need a repo for this. For the naming of the versions to be created, I don't know if there is a convention. If not I would propose: 2.10-patched-HBASE Obviously, it's important to get our changes integrated in the main release: we're not forking surefire junit! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4763) Integrate surefire and junit for category management
[ https://issues.apache.org/jira/browse/HBASE-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152803#comment-13152803 ] nkeywal commented on HBASE-4763: Fix for surefire 793. I've been able to reduce the changes to a minimum. Tested on a full build, seems to run fine. I push it on they jira as well, but it will require some changes to manage some cases we don't care about and to be more integrated in their design, so we should not wait for the trunk here. Integrate surefire and junit for category management Key: HBASE-4763 URL: https://issues.apache.org/jira/browse/HBASE-4763 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: surefire_793_trunk.v3.patch, surefire_hbase.v2.patch, surefire_hbase.v3.patch As of today, Surefire integrates category on the trunk of 2.11 version: http://jira.codehaus.org/browse/SUREFIRE-329 . It may requires private patches as well. It may impact JUnit: https://github.com/KentBeck/junit/issues/359 This jira is about this integration. We will need a repo for this. For the naming of the versions to be created, I don't know if there is a convention. If not I would propose: 2.10-patched-HBASE Obviously, it's important to get our changes integrated in the main release: we're not forking surefire junit! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Open (was: Patch Available) Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Attachment: 4798_trunk_all.v5.patch Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Patch Available (was: Open) Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152857#comment-13152857 ] nkeywal commented on HBASE-4798: v5 contains everything except HRegionServer#stop: use a notification on sleeper to lower shutdown time by 0,5s. I will create a specific Jira for this one as it implies some changes in TestRegionServerCoprocessorExceptionWithAbort. @eugene: with a specific jira it will be simpler :-). But my comment from 17/Nov/11 11:21 contains everything related to TestRegionServerCoprocessorExceptionWithAbort. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152859#comment-13152859 ] Hadoop QA commented on HBASE-4798: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504205/4798_trunk_all.v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/292//console This message is automatically generated. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152886#comment-13152886 ] ramkrishna.s.vasudevan commented on HBASE-4815: --- Will upload a patch..shortly Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Priority: Blocker Fix For: 0.92.0 There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to disable to table to alter). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Attachment: 4798_trunk_all.v6.patch Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Patch Available (was: Open) Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152939#comment-13152939
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/
---
(Updated 2011-11-18 16:05:49.921654)
Review request for hbase, Gary Helmling and Eugene Koontz.
Changes
---
Unconditionally pull in ZK 3.4.
Some of the POM changes might reject, depending on what goes in first, but
would be trivial to fix up.
Summary
---
These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
SASL authentication of ZooKeeper clients with the quorum is handled in the ZK
client independently of HBase concerns. To enable strong ZK authentication, one
must create a suitable JaaS configuration, for example:
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab=/etc/hbase/conf/hbase.keytab
storeKey=true
useTicketCache=false
principal=zookeeper/$HOSTNAME;
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
useTicketCache=false
keyTab=/etc/hbase/conf/hbase.keytab
principal=hbase/$HOSTNAME;
};
and then configure both the client and server processes to use it, for example
in hbase-site.xml:
HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
HBASE_OPTS=${HBASE_OPTS} -Dzookeeper.kerberos.removeHostFromPrincipal=true
HBASE_OPTS=${HBASE_OPTS} -Dzookeeper.kerberos.removeRealmFromPrincipal=true
HBase will then secure all znodes but for a few world-readable read-only ones
needed for clients to look up region locations. All internal cluster operations
will be protected from unauthenticated ZK clients, or clients not authenticated
to the HBase principal. Presumably the only ZK clients authenticated to the
HBase principal will be those embedded in the master and regionservers.
There is extraneous whitespace in code surrounding these changes.
This addresses bug HBASE-2418.
https://issues.apache.org/jira/browse/HBASE-2418
Diffs (updated)
-
pom.xml 382c7c2
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java a75cf87
src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PRE-CREATION
Diff: https://reviews.apache.org/r/2837/diff
Testing
---
These changes are running in production at Trend Micro, using a snapshot build
of ZooKeeper 3.4.0.
New unit test TestZooKeeperACL passes 100 iterations. All test pass not
otherwise currently failing on trunk.
Thanks,
Andrew
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
security and helping to ensure that services don't interact negatively (touch
each other's data).
Today HBase does not have support for authentication or authorization. This
should be added to the HBase clients
that are accessing the ZK cluster. In general it means calling addAuthInfo
once after a session is established:
http://hadoop.apache.org/zookeeper/docs/current/api/org/apache/zookeeper/ZooKeeper.html#addAuthInfo(java.lang.String,
byte[])
with a user specific credential, often times this is a shared secret or
certificate. You may be able to statically configure this
in some cases (config string or file to read from), however in my case in
particular you may need to access it programmatically,
which adds complexity as the
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152941#comment-13152941
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
bq. On 2011-11-18 04:14:48, Andrew Purtell wrote:
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java,
line 338
bq. https://reviews.apache.org/r/2837/diff/3/?file=59201#file59201line338
bq.
bq. I don't see how to work around this. The code has to be recompiled
against 3.3 or 3.4.
bq.
bq. Any ideas?
bq.
bq. I think we have to pull in 3.4 unconditionally.
bq.
bq. Lars Hofhansl wrote:
bq. Might be a good thing anyway. 3.4 is in RC right now, would probably
be GA by the time we GA 0.92. (we're meeting with Ted Dunning tomorrow -
unrelated - can ask him what he thinks about it).
bq.
bq. Andrew Purtell wrote:
bq. We can put in the other two patches and defer this one until 3.4 is
released. Consequence would be that 3025 can be subverted if one allows direct
client access to the ZK cluster.
bq.
bq. Michael Stack wrote:
bq. +1 on fixing this patch so it just upgrades our zk to 3.4. I just
tried hbase w/ a 3.4 client and a 3.3.3. ensemble and it seems to work. I
asked Mahadev to be sure and he says A 3.3.* client should be able to talk to
3.4.0 server and vice versa. Its as yet unreleased but I'm pretty sure it'll
be out before we ship -- maybe even a 3.4.1 (smile).
Say what you will about the Hadoop Record compiler aka Jute, but ZK is 1) the
only project that uses it that I know of 2) yet consistently can answer in the
affirmative regarding cross-version compatibility.
- Andrew
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3344
---
On 2011-11-18 16:05:49, Andrew Purtell wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2837/
bq. ---
bq.
bq. (Updated 2011-11-18 16:05:49)
bq.
bq.
bq. Review request for hbase, Gary Helmling and Eugene Koontz.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
bq.
bq. SASL authentication of ZooKeeper clients with the quorum is handled in the
ZK client independently of HBase concerns. To enable strong ZK authentication,
one must create a suitable JaaS configuration, for example:
bq.
bq.Server {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. storeKey=true
bq. useTicketCache=false
bq. principal=zookeeper/$HOSTNAME;
bq.};
bq.Client {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. useTicketCache=false
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. principal=hbase/$HOSTNAME;
bq.};
bq.
bq. and then configure both the client and server processes to use it, for
example in hbase-site.xml:
bq.
bq.HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeHostFromPrincipal=true
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeRealmFromPrincipal=true
bq.
bq. HBase will then secure all znodes but for a few world-readable read-only
ones needed for clients to look up region locations. All internal cluster
operations will be protected from unauthenticated ZK clients, or clients not
authenticated to the HBase principal. Presumably the only ZK clients
authenticated to the HBase principal will be those embedded in the master and
regionservers.
bq.
bq. There is extraneous whitespace in code surrounding these changes.
bq.
bq.
bq. This addresses bug HBASE-2418.
bq. https://issues.apache.org/jira/browse/HBASE-2418
bq.
bq.
bq. Diffs
bq. -
bq.
bq.pom.xml 382c7c2
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
a75cf87
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2837/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. These
[jira] [Commented] (HBASE-4295) rowcounter does not return the correct number of rows in certain circumstances
[
https://issues.apache.org/jira/browse/HBASE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152956#comment-13152956
]
stack commented on HBASE-4295:
--
Stuti Awasthi on the list confirms this fixed rowcounter for him. Lets get
this into 0.90.5.
rowcounter does not return the correct number of rows in certain circumstances
--
Key: HBASE-4295
URL: https://issues.apache.org/jira/browse/HBASE-4295
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.90.4
Reporter: Wing Yew Poon
Assignee: Dave Revell
Fix For: 0.90.5
Attachments: HBASE-4295-v1.patch
When you run
{noformat}
hadoop jar hbase.jar rowcounter table
{noformat}
the org.apache.hadoop.hbase.mapreduce.RowCounter class is run.
The RowCounterMapper class in the RowCounter mapreduce job contains the
following:
{noformat}
@Override
public void map(ImmutableBytesWritable row, Result values,
Context context)
throws IOException {
for (KeyValue value: values.list()) {
if (value.getValue().length 0) {
context.getCounter(Counters.ROWS).increment(1);
break;
}
}
}
{noformat}
The intention is to go through the column values in the row, and increment
the ROWS counter if some value is non-empty. However, values.list() always
has size 1. This is because the createSubmittableJob static method uses a
Scan as follows:
{noformat}
Scan scan = new Scan();
scan.setFilter(new FirstKeyOnlyFilter());
{noformat}
So the input map splits always contain just the first KV. If the column
corresponding to that first KV is empty, even though other columns are
non-empty, that row is skipped.
This way, rowcounter can return an incorrect result.
One way to reproduce this is to create an hbase table with two columns, say
f1:q1 and f2:q2. Create some (say 2) rows with empty f1:q1 but non-empty
f2:q2, and some (say 3) rows with empty f2:q2 and non-empty f1:q1.
Then run rowcounter (specifying only the table but not any columns). The
count will be either 2 short or 3 short.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4295) rowcounter does not return the correct number of rows in certain circumstances
[
https://issues.apache.org/jira/browse/HBASE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4295:
-
Priority: Critical (was: Major)
rowcounter does not return the correct number of rows in certain circumstances
--
Key: HBASE-4295
URL: https://issues.apache.org/jira/browse/HBASE-4295
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.90.4
Reporter: Wing Yew Poon
Assignee: Dave Revell
Priority: Critical
Fix For: 0.90.5
Attachments: HBASE-4295-v1.patch
When you run
{noformat}
hadoop jar hbase.jar rowcounter table
{noformat}
the org.apache.hadoop.hbase.mapreduce.RowCounter class is run.
The RowCounterMapper class in the RowCounter mapreduce job contains the
following:
{noformat}
@Override
public void map(ImmutableBytesWritable row, Result values,
Context context)
throws IOException {
for (KeyValue value: values.list()) {
if (value.getValue().length 0) {
context.getCounter(Counters.ROWS).increment(1);
break;
}
}
}
{noformat}
The intention is to go through the column values in the row, and increment
the ROWS counter if some value is non-empty. However, values.list() always
has size 1. This is because the createSubmittableJob static method uses a
Scan as follows:
{noformat}
Scan scan = new Scan();
scan.setFilter(new FirstKeyOnlyFilter());
{noformat}
So the input map splits always contain just the first KV. If the column
corresponding to that first KV is empty, even though other columns are
non-empty, that row is skipped.
This way, rowcounter can return an incorrect result.
One way to reproduce this is to create an hbase table with two columns, say
f1:q1 and f2:q2. Create some (say 2) rows with empty f1:q1 but non-empty
f2:q2, and some (say 3) rows with empty f2:q2 and non-empty f1:q1.
Then run rowcounter (specifying only the table but not any columns). The
count will be either 2 short or 3 short.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4295) rowcounter does not return the correct number of rows in certain circumstances
[
https://issues.apache.org/jira/browse/HBASE-4295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152959#comment-13152959
]
stack commented on HBASE-4295:
--
Hmm... it looks like it is in 0.90.5. Ignore above.
rowcounter does not return the correct number of rows in certain circumstances
--
Key: HBASE-4295
URL: https://issues.apache.org/jira/browse/HBASE-4295
Project: HBase
Issue Type: Bug
Components: mapreduce
Affects Versions: 0.90.4
Reporter: Wing Yew Poon
Assignee: Dave Revell
Priority: Critical
Fix For: 0.90.5
Attachments: HBASE-4295-v1.patch
When you run
{noformat}
hadoop jar hbase.jar rowcounter table
{noformat}
the org.apache.hadoop.hbase.mapreduce.RowCounter class is run.
The RowCounterMapper class in the RowCounter mapreduce job contains the
following:
{noformat}
@Override
public void map(ImmutableBytesWritable row, Result values,
Context context)
throws IOException {
for (KeyValue value: values.list()) {
if (value.getValue().length 0) {
context.getCounter(Counters.ROWS).increment(1);
break;
}
}
}
{noformat}
The intention is to go through the column values in the row, and increment
the ROWS counter if some value is non-empty. However, values.list() always
has size 1. This is because the createSubmittableJob static method uses a
Scan as follows:
{noformat}
Scan scan = new Scan();
scan.setFilter(new FirstKeyOnlyFilter());
{noformat}
So the input map splits always contain just the first KV. If the column
corresponding to that first KV is empty, even though other columns are
non-empty, that row is skipped.
This way, rowcounter can return an incorrect result.
One way to reproduce this is to create an hbase table with two columns, say
f1:q1 and f2:q2. Create some (say 2) rows with empty f1:q1 but non-empty
f2:q2, and some (say 3) rows with empty f2:q2 and non-empty f1:q1.
Then run rowcounter (specifying only the table but not any columns). The
count will be either 2 short or 3 short.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152965#comment-13152965
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3355
---
This last version pulls in 3.4? Maybe I'm blind but I don't see it. Mind
putting a patch up in JIRA for hadoopqa patch-build to run Andrew? Good stuff.
- Michael
On 2011-11-18 16:05:49, Andrew Purtell wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2837/
bq. ---
bq.
bq. (Updated 2011-11-18 16:05:49)
bq.
bq.
bq. Review request for hbase, Gary Helmling and Eugene Koontz.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
bq.
bq. SASL authentication of ZooKeeper clients with the quorum is handled in the
ZK client independently of HBase concerns. To enable strong ZK authentication,
one must create a suitable JaaS configuration, for example:
bq.
bq.Server {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. storeKey=true
bq. useTicketCache=false
bq. principal=zookeeper/$HOSTNAME;
bq.};
bq.Client {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. useTicketCache=false
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. principal=hbase/$HOSTNAME;
bq.};
bq.
bq. and then configure both the client and server processes to use it, for
example in hbase-site.xml:
bq.
bq.HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeHostFromPrincipal=true
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeRealmFromPrincipal=true
bq.
bq. HBase will then secure all znodes but for a few world-readable read-only
ones needed for clients to look up region locations. All internal cluster
operations will be protected from unauthenticated ZK clients, or clients not
authenticated to the HBase principal. Presumably the only ZK clients
authenticated to the HBase principal will be those embedded in the master and
regionservers.
bq.
bq. There is extraneous whitespace in code surrounding these changes.
bq.
bq.
bq. This addresses bug HBASE-2418.
bq. https://issues.apache.org/jira/browse/HBASE-2418
bq.
bq.
bq. Diffs
bq. -
bq.
bq.pom.xml 382c7c2
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
a75cf87
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2837/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. These changes are running in production at Trend Micro, using a snapshot
build of ZooKeeper 3.4.0.
bq.
bq. New unit test TestZooKeeperACL passes 100 iterations. All test pass not
otherwise currently failing on trunk.
bq.
bq.
bq. Thanks,
bq.
bq. Andrew
bq.
bq.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
security and helping to ensure that services don't interact negatively (touch
each other's data).
[jira] [Commented] (HBASE-4544) Rename RWCC to MVCC
[ https://issues.apache.org/jira/browse/HBASE-4544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152969#comment-13152969 ] stack commented on HBASE-4544: -- Hurray! Rename RWCC to MVCC --- Key: HBASE-4544 URL: https://issues.apache.org/jira/browse/HBASE-4544 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.94.0 Attachments: 4544-v1.txt, 4544-v2.txt ReadWriteConcurrencyControl should be called MultiVersionConcurrencyControl. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4815: -- Description: There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). was: There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to disable to table to alter). Assignee: ramkrishna.s.vasudevan Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4815: -- Status: Patch Available (was: Open) Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4815: -- Attachment: 4815.patch Patch from Ramkrishna Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13152992#comment-13152992 ] Hadoop QA commented on HBASE-4798: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504222/4798_trunk_all.v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 59 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.zookeeper.TestZKLeaderManager org.apache.hadoop.hbase.client.TestShell org.apache.hadoop.hbase.TestFullLogReconstruction Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/293//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/293//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/293//console This message is automatically generated. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Attachment: 4798_trunk_all.v7.patch Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Open (was: Patch Available) Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4798: --- Status: Patch Available (was: Open) Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153000#comment-13153000
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
bq. On 2011-11-18 16:59:52, Michael Stack wrote:
bq. This last version pulls in 3.4? Maybe I'm blind but I don't see it.
Mind putting a patch up in JIRA for hadoopqa patch-build to run Andrew? Good
stuff.
Patch won't apply until 3025 is committed.
- Andrew
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3355
---
On 2011-11-18 16:05:49, Andrew Purtell wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2837/
bq. ---
bq.
bq. (Updated 2011-11-18 16:05:49)
bq.
bq.
bq. Review request for hbase, Gary Helmling and Eugene Koontz.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
bq.
bq. SASL authentication of ZooKeeper clients with the quorum is handled in the
ZK client independently of HBase concerns. To enable strong ZK authentication,
one must create a suitable JaaS configuration, for example:
bq.
bq.Server {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. storeKey=true
bq. useTicketCache=false
bq. principal=zookeeper/$HOSTNAME;
bq.};
bq.Client {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. useTicketCache=false
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. principal=hbase/$HOSTNAME;
bq.};
bq.
bq. and then configure both the client and server processes to use it, for
example in hbase-site.xml:
bq.
bq.HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeHostFromPrincipal=true
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeRealmFromPrincipal=true
bq.
bq. HBase will then secure all znodes but for a few world-readable read-only
ones needed for clients to look up region locations. All internal cluster
operations will be protected from unauthenticated ZK clients, or clients not
authenticated to the HBase principal. Presumably the only ZK clients
authenticated to the HBase principal will be those embedded in the master and
regionservers.
bq.
bq. There is extraneous whitespace in code surrounding these changes.
bq.
bq.
bq. This addresses bug HBASE-2418.
bq. https://issues.apache.org/jira/browse/HBASE-2418
bq.
bq.
bq. Diffs
bq. -
bq.
bq.pom.xml 382c7c2
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
a75cf87
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2837/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. These changes are running in production at Trend Micro, using a snapshot
build of ZooKeeper 3.4.0.
bq.
bq. New unit test TestZooKeeperACL passes 100 iterations. All test pass not
otherwise currently failing on trunk.
bq.
bq.
bq. Thanks,
bq.
bq. Andrew
bq.
bq.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase and Solr
and Neo4j, or multiple HBase instances, etc... having
authentication/authorization on the znodes is important for both
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153021#comment-13153021 ] stack commented on HBASE-3025: -- The TestAdmin was because of this patch Gary (The TestShell seems to have come in w/ the HTable changes yesterday of Lars?) Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153031#comment-13153031
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
bq. On 2011-11-18 04:14:48, Andrew Purtell wrote:
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java,
line 338
bq. https://reviews.apache.org/r/2837/diff/3/?file=59201#file59201line338
bq.
bq. I don't see how to work around this. The code has to be recompiled
against 3.3 or 3.4.
bq.
bq. Any ideas?
bq.
bq. I think we have to pull in 3.4 unconditionally.
bq.
bq. Lars Hofhansl wrote:
bq. Might be a good thing anyway. 3.4 is in RC right now, would probably
be GA by the time we GA 0.92. (we're meeting with Ted Dunning tomorrow -
unrelated - can ask him what he thinks about it).
bq.
bq. Andrew Purtell wrote:
bq. We can put in the other two patches and defer this one until 3.4 is
released. Consequence would be that 3025 can be subverted if one allows direct
client access to the ZK cluster.
bq.
bq. Michael Stack wrote:
bq. +1 on fixing this patch so it just upgrades our zk to 3.4. I just
tried hbase w/ a 3.4 client and a 3.3.3. ensemble and it seems to work. I
asked Mahadev to be sure and he says A 3.3.* client should be able to talk to
3.4.0 server and vice versa. Its as yet unreleased but I'm pretty sure it'll
be out before we ship -- maybe even a 3.4.1 (smile).
bq.
bq. Andrew Purtell wrote:
bq. Say what you will about the Hadoop Record compiler aka Jute, but ZK is
1) the only project that uses it that I know of 2) yet consistently can answer
in the affirmative regarding cross-version compatibility.
Let me add, Mahadev tried it himself on his end too and it seemed fine and then
he did the caveat that 3.4 has not yet been used in production.
- Michael
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3344
---
On 2011-11-18 16:05:49, Andrew Purtell wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2837/
bq. ---
bq.
bq. (Updated 2011-11-18 16:05:49)
bq.
bq.
bq. Review request for hbase, Gary Helmling and Eugene Koontz.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
bq.
bq. SASL authentication of ZooKeeper clients with the quorum is handled in the
ZK client independently of HBase concerns. To enable strong ZK authentication,
one must create a suitable JaaS configuration, for example:
bq.
bq.Server {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. storeKey=true
bq. useTicketCache=false
bq. principal=zookeeper/$HOSTNAME;
bq.};
bq.Client {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. useTicketCache=false
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. principal=hbase/$HOSTNAME;
bq.};
bq.
bq. and then configure both the client and server processes to use it, for
example in hbase-site.xml:
bq.
bq.HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeHostFromPrincipal=true
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeRealmFromPrincipal=true
bq.
bq. HBase will then secure all znodes but for a few world-readable read-only
ones needed for clients to look up region locations. All internal cluster
operations will be protected from unauthenticated ZK clients, or clients not
authenticated to the HBase principal. Presumably the only ZK clients
authenticated to the HBase principal will be those embedded in the master and
regionservers.
bq.
bq. There is extraneous whitespace in code surrounding these changes.
bq.
bq.
bq. This addresses bug HBASE-2418.
bq. https://issues.apache.org/jira/browse/HBASE-2418
bq.
bq.
bq. Diffs
bq. -
bq.
bq.pom.xml 382c7c2
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
a75cf87
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
bq.
[jira] [Created] (HBASE-4819) TestShell broke in trunk; typo
TestShell broke in trunk; typo -- Key: HBASE-4819 URL: https://issues.apache.org/jira/browse/HBASE-4819 Project: HBase Issue Type: Bug Reporter: stack -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4820) Distributed log splitting coding enhancement to make it easier to understand, no semantics change
Distributed log splitting coding enhancement to make it easier to understand, no semantics change - Key: HBASE-4820 URL: https://issues.apache.org/jira/browse/HBASE-4820 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 0.94.0 Reporter: Jimmy Xiang Priority: Minor Fix For: 0.94.0 In reviewing distributed log splitting feature, we found some cosmetic issues. They make the code hard to understand. It will be great to fix them. For this issue, there should be no semantic change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4819) TestShell broke in trunk; typo
[
https://issues.apache.org/jira/browse/HBASE-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153036#comment-13153036
]
stack commented on HBASE-4819:
--
{code}Index: src/main/ruby/hbase/admin.rb
===
--- src/main/ruby/hbase/admin.rb(revision 1203787)
+++ src/main/ruby/hbase/admin.rb(working copy)
@@ -199,7 +199,7 @@
splits[idx] = split.to_java_bytes
idx = idx + 1
end
-elsif arg.kind_of?(Hash) and (arg.has_key?(NUMREGIONS) or
arg.has_key?(SLITALGO))
+elsif arg.kind_of?(Hash) and (arg.has_key?(NUMREGIONS) or
arg.has_key?(SPLITALGO))
raise(ArgumentError, Number of regions must be specified) unless
arg.has_key?(NUMREGIONS)
raise(ArgumentError, Split algorithm must be specified) unless
arg.has_key?(SPLITALGO)
raise(ArgumentError, Number of regions must be geter than 1)
unless arg[NUMREGIONS] 1
{code}
TestShell broke in trunk; typo
--
Key: HBASE-4819
URL: https://issues.apache.org/jira/browse/HBASE-4819
Project: HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.94.0
Attachments: broke_testshell.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4819) TestShell broke in trunk; typo
[ https://issues.apache.org/jira/browse/HBASE-4819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-4819. -- Resolution: Fixed Fix Version/s: 0.94.0 Assignee: stack Committed to trunk. TestShell broke in trunk; typo -- Key: HBASE-4819 URL: https://issues.apache.org/jira/browse/HBASE-4819 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.94.0 Attachments: broke_testshell.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4820) Distributed log splitting coding enhancement to make it easier to understand, no semantics change
[ https://issues.apache.org/jira/browse/HBASE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HBASE-4820: -- Assignee: Jimmy Xiang Distributed log splitting coding enhancement to make it easier to understand, no semantics change - Key: HBASE-4820 URL: https://issues.apache.org/jira/browse/HBASE-4820 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Labels: newbie Fix For: 0.94.0 In reviewing distributed log splitting feature, we found some cosmetic issues. They make the code hard to understand. It will be great to fix them. For this issue, there should be no semantic change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Attachment: 4213-trunk-v5.txt Created InstantSchemaChangeTestBase and separated some tests into TestInstantSchemaChangeSplit Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Status: Open (was: Patch Available) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504174/4213-trunk-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 64 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestInstantSchemaChange org.apache.hadoop.hbase.regionserver.wal.TestLogRolling org.apache.hadoop.hbase.util.TestFSUtils org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/289//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/289//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/289//console This message is automatically generated.) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153065#comment-13153065 ] Hadoop QA commented on HBASE-4815: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504234/4815.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 60 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestShell Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/294//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/294//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/294//console This message is automatically generated. Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153068#comment-13153068 ] stack commented on HBASE-3025: -- hadoopqa is failing some tests because of 'too many open files' Its failing distributedlogsplitting for same reason. I'd say its not your patch. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153070#comment-13153070 ] Hadoop QA commented on HBASE-4798: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504235/4798_trunk_all.v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 33 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 59 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication org.apache.hadoop.hbase.client.TestShell org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/295//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/295//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/295//console This message is automatically generated. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153074#comment-13153074 ] stack commented on HBASE-2856: -- Whoopdee! (Good on you Amit). TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153081#comment-13153081 ] stack commented on HBASE-2856: -- So remaining acid issues are linked from this one? hbase-3543, etc.? TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153084#comment-13153084 ] stack commented on HBASE-4623: -- The output shows up on jenkins and was just 'fixed' -- was a bad constant define. Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.94.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Status: Patch Available (was: Open) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Status: Open (was: Patch Available) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4798) Sleeps and synchronisation improvements for tests
[ https://issues.apache.org/jira/browse/HBASE-4798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153085#comment-13153085 ] nkeywal commented on HBASE-4798: These ones are likely to be in the flaky zone, but I found something better to do so there will be a v8 soon. Sleeps and synchronisation improvements for tests - Key: HBASE-4798 URL: https://issues.apache.org/jira/browse/HBASE-4798 Project: HBase Issue Type: Improvement Components: master, regionserver, test Affects Versions: 0.94.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 4798_trunk_all.v2.patch, 4798_trunk_all.v5.patch, 4798_trunk_all.v6.patch, 4798_trunk_all.v7.patch Multiple small changes: @commiters: Removing some sleeps made visible a bug on JVMClusterUtil#HMaster#waitForServerOnline, so I had to add a synchro point. You may want to review this. JVMClusterUtil#HMaster#waitForServerOnline: removed, the condition was never met (test on !c !!c). Added a new synchronization point. AssignementManager#waitForAssignment: add a timeout on the wait = not stuck if the notification is received before the wait. HMaster#loop: use a notification instead of a 1s sleep HRegionServer#waitForServerOnline: new method used by JVMClusterUtil#waitForServerOnline() to replace a 1s sleep by a notification HRegionServer#getMaster() 1s sleeps replaced by one 0,1s sleep and one 0,2s sleep HRegionServer#stop: use a notification on sleeper to lower shutdown by 0,5s ZooKeeperNodeTracker#start: replace a recursive call by a loop ZooKeeperNodeTracker#blockUntilAvailable: add a timeout on the wait = not stuck if the notification is received before the wait. HBaseTestingUtility#expireSession: use a timeout of 1s instead of 5s TestZooKeeper#testClientSessionExpired: use a timeout of 1s instead of 5s, with the change on HBaseTestingUtility we are 60s faster TestRegionRebalancing#waitForAllRegionsAssigned: use a sleep of 0,2s instead of 1s TestRestartCluster#testClusterRestart: send all the table creation together, then check creation, should be faster TestHLog: shutdown the whole cluster instead of DFS only (more standard) JVMClusterUtil#startup: lower the sleep from 1s to 0,1s HConnectionManager#close: Zookeeper name in debug message from HConnectionManager after connection close was always null because it was set to null in the delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4816) Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out
[ https://issues.apache.org/jira/browse/HBASE-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4816: - Resolution: Fixed Assignee: stack Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed trunk and 0.92 branch (after testing on cluster). Thanks for review Ram. Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out -- Key: HBASE-4816 URL: https://issues.apache.org/jira/browse/HBASE-4816 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.0 Attachments: 4816-v2.txt, 4816.txt A regionserver wouldn't go down because it was waiting on a user region to close only the user-space region had just been opened as part of a split transaction -- it was a new daughter -- just as we'd issued the bulk close to all user regions on receipt of a cluster shutdown call. We need to add a check for this condition -- user tables that did not get the close. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2742) Provide strong authentication with a secure RPC engine
[
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-2742:
-
Attachment: HBASE-2742_addendum.patch
Here's a trivial addendum patch to fix intermittent failures in
TestZKLeaderManager. Previously the test was always using
{{CANDIDATES[0].getWatcher()}} as the ZooKeeperWatcher for verifying data in zk
matched the expected leader. This could fail if {{CANDIDATES[0]}} was a
previous leader stopped by the test.
Provide strong authentication with a secure RPC engine
--
Key: HBASE-2742
URL: https://issues.apache.org/jira/browse/HBASE-2742
Project: HBase
Issue Type: Improvement
Components: ipc
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
Fix For: 0.92.0
Attachments: HBASE-2742_10.patch, HBASE-2742_addendum.patch
The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off
of Hadoop RPC classes, with some performance tweaks added. Those
optimizations have come at a cost in keeping up with Hadoop RPC changes
however, both bug fixes and improvements/new features.
In particular, this impacts how we implement security features in HBase (see
HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487)
relies heavily on RPC changes to support client authentication via kerberos
and securing and mutual authentication of client/server connections via SASL.
Making use of the built-in Hadoop RPC classes will gain us these pieces for
free in a secure HBase.
So, I'm proposing that we drop the HBase forked version of RPC and convert to
direct use of Hadoop RPC, while working to contribute important fixes back
upstream to Hadoop core. Based on a review of the HBase RPC changes, the key
divergences seem to be:
HBaseClient:
- added use of TCP keepalive (HBASE-1754)
- made connection retries and sleep configurable (HBASE-1815)
- prevent NPE if socket == null due to creation failure (HBASE-2443)
HBaseRPC:
- mapping of method names - codes (removed in HBASE-2219)
HBaseServer:
- use of TCP keep alives (HBASE-1754)
- OOME in server does not trigger abort (HBASE-1198)
HbaseObjectWritable:
- allows List serialization
- includes it's own class - code mapping (HBASE-328)
Proposed process is:
1. open issues with patches on Hadoop core for important fixes/adjustments
from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a
pluggable ObjectWritable implementation in RPC.Invocation to allow use of
HbaseObjectWritable).
2. ship a Hadoop version with RPC patches applied -- ideally we should avoid
another copy-n-paste code fork, subject to ability to isolate changes from
impacting Hadoop internal RPC wire formats
3. if all Hadoop core patches are applied we can drop back to a plain vanilla
Hadoop version
I realize there are many different opinions on how to proceed with HBase RPC,
so I'm hoping this issue will kick off a discussion on what the best approach
might be. My own motivation is maximizing re-use of the authentication and
connection security work that's already gone into Hadoop core. I'll put
together a set of patches around #1 and #2, but obviously we need some
consensus around this to move forward. If I'm missing other differences
between HBase and Hadoop RPC, please list as well. Discuss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153113#comment-13153113 ] Ted Yu commented on HBASE-2742: --- +1 on addendum. Thanks Gary. Provide strong authentication with a secure RPC engine -- Key: HBASE-2742 URL: https://issues.apache.org/jira/browse/HBASE-2742 Project: HBase Issue Type: Improvement Components: ipc Reporter: Gary Helmling Assignee: Gary Helmling Priority: Critical Fix For: 0.92.0 Attachments: HBASE-2742_10.patch, HBASE-2742_addendum.patch The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off of Hadoop RPC classes, with some performance tweaks added. Those optimizations have come at a cost in keeping up with Hadoop RPC changes however, both bug fixes and improvements/new features. In particular, this impacts how we implement security features in HBase (see HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487) relies heavily on RPC changes to support client authentication via kerberos and securing and mutual authentication of client/server connections via SASL. Making use of the built-in Hadoop RPC classes will gain us these pieces for free in a secure HBase. So, I'm proposing that we drop the HBase forked version of RPC and convert to direct use of Hadoop RPC, while working to contribute important fixes back upstream to Hadoop core. Based on a review of the HBase RPC changes, the key divergences seem to be: HBaseClient: - added use of TCP keepalive (HBASE-1754) - made connection retries and sleep configurable (HBASE-1815) - prevent NPE if socket == null due to creation failure (HBASE-2443) HBaseRPC: - mapping of method names - codes (removed in HBASE-2219) HBaseServer: - use of TCP keep alives (HBASE-1754) - OOME in server does not trigger abort (HBASE-1198) HbaseObjectWritable: - allows List serialization - includes it's own class - code mapping (HBASE-328) Proposed process is: 1. open issues with patches on Hadoop core for important fixes/adjustments from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a pluggable ObjectWritable implementation in RPC.Invocation to allow use of HbaseObjectWritable). 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid another copy-n-paste code fork, subject to ability to isolate changes from impacting Hadoop internal RPC wire formats 3. if all Hadoop core patches are applied we can drop back to a plain vanilla Hadoop version I realize there are many different opinions on how to proceed with HBase RPC, so I'm hoping this issue will kick off a discussion on what the best approach might be. My own motivation is maximizing re-use of the authentication and connection security work that's already gone into Hadoop core. I'll put together a set of patches around #1 and #2, but obviously we need some consensus around this to move forward. If I'm missing other differences between HBase and Hadoop RPC, please list as well. Discuss! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153127#comment-13153127 ] Gary Helmling commented on HBASE-2742: -- Addendum patch committed to 0.92 branch and trunk. Thanks for review, Ted. Provide strong authentication with a secure RPC engine -- Key: HBASE-2742 URL: https://issues.apache.org/jira/browse/HBASE-2742 Project: HBase Issue Type: Improvement Components: ipc Reporter: Gary Helmling Assignee: Gary Helmling Priority: Critical Fix For: 0.92.0 Attachments: HBASE-2742_10.patch, HBASE-2742_addendum.patch The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off of Hadoop RPC classes, with some performance tweaks added. Those optimizations have come at a cost in keeping up with Hadoop RPC changes however, both bug fixes and improvements/new features. In particular, this impacts how we implement security features in HBase (see HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487) relies heavily on RPC changes to support client authentication via kerberos and securing and mutual authentication of client/server connections via SASL. Making use of the built-in Hadoop RPC classes will gain us these pieces for free in a secure HBase. So, I'm proposing that we drop the HBase forked version of RPC and convert to direct use of Hadoop RPC, while working to contribute important fixes back upstream to Hadoop core. Based on a review of the HBase RPC changes, the key divergences seem to be: HBaseClient: - added use of TCP keepalive (HBASE-1754) - made connection retries and sleep configurable (HBASE-1815) - prevent NPE if socket == null due to creation failure (HBASE-2443) HBaseRPC: - mapping of method names - codes (removed in HBASE-2219) HBaseServer: - use of TCP keep alives (HBASE-1754) - OOME in server does not trigger abort (HBASE-1198) HbaseObjectWritable: - allows List serialization - includes it's own class - code mapping (HBASE-328) Proposed process is: 1. open issues with patches on Hadoop core for important fixes/adjustments from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a pluggable ObjectWritable implementation in RPC.Invocation to allow use of HbaseObjectWritable). 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid another copy-n-paste code fork, subject to ability to isolate changes from impacting Hadoop internal RPC wire formats 3. if all Hadoop core patches are applied we can drop back to a plain vanilla Hadoop version I realize there are many different opinions on how to proceed with HBase RPC, so I'm hoping this issue will kick off a discussion on what the best approach might be. My own motivation is maximizing re-use of the authentication and connection security work that's already gone into Hadoop core. I'll put together a set of patches around #1 and #2, but obviously we need some consensus around this to move forward. If I'm missing other differences between HBase and Hadoop RPC, please list as well. Discuss! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153132#comment-13153132 ] Hadoop QA commented on HBASE-4213: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504247/4213-trunk-v5.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 68 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.client.TestInstantSchemaChange Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/296//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/296//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/296//console This message is automatically generated. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153140#comment-13153140 ] Ted Yu commented on HBASE-4213: --- The test failures were caused by 'Too many open files'. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153181#comment-13153181 ] jirapos...@reviews.apache.org commented on HBASE-3025: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2041/#review3365 --- src/main/ruby/hbase/admin.rb https://reviews.apache.org/r/2041/#comment7545 This is duplicated code from a merge somewhere along the way. I'll remove on commit. - Gary On 2011-11-17 18:48:44, Gary Helmling wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2041/ bq. --- bq. bq. (Updated 2011-11-17 18:48:44) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This patch implements access control list based authorization of HBase operations. The patch depends on the currently posted patch for HBASE-2742 (secure RPC engine). bq. bq. Key parts of the implementation are: bq. bq. * AccessControlLists - encapsulates storage of permission grants in a metadata table (_acl_). This differs from previous implementation where the .META. table was used to store permissions. bq. bq. * AccessController - bq.- implements MasterObserver and RegionObserver, performing authorization checks in each of the preXXX() hooks. If authorization fails, an AccessDeniedException is thrown. bq.- implements AccessControllerProtocol as a coprocessor endpoint to provide RPC methods for granting, revoking and listing permissions. bq. bq. * ZKPermissionWatcher (and TableAuthManager) - synchronizes ACL entries and updates throughout the cluster nodes using ZK. ACL entries are stored in per-table znodes as /hbase/acl/tablename. bq. bq. * Additional ruby shell scripts providing the grant, revoke and user_permission commands bq. bq. * Support for a new OWNER attribute in HTableDescriptor. I could separate out this change into a new JIRA for discussion, but I don't see it as currently useful outside of security. Alternately, I could handle the OWNER attribute completely in AccessController without changing HTD, but that would make interaction via hbase shell a bit uglier. bq. bq. bq. This addresses bug HBASE-3025. bq. https://issues.apache.org/jira/browse/HBASE-3025 bq. bq. bq. Diffs bq. - bq. bq. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java PRE-CREATION bq. security/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/access/TestTablePermissions.java PRE-CREATION bq. security/src/test/java/org/apache/hadoop/hbase/security/access/TestZKPermissionsWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8 bq. src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserver.java 8a40762 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 bq.src/main/resources/hbase-default.xml 6f98f5d bq.src/main/ruby/hbase.rb 4d27191 bq.src/main/ruby/hbase/admin.rb 33cd208 bq.src/main/ruby/hbase/hbase.rb beb2450 bq.src/main/ruby/hbase/security.rb PRE-CREATION bq.src/main/ruby/shell.rb 9a47600 bq.src/main/ruby/shell/commands.rb a352c2e bq.src/main/ruby/shell/commands/grant.rb PRE-CREATION bq.src/main/ruby/shell/commands/revoke.rb PRE-CREATION bq.src/main/ruby/shell/commands/user_permission.rb PRE-CREATION
[jira] [Created] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
A fully automated comprehensive distributed integration test for HBase -- Key: HBASE-4821 URL: https://issues.apache.org/jira/browse/HBASE-4821 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin To properly verify that a particular version of HBase is good for production deployment we need a better way to do real cluster testing after incremental changes. Running unit tests is good, but we also need to deploy HBase to a cluster, run integration tests, load tests, Thrift server tests, kill some region servers, kill the master, and produce a report. All of this needs to happen in 20-30 minutes with minimal manual intervention. I think this way we can combine agile development with high stability of the codebase. I am envisioning a high-level framework written in a scripting language (e.g. Python) that would abstract external operations such as deploy to test cluster, kill a particular server, run load test A, run load test B (we already have a few kinds of load tests implemented in Java, and we could write a Thrift load test in Python). This tool should also produce intermediate output, allowing to catch problems early and restart the test. No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153195#comment-13153195 ] Jonathan Hsieh commented on HBASE-4623: --- @stack Is that fix in reference to HBASE-4973 or specific to this patch? (This one I'm mentioning is specific to a patch on this one). Is there a place to find this without jenkins? Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.94.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153203#comment-13153203 ] Lars Hofhansl commented on HBASE-4213: -- Seems like this is getting ready for check-in. The most important part to have confidence in is whether this change will impact stability even when the feature is switched off. What do you think Ted? Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153208#comment-13153208 ] Jonathan Hsieh commented on HBASE-4623: --- Found the info -- it is in hbase/target/surefire-reports/TEST-org.apache.hadoop.hbase.client.TestShell.xml Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.94.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153220#comment-13153220 ] Ted Yu commented on HBASE-4213: --- I have tested this feature on dev cluster and observed satisfactory behavior. For a 75 region table with 280K rows, instant schema update took 2.2 seconds. HBASE-4815 is introducing a config parameter which would govern online (current) and instant schema changes. If schema change is totally disabled, impact on stability is minimal. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153223#comment-13153223 ] Roman Shaposhnik commented on HBASE-4821: - Bigtop (http://incubator.apache.org/projects/bigtop.html) aims at providing a platform for the creations of exactly the kind of tests you're talking about *across* all the projects of a Hadoop stack. Granted, we're a young project and our test management framework is nowhere near the scope and quality of something like TestNG, but I think it will be helpful to invest in improving it. To give you a quick idea of the baseline architecture, here's what Bigtop testing framework assumes: 1. all tests are implemented as code running on top of JVM. We don't care what language it is (Java, Groovy, Clojure, etc) as long as at the end of the day there's a bunch of class files generated. 2. all tests are packaged/versioned as Maven artifacts 3. all test data is packaged/versioned as Maven artifacts 4. the entry point into test execution is via Junit/TestNG-style 5. tests NEVER concern themselves with deployment (we've got Puppet for that) 6. tests NEVER concern themselves with configuration (we've got Puppet for that) If you think Bigtop can serve as a reasonable platform for what you're trying to accomplish, lets continue this discussion over at bigtop-dev@incubator (and Bigtop JIRAs). A fully automated comprehensive distributed integration test for HBase -- Key: HBASE-4821 URL: https://issues.apache.org/jira/browse/HBASE-4821 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin To properly verify that a particular version of HBase is good for production deployment we need a better way to do real cluster testing after incremental changes. Running unit tests is good, but we also need to deploy HBase to a cluster, run integration tests, load tests, Thrift server tests, kill some region servers, kill the master, and produce a report. All of this needs to happen in 20-30 minutes with minimal manual intervention. I think this way we can combine agile development with high stability of the codebase. I am envisioning a high-level framework written in a scripting language (e.g. Python) that would abstract external operations such as deploy to test cluster, kill a particular server, run load test A, run load test B (we already have a few kinds of load tests implemented in Java, and we could write a Thrift load test in Python). This tool should also produce intermediate output, allowing to catch problems early and restart the test. No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4623: -- Attachment: hbase-4623-0.92.patch hbase-4623.patch Attached version for trunk and 0.92. Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.94.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Attachment: schema-update.png Task status showing the completion of instant schema change Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch, schema-update.png This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153233#comment-13153233 ] Ted Yu commented on HBASE-4213: --- See schema-update.png for screen snapshot. Note, since the schema change completed so fast, I wasn't able to try to trigger region splitting. Also, in CompactSplitThread.requestSplit(final HRegion r, byte[] midKey), call to waitForInflightSchemaChange() has been added. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch, schema-update.png This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Attachment: 4213-trunk-v6.txt Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk-v6.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch, schema-update.png This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153235#comment-13153235 ] jirapos...@reviews.apache.org commented on HBASE-4213: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1786/ --- (Updated 2011-11-18 23:39:40.651341) Review request for Todd Lipcon, Andrew Purtell and Subbu Iyer. Changes --- Latest patch with updated tests. Summary --- bq. From Subbu: here is the latest patch that support alter_instant, an instant schema change command that supports (Add, Modify, Delete column and Modify table) actions through ZK. 1. This pattern capitalizes on the fact that HRI's are now in HDFS and need not be sent over again from Master to RS cloud on every schema change event. 2. Offers real time instant schema change as we bypass the explicit bulk reassign (unassign + assign) of regions from master to RS. 3. Offers fault tolerant schema change support as schema changes now go through ZK. Secondary master taking over a failed schema change will be addressed through a separate JIRA. Diffs (updated) - /pom.xml 1203789 /src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 1202381 /src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 1202381 /src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1202941 /src/main/java/org/apache/hadoop/hbase/master/HMaster.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/MasterServices.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java 1202381 /src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java 1202523 /src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java 1202381 /src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 1203441 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1202941 /src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java 1202381 /src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java PRE-CREATION /src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 1202381 /src/main/resources/hbase-default.xml 1202381 /src/test/java/org/apache/hadoop/hbase/client/InstantSchemaChangeTestBase.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java PRE-CREATION /src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1202381 /src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 1202381 Diff: https://reviews.apache.org/r/1786/diff Testing --- Unit tests pass. Thanks, Ted Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch,
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153237#comment-13153237 ] stack commented on HBASE-4821: -- +1 on at least trying bigtop till we figure why it just won't work for us Mikhail, Accumulo, the other bigtable clone, has an integration suite that is made of python and c hackings. You can check it out here: http://svn.apache.org/viewvc/incubator/accumulo/trunk/test/ I'd like a single integration suite/framework that could be run on amz, locally, on emc's 1k cluster that they are talking of donating to apache, etc. I'd like to help w/ t his project. A fully automated comprehensive distributed integration test for HBase -- Key: HBASE-4821 URL: https://issues.apache.org/jira/browse/HBASE-4821 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin To properly verify that a particular version of HBase is good for production deployment we need a better way to do real cluster testing after incremental changes. Running unit tests is good, but we also need to deploy HBase to a cluster, run integration tests, load tests, Thrift server tests, kill some region servers, kill the master, and produce a report. All of this needs to happen in 20-30 minutes with minimal manual intervention. I think this way we can combine agile development with high stability of the codebase. I am envisioning a high-level framework written in a scripting language (e.g. Python) that would abstract external operations such as deploy to test cluster, kill a particular server, run load test A, run load test B (we already have a few kinds of load tests implemented in Java, and we could write a Thrift load test in Python). This tool should also produce intermediate output, allowing to catch problems early and restart the test. No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4821: - Priority: Critical (was: Major) A fully automated comprehensive distributed integration test for HBase -- Key: HBASE-4821 URL: https://issues.apache.org/jira/browse/HBASE-4821 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical To properly verify that a particular version of HBase is good for production deployment we need a better way to do real cluster testing after incremental changes. Running unit tests is good, but we also need to deploy HBase to a cluster, run integration tests, load tests, Thrift server tests, kill some region servers, kill the master, and produce a report. All of this needs to happen in 20-30 minutes with minimal manual intervention. I think this way we can combine agile development with high stability of the codebase. I am envisioning a high-level framework written in a scripting language (e.g. Python) that would abstract external operations such as deploy to test cluster, kill a particular server, run load test A, run load test B (we already have a few kinds of load tests implemented in Java, and we could write a Thrift load test in Python). This tool should also produce intermediate output, allowing to catch problems early and restart the test. No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4623: - Status: Patch Available (was: Open) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.94.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4623: - Fix Version/s: (was: 0.94.0) 0.92.0 Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153243#comment-13153243 ] Hadoop QA commented on HBASE-4623: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504293/hbase-4623-0.92.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/298//console This message is automatically generated. Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153246#comment-13153246 ] stack commented on HBASE-3025: -- You want more reviews Gary? Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4816) Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out
[ https://issues.apache.org/jira/browse/HBASE-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153253#comment-13153253 ] Hudson commented on HBASE-4816: --- Integrated in HBase-0.92 #144 (See [https://builds.apache.org/job/HBase-0.92/144/]) HBASE-4816 Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out stack : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/SplitRequest.java Regionserver wouldn't go down because split happened exactly at same time we issued bulk user region close call on our way out -- Key: HBASE-4816 URL: https://issues.apache.org/jira/browse/HBASE-4816 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.92.0 Attachments: 4816-v2.txt, 4816.txt A regionserver wouldn't go down because it was waiting on a user region to close only the user-space region had just been opened as part of a split transaction -- it was a new daughter -- just as we'd issued the bulk close to all user regions on receipt of a cluster shutdown call. We need to add a check for this condition -- user tables that did not get the close. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
[ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153252#comment-13153252 ] Hudson commented on HBASE-2742: --- Integrated in HBase-0.92 #144 (See [https://builds.apache.org/job/HBase-0.92/144/]) Amend HBASE-2742 Fix intermittent TestZKLeaderManager failure garyh : Files : * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java Provide strong authentication with a secure RPC engine -- Key: HBASE-2742 URL: https://issues.apache.org/jira/browse/HBASE-2742 Project: HBase Issue Type: Improvement Components: ipc Reporter: Gary Helmling Assignee: Gary Helmling Priority: Critical Fix For: 0.92.0 Attachments: HBASE-2742_10.patch, HBASE-2742_addendum.patch The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off of Hadoop RPC classes, with some performance tweaks added. Those optimizations have come at a cost in keeping up with Hadoop RPC changes however, both bug fixes and improvements/new features. In particular, this impacts how we implement security features in HBase (see HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487) relies heavily on RPC changes to support client authentication via kerberos and securing and mutual authentication of client/server connections via SASL. Making use of the built-in Hadoop RPC classes will gain us these pieces for free in a secure HBase. So, I'm proposing that we drop the HBase forked version of RPC and convert to direct use of Hadoop RPC, while working to contribute important fixes back upstream to Hadoop core. Based on a review of the HBase RPC changes, the key divergences seem to be: HBaseClient: - added use of TCP keepalive (HBASE-1754) - made connection retries and sleep configurable (HBASE-1815) - prevent NPE if socket == null due to creation failure (HBASE-2443) HBaseRPC: - mapping of method names - codes (removed in HBASE-2219) HBaseServer: - use of TCP keep alives (HBASE-1754) - OOME in server does not trigger abort (HBASE-1198) HbaseObjectWritable: - allows List serialization - includes it's own class - code mapping (HBASE-328) Proposed process is: 1. open issues with patches on Hadoop core for important fixes/adjustments from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a pluggable ObjectWritable implementation in RPC.Invocation to allow use of HbaseObjectWritable). 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid another copy-n-paste code fork, subject to ability to isolate changes from impacting Hadoop internal RPC wire formats 3. if all Hadoop core patches are applied we can drop back to a plain vanilla Hadoop version I realize there are many different opinions on how to proceed with HBase RPC, so I'm hoping this issue will kick off a discussion on what the best approach might be. My own motivation is maximizing re-use of the authentication and connection security work that's already gone into Hadoop core. I'll put together a set of patches around #1 and #2, but obviously we need some consensus around this to move forward. If I'm missing other differences between HBase and Hadoop RPC, please list as well. Discuss! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4821) A fully automated comprehensive distributed integration test for HBase
[ https://issues.apache.org/jira/browse/HBASE-4821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153256#comment-13153256 ] Konstantin Boudnik commented on HBASE-4821: --- +1 on BigTop approach: it has been proven once and again to be a consistent and repeatable environment for stacks' (no pun intended ;) integration testing. It has well thought separation of concerns in place (as has been pointed by Roman). Besides, I am not sure I am buying into obsession with Python (or Ruby, etc.) when it comes to work with software written in JVM languages: why one needs to block himself an access to all nice things the platform provides. If a scripting language is desirable - and I can totally buy that - be Groovy with your Java apps ;) A fully automated comprehensive distributed integration test for HBase -- Key: HBASE-4821 URL: https://issues.apache.org/jira/browse/HBASE-4821 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Critical To properly verify that a particular version of HBase is good for production deployment we need a better way to do real cluster testing after incremental changes. Running unit tests is good, but we also need to deploy HBase to a cluster, run integration tests, load tests, Thrift server tests, kill some region servers, kill the master, and produce a report. All of this needs to happen in 20-30 minutes with minimal manual intervention. I think this way we can combine agile development with high stability of the codebase. I am envisioning a high-level framework written in a scripting language (e.g. Python) that would abstract external operations such as deploy to test cluster, kill a particular server, run load test A, run load test B (we already have a few kinds of load tests implemented in Java, and we could write a Thrift load test in Python). This tool should also produce intermediate output, allowing to catch problems early and restart the test. No implementation has yet been done. Any ideas or suggestions are welcome. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153259#comment-13153259 ] stack commented on HBASE-4623: -- Doesn't apply Jon? Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-3025: - Attachment: HBASE-3025_6.patch Attaching final patch. Only change is removal of a small amount of duplicated code in previous patch from admin.rb. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch, HBASE-3025_6.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4815: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Applied branch and trunk. Thanks for patch Ram. Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4815) Disable online altering by default, create a config for it
[ https://issues.apache.org/jira/browse/HBASE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4815: - Attachment: 4815-v2.txt Here is Ram's patch plus adding below to hbase-default.xml: property +namehbase.online.schema.update.enable/name +valuefalse/value +description +Set true to enable online schema changes. This is an experimental feature. +There are known issues modifying table schemas at the same time a region +split is happening so your table needs to be quiescent or else you have to +be running with splits disabled. +/description + /property I think its important to surface this new feature but w/ the caveats. Disable online altering by default, create a config for it -- Key: HBASE-4815 URL: https://issues.apache.org/jira/browse/HBASE-4815 Project: HBase Issue Type: Task Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.92.0 Attachments: 4815-v2.txt, 4815.patch There's a whole class of bugs that we've been revealing from trying out online altering in conjunction with other operations like splitting. HBASE-4729, HBASE-4794, and HBASE-4814 are examples. It's not so much that the online altering code is buggy, but that it wasn't tested in an environment that permits splitting. I think we should mark online altering as experimental in 0.92 and add a config to enable it (so it would be disabled by default, requiring people to enable for altering table schema). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153267#comment-13153267 ] Hadoop QA commented on HBASE-3025: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504305/HBASE-3025_6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 10 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/299//console This message is automatically generated. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch, HBASE-3025_6.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3025) Coprocessor based simple access control
[ https://issues.apache.org/jira/browse/HBASE-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-3025: - Resolution: Fixed Release Note: This is part of an overall implementation of security features for HBase. This issue adds a new AccessController coprocessor which, when enabled, performs authorization checks on all cluster operations, using stored access control lists. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.92 branch and trunk. Thanks again for the reviews. Coprocessor based simple access control --- Key: HBASE-3025 URL: https://issues.apache.org/jira/browse/HBASE-3025 Project: HBase Issue Type: Sub-task Components: coprocessors Reporter: Andrew Purtell Priority: Critical Fix For: 0.92.0 Attachments: HBASE-3025.1.patch, HBASE-3025_5.patch, HBASE-3025_6.patch Thanks for the clarification Jeff which reminds me to edit this issue. Goals of this issue # Client access to HBase is authenticated # User data is private unless access has been granted # Access to data can be granted at a table or per column family basis. Non-Goals of this issue The following items will be left out of the initial implementation for simplicity: # Row-level or per value (cell) This would require broader changes for storing the ACLs inline with rows. It's still a future goal, but would slow down the initial implementation considerably. # Push down of file ownership to HDFS While table ownership seems like a useful construct to start with (at least to lay the groundwork for future changes), making HBase act as table owners when interacting with HDFS would require more changes. In additional, while HDFS file ownership would make applying quotas easy, and possibly make bulk imports more straightforward, it's not clean it would offer a more secure setup. We'll leave this to evaluate in a later phase. # HBase managed roles as collections of permissions We will not model roles internally in HBase to begin with. We will instead allow group names to be granted permissions, which will allow some external modeling of roles via group memberships. Groups will be created and manipulated externally to HBase. While the assignment of permissions to roles and roles to users (or other roles) allows a great deal of flexibility in security policy, it would add complexity to the initial implementation. After the initial implementation, which will appear on this issue, we will evaluate the addition of role definitions internal to HBase in a new JIRA. In this scheme, administrators could assign permissions specifying HDFS groups, and additionally HBase roles. HBase roles would be created and manipulated internally to HBase, and would appear distinct from HDFS groups via some syntactic sugar. HBase role definitions will be allowed to reference other HBase role definitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153271#comment-13153271 ] Jonathan Hsieh commented on HBASE-4623: --- @stack. hbase-4623-0.92.patch doesn't apply on trunk. The robot tried the 0.92 version on trunk. I did the diff for trunk backwards. Fixing. Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4789:
-
Attachment: 4789-v2.txt
Hmmm... I've been running tests w/ this patch in place and have not seen the
issue since. I'm going to commit this unless objection. I don't think this
the 'real' fix. This patch adds a clear of the raw encoded name after we do
the snp-encodedname remove. We shouldn't be having to do this if our
accounting is correct. This patch logs when we the misaccounting so hopefully
we'll be able to figure whats going on down the line. Will do that in
different issue.
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4789:
-
Status: Patch Available (was: Open)
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4623: -- Attachment: hbase-4623.patch Corrected trunk patch. Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4623: -- Attachment: (was: hbase-4623.patch) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4623: -- Status: Open (was: Patch Available) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4623) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92
[ https://issues.apache.org/jira/browse/HBASE-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-4623: -- Status: Patch Available (was: Open) Remove @deprecated Scan methods in 0.90 from TRUNK and 0.92 --- Key: HBASE-4623 URL: https://issues.apache.org/jira/browse/HBASE-4623 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4623-0.92.patch, hbase-4623.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153276#comment-13153276
]
Hadoop QA commented on HBASE-4789:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504307/4789-v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/300//console
This message is automatically generated.
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153269#comment-13153269
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
bq. On 2011-11-18 16:59:52, Michael Stack wrote:
bq. This last version pulls in 3.4? Maybe I'm blind but I don't see it.
Mind putting a patch up in JIRA for hadoopqa patch-build to run Andrew? Good
stuff.
bq.
bq. Andrew Purtell wrote:
bq. Patch won't apply until 3025 is committed.
3025 just went in to 0.92 and trunk, so we should be good to put the patch up
for testing.
- Gary
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3355
---
On 2011-11-18 16:05:49, Andrew Purtell wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2837/
bq. ---
bq.
bq. (Updated 2011-11-18 16:05:49)
bq.
bq.
bq. Review request for hbase, Gary Helmling and Eugene Koontz.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
bq.
bq. SASL authentication of ZooKeeper clients with the quorum is handled in the
ZK client independently of HBase concerns. To enable strong ZK authentication,
one must create a suitable JaaS configuration, for example:
bq.
bq.Server {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. storeKey=true
bq. useTicketCache=false
bq. principal=zookeeper/$HOSTNAME;
bq.};
bq.Client {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. useTicketCache=false
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. principal=hbase/$HOSTNAME;
bq.};
bq.
bq. and then configure both the client and server processes to use it, for
example in hbase-site.xml:
bq.
bq.HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeHostFromPrincipal=true
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeRealmFromPrincipal=true
bq.
bq. HBase will then secure all znodes but for a few world-readable read-only
ones needed for clients to look up region locations. All internal cluster
operations will be protected from unauthenticated ZK clients, or clients not
authenticated to the HBase principal. Presumably the only ZK clients
authenticated to the HBase principal will be those embedded in the master and
regionservers.
bq.
bq. There is extraneous whitespace in code surrounding these changes.
bq.
bq.
bq. This addresses bug HBASE-2418.
bq. https://issues.apache.org/jira/browse/HBASE-2418
bq.
bq.
bq. Diffs
bq. -
bq.
bq.pom.xml 382c7c2
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
a75cf87
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2837/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. These changes are running in production at Trend Micro, using a snapshot
build of ZooKeeper 3.4.0.
bq.
bq. New unit test TestZooKeeperACL passes 100 iterations. All test pass not
otherwise currently failing on trunk.
bq.
bq.
bq. Thanks,
bq.
bq. Andrew
bq.
bq.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically want to protect
their data (ZK znodes) from access by other services (tenants) on the
cluster. Say you are running HBase
[jira] [Updated] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4789:
-
Attachment: 4789-v3.txt
Trunk has changed a little
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789-v3.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4789) On split, parent region is sticking around in oldest sequenceid to region map though not online; we don't cleanup WALs.
[
https://issues.apache.org/jira/browse/HBASE-4789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-4789:
-
Status: Open (was: Patch Available)
On split, parent region is sticking around in oldest sequenceid to region map
though not online; we don't cleanup WALs.
---
Key: HBASE-4789
URL: https://issues.apache.org/jira/browse/HBASE-4789
Project: HBase
Issue Type: Bug
Reporter: stack
Priority: Critical
Fix For: 0.92.0
Attachments: 4789-v2.txt, 4789-v3.txt, 4789.txt
Here is log for a particular region:
{code}
2011-11-15 05:46:31,382 INFO
org.apache.hadoop.hbase.regionserver.SplitTransaction: Still waiting on the
master to process the split for 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:46:31,483 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
regionserver:7003-0x1337b0b92cd000a-0x1337b0b92cd000a Attempting to
transition node 8bbd7388262dc8cb1ce2cf4f04a7281d from RS_ZK_REGION_SPLIT to
RS_ZK_REG
ION_SPLIT
2011-11-15 05:46:31,484 INFO
org.apache.hadoop.hbase.regionserver.SplitRequest: Region split, META
updated, and report to master.
Parent=TestTable,0862220095,1321335865649.8bbd7388262dc8cb1ce2cf4f04a7281d.,
new regions: TestTab
le,0862220095,1321335989689.f00c683df3182d8ef33e315f77ca539c.,
TestTable,0892568091,1321335989689.a56ca1eff5b4401432fcba04b4e851f8.. Split
took 1sec
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-top,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:37,705 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/a56ca1eff5b4401432fcba04b4e851f8/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-top,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9ce16d8fa94e4938964c04775a6fa1a7.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9ce16d8fa94e4938964c04775a6fa1a7-bottom,
keycount=717559, bloomtype=NONE, size=711.1m
2011-11-15 05:46:53,090 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compacting
hdfs://sv4r11s38:7000/hbase/TestTable/f00c683df3182d8ef33e315f77ca539c/info/9213f4d7ee9b4fda857a97603a001f9e.8bbd7388262dc8cb1ce2cf4f04a7281d-
hdfs://sv4r11s38:7000/hbase/TestTable/8bbd7388262dc8cb1ce2cf4f04a7281d/info/9213f4d7ee9b4fda857a97603a001f9e-bottom,
keycount=416691, bloomtype=NONE, size=412.9m
2011-11-15 05:48:00,690 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog:
Found 3 hlogs to remove out of total 12; oldest outstanding sequenceid is
5699 from region 8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=33, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:57:54,083 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
2011-11-15 05:58:01,358 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
Too many hlogs: logs=34, maxlogs=32; forcing flush of 1 regions(s):
8bbd7388262dc8cb1ce2cf4f04a7281d
2011-11-15 05:58:01,359 WARN org.apache.hadoop.hbase.regionserver.LogRoller:
Failed to schedule flush of 8bbd7388262dc8cb1ce2cf4f04a7281dr=null,
requester=null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2856) TestAcidGuarantee broken on trunk
[ https://issues.apache.org/jira/browse/HBASE-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153282#comment-13153282 ] Jonathan Hsieh commented on HBASE-2856: --- I've been looping TestAcidGuarantee's fro about 6 hours now and it is still chugging along and has not failed. I'm going to let it go overnight. (I believe it used to fail within an hour) What are thoughts on backporting this onto the 0.92 branch? (as a separate issue..) TestAcidGuarantee broken on trunk -- Key: HBASE-2856 URL: https://issues.apache.org/jira/browse/HBASE-2856 Project: HBase Issue Type: Bug Affects Versions: 0.89.20100621 Reporter: ryan rawson Assignee: Amitanand Aiyer Priority: Blocker Fix For: 0.94.0 Attachments: 2856-v2.txt, 2856-v3.txt, 2856-v4.txt, 2856-v5.txt, 2856-v6.txt, 2856-v7.txt, 2856-v8.txt, 2856-v9-all-inclusive.txt, acid.txt TestAcidGuarantee has a test whereby it attempts to read a number of columns from a row, and every so often the first column of N is different, when it should be the same. This is a bug deep inside the scanner whereby the first peek() of a row is done at time T then the rest of the read is done at T+1 after a flush, thus the memstoreTS data is lost, and previously 'uncommitted' data becomes committed and flushed to disk. One possible solution is to introduce the memstoreTS (or similarly equivalent value) to the HFile thus allowing us to preserve read consistency past flushes. Another solution involves fixing the scanners so that peek() is not destructive (and thus might return different things at different times alas). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2418) add support for ZooKeeper authentication
[
https://issues.apache.org/jira/browse/HBASE-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13153284#comment-13153284
]
jirapos...@reviews.apache.org commented on HBASE-2418:
--
bq. On 2011-11-18 16:59:52, Michael Stack wrote:
bq. This last version pulls in 3.4? Maybe I'm blind but I don't see it.
Mind putting a patch up in JIRA for hadoopqa patch-build to run Andrew? Good
stuff.
bq.
bq. Andrew Purtell wrote:
bq. Patch won't apply until 3025 is committed.
bq.
bq. Gary Helmling wrote:
bq. 3025 just went in to 0.92 and trunk, so we should be good to put the
patch up for testing.
Rebased patch against current trunk, running tests, will post patch shortly.
- Andrew
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2837/#review3355
---
On 2011-11-18 16:05:49, Andrew Purtell wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2837/
bq. ---
bq.
bq. (Updated 2011-11-18 16:05:49)
bq.
bq.
bq. Review request for hbase, Gary Helmling and Eugene Koontz.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. These changes add support for protecting the state of HBase znodes on a
multi-tenant ZooKeeper cluster. This support requires ZK 3.4.0, currently at
RC2. It is a companion patch to HBASE-2742 (secure RPC), and HBASE-3025
(Coprocessor based access control).
bq.
bq. SASL authentication of ZooKeeper clients with the quorum is handled in the
ZK client independently of HBase concerns. To enable strong ZK authentication,
one must create a suitable JaaS configuration, for example:
bq.
bq.Server {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. storeKey=true
bq. useTicketCache=false
bq. principal=zookeeper/$HOSTNAME;
bq.};
bq.Client {
bq. com.sun.security.auth.module.Krb5LoginModule required
bq. useKeyTab=true
bq. useTicketCache=false
bq. keyTab=/etc/hbase/conf/hbase.keytab
bq. principal=hbase/$HOSTNAME;
bq.};
bq.
bq. and then configure both the client and server processes to use it, for
example in hbase-site.xml:
bq.
bq.HBASE_OPTS=${HBASE_OPTS}
-Djava.security.auth.login.config=/etc/hbase/conf/jaas.conf
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeHostFromPrincipal=true
bq.HBASE_OPTS=${HBASE_OPTS}
-Dzookeeper.kerberos.removeRealmFromPrincipal=true
bq.
bq. HBase will then secure all znodes but for a few world-readable read-only
ones needed for clients to look up region locations. All internal cluster
operations will be protected from unauthenticated ZK clients, or clients not
authenticated to the HBase principal. Presumably the only ZK clients
authenticated to the HBase principal will be those embedded in the master and
regionservers.
bq.
bq. There is extraneous whitespace in code surrounding these changes.
bq.
bq.
bq. This addresses bug HBASE-2418.
bq. https://issues.apache.org/jira/browse/HBASE-2418
bq.
bq.
bq. Diffs
bq. -
bq.
bq.pom.xml 382c7c2
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
05abeb7
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 74b9e62
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
a75cf87
bq.src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java f613ba9
bq.src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperACL.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2837/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. These changes are running in production at Trend Micro, using a snapshot
build of ZooKeeper 3.4.0.
bq.
bq. New unit test TestZooKeeperACL passes 100 iterations. All test pass not
otherwise currently failing on trunk.
bq.
bq.
bq. Thanks,
bq.
bq. Andrew
bq.
bq.
add support for ZooKeeper authentication
Key: HBASE-2418
URL: https://issues.apache.org/jira/browse/HBASE-2418
Project: HBase
Issue Type: Improvement
Components: master, regionserver
Reporter: Patrick Hunt
Assignee: Eugene Koontz
Priority: Critical
Labels: security, zookeeper
Fix For: 0.92.0
Some users may run a ZooKeeper cluster in multi tenant mode meaning that
more than one client service would
like to share a single ZooKeeper service instance (cluster). In this case the
client services typically
[jira] [Updated] (HBASE-4213) Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK.
[ https://issues.apache.org/jira/browse/HBASE-4213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4213: -- Attachment: 4213-trunk-v7.txt Rebased after HBASE-4815 went in. Support for fault tolerant, instant schema updates with out master's intervention (i.e with out enable/disable and bulk assign/unassign) through ZK. Key: HBASE-4213 URL: https://issues.apache.org/jira/browse/HBASE-4213 Project: HBase Issue Type: Improvement Reporter: Subbu M Iyer Assignee: Subbu M Iyer Fix For: 0.94.0 Attachments: 4213-0.92.txt, 4213-0.92.v2, 4213-0.92.v4, 4213-101211-Support_instant_schema_changes_through_ZK.patch, 4213-102511.patch, 4213-Fixed_NPE_in_RS_during_alter_.patch, 4213-Instant_Schema_change_through_ZK.patch, 4213-Nov-2-2011_patch_.patch, 4213-Nov072011-Patch_to_support_concurrent_split_and_alter__.patch, 4213-V10-Support_instant_schema_changes_through_ZK.patch, 4213-V5-Support_instant_schema_changes_through_ZK.patch, 4213-V7-Support_instant_schema_changes_through_ZK.patch, 4213-V8-Support_instant_schema_changes_through_ZK.patch, 4213-V9-Support_instant_schema_changes_through_ZK.patch, 4213-trunk-v2.txt, 4213-trunk-v3.txt, 4213-trunk-v4.txt, 4213-trunk-v5.txt, 4213-trunk-v6.txt, 4213-trunk-v7.txt, 4213-trunk.txt, 4213-v9.txt, 4213.v6, HBASE-4213-Instant_schema_change.patch, HBASE-4213_Instant_schema_change_-Version_2_.patch, HBASE_Instant_schema_change-version_3_.patch, schema-update.png This Jira is a slight variation in approach to what is being done as part of https://issues.apache.org/jira/browse/HBASE-1730 Support instant schema updates such as Modify Table, Add Column, Modify Column operations: 1. With out enable/disabling the table. 2. With out bulk unassign/assign of regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4820) Distributed log splitting coding enhancement to make it easier to understand, no semantics change
[ https://issues.apache.org/jira/browse/HBASE-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-4820: --- Status: Patch Available (was: Open) Distributed log splitting coding enhancement to make it easier to understand, no semantics change - Key: HBASE-4820 URL: https://issues.apache.org/jira/browse/HBASE-4820 Project: HBase Issue Type: Improvement Components: wal Affects Versions: 0.94.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Labels: newbie Fix For: 0.94.0 In reviewing distributed log splitting feature, we found some cosmetic issues. They make the code hard to understand. It will be great to fix them. For this issue, there should be no semantic change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
