[
https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132283#comment-13132283
]
[email protected] commented on HBASE-2742:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1991/
-----------------------------------------------------------
(Updated 2011-10-21 00:14:15.788894)
Review request for hbase.
Changes
-------
Updated patch against current trunk.
* Cleaned out some unnecessary code duplication in SecureClient vs. HBaseClient
* Some javadoc cleanups and additions
* Addressed Andy's review comments
Running tests using SecureRpcEngine requires manually editing
src/test/resources/hbase-site.xml and adding:
<property>
<name>hbase.rpc.engine</name>
<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
</property>
Then run the tests with "mvn test -P hadoop-0.20S"
I'm hacking at the pom.xml to see if I can get it to override the
hbase-site.xml that's used so this isn't necessary.
Summary
-------
This patch creates a new secure RPC engine for HBase, which provides Kerberos
based authentication of clients, and a token-based authentication mechanism for
mapreduce jobs. Primary components of the patch are:
- a new maven profile for secure Hadoop/HBase: hadoop-0.20S
- Secure Hadoop dependent classes are separated under a pseudo-module in the
security/ directory. These source and test directories are only including if
building the secure Hadoop profile
- Currently the security classes get packaged with the regular HBase build
artifacts. We need a way to at least override project.version, so we can
append something like a "-security" suffix indicating the additional security
components.
- The pseudo-module here is really a half-step forward. It enables the
security code to be optionally included in the build for now, and sets up the
structure for a security module. But we still will want to pursue full
modularization (see HBASE-4336), which will allow packing the security code in
a separate build artifact.
- a new RPC engine providing kerberos and token-based authentication:
org.apache.hadoop.hbase.ipc.SecureRpcEngine
- implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
- The implementation classes extend the existing HBaseClient and HBaseServer
to share as much of the RPC code as possible. The main override is of the
connection classes to allow control over the SASL negotiation of secure
connections
- existing RPC changes
- The existing HBaseClient and HBaseServer have been modified to make
subclassing possible
- All references to Hadoop UserGroupInformation have been replaced with
org.apache.hadoop.hbase.security.User to insulate from future dependencies on
specific Hadoop versions
- a coprocessor endpoint for obtaining new authentication tokens:
TokenProvider, and supporting classes for token generation and synchronization
(incorporating HBASE-3615)
- implementation is under
security/src/main/java/org/apache/hadoop/hbase/security/token/
- Secret keys for token generation and verification are synchronized
throughout the cluster in zookeeper, under /hbase/tokenauth/keys
To enable secure RPC, add the following configuration to hbase-site.xml:
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.rpc.engine</name>
<value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
</property>
<property>
<name>hbase.coprocessor.region.classes</name>
<value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
</property>
In addition, the master and regionserver processes must be configured for
kerberos authentication using the properties:
* hbase.(master|regionserver).keytab.file
* hbase.(master|regionserver).kerberos.principal
* hbase.(master|regionserver).kerberos.https.principal
This addresses bug HBASE-2742.
https://issues.apache.org/jira/browse/HBASE-2742
Diffs (updated)
-----
conf/hbase-policy.xml PRE-CREATION
pom.xml bfe8103
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/ZKLeaderManager.java
PRE-CREATION
security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java
PRE-CREATION
security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java
PRE-CREATION
security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HServerAddress.java 94977e0
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java f7fac44
src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107
src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1365411
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 18310d6
src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1db05cb
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 000f99c
src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 965102c
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1f8b629
src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a
src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 60a9248
src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1
src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
ad88b76
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 50b49a6
src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bb67e53
src/main/resources/hbase-default.xml a35e7c7
src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 7194c02
src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java 3982eff
Diff: https://reviews.apache.org/r/1991/diff
Testing
-------
A full test suite run showed failures in TestMasterFailover (timeout) and
TestAvroServer. I'll look into those.
Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and
map reduce token authentication, using YCSB, RowCounter, PerformanceEvaluation.
Thanks,
Gary
> Provide strong authentication with a secure RPC engine
> ------------------------------------------------------
>
> Key: HBASE-2742
> URL: https://issues.apache.org/jira/browse/HBASE-2742
> Project: HBase
> Issue Type: Improvement
> Components: ipc
> Reporter: Gary Helmling
>
> The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off
> of Hadoop RPC classes, with some performance tweaks added. Those
> optimizations have come at a cost in keeping up with Hadoop RPC changes
> however, both bug fixes and improvements/new features.
> In particular, this impacts how we implement security features in HBase (see
> HBASE-1697 and HBASE-2016). The secure Hadoop implementation (HADOOP-4487)
> relies heavily on RPC changes to support client authentication via kerberos
> and securing and mutual authentication of client/server connections via SASL.
> Making use of the built-in Hadoop RPC classes will gain us these pieces for
> free in a secure HBase.
> So, I'm proposing that we drop the HBase forked version of RPC and convert to
> direct use of Hadoop RPC, while working to contribute important fixes back
> upstream to Hadoop core. Based on a review of the HBase RPC changes, the key
> divergences seem to be:
> HBaseClient:
> - added use of TCP keepalive (HBASE-1754)
> - made connection retries and sleep configurable (HBASE-1815)
> - prevent NPE if socket == null due to creation failure (HBASE-2443)
> HBaseRPC:
> - mapping of method names <-> codes (removed in HBASE-2219)
> HBaseServer:
> - use of TCP keep alives (HBASE-1754)
> - OOME in server does not trigger abort (HBASE-1198)
> HbaseObjectWritable:
> - allows List<> serialization
> - includes it's own class <-> code mapping (HBASE-328)
> Proposed process is:
> 1. open issues with patches on Hadoop core for important fixes/adjustments
> from HBase RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a
> pluggable ObjectWritable implementation in RPC.Invocation to allow use of
> HbaseObjectWritable).
> 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid
> another copy-n-paste code fork, subject to ability to isolate changes from
> impacting Hadoop internal RPC wire formats
> 3. if all Hadoop core patches are applied we can drop back to a plain vanilla
> Hadoop version
> I realize there are many different opinions on how to proceed with HBase RPC,
> so I'm hoping this issue will kick off a discussion on what the best approach
> might be. My own motivation is maximizing re-use of the authentication and
> connection security work that's already gone into Hadoop core. I'll put
> together a set of patches around #1 and #2, but obviously we need some
> consensus around this to move forward. If I'm missing other differences
> between HBase and Hadoop RPC, please list as well. Discuss!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
