-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30817/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 7:13 p.m.)
Review request for accumulo and Christopher Tubbs.
Changes
-------
Introduces DelegationTokenConfig and automatically tries to fetch a DT from a
KT in MapReduce (Thanks to Keith for both suggestions).
Need to re-test mapreduce on a real system and add some more
equals/toString/hashCode and tests for DelegationTokenConfig.
Getting close -- thanks to all for the reviews.
Bugs: ACCUMULO-3513
https://issues.apache.org/jira/browse/ACCUMULO-3513
Repository: accumulo
Description
-------
DelegationTokens are a way to enable MapReduce jobs (or any distributed task
which does not have direct access to the user's kerberos credentials) to
identify themself and communicate with Accumulo. This is implemented in the
same style of Hadoop which does this to provide HDFS and YARN access in YARN
jobs, as well as HBase (HBase also served as the base when I started coding).
A DelegationToken (nothing more than a short-lived, on-demand password
generated by Accumulo) is provided to users who request one through the
SecurityOperations API. Secret keys are created, distributed and expired within
Accumulo itself (Master creates and expires keys, distributes via ZK to
tservers), which allow the master to create delegation tokens upon request and
tabletservers to validate delegation tokens provided as the authentication
mechanism by users. Tokens are valid for 7 days by default and new secret keys
are rolled every day (configurable).
RPC is implemented SASL's DIGEST-MD5 mechanism (in addition to the GSSAPI
support we already have for "normal" kerberos -- thrift makes this really
easy). YARN provides a secure way to pass delegation tokens from the client JVM
to each YARN task (mapper/reducer) to prevent unwanted parties from stealing
the token (password) and using it for their own (nefarious) purposes. The
feature is transparent for users as long as they as using our MapReduce
JCommander options classes.
User documentation has been added to the kerberos chapter.
Diffs (updated)
-----
core/src/main/java/org/apache/accumulo/core/Constants.java 0229d4e
core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java 216f32d
core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnDefaultTable.java
0cf081f
core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnRequiredTable.java
7719e92
core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOpts.java
4b3b7ed
core/src/main/java/org/apache/accumulo/core/client/admin/DelegationTokenConfig.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java
efeafc0
core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java
8470da4
core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java
feb1ee7
core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java
072724b
core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java
b83a024
core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java
f877ec6
core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java
5c7b780
core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java
5e0aa73
core/src/main/java/org/apache/accumulo/core/client/mapreduce/impl/DelegationTokenStub.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java
b2b5150
core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java
5405ac0
core/src/main/java/org/apache/accumulo/core/client/mock/MockSecurityOperations.java
db88cfb
core/src/main/java/org/apache/accumulo/core/client/security/tokens/DelegationToken.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/conf/Property.java 68fac73
core/src/main/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandler.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java
e067e23
core/src/main/java/org/apache/accumulo/core/rpc/SaslDigestCallbackHandler.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java d880fb3
core/src/main/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifier.java
PRE-CREATION
core/src/main/java/org/apache/accumulo/core/security/SystemPermission.java
b998179
core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java
PRE-CREATION
core/src/main/thrift/master.thrift d89e381
core/src/main/thrift/security.thrift 66235a8
core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java
2723273
core/src/test/java/org/apache/accumulo/core/client/security/tokens/DelegationTokenTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandlerTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java
3910f34
core/src/test/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifierTest.java
PRE-CREATION
core/src/test/java/org/apache/accumulo/core/util/ThriftMessageUtilTest.java
PRE-CREATION
docs/src/main/asciidoc/chapters/kerberos.txt 05d7384
fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReader.java 610b1bd
fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java 5706cf3
fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooUtil.java 805bfff
proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java f9039be
server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java
84c3853
server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
bf56a7a
server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerConnectionParams.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerDigestCallbackHandler.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java
f85505d
server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java
f1f8963
server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java
ab106a6
server/base/src/main/java/org/apache/accumulo/server/security/AuditedSecurityOperation.java
cc7a7cd
server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java
7adb46e
server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java
51d50a1
server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationKey.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManager.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManager.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributor.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcher.java
PRE-CREATION
server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java
08fa55b
server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java
49a60a6
server/base/src/test/java/org/apache/accumulo/server/rpc/SaslDigestCallbackHandlerTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/rpc/SaslServerConnectionParamsTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationKeyTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManagerTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManagerTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributorTest.java
PRE-CREATION
server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcherTest.java
PRE-CREATION
server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java
da0b07c
server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java
1d7f90f
server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java
6fcdd37
server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
120692a
server/master/src/main/java/org/apache/accumulo/master/Master.java be476de
server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java
72cba26
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
a5675dc
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousBatchWalker.java
a2687bb
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java
dba6ac9
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousMoru.java
4b5c3e7
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousOpts.java
PRE-CREATION
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousQuery.java
73048f6
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousScanner.java
f68377a
test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java
60f8ec2
test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java
06b4303
test/src/test/java/org/apache/accumulo/test/ShellServerIT.java 2d9166e
test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java
3d48657
Diff: https://reviews.apache.org/r/30817/diff/
Testing
-------
Many new unit tests. Additional integration tests. Standalone mapreduce jobs
(continuous verify and terasort ingest) on a fully secure cluster.
Thanks,
Josh Elser
