> On Feb. 10, 2015, 6:07 p.m., Christopher Tubbs wrote: > > core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java, > > line 173 > > <https://reviews.apache.org/r/30817/diff/1/?file=859803#file859803line173> > > > > Why does OutputFormat depend on utils for InputFormat? Should probably > > push this method down to ConfiguratorBase.
Good point. Done. > On Feb. 10, 2015, 6:07 p.m., Christopher Tubbs wrote: > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java, > > line 88 > > <https://reviews.apache.org/r/30817/diff/1/?file=859804#file859804line88> > > > > It's not just for PasswordToken. It's for any serialized token, which > > might be provided by an add-on package. Do we care about any hypothetical tokens that may or may not exist? My main concern is noting that DelegationTokens (and KerberosTokens, although they're not relevant at all really) are not serialized in the Configuration. Recommendations on how you would prefer this to read? > On Feb. 10, 2015, 6:07 p.m., Christopher Tubbs wrote: > > core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java, > > line 87 > > <https://reviews.apache.org/r/30817/diff/1/?file=859812#file859812line87> > > > > Would be helpful if review did not include thrift generated classes > > (easier to evaluate). Uhh, I'll see if I can rip that out of the patch.. - Josh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30817/#review71810 ----------------------------------------------------------- On Feb. 10, 2015, 6:22 p.m., Josh Elser wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30817/ > ----------------------------------------------------------- > > (Updated Feb. 10, 2015, 6:22 p.m.) > > > Review request for accumulo and Christopher Tubbs. > > > Bugs: ACCUMULO-3513 > https://issues.apache.org/jira/browse/ACCUMULO-3513 > > > Repository: accumulo > > > Description > ------- > > DelegationTokens are a way to enable MapReduce jobs (or any distributed task > which does not have direct access to the user's kerberos credentials) to > identify themself and communicate with Accumulo. This is implemented in the > same style of Hadoop which does this to provide HDFS and YARN access in YARN > jobs, as well as HBase (HBase also served as the base when I started coding). > > A DelegationToken (nothing more than a short-lived, on-demand password > generated by Accumulo) is provided to users who request one through the > SecurityOperations API. Secret keys are created, distributed and expired > within Accumulo itself (Master creates and expires keys, distributes via ZK > to tservers), which allow the master to create delegation tokens upon request > and tabletservers to validate delegation tokens provided as the > authentication mechanism by users. Tokens are valid for 7 days by default and > new secret keys are rolled every day (configurable). > > RPC is implemented SASL's DIGEST-MD5 mechanism (in addition to the GSSAPI > support we already have for "normal" kerberos -- thrift makes this really > easy). YARN provides a secure way to pass delegation tokens from the client > JVM to each YARN task (mapper/reducer) to prevent unwanted parties from > stealing the token (password) and using it for their own (nefarious) > purposes. The feature is transparent for users as long as they as using our > MapReduce JCommander options classes. > > User documentation has been added to the kerberos chapter. > > > Diffs > ----- > > core/src/main/java/org/apache/accumulo/core/Constants.java 0229d4e > core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java 216f32d > > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnDefaultTable.java > 0cf081f > > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnRequiredTable.java > 7719e92 > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOpts.java > 4b3b7ed > > core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java > efeafc0 > core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java > 8470da4 > > core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java > feb1ee7 > > core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java > 072724b > > core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java > b83a024 > > core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java > f877ec6 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java > 5c7b780 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java > 5e0aa73 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/impl/DelegationTokenStub.java > PRE-CREATION > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java > b2b5150 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java > 5405ac0 > > core/src/main/java/org/apache/accumulo/core/client/mock/MockSecurityOperations.java > db88cfb > > core/src/main/java/org/apache/accumulo/core/client/security/tokens/DelegationToken.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/conf/Property.java 68fac73 > > core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java > f5cfdd5 > > core/src/main/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandler.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java > e067e23 > > core/src/main/java/org/apache/accumulo/core/rpc/SaslDigestCallbackHandler.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java d880fb3 > > core/src/main/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifier.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/security/SystemPermission.java > b998179 > > core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationKey.java > PRE-CREATION > > core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationTokenIdentifier.java > PRE-CREATION > > core/src/main/java/org/apache/accumulo/core/security/thrift/TDelegationToken.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java > PRE-CREATION > core/src/main/thrift/master.thrift d89e381 > core/src/main/thrift/security.thrift 66235a8 > > core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java > 2723273 > > core/src/test/java/org/apache/accumulo/core/client/security/tokens/DelegationTokenTest.java > PRE-CREATION > > core/src/test/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandlerTest.java > PRE-CREATION > > core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java > 3910f34 > > core/src/test/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifierTest.java > PRE-CREATION > core/src/test/java/org/apache/accumulo/core/util/ThriftMessageUtilTest.java > PRE-CREATION > docs/src/main/asciidoc/chapters/kerberos.txt 05d7384 > fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReader.java > 610b1bd > fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java > 5706cf3 > proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java f9039be > > server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java > 84c3853 > > server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java > bf56a7a > > server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerConnectionParams.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerDigestCallbackHandler.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java > f85505d > server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java > f1f8963 > > server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java > ab106a6 > > server/base/src/main/java/org/apache/accumulo/server/security/AuditedSecurityOperation.java > cc7a7cd > > server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java > 7adb46e > > server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java > 51d50a1 > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationKey.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManager.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManager.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributor.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcher.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java > 08fa55b > > server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java > 49a60a6 > > server/base/src/test/java/org/apache/accumulo/server/rpc/SaslDigestCallbackHandlerTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/rpc/SaslServerConnectionParamsTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationKeyTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManagerTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManagerTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributorTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcherTest.java > PRE-CREATION > server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java > da0b07c > > server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java > 1d7f90f > > server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java > 6fcdd37 > > server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java > 120692a > server/master/src/main/java/org/apache/accumulo/master/Master.java be476de > > server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java > 72cba26 > server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java > a5675dc > > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousBatchWalker.java > a2687bb > > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java > dba6ac9 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousMoru.java > 4b5c3e7 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousOpts.java > PRE-CREATION > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousQuery.java > 73048f6 > > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousScanner.java > f68377a > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java > 60f8ec2 > test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java > 06b4303 > test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java > 3d48657 > > Diff: https://reviews.apache.org/r/30817/diff/ > > > Testing > ------- > > Many new unit tests. Additional integration tests. Standalone mapreduce jobs > (continuous verify and terasort ingest) on a fully secure cluster. > > > Thanks, > > Josh Elser > >
