> On Feb. 11, 2015, 7:33 p.m., kturner wrote: > > docs/src/main/asciidoc/chapters/kerberos.txt, line 318 > > <https://reviews.apache.org/r/30817/diff/4/?file=860348#file860348line318> > > > > these classes are not part of public API
o.a.a.core.cli and o.a.a.core.client get me every time... > On Feb. 11, 2015, 7:33 p.m., kturner wrote: > > docs/src/main/asciidoc/chapters/kerberos.txt, line 320 > > <https://reviews.apache.org/r/30817/diff/4/?file=860348#file860348line320> > > > > What `getToken()` method? From which class/interface? Good catch. > On Feb. 11, 2015, 7:33 p.m., kturner wrote: > > docs/src/main/asciidoc/chapters/kerberos.txt, line 343 > > <https://reviews.apache.org/r/30817/diff/4/?file=860348#file860348line343> > > > > And I suppose Accumulo input and output formats are leveraging this > > secure transport mechanism? Is there every a situation or configuration > > where yarn would not securely transport the token? > > kturner wrote: > I did not mean to make this an issue, it was just a question. Josh feel > free to close this at any time. In the case where Kerberos is being used, these should always be protected by YARN in memory and then back again as the original user when the job runs. I'm not sure how YARN's protection on delegation tokens works if Hadoop/YARN were not running with Kerberos (but Accumulo was). I believe everything would be the same except that the locally stored tokens for the MapReduce job would be stored as the "yarn" user instead of the submitting user which means that a Hadoop administrator would have access to them. I am not 100% sure if that's all of the corner-cases though. - Josh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30817/#review72003 ----------------------------------------------------------- On Feb. 10, 2015, 7:15 p.m., Josh Elser wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30817/ > ----------------------------------------------------------- > > (Updated Feb. 10, 2015, 7:15 p.m.) > > > Review request for accumulo and Christopher Tubbs. > > > Bugs: ACCUMULO-3513 > https://issues.apache.org/jira/browse/ACCUMULO-3513 > > > Repository: accumulo > > > Description > ------- > > DelegationTokens are a way to enable MapReduce jobs (or any distributed task > which does not have direct access to the user's kerberos credentials) to > identify themself and communicate with Accumulo. This is implemented in the > same style of Hadoop which does this to provide HDFS and YARN access in YARN > jobs, as well as HBase (HBase also served as the base when I started coding). > > A DelegationToken (nothing more than a short-lived, on-demand password > generated by Accumulo) is provided to users who request one through the > SecurityOperations API. Secret keys are created, distributed and expired > within Accumulo itself (Master creates and expires keys, distributes via ZK > to tservers), which allow the master to create delegation tokens upon request > and tabletservers to validate delegation tokens provided as the > authentication mechanism by users. Tokens are valid for 7 days by default and > new secret keys are rolled every day (configurable). > > RPC is implemented SASL's DIGEST-MD5 mechanism (in addition to the GSSAPI > support we already have for "normal" kerberos -- thrift makes this really > easy). YARN provides a secure way to pass delegation tokens from the client > JVM to each YARN task (mapper/reducer) to prevent unwanted parties from > stealing the token (password) and using it for their own (nefarious) > purposes. The feature is transparent for users as long as they as using our > MapReduce JCommander options classes. > > User documentation has been added to the kerberos chapter. > > > Diffs > ----- > > core/src/main/java/org/apache/accumulo/core/Constants.java 0229d4e > core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java 216f32d > > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnDefaultTable.java > 0cf081f > > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnRequiredTable.java > 7719e92 > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOpts.java > 4b3b7ed > > core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java > efeafc0 > core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java > 8470da4 > > core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java > feb1ee7 > > core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java > 072724b > > core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java > b83a024 > > core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java > f877ec6 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java > 5c7b780 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java > 5e0aa73 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/impl/DelegationTokenStub.java > PRE-CREATION > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java > b2b5150 > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java > 5405ac0 > > core/src/main/java/org/apache/accumulo/core/client/mock/MockSecurityOperations.java > db88cfb > > core/src/main/java/org/apache/accumulo/core/client/security/tokens/DelegationToken.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/conf/Property.java 68fac73 > > core/src/main/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandler.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java > e067e23 > > core/src/main/java/org/apache/accumulo/core/rpc/SaslDigestCallbackHandler.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java d880fb3 > > core/src/main/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifier.java > PRE-CREATION > core/src/main/java/org/apache/accumulo/core/security/SystemPermission.java > b998179 > core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java > PRE-CREATION > core/src/main/thrift/master.thrift d89e381 > core/src/main/thrift/security.thrift 66235a8 > > core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java > 2723273 > > core/src/test/java/org/apache/accumulo/core/client/security/tokens/DelegationTokenTest.java > PRE-CREATION > > core/src/test/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandlerTest.java > PRE-CREATION > > core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java > 3910f34 > > core/src/test/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifierTest.java > PRE-CREATION > core/src/test/java/org/apache/accumulo/core/util/ThriftMessageUtilTest.java > PRE-CREATION > docs/src/main/asciidoc/chapters/kerberos.txt 05d7384 > fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReader.java > 610b1bd > fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java > 5706cf3 > proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java f9039be > > server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java > 84c3853 > > server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java > bf56a7a > > server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerConnectionParams.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerDigestCallbackHandler.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java > f85505d > server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java > f1f8963 > > server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java > ab106a6 > > server/base/src/main/java/org/apache/accumulo/server/security/AuditedSecurityOperation.java > cc7a7cd > > server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java > 7adb46e > > server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java > 51d50a1 > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationKey.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManager.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManager.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributor.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcher.java > PRE-CREATION > > server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java > 08fa55b > > server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java > 49a60a6 > > server/base/src/test/java/org/apache/accumulo/server/rpc/SaslDigestCallbackHandlerTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/rpc/SaslServerConnectionParamsTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationKeyTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManagerTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManagerTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributorTest.java > PRE-CREATION > > server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcherTest.java > PRE-CREATION > server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java > da0b07c > > server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java > 1d7f90f > > server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java > 6fcdd37 > > server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java > 120692a > server/master/src/main/java/org/apache/accumulo/master/Master.java be476de > > server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java > 72cba26 > server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java > a5675dc > > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousBatchWalker.java > a2687bb > > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java > dba6ac9 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousMoru.java > 4b5c3e7 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousOpts.java > PRE-CREATION > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousQuery.java > 73048f6 > > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousScanner.java > f68377a > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java > 60f8ec2 > test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java > 06b4303 > test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java > 3d48657 > > Diff: https://reviews.apache.org/r/30817/diff/ > > > Testing > ------- > > Many new unit tests. Additional integration tests. Standalone mapreduce jobs > (continuous verify and terasort ingest) on a fully secure cluster. > > > Thanks, > > Josh Elser > >
