Hi, thank you for the quick reply. I'll look into the links to see if we can implement a similar mechanism.
Robert On Thu, Feb 12, 2015 at 6:19 PM, Alexander Alten-Lorenz <[email protected] > wrote: > Hi Robert, > > forgive me if I’m wrong, but so far as I understand Flink uses nearly the > same model as HDFS (not at all). Means the master receives an action and > distribute that to the workers (more or less ;)) > HDFS as example uses not an push mechanism, the DN clients fetch the token > from the NN when they need them. Could that be a solution, too? > > > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java > > MapReduce gets the token from the JT with getDelegationToken() > > > http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text) > > > BR, > Alexander > > > On 12 Feb 2015, at 15:28, Robert Metzger <[email protected]> wrote: > > Hi, > > I'm a committer at the Apache Flink project. > One of our users asked for adding support for reading from a secured HDFS > cluster. > > Flink has a master-worker model. Since its not really feasible for users > to login with their kerberos credentials on all workers, I wanted to > acquire the security token on the master and send it to all workers. > For that, I wrote the following code to get the tokens in to a byte array: > > UserGroupInformation.setConfiguration(hdConf); > Credentials credentials = new Credentials(); > UserGroupInformation currUsr = UserGroupInformation.getCurrentUser(); > > Collection<Token<? extends TokenIdentifier>> usrTok = currUsr.getTokens(); > for(Token<? extends TokenIdentifier> token : usrTok) { > final Text id = new Text(token.getIdentifier()); > credentials.addToken(id, token); > } > DataOutputBuffer dob = new DataOutputBuffer(); > credentials.writeTokenStorageToStream(dob); > dob.flush(); > > However, the collection currUsr.getTokens() is empty, hence the output buffer > doesn't contain much data. > > I suspect that I didn't fully understand the Hadoop security concepts yet. > It would be great if somebody from the list could clarify how to properly > acquire the tokens. > > Also, I was wondering if there is any document describing how the > UserGroupInformation class is working (when is it loading the credentials, > does it only work for Kerberos, ...) > > > Best, > > Robert > > > >
