Hi Robert, forgive me if I’m wrong, but so far as I understand Flink uses nearly the same model as HDFS (not at all). Means the master receives an action and distribute that to the workers (more or less ;)) HDFS as example uses not an push mechanism, the DN clients fetch the token from the NN when they need them. Could that be a solution, too?
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java <https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java> MapReduce gets the token from the JT with getDelegationToken() http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text) <http://hadoop.apache.org/docs/r2.5.2/api/org/apache/hadoop/mapreduce/Cluster.html#getDelegationToken(org.apache.hadoop.io.Text)> BR, Alexander > On 12 Feb 2015, at 15:28, Robert Metzger <[email protected]> wrote: > > Hi, > > I'm a committer at the Apache Flink project. > One of our users asked for adding support for reading from a secured HDFS > cluster. > > Flink has a master-worker model. Since its not really feasible for users to > login with their kerberos credentials on all workers, I wanted to acquire the > security token on the master and send it to all workers. > For that, I wrote the following code to get the tokens in to a byte array: > > UserGroupInformation.setConfiguration(hdConf); > Credentials credentials = new Credentials(); > UserGroupInformation currUsr = UserGroupInformation.getCurrentUser(); > > Collection<Token<? extends TokenIdentifier>> usrTok = currUsr.getTokens(); > for(Token<? extends TokenIdentifier> token : usrTok) { > final Text id = new Text(token.getIdentifier()); > credentials.addToken(id, token); > } > DataOutputBuffer dob = new DataOutputBuffer(); > credentials.writeTokenStorageToStream(dob); > dob.flush(); > However, the collection currUsr.getTokens() is empty, hence the output buffer > doesn't contain much data. > I suspect that I didn't fully understand the Hadoop security concepts yet. > It would be great if somebody from the list could clarify how to properly > acquire the tokens. > > Also, I was wondering if there is any document describing how the > UserGroupInformation class is working (when is it loading the credentials, > does it only work for Kerberos, ...) > > Best, > Robert >
