On 13 Sep 2017, at 14:03, Srikrishan Malik <malik....@gmail.com<mailto:malik....@gmail.com>> wrote:
Hello, I was trying to understand how impersonation works in hadoop environment. I found a few resources like: About doAs and proxy users: http://dewoods.com/blog/hadoop-kerberos-guide and about tokens: https://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/ .. alsohttps://www.gitbook.com/book/steveloughran/kerberos_and_hadoop/details But I was not able to connect all the dots wrt the full flow of operations. My current understanding is : 1. user does a kinit and executes a end user facing program like beeline, spark-submit etc. 2. The program is app specific and gets service tickets for HDFS yes 3. It then gets tokens for all the services it may need during the job yes exeution and saves the tokens in an HDFS directory. or includes them in IPC calls 4. The program then connects a job executer(using a service ticket for the job executer??) e.g. yarn with the job info and the token path. If you are using Hadoop RPC, you can include credentials (Kerberos tickets, hadoop tokens) in the IPC call (which are encrypted if wire encryption is turned on) 5. The job executor get the tocken and initializes UGI and all communication with HDFS is done using the token and kerberos ticket are not used. if you want to talk with the identify of the user from an RPC call, you just use UGI.getCurrentUser.doAs(), as the RPC call will be mapped to the caller before your code is invoked Is the above high level understanding correct? (I have more follow up queries.) nobody really understands it. Nobody understands Kerberos either. Stepping through with a debugger always helps. Can the token mecahnism be skipped and use only kerberos at each layer, if so, any resources will help. yes My final aim is to write a spark connector with impersonation support for an data storage system which does not use hadoop(tokens) but supports kerberos. Spark yarn job submit takes a list of filesystems to get K-tickets for, includes it in job setup. Also looks for Hive and HBase if configured. Long lived spark jobs (streaming) take a keytab; the app master re-auths with the KDC reguarly, then pushes tokens out to the workers
