otterc commented on PR #58190: URL: https://github.com/apache/spark/pull/58190#issuecomment-5373339653
> UGI also carries credentials. Without properly set up UGI doAs, how are the credentials propagated then? Yes, credentials are still correct. Credentials are attached to whatever UGI transferCredentials copies them into - changing the UGI's name does not change the delegation tokens it holds or the identity encoded in those tokens. Flow: - When the driver's login UGI is initialized, it reads HADOOP_TOKEN_FILE_LOCATION and loads the delegation tokens. For Spark-generated credentials, those tokens were obtained at submission time for the proxy user: the submitter authenticates with Kerberos and Spark obtains delegation tokens for the impersonated user. - SparkHadoopUtil.createSparkUser() calls UGI.createRemoteUser(<name>) and transferCredentials(UGI.getCurrentUser(), newUgi). - **Without the fix**: `SPARK_USER=submitter`, so Spark creates a UGI named submitter, but that UGI contains delegation tokens issued for alice. The credentials still authenticate as alice, but the UGI's local identity is submitter. - **With the fix**: `SPARK_USER=alice`, so Spark creates a UGI named alice containing delegation tokens issued for alice. The UGI identity and token identity are aligned, with no change to credential propagation. I can look into adding an integration test for this. Need to check what's feasible with the current K8s integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
