[
https://issues.apache.org/jira/browse/YARN-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276106#comment-17276106
]
Prabhu Joseph commented on YARN-10601:
--------------------------------------
[~fritsi] Thanks for the details.
>> As you can see submitApplication is not invoked inside an ugi.doAs block
Why submitApplication is not invoked inside ugi.doAs block. If we need the log
aggregation to happen as per submitterUser, the job also has to be submitted by
submitterUser right?
> The Yarn client should use the UGI who created the Yarn client for obtaining
> a delegation token for the remote log dir
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-10601
> URL: https://issues.apache.org/jira/browse/YARN-10601
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
> Affects Versions: 3.3.0, 3.4.0
> Reporter: Daniel Fritsi
> Priority: Critical
>
> It seems there was a bug introduced in YARN-10333 in this section of
> *{color:#0747A6}{{addLogAggregationDelegationToken}}{color}*:
> {code:java}
> Path remoteRootLogDir = fileController.getRemoteRootLogDir();
> FileSystem fs = remoteRootLogDir.getFileSystem(conf);
> final org.apache.hadoop.security.token.Token<?>[] finalTokens =
> fs.addDelegationTokens(masterPrincipal, credentials);
> {code}
> *{color:#0747A6}{{remoteRootLogDir.getFileSystem}}{color}* simply does this:
> {code:java}
> public FileSystem getFileSystem(Configuration conf) throws IOException {
> return FileSystem.get(this.toUri(), conf);
> }
> {code}
> As far as I know it's customary to create a YarnClient instance via
> *{color:#0747A6}{{YarnClient.createYarnClient()}}{color}* in a
> UserGroupInformation.doAs block if you would like to use it with a different
> user then the current one. E.g.:
> {code:java}
> YarnClient yarnClient = ugi.doAs(new PrivilegedExceptionAction<YarnClient>() {
> @Override
> public YarnClient run() throws Exception {
> YarnClient yarnClient = YarnClient.createYarnClient();
> yarnClient.init(conf);
> yarnClient.start();
> return yarnClient;
> }
> });
> {code}
> If this statement is correct then I think YarnClient should save the
> *{color:#0747A6}{{UserGroupInformation.getCurrentUser()}}{color}* when the
> YarnClient is being created and the
> *{color:#0747A6}{{remoteRootLogDir.getFileSystem(conf)}}{color}* call should
> be made inside an ugi.doAs block with that saved user.
> A more concrete example:
> {code:java}
> public YarnClient createYarnClient(UserGroupInformation ugi, Configuration
> conf) throws Exception {
> return ugi.doAs((PrivilegedExceptionAction<YarnClient>) () -> {
> // Her I am the submitterUser (see below)
> YarnClient yarnClient = YarnClient.createYarnClient();
> yarnClient.init(conf);
> yarnClient.start();
> return yarnClient;
> });
> }
> public void run() {
> // Here I am the serviceUser
> // ...
> Configuration conf = ...
> // ...
> UserGroupInformation ugi = getSubmitterUser();
> // ...
> YarnClient yarnClient = createYarnClient(ugi);
> // ...
> ApplicationSubmissionContext context = ...
> // ...
> yarnClient.submitApplication(context);
> }
> {code}
> As you can see *{color:#0747A6}{{submitApplication}}{color}* is not invoked
> inside an ugi.doAs block and submitApplication is the one who will eventually
> invoke *{color:#0747A6}{{addLogAggregationDelegationToken}}{color}*. That's
> why we need to save the UGI during the YarnClient creation and create the
> FileSystem instance inside an ugi.doAs with that saved user. Otherwise Yarn
> will try to get a delegation token with an incorrect user (serviceUser)
> instead of the submitterUser.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
