[
https://issues.apache.org/jira/browse/TEZ-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955549#comment-14955549
]
Jason Lowe commented on TEZ-2886:
---------------------------------
For example, the RM will automatically add tokens for the log aggregation
filesystem if they are missing from the job's credentials. We have some
clusters setup where the log aggregation filesystem is not the default
filesystem, and the RM is automatically adding the token for the remote log
aggregation filesystem. In this case Tez logs for any tasks are not aggregated
because only the AM container has the necessary log aggregation filesystem
token. Other containers only have the DAG credentials, and since the client
was unaware of the log aggregation filesystem it doesn't pass it as part of the
DAG credentials.
One could argue that the client should look to see if the log aggregation
filesystem is not the default filesystem, but clients don't necessarily have
the correct setting for those configs (it's a YARN server config and not a
client-side config). Therefore it's not appropriate for the client to try to
work around this.
This may be related to TEZ-1588 since I see that patch was also proposing to
merge the AM credentials with the DAG credentials in the DAGAppMaster.
> Ability to merge AM credentials with DAG credentials
> ----------------------------------------------------
>
> Key: TEZ-2886
> URL: https://issues.apache.org/jira/browse/TEZ-2886
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Jason Lowe
>
> Currently AM credentials are explicitly kept separate from DAG credentials,
> but this can cause problems when credentials are automatically added to the
> application as part of the submission process but outside of the client's
> knowledge. We need the ability for the AM's credentials to be merged with
> the DAG credentials so DAGs can pick up important credentials that were not
> submitted by the client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
