[
https://issues.apache.org/jira/browse/TEZ-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511817#comment-16511817
]
ASF GitHub Bot commented on TEZ-3904:
-------------------------------------
GitHub user bjmb opened a pull request:
https://github.com/apache/tez/pull/22
[WIP][TEZ-3904] An API to update tokens for Tez AM and the DAG
This PR has ended up bigger than what I expected so I wanted to ask for
some feedback before going forward. This is what is does mainly:
* Adds `updateAMCredentials` and `updateDAGCredentials` to `TezClient`.
This is not the final API but this functions will be useful anyway to send the
credentials
* Add two corresponding functions to the DAGClientAMProtocol
* Add a `SystemEventHandler` in the runtime internals to process the
`UpdateCredentialsEvent`s
* Add the `UpdateCredentialsEvent` event and some transitions to the
`TaskImpl`'s state machine that represent the credentials being updated.
TODO:
* Add more tests. Will do after the feedback
* Log to history the credentials change?
* I've added some logic to renew the session credentials, these credentials
should be updated as well?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bjmb/tez TEZ-3904
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tez/pull/22.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22
----
commit 6b50f27fda0c1a8698f0aed0f5e63f5f8413ba78
Author: Jaume Marhuenda <jaumemarhuenda@...>
Date: 2018-06-08T18:12:44Z
[TEZ-3904] An API to update tokens for Tez AM and the DAG
----
> an API to update tokens for Tez AM and the DAG
> ----------------------------------------------
>
> Key: TEZ-3904
> URL: https://issues.apache.org/jira/browse/TEZ-3904
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Priority: Major
>
> Nothing is permanent in this world, lest of all delegation tokens.
> The current way around token expiration (the one where you cannot keep
> renewing anymore) in Hive when Tez AM is used in session mode is to cycle Tez
> AM. It may happen though that a query is running at that time, and so the AM
> cannot be restarted with new tokens. We let the query run its course and it
> usually dies because it tries to do something with an expired token.
> To get around that, we cycle AMs a few hours before tokens are going to
> expire.
> However, that is still not ideal because it puts an upper bound on safe Hive
> query runtime (a query longer than 3 hours with current config may fail due
> to an expired token if its timing is unlucky), and also precludes setting
> tokens to expire much faster than the standard 7-day time frame.
> There should be a mechanism to replace tokens in the AM, including for a
> running DAG.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
