[
https://issues.apache.org/jira/browse/OOZIE-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy updated OOZIE-2797:
--------------------------------------
Description:
Pig and hive clients when run from command line take care of canceling HCatalog
delegation tokens. But they do not get cancelled when run via Oozie. RM only
supports cancellation of hdfs and yarn tokens on job completion. So when
running with Oozie, it is left to the HCatalog server to remove the token after
the expiry. When there are too many jobs, the number of unexpired tokens can
overwhelm the hcat server and cause outages. Oozie should take care of
canceling these tokens after the job is done to avoid that.
We should add a token cancellation framework in the new Oozie AM to support
implementations for token cancellation for different credential types similar
to CredentialsProvider.
Attaching a reference patch that we used internally for directly canceling
hcat tokens for just pig and hive actions that could be used by anyone who
needs it till OOZIE-1770 is done and the new token cancellation framework is
added by this jira.
was:
HCat delegation token does not get cancelled unless HCat cancels the token on
its own after several days (i.e. depending on its configuration).
Hive Action and Pig Action should cancel delegation token after completion.
Currently, we can fix it by cancelling the tokens in finally block for Hive and
Pig Action. I am putting patch for this.
But, a cleaner solution would be to have a class specifying APIs for
cancellation and for each type of token, there would be a separate
implementation. We can do it for Oozie 5.0.0.
bq. Maybe a follow-up JIRA should be linked to this one to be clear about what
is the purpose of OOZIE-2797 and what will be handled separately.
Updated jira title and description to reflect what this jira is for.
> Add a framework for cancellation of Delegation Tokens in Oozie AM
> -----------------------------------------------------------------
>
> Key: OOZIE-2797
> URL: https://issues.apache.org/jira/browse/OOZIE-2797
> Project: Oozie
> Issue Type: Bug
> Reporter: Satish Subhashrao Saley
> Assignee: Satish Subhashrao Saley
> Fix For: 5.0.0
>
> Attachments: OOZIE-2797-1.patch
>
>
> Pig and hive clients when run from command line take care of canceling
> HCatalog delegation tokens. But they do not get cancelled when run via Oozie.
> RM only supports cancellation of hdfs and yarn tokens on job completion. So
> when running with Oozie, it is left to the HCatalog server to remove the
> token after the expiry. When there are too many jobs, the number of unexpired
> tokens can overwhelm the hcat server and cause outages. Oozie should take
> care of canceling these tokens after the job is done to avoid that.
> We should add a token cancellation framework in the new Oozie AM to support
> implementations for token cancellation for different credential types similar
> to CredentialsProvider.
> Attaching a reference patch that we used internally for directly canceling
> hcat tokens for just pig and hive actions that could be used by anyone who
> needs it till OOZIE-1770 is done and the new token cancellation framework is
> added by this jira.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
