[
https://issues.apache.org/jira/browse/OOZIE-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898343#comment-15898343
]
Hadoop QA commented on OOZIE-2807:
----------------------------------
Testing JIRA OOZIE-2807
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
. {color:red}-1{color} HEAD does not compile
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3672/
> Oozie gets RM delegation token even for checking job status
> -----------------------------------------------------------
>
> Key: OOZIE-2807
> URL: https://issues.apache.org/jira/browse/OOZIE-2807
> Project: Oozie
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Satish Subhashrao Saley
> Fix For: 5.0.0
>
> Attachments: OOZIE-2807-1.patch, OOZIE-2807-2.patch,
> OOZIE-2807-3.patch
>
>
> We had one user submitting way too many workflows with single hive query -
> ~3600 workflows running concurrently. Surprisingly Oozie held up well without
> issues.
> But [~daryn] from our hadoop team saw that the amount of delegation tokens
> fetched by Oozie was very high compared to actual number of jobs submitted
> and was stressing RM with the calls and also pushing it close to its memory
> limits. This is because we are fetching the delegation token every time we
> create a JobClient instead of only during job submission.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L503-L519
> So for one job we fetch
> 1) 1 token during submission
> 2) 1 token every 5 minutes when we check status of job
> 3) 1 token after the job ends to retrieve status.
> 4) 1 token if we are killing the job.
> So for a job running for 11 minutes, we would have fetched the token 4 times.
> May be more in other cases like mapreduce where we check for end of launcher
> and child job.
> Only 1 out of the token (used in the job submission) will be cancelled after
> job completes. Other tokens are kind of leaked and will only be cleaned up by
> RM after the expiry period (24 hrs is default). This can make RM go out of
> memory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
