[
https://issues.apache.org/jira/browse/OOZIE-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881914#comment-15881914
]
Hadoop QA commented on OOZIE-2807:
----------------------------------
Testing JIRA OOZIE-2807
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [50] new bugs found below threshold in total
that must be fixed.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:red}-1{color} There are [2] new bugs found below threshold in
[sharelib/spark] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/spark/findbugs-new.html
. The most important FindBugs errors are:
. At SparkMain.java:[line 477]:
org.apache.oozie.action.hadoop.SparkMain.getJarVersion(File) may fail to close
stream
. At SparkMain.java:[line 300]: Exceptional return value of
java.io.File.mkdirs() ignored in
org.apache.oozie.action.hadoop.SparkMain.createPySparkLibFolder()
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in
[hadooplibs/hadoop-utils-2].
. {color:red}-1{color} There are [6] new bugs found below threshold in
[examples] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
examples/findbugs-new.html
. The most important FindBugs errors are:
. At DateList.java:[line 55]: Nullcheck of date at line 55 of value
previously dereferenced in org.apache.oozie.example.DateList.main(String[])
. Private method org.apache.oozie.example.DateList.formatDateUTC(Calendar)
is never called: Redundant null check at DateList.java:[line 62]
. org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail to
clean up java.io.InputStream: At DateList.java:[line 97]
. Path continues at LocalOozieExample.java:[line 76]: Obligation to clean up
resource created at LocalOozieExample.java:[line 72] is not discharged
. Path continues at LocalOozieExample.java:[line 78]: Path continues at
LocalOozieExample.java:[line 77]
. Path continues at LocalOozieExample.java:[line 88]: Path continues at
LocalOozieExample.java:[line 81]
. Path continues at LocalOozieExample.java:[line 91]: Path continues at
LocalOozieExample.java:[line 89]
. org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail to
close stream: Path continues at LocalOozieExample.java:[line 100]
. org.apache.oozie.example.Repeatable.getBaseline() may expose internal
representation by returning Repeatable.baseline: At
LocalOozieExample.java:[line 72]
. org.apache.oozie.example.Repeatable.setBaseline(Date) may expose internal
representation by storing an externally mutable object into
Repeatable.baseline: At Repeatable.java:[line 168]
. At Repeatable.java:[line 172]
. {color:red}-1{color} There are [42] new bugs found below threshold in
[tools] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
. The most important FindBugs errors are:
. At OozieDBCLI.java:[line 548]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInMysql(String, Connection):
new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 577]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInPostgres(String,
Connection, String): new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 983]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.createOozieSysTable(String, boolean, String):
new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 759]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.ddlTweaks(String, boolean): new
java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 712]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.ddlTweaksFor50(String, boolean, String): new
java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 344]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.postUpgradeTasks(String, boolean, boolean):
new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 420]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.postUpgradeTasksFor40(String, boolean): new
java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 258]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.upgradeOozieDBVersion(String, boolean,
String): new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 679]:
org.apache.oozie.tools.OozieDBCLI.getAddColumnQuery(String, String, String)
invokes inefficient new String(String) constructor
. At OozieDBCLI.java:[line 675]:
org.apache.oozie.tools.OozieDBCLI.getDropColumnQuery(String, String) invokes
inefficient new String(String) constructor
. At OozieDBCLI.java:[line 671]:
org.apache.oozie.tools.OozieDBCLI.getRenameColumnQuery(String, String, String)
invokes inefficient new String(String) constructor
. At OozieDBCLI.java:[line 69]: org.apache.oozie.tools.OozieDBCLI.HELP_INFO
should be package protected
. Obligation to clean up resource created at OozieDBCLI.java:[line 721] is
not discharged: org.apache.oozie.tools.OozieDBCLI.ddlTweaksFor50(String,
boolean, String) may fail to clean up java.sql.Statement
. Path continues at OozieDBCLI.java:[line 728]: Path continues at
OozieDBCLI.java:[line 722]
. Path continues at OozieDBCLI.java:[line 730]: Path continues at
OozieDBCLI.java:[line 729]
. org.apache.oozie.tools.OozieDBCLI.checkDBExists() may fail to clean up
java.sql.ResultSet on checked exception: Path continues at
OozieDBCLI.java:[line 734]
. Path continues at OozieDBCLI.java:[line 892]: Obligation to clean up
resource created at OozieDBCLI.java:[line 891] is not discharged
. Path continues at OozieDBCLI.java:[line 898]: Path continues at
OozieDBCLI.java:[line 897]
. org.apache.oozie.tools.OozieDBCLI.checkDBExists() may fail to clean up
java.sql.Statement on checked exception: Path continues at
OozieDBCLI.java:[line 901]
. Path continues at OozieDBCLI.java:[line 891]: Obligation to clean up
resource created at OozieDBCLI.java:[line 890] is not discharged
. Path continues at OozieDBCLI.java:[line 897]: Path continues at
OozieDBCLI.java:[line 892]
. Path continues at OozieDBCLI.java:[line 901]: Path continues at
OozieDBCLI.java:[line 898]
. Obligation to clean up resource created at OozieDBCLI.java:[line 550] is
not discharged:
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInMysql(String, Connection)
may fail to clean up java.sql.Statement on checked exception
. Path continues at OozieDBCLI.java:[line 552]: Path continues at
OozieDBCLI.java:[line 551]
. Path continues at OozieDBCLI.java:[line 554]: Path continues at
OozieDBCLI.java:[line 553]
. Path continues at OozieDBCLI.java:[line 556]: Path continues at
OozieDBCLI.java:[line 555]
. Path continues at OozieDBCLI.java:[line 560]: Path continues at
OozieDBCLI.java:[line 559]
. Path continues at OozieDBCLI.java:[line 562]: Path continues at
OozieDBCLI.java:[line 561]
. Obligation to clean up resource created at OozieDBCLI.java:[line 998] is
not discharged: org.apache.oozie.tools.OozieDBCLI.createOozieSysTable(String,
boolean, String) may fail to clean up java.sql.Statement on checked exception
. Path continues at OozieDBCLI.java:[line 1007]: Path continues at
OozieDBCLI.java:[line 999]
. Path continues at OozieDBCLI.java:[line 1011]: Path continues at
OozieDBCLI.java:[line 1008]
. org.apache.oozie.tools.OozieDBCLI.ddlTweaks(String, boolean) may fail to
clean up java.sql.Statement on checked exception: Path continues at
OozieDBCLI.java:[line 1012]
. Path continues at OozieDBCLI.java:[line 835]: Obligation to clean up
resource created at OozieDBCLI.java:[line 834] is not discharged
. Path continues at OozieDBCLI.java:[line 856]: Path continues at
OozieDBCLI.java:[line 855]
. Obligation to clean up resource created at OozieDBCLI.java:[line 946] is
not discharged: org.apache.oozie.tools.OozieDBCLI.getOozieDBVersion() may fail
to clean up java.sql.ResultSet on checked exception
. Path continues at OozieDBCLI.java:[line 956]: Path continues at
OozieDBCLI.java:[line 947]
. Path continues at OozieDBCLI.java:[line 960]: Path continues at
OozieDBCLI.java:[line 957]
. Obligation to clean up resource created at OozieDBCLI.java:[line 945] is
not discharged: org.apache.oozie.tools.OozieDBCLI.getOozieDBVersion() may fail
to clean up java.sql.Statement on checked exception
. Path continues at OozieDBCLI.java:[line 947]: Path continues at
OozieDBCLI.java:[line 946]
. Path continues at OozieDBCLI.java:[line 957]: Path continues at
OozieDBCLI.java:[line 956]
. org.apache.oozie.tools.OozieDBCLI.postUpgradeTasks(String, boolean,
boolean) may fail to clean up java.sql.Statement on checked exception: Path
continues at OozieDBCLI.java:[line 960]
. Path continues at OozieDBCLI.java:[line 354]: Obligation to clean up
resource created at OozieDBCLI.java:[line 353] is not discharged
. Path continues at OozieDBCLI.java:[line 414]: Path continues at
OozieDBCLI.java:[line 413]
. Obligation to clean up resource created at OozieDBCLI.java:[line 435] is
not discharged: org.apache.oozie.tools.OozieDBCLI.postUpgradeTasksFor40(String,
boolean) may fail to clean up java.sql.Statement on checked exception
. Path continues at OozieDBCLI.java:[line 448]: Path continues at
OozieDBCLI.java:[line 436]
. org.apache.oozie.tools.OozieDBCLI.showOozieSysInfo() may fail to clean up
java.sql.ResultSet on checked exception: Path continues at
OozieDBCLI.java:[line 449]
. Path continues at OozieDBCLI.java:[line 1029]: Obligation to clean up
resource created at OozieDBCLI.java:[line 1028] is not discharged
. Path continues at OozieDBCLI.java:[line 1038]: Path continues at
OozieDBCLI.java:[line 1037]
. org.apache.oozie.tools.OozieDBCLI.showOozieSysInfo() may fail to clean up
java.sql.Statement on checked exception: Path continues at
OozieDBCLI.java:[line 1041]
. Path continues at OozieDBCLI.java:[line 1028]: Obligation to clean up
resource created at OozieDBCLI.java:[line 1027] is not discharged
. Path continues at OozieDBCLI.java:[line 1037]: Path continues at
OozieDBCLI.java:[line 1029]
. Path continues at OozieDBCLI.java:[line 1041]: Path continues at
OozieDBCLI.java:[line 1038]
. Obligation to clean up resource created at OozieDBCLI.java:[line 268] is
not discharged: org.apache.oozie.tools.OozieDBCLI.upgradeOozieDBVersion(String,
boolean, String) may fail to clean up java.sql.Statement on checked exception
. Path continues at OozieDBCLI.java:[line 273]: Path continues at
OozieDBCLI.java:[line 269]
. Path continues at OozieDBCLI.java:[line 277]: Path continues at
OozieDBCLI.java:[line 274]
. Obligation to clean up resource created at OozieDBCLI.java:[line 1050] is
not discharged: org.apache.oozie.tools.OozieDBCLI.verifyDBState() may fail to
clean up java.sql.ResultSet on checked exception
. Path continues at OozieDBCLI.java:[line 1062]: Path continues at
OozieDBCLI.java:[line 1051]
. Obligation to clean up resource created at OozieDBCLI.java:[line 1049] is
not discharged: org.apache.oozie.tools.OozieDBCLI.verifyDBState() may fail to
clean up java.sql.Statement on checked exception
. Path continues at OozieDBCLI.java:[line 1051]: Path continues at
OozieDBCLI.java:[line 1050]
. org.apache.oozie.tools.OozieDBCLI.verifyOozieSysTable(boolean, boolean)
may fail to clean up java.sql.ResultSet on checked exception: Path continues at
OozieDBCLI.java:[line 1062]
. Path continues at OozieDBCLI.java:[line 920]: Obligation to clean up
resource created at OozieDBCLI.java:[line 919] is not discharged
. Path continues at OozieDBCLI.java:[line 926]: Path continues at
OozieDBCLI.java:[line 925]
. org.apache.oozie.tools.OozieDBCLI.verifyOozieSysTable(boolean, boolean)
may fail to clean up java.sql.Statement on checked exception: Path continues at
OozieDBCLI.java:[line 929]
. Path continues at OozieDBCLI.java:[line 919]: Obligation to clean up
resource created at OozieDBCLI.java:[line 918] is not discharged
. Path continues at OozieDBCLI.java:[line 925]: Path continues at
OozieDBCLI.java:[line 920]
. Path continues at OozieDBCLI.java:[line 929]: Path continues at
OozieDBCLI.java:[line 926]
. At OozieDBCLI.java:[line 721]:
org.apache.oozie.tools.OozieDBCLI.ddlTweaksFor50(String, boolean, String) may
fail to close Statement
. At OozieDBCLI.java:[line 460]:
org.apache.oozie.tools.OozieDBCLI.replaceForDerby(String, String) may fail to
close Statement
. At OozieDBCLI.java:[line 1003]:
org.apache.oozie.tools.OozieDBCLI.createOozieSysTable(String, boolean, String)
passes a nonconstant String to an execute method on an SQL statement
. At OozieDBCLI.java:[line 269]:
org.apache.oozie.tools.OozieDBCLI.upgradeOozieDBVersion(String, boolean,
String) passes a nonconstant String to an execute method on an SQL statement
. At OozieDBCLI.java:[line 562]:
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInMysql(String, Connection)
passes a nonconstant String to an execute method on an SQL statement
. At OozieDBCLI.java:[line 604]:
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInPostgres(String,
Connection, String) passes a nonconstant String to an execute method on an SQL
statement
. Another occurrence at OozieDBCLI.java:[line 606]: Another occurrence at
OozieDBCLI.java:[line 605]
. org.apache.oozie.tools.OozieDBCLI.postUpgradeTasks(String, boolean,
boolean) passes a nonconstant String to an execute method on an SQL statement:
Another occurrence at OozieDBCLI.java:[line 607]
. Exception is caught when Exception is not thrown in
org.apache.oozie.tools.OozieDBExportCLI.queryAllDBTables(String): At
OozieDBCLI.java:[line 401]
. Exceptional return value of java.io.File.delete() ignored in
org.apache.oozie.tools.OozieDBExportCLI.queryAllDBTables(String): At
OozieDBExportCLI.java:[line 162]
. Boxing/unboxing to parse a primitive
org.apache.oozie.tools.OozieSharelibCLI.run(String[]): At
OozieDBExportCLI.java:[line 168]
. org.apache.oozie.tools.OozieSharelibCLI.HELP_INFO should be package
protected: At OozieSharelibCLI.java:[line 114]
. Exceptional return value of java.io.File.delete() ignored in
org.apache.oozie.tools.OozieSharelibCLI.run(String[]): At
OozieSharelibCLI.java:[line 51]
. Exceptional return value of java.io.File.mkdir() ignored in
org.apache.oozie.tools.OozieSharelibCLI.run(String[]): At
OozieSharelibCLI.java:[line 138]
. At OozieSharelibCLI.java:[line 139]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3657/
> Oozie gets RM delegation token even for checking job status
> -----------------------------------------------------------
>
> Key: OOZIE-2807
> URL: https://issues.apache.org/jira/browse/OOZIE-2807
> Project: Oozie
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Satish Subhashrao Saley
> Fix For: 5.0.0
>
> Attachments: OOZIE-2807-1.patch
>
>
> We had one user submitting way too many workflows with single hive query -
> ~3600 workflows running concurrently. Surprisingly Oozie held up well without
> issues.
> But [~daryn] from our hadoop team saw that the amount of delegation tokens
> fetched by Oozie was very high compared to actual number of jobs submitted
> and was stressing RM with the calls and also pushing it close to its memory
> limits. This is because we are fetching the delegation token every time we
> create a JobClient instead of only during job submission.
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java#L503-L519
> So for one job we fetch
> 1) 1 token during submission
> 2) 1 token every 5 minutes when we check status of job
> 3) 1 token after the job ends to retrieve status.
> 4) 1 token if we are killing the job.
> So for a job running for 11 minutes, we would have fetched the token 4 times.
> May be more in other cases like mapreduce where we check for end of launcher
> and child job.
> Only 1 out of the token (used in the job submission) will be cancelled after
> job completes. Other tokens are kind of leaked and will only be cleaned up by
> RM after the expiry period (24 hrs is default). This can make RM go out of
> memory.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
