[
https://issues.apache.org/jira/browse/OOZIE-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16002640#comment-16002640
]
Hadoop QA commented on OOZIE-2854:
----------------------------------
Testing JIRA OOZIE-2854
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any line longer than
132
. {color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:red}-1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:red}-1{color} patch does not compile
. {color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [181] new bugs found below threshold in total
that must be fixed.
. {color:red}-1{color} There are [98] new bugs found below threshold in [core]
that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones):
core/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At CLIParser.java:[line 169]: Found reliance on default encoding in
org.apache.oozie.cli.CLIParser.showHelp(CommandLine): new
java.io.PrintWriter(OutputStream)
. At CLIParser.java:[line 57]: new org.apache.oozie.cli.CLIParser(String,
String[]) may expose internal representation by storing an externally mutable
object into CLIParser.cliHelp
. At CLIParser.java:[lines 80-104]: Should
org.apache.oozie.cli.CLIParser$Command be a _static_ inner class?
. At OozieCLI.java:[line 845]: Found reliance on default encoding in
org.apache.oozie.cli.OozieCLI.getConfiguration(OozieClient, CommandLine): new
java.io.FileReader(File)
. At OozieCLI.java:[line 2175]: Found reliance on default encoding in
org.apache.oozie.cli.OozieCLI.validateCommandV41(CommandLine): new
java.io.FileReader(File)
. {color:green}+1{color} There are no new bugs found in [client].
. {color:red}-1{color} There are [1] new bugs found below threshold in [docs]
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
docs/findbugs-new.html
. The most important FindBugs errors are:
. At StreamingMain.java:[line 70]:
org.apache.oozie.action.hadoop.StreamingMain.addActionConf(JobConf,
Configuration) concatenates strings using + in a loop
. {color:red}-1{color} There are [2] new bugs found below threshold in
[sharelib/streaming] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/streaming/findbugs-new.html
. The most important FindBugs errors are:
. At HiveMain.java:[line 340]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.HiveMain.readStringFromFile(String): new
java.io.FileReader(String)
. Obligation to clean up resource created at HiveMain.java:[line 184] is not
discharged: org.apache.oozie.action.hadoop.HiveMain.setUpHiveSite() may fail to
clean up java.io.OutputStream on checked exception
. Path continues at HiveMain.java:[line 185]
. {color:red}-1{color} There are [3] new bugs found below threshold in
[sharelib/hive] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/hive/findbugs-new.html
. The most important FindBugs errors are:
. At Hive2Main.java:[line 281]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.Hive2Main.readStringFromFile(String): new
java.io.FileReader(String)
. At Hive2Main.java:[line 270]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.Hive2Main.runBeeline(String[], String): new
java.io.PrintStream(OutputStream)
. At Hive2Main.java:[line 273]:
org.apache.oozie.action.hadoop.Hive2Main.runBeeline(String[], String) invokes
System.exit(...), which shuts down the entire virtual machine
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:red}-1{color} There are [19] new bugs found below threshold in
[sharelib/sqoop] that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/sqoop/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At LauncherMain.java:[line 124]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.LauncherMain.getHadoopJobIds(String, Pattern[]):
new java.io.FileReader(String)
. At LauncherMain.java:[line 160]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.LauncherMain.writeExternalChildIDs(String,
Pattern[], String): String.getBytes()
. At LauncherMain.java:[line 61]:
org.apache.oozie.action.hadoop.LauncherMain.HADOOP_SITE_FILES should be both
final and package protected
. At LauncherMain.java:[line 327]: Exceptional return value of
java.io.File.mkdirs() ignored in
org.apache.oozie.action.hadoop.LauncherMain.writeHadoopConfig(String, File)
. At LauncherMapper.java:[line 407]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.LauncherMapper.getLocalFileContentStr(File,
String, int): new java.io.FileReader(File)
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:red}-1{color} There are [8] new bugs found below threshold in
[sharelib/hcatalog] that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/hcatalog/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At OoziePigStats.java:[line 130]:
org.apache.oozie.action.hadoop.OoziePigStats.toJSONFromMultiStoreCounters(Map)
makes inefficient use of keySet iterator instead of entrySet iterator
. At PigMain.java:[line 334]: Dead store to klass in
org.apache.oozie.action.hadoop.PigMain.runPigJob(String[], String, boolean,
boolean)
. At PigMain.java:[line 306]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.PigMain.handleError(String): new
java.io.FileReader(String)
. At PigMain.java:[line 407]: Found reliance on default encoding in
org.apache.oozie.action.hadoop.PigMain.writeExternalData(String, File): new
java.io.FileWriter(File)
. Obligation to clean up resource created at PigMain.java:[line 130] is not
discharged: org.apache.oozie.action.hadoop.PigMain.run(String[]) may fail to
clean up java.io.OutputStream on checked exception
. {color:red}-1{color} There are [2] new bugs found below threshold in
[sharelib/pig] that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/pig/findbugs-new.html
. The most important FindBugs errors are:
. At SparkMain.java:[line 493]:
org.apache.oozie.action.hadoop.SparkMain.getJarVersion(File) may fail to close
stream
. At SparkMain.java:[line 326]: Exceptional return value of
java.io.File.mkdirs() ignored in
org.apache.oozie.action.hadoop.SparkMain.createPySparkLibFolder()
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:red}-1{color} There are [6] new bugs found below threshold in
[sharelib/distcp] that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones):
sharelib/distcp/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At DateList.java:[line 55]: Nullcheck of date at line 55 of value previously
dereferenced in org.apache.oozie.example.DateList.main(String[])
. Private method org.apache.oozie.example.DateList.formatDateUTC(Calendar) is
never called: Redundant null check at DateList.java:[line 62]
. org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail to
clean up java.io.InputStream: At DateList.java:[line 97]
. Path continues at LocalOozieExample.java:[line 76]: Obligation to clean up
resource created at LocalOozieExample.java:[line 72] is not discharged
. Path continues at LocalOozieExample.java:[line 78]: Path continues at
LocalOozieExample.java:[line 77]
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:red}-1{color} There are [42] new bugs found below threshold in
[hadooplibs/hadoop-utils-2] that must be fixed, listing only the first [5] ones.
. You can find the FindBugs diff here (look for the red and orange ones):
hadooplibs/hadoop-utils-2/findbugs-new.html
. The top [5] most important FindBugs errors are:
. At OozieDBCLI.java:[line 548]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInMysql(String, Connection):
new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 577]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.convertClobToBlobInPostgres(String,
Connection, String): new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 983]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.createOozieSysTable(String, boolean, String):
new java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 759]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.ddlTweaks(String, boolean): new
java.io.FileWriter(String, boolean)
. At OozieDBCLI.java:[line 712]: Found reliance on default encoding in
org.apache.oozie.tools.OozieDBCLI.ddlTweaksFor50(String, boolean, String): new
java.io.FileWriter(String, boolean)
. {color:orange}0{color} There are [1] new bugs found in [tools] that would be
nice to have fixed.
. You can find the FindBugs diff here: tools/findbugs-new.html
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
. {color:red}-1{color} distro tarball fails with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/3812/
> Oozie should handle transient DB problems
> -----------------------------------------
>
> Key: OOZIE-2854
> URL: https://issues.apache.org/jira/browse/OOZIE-2854
> Project: Oozie
> Issue Type: Improvement
> Components: core
> Reporter: Peter Bacsko
> Assignee: Peter Bacsko
> Attachments: OOZIE-2854-001.patch, OOZIE-2854-002.patch,
> OOZIE-2854-POC-001.patch
>
>
> There can be problems when Oozie cannot update the database properly.
> Recently, we have experienced erratic behavior with two setups:
> * MySQL with the Galera cluster manager. Galera uses cluster-wide optimistic
> locking which might cause a transaction to rollback if there are two or more
> parallel transaction running and one of them cannot complete because of a
> conflict.
> * MySQL with Percona XtraDB Cluster. If one of the MySQL instances is killed,
> Oozie might get "Communications link failure" exception during the failover.
> The problem is that failed DB transactions later might cause a workflow
> (which are started/re-started by RecoveryService) to get stuck. It's not
> clear to us how this happens but it has to do with the fact that certain DB
> updates are not executed.
> The solution is to use some sort of retry logic with exponential backoff if
> the DB update fails. We could start with a 100ms wait time which is doubled
> at every retry. The operation can be considered a failure if it still fails
> after 10 attempts. These values could be configurable. We should discuss
> initial values in the scope of this JIRA.
> Note that this solution is to handle *transient* failures. If the DB is down
> for a longer period of time, we have to accept that the internal state of
> Oozie is corrupted.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
