[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (93 issues) Subscriber: ooziedaily Key Summary OOZIE-3606 Extend file system EL functions to use custom file system properties https://issues.apache.org/jira/browse/OOZIE-3606 OOZIE-3605 ShareLib installation does not honor dfs.replication in HDFS configuration https://issues.apache.org/jira/browse/OOZIE-3605 OOZIE-3596 When the SSH action is killed, it must be changed to the kill command that can terminate the related subprocess. https://issues.apache.org/jira/browse/OOZIE-3596 OOZIE-3568 Have large amount of log information “WARN messages [main] openjpa.MetaData” in jetty.log need to clean https://issues.apache.org/jira/browse/OOZIE-3568 OOZIE-3567 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3567 OOZIE-3560 IDEA shows have some error in index.jsp https://issues.apache.org/jira/browse/OOZIE-3560 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3434 Filtering for invalid jobtype should give error message https://issues.apache.org/jira/browse/OOZIE-3434 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib direct
Failed: OOZIE-3606 PreCommit Build #1323
Jira: https://issues.apache.org/jira/browse/OOZIE-3606
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1323/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 2.16 MB...]
Running test-patch task MODERNIZER
Running modernizer op=report
[TRACE] grep -c '.*ERROR.*Prefer'
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/reports/MODERNIZER-clean.txt
[TRACE] Modernizer errors before patch: 52
[TRACE] grep -c '.*ERROR.*Prefer'
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/reports/MODERNIZER-patch.txt
[TRACE] Modernizer errors after patch: 52
Testing JIRA OOZIE-3606
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
-1 the patch contains 2 line(s) with trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [21] new bugs found below threshold in total that must be fixed.
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [webapp].
-1 There are [6] new bugs found below threshold in [core] that must be
fixed, listing only the first [5] ones.
You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
The top [5] most important SpotBugs errors are:
At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
-1 There are [15] new bugs found below threshold in [tools] that must be
fixed, listing only the first [5] ones.
You can find the SpotBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
The top [5] most important SpotBugs errors are:
At OozieDBCLI.java:[line 584]: This use of
java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL
injection
At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573]
At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575]
At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578]
At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581]
+1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3218
Tests failed at first run:
TestBlockingInputStream#testFastWritingBlockingInputStream
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch
+1 MODERNIZER
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1323/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 48930 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13319682/c
[jira] [Commented] (OOZIE-3606) Extend file system EL functions to use custom file system properties
[
https://issues.apache.org/jira/browse/OOZIE-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166274#comment-17166274
]
Hadoop QA commented on OOZIE-3606:
--
Testing JIRA OOZIE-3606
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 2 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [21] new bugs found below threshold in total
that must be fixed.
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:red}-1{color} There are [6] new bugs found below threshold in
[core] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
.{color:red}-1{color} There are [15] new bugs found below threshold in
[tools] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At OozieDBCLI.java:[line 584]: This use of
java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL
injection
.At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573]
.At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575]
.At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578]
.At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581]
.{color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3218
.{color:orange}Tests failed at first run:{color}
TestBlockingInputStream#testFastWritingBlockingInputStream
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is availab
Review Request 72711: Extend file system EL functions to use custom file system properties
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72711/ --- Review request for oozie. Bugs: OOZIE-3606 https://issues.apache.org/jira/browse/OOZIE-3606 Repository: oozie-git Description --- In case we want to check file availability on abfss file system in scope of a decision node and we have access to that file system only using a key file which is password protected then currently we are blocked as we cannot allow configuring hadoop.security.credstore.java-keystore-provider.password-file globally. This is why this ticket is created to propose a solution for this issue: allow special file system configurations only for file system EL functions let them configure globally via oozie-site.xml and for every workflow independently Diffs - core/src/main/java/org/apache/oozie/ErrorCode.java 8f4e21d03 core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 0f81d7633 core/src/main/resources/oozie-default.xml ab7b8d358 core/src/test/java/org/apache/oozie/action/hadoop/TestFsELFunctions.java 7b8187e0f docs/src/site/markdown/WorkflowFunctionalSpec.md 136f1d77c Diff: https://reviews.apache.org/r/72711/diff/1/ Testing --- Thanks, Denes Bodo
[jira] [Commented] (OOZIE-3606) Extend file system EL functions to use custom file system properties
[ https://issues.apache.org/jira/browse/OOZIE-3606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166198#comment-17166198 ] Hadoop QA commented on OOZIE-3606: -- PreCommit-OOZIE-Build started > Extend file system EL functions to use custom file system properties > > > Key: OOZIE-3606 > URL: https://issues.apache.org/jira/browse/OOZIE-3606 > Project: Oozie > Issue Type: Improvement > Components: core >Affects Versions: 5.2.0 >Reporter: Dénes Bodó >Assignee: Dénes Bodó >Priority: Major > Attachments: OOZIE-3606-001.patch > > > In case we want to check file availability on abfss file system in scope of a > decision node and we have access to that file system only using a key file > which is password protected then currently we are blocked as we cannot allow > configuring hadoop.security.credstore.java-keystore-provider.password-file > globally. > This is why this ticket is created to propose a solution for this issue: > * allow special file system configurations only for file system EL functions > * let them configure globally via oozie-site.xml and for every workflow > independently -- This message was sent by Atlassian Jira (v8.3.4#803005)
