[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (92 issues) Subscriber: ooziedaily Key Summary OOZIE-3504 [Java 11] Fix tests using PowerMock https://issues.apache.org/jira/browse/OOZIE-3504 OOZIE-3486 duplicate code in ControlNodeHandler https://issues.apache.org/jira/browse/OOZIE-3486 OOZIE-3482 Fix bug in CoordSubmitXCommand#validateCoordinatorJob https://issues.apache.org/jira/browse/OOZIE-3482 OOZIE-3480 Add windowactionstatus metrics in DBLiteWorkflowStoreService https://issues.apache.org/jira/browse/OOZIE-3480 OOZIE-3468 Use modernizer plugin https://issues.apache.org/jira/browse/OOZIE-3468 OOZIE-3461 CoordMaterializeTriggerService code cleanup https://issues.apache.org/jira/browse/OOZIE-3461 OOZIE-3449 Make spark-2 as the default profile https://issues.apache.org/jira/browse/OOZIE-3449 OOZIE-3447 Run test case in local : It shows oozie-hsqldb-orm.xml exception https://issues.apache.org/jira/browse/OOZIE-3447 OOZIE-3418 Upgrade to Guava 27 https://issues.apache.org/jira/browse/OOZIE-3418 OOZIE-3404 The env variable of SPARK_HOME needs to be set when running pySpark https://issues.apache.org/jira/browse/OOZIE-3404 OOZIE-3375 Can't use empty in coordinator https://issues.apache.org/jira/browse/OOZIE-3375 OOZIE-3367 Using && in EL expressions in oozie bundle.xml files generates parse errors https://issues.apache.org/jira/browse/OOZIE-3367 OOZIE-3366 Update workflow status and subworkflow status on suspend command https://issues.apache.org/jira/browse/OOZIE-3366 OOZIE-3364 Rerunning Oozie bundle jobs starts the coordinators in indeterminate order https://issues.apache.org/jira/browse/OOZIE-3364 OOZIE-3362 When killed, SSH action should kill the spawned processes on target host https://issues.apache.org/jira/browse/OOZIE-3362 OOZIE-3335 Cleanup parseFilter methods https://issues.apache.org/jira/browse/OOZIE-3335 OOZIE-3328 Create Hive compatibility action executor to run hive actions using beeline https://issues.apache.org/jira/browse/OOZIE-3328 OOZIE-3320 Oozie ShellAction should support absolute bash file path https://issues.apache.org/jira/browse/OOZIE-3320 OOZIE-3319 Log SSH action callback error output https://issues.apache.org/jira/browse/OOZIE-3319 OOZIE-3301 Update NOTICE file https://issues.apache.org/jira/browse/OOZIE-3301 OOZIE-3274 Remove slf4j https://issues.apache.org/jira/browse/OOZIE-3274 OOZIE-3266 Coord action rerun support RERUN_SKIP_NODES option https://issues.apache.org/jira/browse/OOZIE-3266 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3254 [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize https://issues.apache.org/jira/browse/OOZIE-3254 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3137 Add support for log4j2 in HiveMain https://issues.apache.org/jira/browse/OOZIE-3137 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2784 Include WEEK as a parameter in th
[jira] [Commented] (OOZIE-3504) [Java 11] Fix tests using PowerMock
[
https://issues.apache.org/jira/browse/OOZIE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891294#comment-16891294
]
Hadoop QA commented on OOZIE-3504:
--
Testing JIRA OOZIE-3504
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:orange}0{color} There are [4] new bugs found in total that would be nice
to have fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2975
.Tests failed : 1
.Tests in error : 0
.Tests timed out : 0
{color:red}-1{color} [ERROR] There are [1] test failures in [core]. Listing
only the first [5] ones
testPurgeableWorkflowPurgeableSubWorkflowPurgeableSubSubWorkflow:org.apache.oozie.command.TestPurgeXCommand
Check console output for the full list of errors/failures
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1193/
> [Java 11] Fix tests using PowerMock
> ---
>
> Key: OOZIE-3504
> URL: https://issues.apache.org/jira/browse/OOZIE-3504
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3504-001.patch
>
>
> With JDK 11 the following tests are failing with initialisation error:
> * TestScriptLanguageActionExecutor
> * TestHCatCredentials
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
Failed: OOZIE-3504 PreCommit Build #1193
Jira: https://issues.apache.org/jira/browse/OOZIE-3504
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1193/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.99 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[WARN] There are [4] new bugs found in total that would be nice to have fixed.
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 2670 bytes
[TRACE] Full summary file size is 1660 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
xargs: WARNING: a NUL character occurred in the input. It cannot be passed
through in the argument list. Did you mean to use the --null option?
Running test-patch task DISTRO
Testing JIRA OOZIE-3504
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 Javadoc generation succeeded with the patch
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
0 There are [4] new bugs found in total that would be nice to have fixed.
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
0 There are [4] new bugs found in [server] that would be nice to have fixed.
You can find the SpotBugs diff here: server/findbugs-new.html
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [core].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2975
Tests failed : 1
Tests in error : 0
Tests timed out : 0
-1 [ERROR] There are [1] test failures in [core]. Listing only the first [5]
ones
testPurgeableWorkflowPurgeableSubWorkflowPurgeableSubSubWorkflow:org.apache.oozie.command.TestPurgeXCommand
Check console output for the full list of errors/failures
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1193/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13236444/comment/16891294","id":"16891294","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa&avatarId=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small&ownerId=hadoopqa&avatarId=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall&ownerId=hadoopqa&avatarId=10393","32x32":"https://issues.apache.org/jira/secure/useravatar
[GitHub] [oozie] pkuwm closed pull request #33: [OOZIE-3141] Expose external child job IDs to EL functions: wf:extern…
pkuwm closed pull request #33: [OOZIE-3141] Expose external child job IDs to EL functions: wf:extern… URL: https://github.com/apache/oozie/pull/33 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (OOZIE-3504) [Java 11] Fix tests using PowerMock
[ https://issues.apache.org/jira/browse/OOZIE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891251#comment-16891251 ] Hadoop QA commented on OOZIE-3504: -- PreCommit-OOZIE-Build started > [Java 11] Fix tests using PowerMock > --- > > Key: OOZIE-3504 > URL: https://issues.apache.org/jira/browse/OOZIE-3504 > Project: Oozie > Issue Type: Sub-task >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3504-001.patch > > > With JDK 11 the following tests are failing with initialisation error: > * TestScriptLanguageActionExecutor > * TestHCatCredentials > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3529) Oozie not supported for s3 as filesystem
[
https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891172#comment-16891172
]
Steve Loughran commented on OOZIE-3529:
---
It's the only approach which will work other than have the S3A code fall back
to using java.io.tmp if it can't instantiate a dir allocator. We only need that
Dir allocator for two services HDFS needs; rotating use of multiple disks and
resilience to offline storage. There's no fundamental reason why downgrading to
the normal file temp dir shouldn't be blocked.
Let me know how using bytebuffers work
> Oozie not supported for s3 as filesystem
>
>
> Key: OOZIE-3529
> URL: https://issues.apache.org/jira/browse/OOZIE-3529
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Priority: Critical
> Labels: S3
> Attachments: id.pig, job.properties, workflow.xml
>
>
> Many customer who uses s3 file system as secondary one experiences the
> following error when Oozie tries to submit the Yarn application:
> {noformat}
> 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
> SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
> APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W]
> ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
> at
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
> at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
> at
> org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.ha
[jira] [Resolved] (OOZIE-452) OOZIE-35: Remove ssh action from oozie-default configuration
[ https://issues.apache.org/jira/browse/OOZIE-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon resolved OOZIE-452. -- Resolution: Won't Do No need for this, SSH action is back. > OOZIE-35: Remove ssh action from oozie-default configuration > > > Key: OOZIE-452 > URL: https://issues.apache.org/jira/browse/OOZIE-452 > Project: Oozie > Issue Type: Task >Reporter: Mohammad Kamrul Islam >Assignee: Roman Shaposhnik >Priority: Major > > Since SSH action is discontinued. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (OOZIE-251) GH-338: Provide a way of visually presenting a workflow in diagram
[ https://issues.apache.org/jira/browse/OOZIE-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon resolved OOZIE-251. -- Resolution: Not A Problem Oozie can already draw a graph representing the workflow. > GH-338: Provide a way of visually presenting a workflow in diagram > -- > > Key: OOZIE-251 > URL: https://issues.apache.org/jira/browse/OOZIE-251 > Project: Oozie > Issue Type: Bug >Reporter: Hadoop QA >Priority: Major > > It could be done as an utility or as a part of oozie UI or both. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3254) [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize
[
https://issues.apache.org/jira/browse/OOZIE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891036#comment-16891036
]
Hadoop QA commented on OOZIE-3254:
--
Testing JIRA OOZIE-3254
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:red}-1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:red}-1{color} the patch seems to introduce 1 new Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [5] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
.At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
.At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3179
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1192/
> [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError
> when there are too many coordinator actions to materialize
> --
>
> Key: OOZIE-3254
> URL: https://issues.apache.org/jira/browse/OOZIE-3254
> Project: Oozie
> Issue Type: Bug
> Components: coordinator
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3254-01-wip.patch
>
>
> If there is a coordinator job defined with a {{frequency}} by
Failed: OOZIE-3254 PreCommit Build #1192
Jira: https://issues.apache.org/jira/browse/OOZIE-3254
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/1192/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 2.03 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[ERROR] There are [5] new bugs found below threshold in [core].
[DEBUG] You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
[ERROR] There are [5] new bugs found below threshold in total that must be
fixed.
[TRACE] SpotBugs diffs checked and reports created
[TRACE] Summary file size is 3431 bytes
[TRACE] Full summary file size is 3572 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/SPOTBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3254
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any star imports
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
-1 JAVADOC
+1 Javadoc generation succeeded with the patch
-1 the patch seems to introduce 1 new Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
-1 There are [5] new bugs found below threshold in total that must be fixed.
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/git].
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [fluent-job/fluent-job-api].
0 There are [4] new bugs found in [server] that would be nice to have fixed.
You can find the SpotBugs diff here: server/findbugs-new.html
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [examples].
-1 There are [5] new bugs found below threshold in [core] that must be
fixed.
You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
The most important SpotBugs errors are:
At BulkJPAExecutor.java:[line 206]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
At BulkJPAExecutor.java:[line 176]: At BulkJPAExecutor.java:[line 175]
At BulkJPAExecutor.java:[line 205]: At BulkJPAExecutor.java:[line 199]
This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 206]
At BulkJPAExecutor.java:[line 111]: At BulkJPAExecutor.java:[line 127]
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 3179
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/1192/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13161386/comment/16891036","id":"1689
[jira] [Updated] (OOZIE-3504) [Java 11] Fix tests using PowerMock
[ https://issues.apache.org/jira/browse/OOZIE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julia Kinga Marton updated OOZIE-3504: -- Attachment: OOZIE-3504-001.patch > [Java 11] Fix tests using PowerMock > --- > > Key: OOZIE-3504 > URL: https://issues.apache.org/jira/browse/OOZIE-3504 > Project: Oozie > Issue Type: Sub-task >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3504-001.patch > > > With JDK 11 the following tests are failing with initialisation error: > * TestScriptLanguageActionExecutor > * TestHCatCredentials > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3504) [Java 11] Fix tests using PowerMock
[ https://issues.apache.org/jira/browse/OOZIE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890971#comment-16890971 ] Julia Kinga Marton commented on OOZIE-3504: --- TestHCatCredentials is failing because of classloading issues: it seems that PowerMock has a custom classloader what nows nothing about the modules in JDK11. We can instruct Powermock to let the parent classloader to load some classes by using @PowerMockIgnore annotation. > [Java 11] Fix tests using PowerMock > --- > > Key: OOZIE-3504 > URL: https://issues.apache.org/jira/browse/OOZIE-3504 > Project: Oozie > Issue Type: Sub-task >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > Attachments: OOZIE-3504-001.patch > > > With JDK 11 the following tests are failing with initialisation error: > * TestScriptLanguageActionExecutor > * TestHCatCredentials > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3254) [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError when there are too many coordinator actions to materialize
[
https://issues.apache.org/jira/browse/OOZIE-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890928#comment-16890928
]
Hadoop QA commented on OOZIE-3254:
--
PreCommit-OOZIE-Build started
> [coordinator] LAST_ONLY and NONE execution modes: possible OutOfMemoryError
> when there are too many coordinator actions to materialize
> --
>
> Key: OOZIE-3254
> URL: https://issues.apache.org/jira/browse/OOZIE-3254
> Project: Oozie
> Issue Type: Bug
> Components: coordinator
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-3254-01-wip.patch
>
>
> If there is a coordinator job defined with a {{frequency}} by the minute
> (e.g. {{frequency="* * * * *"}}), and {{start-time}} lies well in the past,
> and the coordinator job's {{execution-mode}} is {{LAST_ONLY}} or {{NONE}}, it
> can happen that too many {{CoordinatorActionBean}} instances are kept on JVM
> heap within {{CoordMaterializeTransitionXCommand#insertList}} as those
> execution modes [*omit the check for the {{throttle}}
> value*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java#L439-L443].
> As a consequence, we can see as many as multiple hundred thousands of log
> entries [*trying to increase
> {{CoordMaterializeTransitionXCommand#insertList}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java#L560-L566]:
> {noformat}
> [user@host ~]$ grep 'In storeToDB() coord action id'
> /var/log/oozie/oozie-HOSTNAME.log.out | wc -l
> 478408
> {noformat}
> A much worse consequence is that those {{CoordinatorActionBean}} instances
> are attached to GC root (the {{insertList}} itself), and thus, JVM is unable
> to free them until a consequent call to {{insertList.clear()}}. This will
> result in {{OutOfMemoryError}} occurrence in worst case.
> {{CoordMaterializeTransitionXCommand#insertList}} should be watched for a
> configurable limit parameter (default value something like 1000), and
> persisted / cleared when that limit is reached.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (OOZIE-3504) [Java 11] Fix tests using PowerMock
[ https://issues.apache.org/jira/browse/OOZIE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890889#comment-16890889 ] Julia Kinga Marton commented on OOZIE-3504: --- OOZIE-3528 solved the TestScriptLanguageActionExecutor failure, but TestHCatCredentials is still failing. > [Java 11] Fix tests using PowerMock > --- > > Key: OOZIE-3504 > URL: https://issues.apache.org/jira/browse/OOZIE-3504 > Project: Oozie > Issue Type: Sub-task >Reporter: Julia Kinga Marton >Assignee: Julia Kinga Marton >Priority: Major > > With JDK 11 the following tests are failing with initialisation error: > * TestScriptLanguageActionExecutor > * TestHCatCredentials > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (OOZIE-3392) Refactor ShellMain to use BufferDrainer class
[
https://issues.apache.org/jira/browse/OOZIE-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon reassigned OOZIE-3392:
-
Assignee: (was: Andras Salamon)
> Refactor ShellMain to use BufferDrainer class
> -
>
> Key: OOZIE-3392
> URL: https://issues.apache.org/jira/browse/OOZIE-3392
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: trunk
>Reporter: Andras Salamon
>Priority: Major
> Fix For: 5.2.0
>
>
> OOZIE-3382 introduced a new {{BufferDrainer}} class to drain the standard
> output and standard error stream of an ssh process. {{ShellMain}} should also
> use this class for reading the standard output and standard error of a shell
> process to avoid code duplication.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (OOZIE-2162) Remove WorkflowStore
[ https://issues.apache.org/jira/browse/OOZIE-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Salamon reassigned OOZIE-2162: - Assignee: (was: Andras Salamon) > Remove WorkflowStore > > > Key: OOZIE-2162 > URL: https://issues.apache.org/jira/browse/OOZIE-2162 > Project: Oozie > Issue Type: Sub-task >Reporter: Bowen Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (OOZIE-3529) Oozie not supported for s3 as filesystem
[
https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890728#comment-16890728
]
Andras Salamon commented on OOZIE-3529:
---
Thanks for checking [~ste...@apache.org]
Oozie has its own
[RawLocalFileSystem|https://github.com/apache/oozie/blob/master/webapp/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java]
implementation to protect agains CVE-2017-15712.
Setting {{fs.s3a.fast.upload.buffer}} would eliminate the error but it would
keep oozie protected agains the CVS, I like this approach.
> Oozie not supported for s3 as filesystem
>
>
> Key: OOZIE-3529
> URL: https://issues.apache.org/jira/browse/OOZIE-3529
> Project: Oozie
> Issue Type: Bug
> Components: core
>Affects Versions: 4.3.1, 5.1.0
>Reporter: Denes Bodo
>Priority: Critical
> Labels: S3
> Attachments: id.pig, job.properties, workflow.xml
>
>
> Many customer who uses s3 file system as secondary one experiences the
> following error when Oozie tries to submit the Yarn application:
> {noformat}
> 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
> SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
> APP[demo-wf] JOB[001-190423141707256-oozie-oozi-W]
> ACTION[001-190423141707256-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
> at
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
> at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
> at
> org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.(S3ABlockOutputStream.java:168)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(F
