[jira] [Created] (OOZIE-3264) Flaky test TestCoordMaterializeTransitionXCommand#testLastOnlyMaterialization
Peter Bacsko created OOZIE-3264:
---
Summary: Flaky test
TestCoordMaterializeTransitionXCommand#testLastOnlyMaterialization
Key: OOZIE-3264
URL: https://issues.apache.org/jira/browse/OOZIE-3264
Project: Oozie
Issue Type: Sub-task
Reporter: Peter Bacsko
The test {{testLastOnlyMaterialization}} failed.
The expected status was {{WAITING}} but it was different. The method
{{checkCoordActionStatus()}} failed at line 1114.
TODO: add stack trace.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3263) Flaky test TestZKUUIDService#testResetSequence_withMultiThread
[
https://issues.apache.org/jira/browse/OOZIE-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492826#comment-16492826
]
Peter Bacsko commented on OOZIE-3263:
-
I have a theory for this.
1. Thread#1 reaches end of the sequence (5000) and enters {{resetSequence()}}
2. Thread#2 also reaches the end of the sequence and enterd {{resetSequence()}}
too
3. Thread#1 successfully acquires ZK lock and begins to perform resetting the
distributed counter
4. Thread#2 tries to acquire the ZK lock in a loop but fails after multiple
attempts.
5. An exception is thrown from {{resetSequence()}} which is caught by
{{createSequence()}} and calls {{UUIDService.getCounter()}} as a fallback
mechanism, which returns 0.
6. In the meantime, Thread #1 has returned 0 and the respective
{{AtomicInteger}} has been incremented
7. Thread#2 also increments {{AtomicInteger}} at position 0 in the array
Step #4 seems to be highly unlikely. Lock timeout for ZK is 5 seconds and there
are 3 attempts to acquire the lock. Even on a slow computer, it shouldn't take
15 seconds, it sounds unacceptably slow.
Regardless, we need more logs to find out if this sequence of events causes the
test error.
> Flaky test TestZKUUIDService#testResetSequence_withMultiThread
> --
>
> Key: OOZIE-3263
> URL: https://issues.apache.org/jira/browse/OOZIE-3263
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> The test case {{testResetSequence_withMultiThread}} failed with the following
> error:
> {noformat}
> junit.framework.AssertionFailedError: expected:<3> but was:<2>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.oozie.service.TestZKUUIDService.testResetSequence_withMultiThread(TestZKUUIDService.java:214)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OOZIE-3263) Flaky test TestZKUUIDService#testResetSequence_withMultiThread
[
https://issues.apache.org/jira/browse/OOZIE-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492826#comment-16492826
]
Peter Bacsko edited comment on OOZIE-3263 at 5/28/18 4:52 PM:
--
I have a theory for this.
1. Thread#1 reaches end of the sequence (5000) and enters {{resetSequence()}}
2. Thread#2 also reaches the end of the sequence and enterd {{resetSequence()}}
too
3. Thread#1 successfully acquires ZK lock and begins to perform resetting the
distributed counter
4. Thread#2 tries to acquire the ZK lock in a loop but fails after multiple
attempts.
5. An exception is thrown from {{resetSequence()}} which is caught by
{{createSequence()}} and calls {{UUIDService.getCounter()}} as a fallback
mechanism, which returns 0.
6. In the meantime, Thread #1 has returned 0 and the respective
{{AtomicInteger}} has been incremented
7. Thread#2 also increments {{AtomicInteger}} at position 0 in the array
Step #4 seems to be highly unlikely. Lock timeout for ZK is 5 seconds and there
are 3 attempts to acquire the lock. Even on a slow computer, it shouldn't take
15 seconds to complete {{resetSequence()}}, it sounds absurdly slow.
Regardless, we need more logs to find out if this sequence of events causes the
test error.
was (Author: pbacsko):
I have a theory for this.
1. Thread#1 reaches end of the sequence (5000) and enters {{resetSequence()}}
2. Thread#2 also reaches the end of the sequence and enterd {{resetSequence()}}
too
3. Thread#1 successfully acquires ZK lock and begins to perform resetting the
distributed counter
4. Thread#2 tries to acquire the ZK lock in a loop but fails after multiple
attempts.
5. An exception is thrown from {{resetSequence()}} which is caught by
{{createSequence()}} and calls {{UUIDService.getCounter()}} as a fallback
mechanism, which returns 0.
6. In the meantime, Thread #1 has returned 0 and the respective
{{AtomicInteger}} has been incremented
7. Thread#2 also increments {{AtomicInteger}} at position 0 in the array
Step #4 seems to be highly unlikely. Lock timeout for ZK is 5 seconds and there
are 3 attempts to acquire the lock. Even on a slow computer, it shouldn't take
15 seconds, it sounds unacceptably slow.
Regardless, we need more logs to find out if this sequence of events causes the
test error.
> Flaky test TestZKUUIDService#testResetSequence_withMultiThread
> --
>
> Key: OOZIE-3263
> URL: https://issues.apache.org/jira/browse/OOZIE-3263
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> The test case {{testResetSequence_withMultiThread}} failed with the following
> error:
> {noformat}
> junit.framework.AssertionFailedError: expected:<3> but was:<2>
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.Assert.failNotEquals(Assert.java:329)
> at junit.framework.Assert.assertEquals(Assert.java:78)
> at junit.framework.Assert.assertEquals(Assert.java:234)
> at junit.framework.Assert.assertEquals(Assert.java:241)
> at junit.framework.TestCase.assertEquals(TestCase.java:409)
> at
> org.apache.oozie.service.TestZKUUIDService.testResetSequence_withMultiThread(TestZKUUIDService.java:214)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3228) [Spark action] Can't load properties from spark-defaults.conf
[
https://issues.apache.org/jira/browse/OOZIE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492488#comment-16492488
]
Peter Cseh commented on OOZIE-3228:
---
+1 (pending Jenkins) for the amend patch
> [Spark action] Can't load properties from spark-defaults.conf
> -
>
> Key: OOZIE-3228
> URL: https://issues.apache.org/jira/browse/OOZIE-3228
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Tang Yan
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3228.001.patch, OOZIE-3228.002.patch,
> OOZIE-3228.003.patch, OOZIE-3228.004.patch, OOZIE-3228.005.patch,
> OOZIE-3228.amend.001.patch
>
>
> When I create a Oozie workflow to launch a spark action, the spark job can't
> load the configured properties in {{spark-defaults.conf}}. I've configured
> each NodeManager as the Spark gateway role, so {{spark-defaults.conf}} is
> generated in {{/etc/spark/conf/}} on each worker node.
> In {{spark-defaults.conf}} some configuration I've set into.
> {noformat}
> spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> {noformat}
> But in the Oozie spark job, they're not loaded automatically.
> {noformat}
> --conf spark.executor.extraClassPath=$PWD/*
> --conf spark.driver.extraClassPath=$PWD/*
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Success: OOZIE-3228 PreCommit Build #583
Jira: https://issues.apache.org/jira/browse/OOZIE-3228
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/583/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.70 MB...]
[DEBUG] There are no new bugs found in [sharelib/pig].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2365 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3228
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2132
+1 DISTRO
+1 distro tarball builds with the patch
+1 Overall result, good!, no -1s
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/583/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13154670/comment/16492569","id":"16492569","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
at hadoop dot apache dot
[jira] [Commented] (OOZIE-3228) [Spark action] Can't load properties from spark-defaults.conf
[
https://issues.apache.org/jira/browse/OOZIE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492569#comment-16492569
]
Hadoop QA commented on OOZIE-3228:
--
Testing JIRA OOZIE-3228
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2132
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}*+1 Overall result, good!, no -1s*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/583/
> [Spark action] Can't load properties from spark-defaults.conf
> -
>
> Key: OOZIE-3228
> URL: https://issues.apache.org/jira/browse/OOZIE-3228
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Tang Yan
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3228.001.patch, OOZIE-3228.002.patch,
> OOZIE-3228.003.patch, OOZIE-3228.004.patch, OOZIE-3228.005.patch,
> OOZIE-3228.amend.001.patch
>
>
> When I create a Oozie workflow to launch a spark action, the spark job can't
> load the configured properties in {{spark-defaults.conf}}. I've configured
> each NodeManager as the Spark gateway role, so {{spark-defaults.conf}} is
> generated in {{/etc/spark/conf/}} on each worker node.
> In {{spark-defaults.conf}} some configuration I've set into.
> {noformat}
> spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> {noformat}
> But in the Oozie spark job, they're not loaded automatically.
> {noformat}
> --conf spark.executor.extraClassPath=$PWD/*
> --conf spark.driver.extraClassPath=$PWD/*
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3228) [Spark action] Can't load properties from spark-defaults.conf
[
https://issues.apache.org/jira/browse/OOZIE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492480#comment-16492480
]
Hadoop QA commented on OOZIE-3228:
--
PreCommit-OOZIE-Build started
> [Spark action] Can't load properties from spark-defaults.conf
> -
>
> Key: OOZIE-3228
> URL: https://issues.apache.org/jira/browse/OOZIE-3228
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Tang Yan
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3228.001.patch, OOZIE-3228.002.patch,
> OOZIE-3228.003.patch, OOZIE-3228.004.patch, OOZIE-3228.005.patch,
> OOZIE-3228.amend.001.patch
>
>
> When I create a Oozie workflow to launch a spark action, the spark job can't
> load the configured properties in {{spark-defaults.conf}}. I've configured
> each NodeManager as the Spark gateway role, so {{spark-defaults.conf}} is
> generated in {{/etc/spark/conf/}} on each worker node.
> In {{spark-defaults.conf}} some configuration I've set into.
> {noformat}
> spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> {noformat}
> But in the Oozie spark job, they're not loaded automatically.
> {noformat}
> --conf spark.executor.extraClassPath=$PWD/*
> --conf spark.driver.extraClassPath=$PWD/*
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3109) Escape log-streaming's HTML-specific characters
[ https://issues.apache.org/jira/browse/OOZIE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492629#comment-16492629 ] Hadoop QA commented on OOZIE-3109: -- PreCommit-OOZIE-Build started > Escape log-streaming's HTML-specific characters > --- > > Key: OOZIE-3109 > URL: https://issues.apache.org/jira/browse/OOZIE-3109 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Major > Attachments: OOZIE-3109-v001.patch, OOZIE-3109-v002.patch, > OOZIE-3109-v003.patch > > > That can be a security problem if we do not encode the stream read from log > files before transferred to the web browser. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3109) Escape log-streaming's HTML-specific characters
[ https://issues.apache.org/jira/browse/OOZIE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492636#comment-16492636 ] Hadoop QA commented on OOZIE-3109: -- PreCommit-OOZIE-Build started > Escape log-streaming's HTML-specific characters > --- > > Key: OOZIE-3109 > URL: https://issues.apache.org/jira/browse/OOZIE-3109 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Major > Attachments: OOZIE-3109-v001.patch, OOZIE-3109-v002.patch, > OOZIE-3109-v003.patch > > > That can be a security problem if we do not encode the stream read from log > files before transferred to the web browser. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3262) [Spark action] Improve TestSparkArgsExtractor
[
https://issues.apache.org/jira/browse/OOZIE-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3262:
Description:
Similar to the text fixture
{{TestSparkArgsExtractor#testPropertiesArePrependedToSparkArgs()}} covered by
OOZIE-3228, extend the usage of {{assertContainsSublist()}} for every other
test case as well.
This has some advantages:
* test runs become more stable when parameterized differently
* code gets more readable
* {{assert}} only what's necessary instead of always the whole Spark argument
list. For most test cases it's not needed
was:
Similar to the text fixture
{{TestSparkArgsExtractor#testPropertiesArePrependedToSparkArgs}} covered by
OOZIE-3228, extend the usage of {{assertContainsSublist()}} for every other
test case as well.
This has some advantages:
* test runs become more stable when parameterized differently
* code gets more readable
* {{assert}} only what's necessary instead of always the whole Spark argument
list. For most test cases it's not needed
> [Spark action] Improve TestSparkArgsExtractor
> -
>
> Key: OOZIE-3262
> URL: https://issues.apache.org/jira/browse/OOZIE-3262
> Project: Oozie
> Issue Type: Improvement
> Components: tests
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Priority: Major
>
> Similar to the text fixture
> {{TestSparkArgsExtractor#testPropertiesArePrependedToSparkArgs()}} covered by
> OOZIE-3228, extend the usage of {{assertContainsSublist()}} for every other
> test case as well.
> This has some advantages:
> * test runs become more stable when parameterized differently
> * code gets more readable
> * {{assert}} only what's necessary instead of always the whole Spark argument
> list. For most test cases it's not needed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3262) [Spark action] Improve TestSparkArgsExtractor
Andras Piros created OOZIE-3262:
---
Summary: [Spark action] Improve TestSparkArgsExtractor
Key: OOZIE-3262
URL: https://issues.apache.org/jira/browse/OOZIE-3262
Project: Oozie
Issue Type: Improvement
Components: tests
Affects Versions: 5.0.0
Reporter: Andras Piros
Similar to the text fixture
{{TestSparkArgsExtractor#testPropertiesArePrependedToSparkArgs}} covered by
OOZIE-3228, extend the usage of {{assertContainsSublist()}} for every other
test case as well.
This has some advantages:
* test runs become more stable when parameterized differently
* code gets more readable
* {{assert}} only what's necessary instead of always the whole Spark argument
list. For most test cases it's not needed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3261) Set up mvn clean to delete the "build" directory as well
[
https://issues.apache.org/jira/browse/OOZIE-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Cseh updated OOZIE-3261:
--
Labels: newbie (was: )
> Set up mvn clean to delete the "build" directory as well
>
>
> Key: OOZIE-3261
> URL: https://issues.apache.org/jira/browse/OOZIE-3261
> Project: Oozie
> Issue Type: Improvement
> Components: build
>Reporter: Peter Cseh
>Priority: Major
> Labels: newbie
>
> There are folders named build in every project we have. They are taking up a
> ton of space and not cleaned away when executing "mvn clean"
> {noformat}
> >> [84] 13:52 : oozie (master) :: du -sh */*/build
> 6.9M sharelib/distcp/build
> 174M sharelib/hive/build
> 1.8G sharelib/pig/build
> 665M sharelib/spark/build
> 498M sharelib/sqoop/build
> 7.3M sharelib/streaming/build
> >> [85] 13:53 : oozie (master) :: du -sh */build
> 23M core/build
> 7.8M minitest/build
> 6.2M tools/build
> {noformat}
> They are containing a bunch of files generated during executions in the
> MiniCluster:
> {noformat}
> sharelib/spark/build/test/data/dfs//data/data1/current/BP-1306731421-172.30.65.43-1527256082477/current/finalized/subdir0/subdir4/blk_1073742879
> sharelib/spark/build/test/data/dfs//data/data1/current/BP-1306731421-172.30.65.43-1527256082477/current/finalized/subdir0/subdir4/blk_1073742863_2039.meta
> sharelib/spark/build/test/data/dfs//data/data1/current/BP-1306731421-172.30.65.43-1527256082477/current/finalized/subdir0/subdir4/blk_1073743029_2205.meta
> sharelib/spark/build/test/data/dfs//data/data1/current/BP-1306731421-172.30.65.43-1527256082477/current/finalized/subdir0/subdir4/blk_1073742975_2151.meta
> {noformat}
> "mvn clean" should be configured to clean these folders up as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (OOZIE-3178) ./bin/mkdistro.sh -Papache-release fails due to javadoc errors
[
https://issues.apache.org/jira/browse/OOZIE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Cseh reopened OOZIE-3178:
---
For the pre-commit hook for OOZIE-3228 we got:
https://builds.apache.org/job/PreCommit-OOZIE-Build/583/consoleFull
{quote}
[INFO]
[ERROR] Failed to execute goal on project oozie-sharelib-hcatalog: Could not
resolve dependencies for project
org.apache.oozie:oozie-sharelib-hcatalog:jar:5.1.0-SNAPSHOT: Failure to find
org.apache.oozie:oozie-sharelib-oozie:jar:5.1.0-SNAPSHOT in
https://repository.apache.org/content/groups/snapshots was cached in the local
repository, resolution will not be reattempted until the update interval of
apache.snapshots.repo has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :oozie-sharelib-hcatalog
{quote}
> ./bin/mkdistro.sh -Papache-release fails due to javadoc errors
> --
>
> Key: OOZIE-3178
> URL: https://issues.apache.org/jira/browse/OOZIE-3178
> Project: Oozie
> Issue Type: Bug
> Components: build
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.1.0
>
> Attachments: JAVADOC-patch.txt, OOZIE-3178-00.patch,
> OOZIE-3178-01.patch, OOZIE-3178-02.patch
>
>
> The distro creation script fails if it's called using the profile
> apache-release.
> The first batch of errors are javadoc-related.
> We should fix them and also enhance our pre-commit hook to prevent this from
> happening in the future.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3178) ./bin/mkdistro.sh -Papache-release fails due to javadoc errors
[ https://issues.apache.org/jira/browse/OOZIE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Cseh updated OOZIE-3178: -- Attachment: JAVADOC-patch.txt > ./bin/mkdistro.sh -Papache-release fails due to javadoc errors > -- > > Key: OOZIE-3178 > URL: https://issues.apache.org/jira/browse/OOZIE-3178 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Artem Ervits >Priority: Blocker > Fix For: 5.1.0 > > Attachments: JAVADOC-patch.txt, OOZIE-3178-00.patch, > OOZIE-3178-01.patch, OOZIE-3178-02.patch > > > The distro creation script fails if it's called using the profile > apache-release. > The first batch of errors are javadoc-related. > We should fix them and also enhance our pre-commit hook to prevent this from > happening in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OOZIE-3260) Remove item after first unsuccessful attempt from in-memory SLA map
[
https://issues.apache.org/jira/browse/OOZIE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3260:
Summary: Remove item after first unsuccessful attempt from in-memory SLA
map (was: Remove item after first try from in-memory SLA map)
> Remove item after first unsuccessful attempt from in-memory SLA map
> ---
>
> Key: OOZIE-3260
> URL: https://issues.apache.org/jira/browse/OOZIE-3260
> Project: Oozie
> Issue Type: Bug
> Components: coordinator, core, workflow
>Affects Versions: 5.0.0
>Reporter: Andras Piros
>Assignee: Andras Piros
>Priority: Major
>
> Despite having implemented OOZIE-3134, there are still cases where
> {{SLACalculatorMemory#slaMap}} and database contents still get out of sync.
> E.g. database contents of {{SLA_SUMMARY}} table have been purged manually
> from DB, or no corresponding {{WF_JOBS}} or {{COORD_JOBS}} entries exist
> anymore in DB.
> In those rare cases, we see {{JPAExecutorException}} instances like:
> {noformat}
> 2017-10-09 17:00:18,185 DEBUG openjpa.jdbc.SQL: SERVER[HOST] conn 1584126245> [0 ms] spent
> 2017-10-09 17:00:18,185 ERROR org.apache.oozie.sla.SLACalculatorMemory:
> SERVER[tplhc01c001.iuser.iroot.adidom.com] USER[-] GROUP[-] TOKEN[-] APP[-]
> JOB[438-170916014916144-oozie-oozi-C@556] ACTION[-] Exception in SLA
> processing for job [438-170916014916144-oozie-oozi-C@556]
> org.apache.oozie.executor.jpa.JPAExecutorException: E0604: Job does not exist
> [select w.eventProcessed from SLASummaryBean w where w.jobId = :id]
> at
> org.apache.oozie.executor.jpa.SLASummaryQueryExecutor.getSingleValue(SLASummaryQueryExecutor.java:161)
> at
> org.apache.oozie.sla.SLACalculatorMemory.updateJobSla(SLACalculatorMemory.java:480)
> at
> org.apache.oozie.sla.SLACalculatorMemory.updateAllSlaStatus(SLACalculatorMemory.java:601)
> {noformat}
> or
> {noformat}
> 2017-10-09 17:00:53,085 WARN
> org.apache.oozie.service.CallableQueueService$CompositeCallable: SERVER[HOST]
> USER[-] GROUP[-] TOKEN[-] APP[-] JOB[011-170813033731256-oozie-oozi-W]
> ACTION[011-170813033731256-oozie-oozi-W@sqoop_full_tbl_unload] exception
> callable [action.check], E0604: Job does not exist [select w.statusStr from
> WorkflowJobBean w where w.id = :id]
> org.apache.oozie.command.CommandException: E0604: Job does not exist [select
> w.statusStr from WorkflowJobBean w where w.id = :id]
> at
> org.apache.oozie.command.wf.ActionCheckXCommand.eagerLoadState(ActionCheckXCommand.java:97)
> at org.apache.oozie.command.XCommand.call(XCommand.java:256)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
> at
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.oozie.executor.jpa.JPAExecutorException: E0604: Job
> does not exist [select w.statusStr from WorkflowJobBean w where w.id = :id]
> at
> org.apache.oozie.executor.jpa.WorkflowJobQueryExecutor.get(WorkflowJobQueryExecutor.java:345)
> at
> org.apache.oozie.executor.jpa.WorkflowJobQueryExecutor.get(WorkflowJobQueryExecutor.java:38)
> at
> org.apache.oozie.command.wf.ActionCheckXCommand.eagerLoadState(ActionCheckXCommand.java:90)
> {noformat}
> Solution here is to remove any {{SLACalculatorMemory#slaMap}} entries that
> are causing those {{JPAExecutorException}} instances after the first
> unsuccessful run, to not cause huge logfiles. The items to be logged don't
> exist anymore, anyways.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3178) ./bin/mkdistro.sh -Papache-release fails due to javadoc errors
[ https://issues.apache.org/jira/browse/OOZIE-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492598#comment-16492598 ] Peter Cseh commented on OOZIE-3178: --- Also attached the output for the JAVADOC step. Both pre-patch and post-patch steps are failing with the same error. [~dbist13], can you take a look at it? > ./bin/mkdistro.sh -Papache-release fails due to javadoc errors > -- > > Key: OOZIE-3178 > URL: https://issues.apache.org/jira/browse/OOZIE-3178 > Project: Oozie > Issue Type: Bug > Components: build >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Artem Ervits >Priority: Blocker > Fix For: 5.1.0 > > Attachments: JAVADOC-patch.txt, OOZIE-3178-00.patch, > OOZIE-3178-01.patch, OOZIE-3178-02.patch > > > The distro creation script fails if it's called using the profile > apache-release. > The first batch of errors are javadoc-related. > We should fix them and also enhance our pre-commit hook to prevent this from > happening in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3109) Escape log-streaming's HTML-specific characters
[ https://issues.apache.org/jira/browse/OOZIE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492631#comment-16492631 ] Peter Cseh commented on OOZIE-3109: --- Thanks for the contribution! Can you extend the test to check that the message does contain something and not just check that the tag "script" is missing? > Escape log-streaming's HTML-specific characters > --- > > Key: OOZIE-3109 > URL: https://issues.apache.org/jira/browse/OOZIE-3109 > Project: Oozie > Issue Type: Bug >Affects Versions: 4.3.0 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Major > Attachments: OOZIE-3109-v001.patch, OOZIE-3109-v002.patch, > OOZIE-3109-v003.patch > > > That can be a security problem if we do not encode the stream read from log > files before transferred to the web browser. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Failed: OOZIE-3109 PreCommit Build #584
Jira: https://issues.apache.org/jira/browse/OOZIE-3109
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/584/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.71 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hcatalog].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/sqoop].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2363 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3109
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
-1 the patch contains 1 line(s) longer than 132 characters
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [tools].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2133
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/584/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 30160 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13116394/comment/16492720","id":"16492720","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
at hadoop dot apache dot
[jira] [Commented] (OOZIE-2339) Fluent Job API
[
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492730#comment-16492730
]
Peter Cseh commented on OOZIE-2339:
---
I was lurking around patch-13 and I found the workflow creation api quite handy.
I could not find a way to use the defined credentials in any of the actions
though.
> Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
> Issue Type: New Feature
> Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch,
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch,
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch,
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch,
> OOZIE-2339.014.patch, OOZIE-2339.015.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs. It
> would be nice if they could write them in something like Java, but we don't
> want to have to maintain a separate Java API for this. I was looking around
> and saw that JAXB might be the right thing here. From what I can tell, it
> lets you create Java classes from XSD schemas. So, we should be able to
> auto-generate a Java API for writing Oozie jobs, without having to really
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: OOZIE-3109 PreCommit Build #585
Jira: https://issues.apache.org/jira/browse/OOZIE-3109
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/585/
###
## LAST 100 LINES OF THE CONSOLE
###
[...truncated 1.71 MB...]
[DEBUG] There are no new bugs found in [sharelib/distcp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [core].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2363 bytes
[TRACE] Full summary file size is 1314 bytes
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
removed
[TRACE] File
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
removed
Running test-patch task BACKWARDS_COMPATIBILITY
Running test-patch task TESTS
Running test-patch task DISTRO
Testing JIRA OOZIE-3109
Cleaning local git workspace
+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
-1 the patch contains 1 line(s) longer than 132 characters
+1 the patch adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
+1 There are no new bugs found in [client].
+1 There are no new bugs found in [sharelib/hive].
+1 There are no new bugs found in [sharelib/sqoop].
+1 There are no new bugs found in [sharelib/spark].
+1 There are no new bugs found in [sharelib/streaming].
+1 There are no new bugs found in [sharelib/hive2].
+1 There are no new bugs found in [sharelib/oozie].
+1 There are no new bugs found in [sharelib/hcatalog].
+1 There are no new bugs found in [sharelib/pig].
+1 There are no new bugs found in [sharelib/distcp].
+1 There are no new bugs found in [docs].
+1 There are no new bugs found in [server].
+1 There are no new bugs found in [core].
+1 There are no new bugs found in [examples].
+1 There are no new bugs found in [webapp].
+1 There are no new bugs found in [tools].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2133
Tests failed at first run:
TestZKUUIDService#testResetSequence_withMultiThread
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch
-1 Overall result, please check the reported -1(s)
The full output of the test-patch run is available at
https://builds.apache.org/job/PreCommit-OOZIE-Build/585/
Adding comment to JIRA
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13116394/comment/16492748","id":"16492748","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","emailAddress":"blackhole
at hadoop dot apache dot
org","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA
OOZIE-3109\n\nCleaning local git
[jira] [Commented] (OOZIE-3109) Escape log-streaming's HTML-specific characters
[
https://issues.apache.org/jira/browse/OOZIE-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492748#comment-16492748
]
Hadoop QA commented on OOZIE-3109:
--
Testing JIRA OOZIE-3109
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2133
.{color:orange}Tests failed at first run:{color}
TestZKUUIDService#testResetSequence_withMultiThread
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/585/
> Escape log-streaming's HTML-specific characters
> ---
>
> Key: OOZIE-3109
> URL: https://issues.apache.org/jira/browse/OOZIE-3109
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 4.3.0
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Major
> Attachments: OOZIE-3109-v001.patch, OOZIE-3109-v002.patch,
> OOZIE-3109-v003.patch
>
>
> That can be a security problem if we do not encode the stream read from log
> files before transferred to the web browser.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (OOZIE-3263) Flaky test TestZKUUIDService#testResetSequence_withMultiThread
Peter Bacsko created OOZIE-3263:
---
Summary: Flaky test
TestZKUUIDService#testResetSequence_withMultiThread
Key: OOZIE-3263
URL: https://issues.apache.org/jira/browse/OOZIE-3263
Project: Oozie
Issue Type: Sub-task
Reporter: Peter Bacsko
Assignee: Peter Bacsko
The test case {{testResetSequence_withMultiThread}} failed with the following
error:
{noformat}
junit.framework.AssertionFailedError: expected:<3> but was:<2>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at junit.framework.TestCase.assertEquals(TestCase.java:409)
at
org.apache.oozie.service.TestZKUUIDService.testResetSequence_withMultiThread(TestZKUUIDService.java:214)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] Subscription: Oozie Patch Available
Issue Subscription Filter: Oozie Patch Available (98 issues) Subscriber: ooziedaily Key Summary OOZIE-3258 Surefire should not trim stack traces https://issues.apache.org/jira/browse/OOZIE-3258 OOZIE-3256 refactor OozieCLI class https://issues.apache.org/jira/browse/OOZIE-3256 OOZIE-3249 [tools] Instrumentation log parser https://issues.apache.org/jira/browse/OOZIE-3249 OOZIE-3218 Oozie Sqoop action with command splits the select clause into multiple parts due to delimiter being space https://issues.apache.org/jira/browse/OOZIE-3218 OOZIE-3199 Let system property restriction configurable https://issues.apache.org/jira/browse/OOZIE-3199 OOZIE-3196 Authorization: restrict world readability by user https://issues.apache.org/jira/browse/OOZIE-3196 OOZIE-3194 Oozie should set proper permissions to sharelib after upload https://issues.apache.org/jira/browse/OOZIE-3194 OOZIE-3193 Applications are not killed when submitted via subworkflow https://issues.apache.org/jira/browse/OOZIE-3193 OOZIE-3186 Oozie is unable to use configuration linked using jceks://file/... https://issues.apache.org/jira/browse/OOZIE-3186 OOZIE-3179 Adding a configurable config-default.xml location to a workflow https://issues.apache.org/jira/browse/OOZIE-3179 OOZIE-3170 Oozie Diagnostic Bundle tool fails with NPE due to missing service class https://issues.apache.org/jira/browse/OOZIE-3170 OOZIE-3160 PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting https://issues.apache.org/jira/browse/OOZIE-3160 OOZIE-3135 Configure log4j2 in SqoopMain https://issues.apache.org/jira/browse/OOZIE-3135 OOZIE-3109 Escape log-streaming's HTML-specific characters https://issues.apache.org/jira/browse/OOZIE-3109 OOZIE-3091 Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper" https://issues.apache.org/jira/browse/OOZIE-3091 OOZIE-3071 Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 than Spark 2.2.0 https://issues.apache.org/jira/browse/OOZIE-3071 OOZIE-3063 Sanitizing variables that are part of openjpa.ConnectionProperties https://issues.apache.org/jira/browse/OOZIE-3063 OOZIE-3062 Set HADOOP_CONF_DIR for spark action https://issues.apache.org/jira/browse/OOZIE-3062 OOZIE-3061 Kill only those child jobs which are not already killed https://issues.apache.org/jira/browse/OOZIE-3061 OOZIE-2956 Fix Findbugs warnings related to reliance on default encoding in oozie-core https://issues.apache.org/jira/browse/OOZIE-2956 OOZIE-2955 Fix Findbugs warnings related to reliance on default encoding in oozie-client https://issues.apache.org/jira/browse/OOZIE-2955 OOZIE-2954 Fix Checkstyle issues in oozie-client https://issues.apache.org/jira/browse/OOZIE-2954 OOZIE-2953 Fix Checkstyle issues in oozie-tools https://issues.apache.org/jira/browse/OOZIE-2953 OOZIE-2952 Fix Findbugs warnings in oozie-sharelib-oozie https://issues.apache.org/jira/browse/OOZIE-2952 OOZIE-2949 Escape quotes whitespaces in Sqoop field https://issues.apache.org/jira/browse/OOZIE-2949 OOZIE-2942 [examples] Fix Findbugs warnings https://issues.apache.org/jira/browse/OOZIE-2942 OOZIE-2927 Append new line character for Hive2 query using query tag https://issues.apache.org/jira/browse/OOZIE-2927 OOZIE-2877 Oozie Git Action https://issues.apache.org/jira/browse/OOZIE-2877 OOZIE-2834 ParameterVerifier logging non-useful warning for workflow definition https://issues.apache.org/jira/browse/OOZIE-2834 OOZIE-2833 when using uber mode the regex pattern used in the extractHeapSizeMB method does not allow heap sizes specified in bytes. https://issues.apache.org/jira/browse/OOZIE-2833 OOZIE-2829 Improve sharelib upload to accept multiple source folders https://issues.apache.org/jira/browse/OOZIE-2829 OOZIE-2812 SparkConfigurationService should support loading configurations from multiple Spark versions https://issues.apache.org/jira/browse/OOZIE-2812 OOZIE-2795 Create lib directory or symlink for Oozie CLI during packaging https://issues.apache.org/jira/browse/OOZIE-2795 OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters https://issues.apache.org/jira/browse/OOZIE-2791 OOZIE-2784 Include WEEK as a parameter in the Coordinator Expression Language Evaulator https://issues.apache.org/jira/browse/OOZIE-2784 OOZIE-2779 Mask Hive2 action Beeline JDBC password https://issues.apache.org/jira/browse/OOZIE-2779 OOZIE-2736 Reduce the number of threads during test execution
[jira] [Commented] (OOZIE-3256) refactor OozieCLI class
[
https://issues.apache.org/jira/browse/OOZIE-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492415#comment-16492415
]
Andras Piros commented on OOZIE-3256:
-
[~dbist13] good idea, and thanks for the contribution!
Can you hold on while OOZIE-2339 finds its way to {{master}} (hope in a few
weeks)? It makes a lot of noise extending also {{OozieCLI}} / {{TestOozieCLI}}
with new functionality, and your work would otherwise surely interfere. Thanks!
> refactor OozieCLI class
> ---
>
> Key: OOZIE-3256
> URL: https://issues.apache.org/jira/browse/OOZIE-3256
> Project: Oozie
> Issue Type: Improvement
>Affects Versions: 5.0.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Minor
> Attachments: OOZIE-3256.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64029/ --- (Updated May 28, 2018, 8:50 a.m.) Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter. Changes --- Pre-commit build and unit test fixtures. Repository: oozie-git Description --- OOZIE-2339 Provide an API for writing jobs based on the XSD schemas Diffs (updated) - client/pom.xml 7d36e2d5941e598520a653a9eb4b293c763378e3 client/src/main/java/org/apache/oozie/cli/OozieCLI.java 08e2b91e6d9209278f9e086e23cfb4e555f7bd52 client/src/main/java/org/apache/oozie/client/ApiJarFactory.java PRE-CREATION client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION client/src/main/java/org/apache/oozie/client/OozieClient.java 4c8175628ba5ee346d97ad984532662032b88acc core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 38fb006980c0acb582b4f91175be1ced8a1afa79 core/src/main/java/org/apache/oozie/local/LocalOozie.java 9ab646cd53223d903e387eb92aa103a7b6577d0f core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java e1bd3cf61aabfa4311881b2944fc9f10ffe9145d core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java d4b08711223356c8d4f58a3391b6b87efa0a2b73 core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java b86fa6ace43aead115302aad3ae15368b15f1923 core/src/main/resources/oozie-default.xml c54db34ffeb560eef3d87abd4d926b4e6a6b5300 core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 3395cc25d27805444ba0b6463357c4ecff977ac1 core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java fb481e367653187a94199069199f1d1b06d1cecf core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 0a030d660f9c545b520afcce458a27ba2a367658 docs/src/site/twiki/DG_CommandLineTool.twiki 9a174596dd78fb15989daf4e88936a9a2b09ef13 docs/src/site/twiki/DG_CustomActionExecutor.twiki 4acbf0d40ae36c441f8510e107243603fa0e001b docs/src/site/twiki/DG_Examples.twiki 13dfa28c0eecab70321886b2806a791346c46d3b docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION examples/pom.xml c13febc81cf5b26b4af90885787113e247ae7cb8 examples/src/main/java/org/apache/oozie/example/fluentjob/Credentials.java PRE-CREATION examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java PRE-CREATION examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java PRE-CREATION examples/src/main/java/org/apache/oozie/example/fluentjob/MultipleShellActions.java PRE-CREATION examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java PRE-CREATION examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java PRE-CREATION examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java PRE-CREATION findbugs-filter.xml 03ee4d103f46487493fd725224cf4912ac7bc7dd fluent-job/fluent-job-api/pom.xml PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBaseBuilder.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chgrp.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChgrpBuilder.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chmod.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChmodBuilder.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Delete.java PRE-CREATION fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/DistcpAction.java PRE-CREATION
[jira] [Updated] (OOZIE-2339) Fluent Job API
[
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-2339:
Attachment: OOZIE-2339.015.patch
> Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
> Issue Type: New Feature
> Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch,
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch,
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch,
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch,
> OOZIE-2339.014.patch, OOZIE-2339.015.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs. It
> would be nice if they could write them in something like Java, but we don't
> want to have to maintain a separate Java API for this. I was looking around
> and saw that JAXB might be the right thing here. From what I can tell, it
> lets you create Java classes from XSD schemas. So, we should be able to
> auto-generate a Java API for writing Oozie jobs, without having to really
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2339) Fluent Job API
[
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492419#comment-16492419
]
Hadoop QA commented on OOZIE-2339:
--
PreCommit-OOZIE-Build started
> Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
> Issue Type: New Feature
> Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch,
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch,
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch,
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch,
> OOZIE-2339.014.patch, OOZIE-2339.015.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs. It
> would be nice if they could write them in something like Java, but we don't
> want to have to maintain a separate Java API for this. I was looking around
> and saw that JAXB might be the right thing here. From what I can tell, it
> lets you create Java classes from XSD schemas. So, we should be able to
> auto-generate a Java API for writing Oozie jobs, without having to really
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-2339) Fluent Job API
[
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492429#comment-16492429
]
Hadoop QA commented on OOZIE-2339:
--
Testing JIRA OOZIE-2339
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 93 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 1 new javac warning(s)
{color:red}-1{color} There are [2] new bugs found below threshold in total that
must be fixed.
. {color:orange}0{color} There are [1] new bugs found in [examples] that would
be nice to have fixed.
. You can find the FindBugs diff here: examples/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:red}-1{color} There are [2] new bugs found below threshold in [client]
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
client/findbugs-new.html
. The most important FindBugs errors are:
. At OozieCLI.java:[line 1420]: java/io/File.init(Ljava/lang/String;)V
reads a file whose location might be specified by user input
. java/io/FileOutputStream.init(Ljava/lang/String;)V writes to a file
whose location might be specified by user input: At OozieCLI.java:[line 1420]
. At ApiJarFactory.java:[line 72]: At ApiJarFactory.java:[line 72]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/581/
> Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
> Issue Type: New Feature
> Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch,
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch,
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch,
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch,
> OOZIE-2339.014.patch, OOZIE-2339.015.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs. It
> would be nice if they could write them in something like Java, but we don't
> want to have to maintain a separate Java API for this. I was looking around
> and saw that JAXB might be the right thing here. From what I can tell, it
> lets you create
Failed: OOZIE-2339 PreCommit Build #581
Jira: https://issues.apache.org/jira/browse/OOZIE-2339 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/581/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.73 MB...] [DEBUG] There are no new bugs found in [sharelib/sqoop]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [sharelib/oozie]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [sharelib/distcp]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [sharelib/spark]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [ERROR] There are [2] new bugs found below threshold in [client]. [DEBUG] You can find the FindBugs diff here (look for the red and orange ones): client/findbugs-new.html [ERROR] There are [2] new bugs found below threshold in total that must be fixed. [TRACE] FindBugs diffs checked and reports created [TRACE] Summary file size is 3115 bytes [TRACE] Full summary file size is 2071 bytes [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar] removed [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum] removed Running test-patch task BACKWARDS_COMPATIBILITY Running test-patch task TESTS Running test-patch task DISTRO Testing JIRA OOZIE-2339 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any line longer than 132 +1 the patch adds/modifies 93 testcase(s) +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 JAVADOC +1 the patch does not seem to introduce new Javadoc warning(s) +1 the patch does not seem to introduce new Javadoc error(s) ERROR: the current HEAD has 2 Javadoc error(s) -1 COMPILE +1 HEAD compiles +1 patch compiles -1 the patch seems to introduce 1 new javac warning(s) -1 There are [2] new bugs found below threshold in total that must be fixed. 0 There are [1] new bugs found in [examples] that would be nice to have fixed. You can find the FindBugs diff here: examples/findbugs-new.html +1 There are no new bugs found in [webapp]. +1 There are no new bugs found in [core]. +1 There are no new bugs found in [tools]. +1 There are no new bugs found in [fluent-job/fluent-job-api]. +1 There are no new bugs found in [server]. +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/oozie]. +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [sharelib/spark]. -1 There are [2] new bugs found below threshold in [client] that must be fixed. You can find the FindBugs diff here (look for the red and orange ones): client/findbugs-new.html The most important FindBugs errors are: At OozieCLI.java:[line 1420]: java/io/File.init(Ljava/lang/String;)V reads a file whose location might be specified by user input java/io/FileOutputStream.init(Ljava/lang/String;)V writes to a file whose location might be specified by user input: At OozieCLI.java:[line 1420] At ApiJarFactory.java:[line 72]: At ApiJarFactory.java:[line 72] +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files -1 TESTS - patch does not compile, cannot run testcases -1 DISTRO -1 distro tarball fails with the patch -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/581/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
[jira] [Commented] (OOZIE-2339) Fluent Job API
[
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492432#comment-16492432
]
Hadoop QA commented on OOZIE-2339:
--
PreCommit-OOZIE-Build started
> Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
> Issue Type: New Feature
> Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch,
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch,
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch,
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch,
> OOZIE-2339.014.patch, OOZIE-2339.015.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs. It
> would be nice if they could write them in something like Java, but we don't
> want to have to maintain a separate Java API for this. I was looking around
> and saw that JAXB might be the right thing here. From what I can tell, it
> lets you create Java classes from XSD schemas. So, we should be able to
> auto-generate a Java API for writing Oozie jobs, without having to really
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Failed: OOZIE-2339 PreCommit Build #582
Jira: https://issues.apache.org/jira/browse/OOZIE-2339 Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/582/ ### ## LAST 100 LINES OF THE CONSOLE ### [...truncated 1.73 MB...] [DEBUG] There are no new bugs found in [sharelib/sqoop]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [sharelib/oozie]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [sharelib/distcp]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [DEBUG] There are no new bugs found in [sharelib/spark]. [TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts [ERROR] There are [2] new bugs found below threshold in [client]. [DEBUG] You can find the FindBugs diff here (look for the red and orange ones): client/findbugs-new.html [ERROR] There are [2] new bugs found below threshold in total that must be fixed. [TRACE] FindBugs diffs checked and reports created [TRACE] Summary file size is 3115 bytes [TRACE] Full summary file size is 2071 bytes [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar] removed [TRACE] File [/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum] removed Running test-patch task BACKWARDS_COMPATIBILITY Running test-patch task TESTS Running test-patch task DISTRO Testing JIRA OOZIE-2339 Cleaning local git workspace +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS +1 the patch does not introduce any @author tags +1 the patch does not introduce any tabs +1 the patch does not introduce any trailing spaces +1 the patch does not introduce any line longer than 132 +1 the patch adds/modifies 93 testcase(s) +1 RAT +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC +1 JAVADOC +1 the patch does not seem to introduce new Javadoc warning(s) +1 the patch does not seem to introduce new Javadoc error(s) ERROR: the current HEAD has 2 Javadoc error(s) -1 COMPILE +1 HEAD compiles +1 patch compiles -1 the patch seems to introduce 1 new javac warning(s) -1 There are [2] new bugs found below threshold in total that must be fixed. 0 There are [1] new bugs found in [examples] that would be nice to have fixed. You can find the FindBugs diff here: examples/findbugs-new.html +1 There are no new bugs found in [webapp]. +1 There are no new bugs found in [core]. +1 There are no new bugs found in [tools]. +1 There are no new bugs found in [fluent-job/fluent-job-api]. +1 There are no new bugs found in [server]. +1 There are no new bugs found in [docs]. +1 There are no new bugs found in [sharelib/hive2]. +1 There are no new bugs found in [sharelib/pig]. +1 There are no new bugs found in [sharelib/streaming]. +1 There are no new bugs found in [sharelib/hive]. +1 There are no new bugs found in [sharelib/hcatalog]. +1 There are no new bugs found in [sharelib/sqoop]. +1 There are no new bugs found in [sharelib/oozie]. +1 There are no new bugs found in [sharelib/distcp]. +1 There are no new bugs found in [sharelib/spark]. -1 There are [2] new bugs found below threshold in [client] that must be fixed. You can find the FindBugs diff here (look for the red and orange ones): client/findbugs-new.html The most important FindBugs errors are: At OozieCLI.java:[line 1420]: java/io/File.init(Ljava/lang/String;)V reads a file whose location might be specified by user input java/io/FileOutputStream.init(Ljava/lang/String;)V writes to a file whose location might be specified by user input: At OozieCLI.java:[line 1420] At ApiJarFactory.java:[line 72]: At ApiJarFactory.java:[line 72] +1 BACKWARDS_COMPATIBILITY +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations +1 the patch does not modify JPA files -1 TESTS - patch does not compile, cannot run testcases -1 DISTRO -1 distro tarball fails with the patch -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at https://builds.apache.org/job/PreCommit-OOZIE-Build/582/ Adding comment to JIRA % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 0 00 00 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 00 00 0 0 0 --:--:-- --:--:-- --:--:--
[jira] [Commented] (OOZIE-2339) Fluent Job API
[
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492436#comment-16492436
]
Hadoop QA commented on OOZIE-2339:
--
Testing JIRA OOZIE-2339
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 93 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:red}-1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:red}-1{color} the patch seems to introduce 1 new javac warning(s)
{color:red}-1{color} There are [2] new bugs found below threshold in total that
must be fixed.
. {color:orange}0{color} There are [1] new bugs found in [examples] that would
be nice to have fixed.
. You can find the FindBugs diff here: examples/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:red}-1{color} There are [2] new bugs found below threshold in [client]
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones):
client/findbugs-new.html
. The most important FindBugs errors are:
. At OozieCLI.java:[line 1420]: java/io/File.init(Ljava/lang/String;)V
reads a file whose location might be specified by user input
. java/io/FileOutputStream.init(Ljava/lang/String;)V writes to a file
whose location might be specified by user input: At OozieCLI.java:[line 1420]
. At ApiJarFactory.java:[line 72]: At ApiJarFactory.java:[line 72]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:red}-1 DISTRO{color}
.{color:red}-1{color} distro tarball fails with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/582/
> Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
> Issue Type: New Feature
> Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch,
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch,
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch,
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch,
> OOZIE-2339.014.patch, OOZIE-2339.015.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs. It
> would be nice if they could write them in something like Java, but we don't
> want to have to maintain a separate Java API for this. I was looking around
> and saw that JAXB might be the right thing here. From what I can tell, it
> lets you create
[jira] [Commented] (OOZIE-2105) Make version of submodules configurable with parent version
[
https://issues.apache.org/jira/browse/OOZIE-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492440#comment-16492440
]
Peter Cseh commented on OOZIE-2105:
---
I still see value in removing superfluous lines here and there.
When we do a rebase in CDH, *every* pom.xml has a conflict in it as we're
changing the version from 5.0.0 to something like 5.0.0-cdh6.0.0. It can be
resolved easily, but still, it makes things like automated testing more
difficult.
> Make version of submodules configurable with parent version
>
>
> Key: OOZIE-2105
> URL: https://issues.apache.org/jira/browse/OOZIE-2105
> Project: Oozie
> Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2105-02.patch, OOZIE-2105-03.patch,
> OOZIE-6252.1.patch
>
>
> Currently the versions of the Oozie sub-modules are hard-coded with parent
> version. If someone changes the parent version, all sub-modules versions will
> need to be explicitly updated. For example, in hadooplibs/hadoop-1/pom.xml we
> use
> org.apache.oozie
> oozie-hadoop
> 1.1.1.oozie-4.1.0
> ...
> If you want to modify the Oozie version to 4.1.1 (say), you need to go to all
> pom files and manually change replace 4.1.0 to 4.1.1.
> This JIRA is to use "parent.version" instead of hard-coding. For example, use
> this :
> 1.1.1.oozie-${parent.version}
>
> This will allow to change only the parent version in root pom.xml.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OOZIE-3228) [Spark action] Can't load properties from spark-defaults.conf
[
https://issues.apache.org/jira/browse/OOZIE-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-3228:
Attachment: OOZIE-3228.amend.001.patch
> [Spark action] Can't load properties from spark-defaults.conf
> -
>
> Key: OOZIE-3228
> URL: https://issues.apache.org/jira/browse/OOZIE-3228
> Project: Oozie
> Issue Type: Bug
> Components: action
>Affects Versions: 5.0.0, 4.3.1
>Reporter: Tang Yan
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-3228.001.patch, OOZIE-3228.002.patch,
> OOZIE-3228.003.patch, OOZIE-3228.004.patch, OOZIE-3228.005.patch,
> OOZIE-3228.amend.001.patch
>
>
> When I create a Oozie workflow to launch a spark action, the spark job can't
> load the configured properties in {{spark-defaults.conf}}. I've configured
> each NodeManager as the Spark gateway role, so {{spark-defaults.conf}} is
> generated in {{/etc/spark/conf/}} on each worker node.
> In {{spark-defaults.conf}} some configuration I've set into.
> {noformat}
> spark.executor.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> spark.driver.extraClassPath=/etc/hbase/conf:/etc/hive/conf
> {noformat}
> But in the Oozie spark job, they're not loaded automatically.
> {noformat}
> --conf spark.executor.extraClassPath=$PWD/*
> --conf spark.driver.extraClassPath=$PWD/*
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
