Review Request 48469: OOZIE-2517 Add support for startCreatedTime and endCreatedTime filters for coord and bundles

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48469/
---

Review request for oozie.


Bugs: OOZIE-2517
https://issues.apache.org/jira/browse/OOZIE-2517


Repository: oozie-git


Description
---

Add support for startCreatedTime and endCreatedTime filters for coord and 
bundles


Diffs
-

  core/src/main/java/org/apache/oozie/CoordinatorEngine.java 79d0d97 
  
core/src/main/java/org/apache/oozie/executor/jpa/BundleJobInfoGetJPAExecutor.java
 4b631c1 
  core/src/main/java/org/apache/oozie/executor/jpa/BundleJobQueryExecutor.java 
bfd1dae 
  
core/src/main/java/org/apache/oozie/executor/jpa/CoordJobInfoGetJPAExecutor.java
 c924f44 
  core/src/main/java/org/apache/oozie/store/StoreStatusFilter.java d442aa8 
  core/src/main/java/org/apache/oozie/util/JobsFilterUtils.java 52d3e9d 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestBundleJobInfoGetJPAExecutor.java
 5714040 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestBundleJobQueryExecutor.java
 97cbb7f 
  
core/src/test/java/org/apache/oozie/executor/jpa/TestCoordJobInfoGetJPAExecutor.java
 5335213 

Diff: https://reviews.apache.org/r/48469/diff/


Testing
---

unit tests done. Manual testing done.


Thanks,

Abhishek Bafna

Review Request 48468: OOZIE-2516 Update web service documentation for jobs API

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48468/
---

Review request for oozie.


Bugs: OOZIE-2516
https://issues.apache.org/jira/browse/OOZIE-2516


Repository: oozie-git


Description
---

Update web service documentation for jobs API


Diffs
-

  docs/src/site/twiki/WebServicesAPI.twiki a34f2d3 

Diff: https://reviews.apache.org/r/48468/diff/


Testing
---

Doc changes only.


Thanks,

Abhishek Bafna

Review Request 48467: OOZIE-2515 Duplicate information for "Changing endtime/pausetime of a Bundle Job" in CommandLineTool wiki

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48467/
---

Review request for oozie.


Bugs: OOZIE-2515
https://issues.apache.org/jira/browse/OOZIE-2515


Repository: oozie-git


Description
---

Duplicate information for "Changing endtime/pausetime of a Bundle Job" in 
CommandLineTool wiki


Diffs
-

  docs/src/site/twiki/DG_CommandLineTool.twiki ff1cce5 

Diff: https://reviews.apache.org/r/48467/diff/


Testing
---

Wiki change only.


Thanks,

Abhishek Bafna

Review Request 48466: OOZIE-2506 Add logs into RecoverService for logging information about queued commnads

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48466/
---

Review request for oozie.


Bugs: OOZIE-2506
https://issues.apache.org/jira/browse/OOZIE-2506


Repository: oozie-git


Description
---

Add logs into RecoverService for logging information about queued commnads


Diffs
-

  core/src/main/java/org/apache/oozie/service/RecoveryService.java 49f47d0 

Diff: https://reviews.apache.org/r/48466/diff/


Testing
---


Thanks,

Abhishek Bafna

Review Request 48465: OOZIE-2500 -DtestJarSimple option mentioned in minioozie doc does not work

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48465/
---

Review request for oozie.


Bugs: OOZIE-2500
https://issues.apache.org/jira/browse/OOZIE-2500


Repository: oozie-git


Description
---

-DtestJarSimple option mentioned in minioozie doc does not work


Diffs
-

  core/pom.xml b72ea7d 

Diff: https://reviews.apache.org/r/48465/diff/


Testing
---


Thanks,

Abhishek Bafna

Review Request 48464: OOZIE-2497 Testcase fails on windows with hard coded URIs

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48464/
---

Review request for oozie.


Bugs: OOZIE-2497
https://issues.apache.org/jira/browse/OOZIE-2497


Repository: oozie-git


Description
---

Testcase fails on windows with hard coded URIs


Diffs
-

  
core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 
52eb9dd 
  core/src/test/java/org/apache/oozie/command/wf/TestSignalXCommand.java 
810bc1e 
  
sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java
 d4095da 

Diff: https://reviews.apache.org/r/48464/diff/


Testing
---

Unit tests passed.


Thanks,

Abhishek Bafna

Review Request 48463: OOZIE-2496 Testcase failure due to environment specific line.separator value

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48463/
---

Review request for oozie.


Bugs: OOZIE-2496
https://issues.apache.org/jira/browse/OOZIE-2496


Repository: oozie-git


Description
---

Testcase failure due to environment specific line.separator value


Diffs
-

  core/src/main/java/org/apache/oozie/util/TimestampedMessageParser.java 
f691409 
  core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 54bfc16 
  core/src/test/java/org/apache/oozie/util/TestTimestampedMessageParser.java 
06af822 

Diff: https://reviews.apache.org/r/48463/diff/


Testing
---

unit tests passed.


Thanks,

Abhishek Bafna

Review Request 48462: OOZIE-2491 oozie acl cannot specify group, it does`t work

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48462/
---

Review request for oozie.


Bugs: OOZIE-2491
https://issues.apache.org/jira/browse/OOZIE-2491


Repository: oozie-git


Description
---

oozie acl cannot specify group,it does`t work


Diffs
-

  core/src/main/java/org/apache/oozie/service/AuthorizationService.java f281873 
  core/src/test/java/org/apache/oozie/service/TestAuthorizationService.java 
ce461c1 

Diff: https://reviews.apache.org/r/48462/diff/


Testing
---

Updated the unit test accordingly.


Thanks,

Abhishek Bafna

Review Request 48461: OOZIE-2493 Remove action.xml file from local file system instead of hadoop file system

[Abhishek Bafna]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48461/
---

Review request for oozie.


Bugs: OOZIE-2493
https://issues.apache.org/jira/browse/OOZIE-2493


Repository: oozie-git


Description
---

Remove action.xml file from local file system instead of hadoop file system


Diffs
-

  core/src/test/java/org/apache/oozie/action/hadoop/TestDistcpMain.java 9581c5f 

Diff: https://reviews.apache.org/r/48461/diff/


Testing
---

Unit test passed.


Thanks,

Abhishek Bafna

[jira] [Commented] (OOZIE-2550) Flaky tests in TestZKUUIDService.java

[abhishek bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321875#comment-15321875
 ] 

abhishek bafna commented on OOZIE-2550:
---

In the same context as [~rkanter] mentioned regarding making {{setMaxSequence}} 
a static method. We can also reduce it's visibility from {{public}} to 
{{default}}. Because this is accessed in the {{TestZKUUIDService}} only, which 
is in the same package as {{ZKUUIDService}} so scope can be reduce.

> Flaky tests in TestZKUUIDService.java
> -
>
> Key: OOZIE-2550
> URL: https://issues.apache.org/jira/browse/OOZIE-2550
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2550-001.patch, OOZIE-2550-002.patch, 
> OOZIE-2550-003.patch, OOZIE-2550-004.patch
>
>
> 1. Test case testMultipleIDGeneration_withMultiThread in TestZKUUIDService 
> uses an ArrayList which is written by two threads simultaneously. This is 
> dangerous and the list must be externally synchronized to prevent race 
> conditions.
> Another problem is that you cannot put items at arbitrary indexes in an 
> ArrayList. For example, if the list is empty, the following code throws 
> ArrayIndexOutOfBoundException:
> {code}
> List test = new ArrayList<>(1);
> test.add(22, true);
> {code}
> In an unlucky scheduling event, the following can happen:
> 1. The list has a certain number of elements, the value of "size" inside the 
> list is 1000
> 2. Thread-1 retrieves the next ID from ZK UUID service, which is 1001
> 3. Thread-2 retrieves the next ID from ZK UUID service, which is 1002
> 4. Thread-2 runs faster than Thread-1 and tries to call {{list.add(1002, 
> true)}} which fails. 
> The following error was caught during a test run:
> {code}
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 53.571 sec 
> <<< FAILURE!
> testMultipleIDGeneration_withMultiThread(org.apache.oozie.service.TestZKUUIDService)
>   Time elapsed: 0.02 sec  <<< ERROR!
> java.lang.IndexOutOfBoundsException: Index: 89, Size: 89
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at 
> org.apache.oozie.service.TestZKUUIDService.testMultipleIDGeneration_withMultiThread(TestZKUUIDService.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2. The order in which the tests are executed is random. The problem is that 
> testResetSequence sets the maximum sequence number to 900. Because this value 
> is stored in a static variable inside ZKUUIDService, it affects 
> testIDGeneration and testMultipleIDGeneration if these tests are run after 
> testResetSequence.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2541) Possible resource leak in Hive2Credentials

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2541:
-
Summary: Possible resource leak in Hive2Credentials  (was: Resource leak in 
Hive2Credentials)

> Possible resource leak in Hive2Credentials
> --
>
> Key: OOZIE-2541
> URL: https://issues.apache.org/jira/browse/OOZIE-2541
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Trivial
> Attachments: OOZIE-2541-001.patch, OOZIE-2541-002.patch, 
> OOZIE-2541-003.patch
>
>
> Connection to Hive is not closed in case of an SQLException:
> {code}
> Connection con = DriverManager.getConnection(url);
> XLog.getLog(getClass()).debug("Connected successfully to " + url);
> // get delegation token for the given proxy user
> String tokenStr = 
> ((HiveConnection)con).getDelegationToken(jobconf.get(USER_NAME), principal);
> XLog.getLog(getClass()).debug("Got token");
> {code}
> Close the Connection in a finally block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2550) Flaky tests in TestZKUUIDService.java

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321776#comment-15321776
 ] 

Robert Kanter commented on OOZIE-2550:
--

I think the {{AtomicReferenceArray}} is a bit overkill here.  The two threads 
should get non-overlapping sets of IDs, which are used as indexes.  So there 
should never be any race conditions on the individual array elements 
themselves.  If that somehow does happen, then we'll know because the test will 
fail because at least one of the array elements will be {{false}}.  In fact, it 
would be helpful if this test does fail, if we were to log the IDs generated by 
each thread.  
So, I think we should:
# Replace the {{AtomicReferenceArray}} with a boolean array:
{code:java}
boolean[] result = new boolean[size];
{code}
This should default to {{false}} for all elements, but if you want to be extra 
sure you can always add {{Arrays.fill(result, false);}}
# Print out the IDs generated by each thread and which thread generated them 
for easier debugging.
# In {{setUp()}}, use a try-finally block to make sure the uuid service there 
is always destroyed.  
#- Actually, I think we'd be better off making {{setMaxSequence}} a static 
method.  It's setting a static variable, so that makes sense.  Then you don't 
have to create a new {{ZKUUIDService}} here.
# I might be missing something here, but don't we need to reset the sequence to 
start at 0 somewhere?  The array is from 0 to 1, but what if a previous 
test had left the sequence at, say 50?  We use the IDs as indexes, so this 
wouldn't work correctly, right?

> Flaky tests in TestZKUUIDService.java
> -
>
> Key: OOZIE-2550
> URL: https://issues.apache.org/jira/browse/OOZIE-2550
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2550-001.patch, OOZIE-2550-002.patch, 
> OOZIE-2550-003.patch, OOZIE-2550-004.patch
>
>
> 1. Test case testMultipleIDGeneration_withMultiThread in TestZKUUIDService 
> uses an ArrayList which is written by two threads simultaneously. This is 
> dangerous and the list must be externally synchronized to prevent race 
> conditions.
> Another problem is that you cannot put items at arbitrary indexes in an 
> ArrayList. For example, if the list is empty, the following code throws 
> ArrayIndexOutOfBoundException:
> {code}
> List test = new ArrayList<>(1);
> test.add(22, true);
> {code}
> In an unlucky scheduling event, the following can happen:
> 1. The list has a certain number of elements, the value of "size" inside the 
> list is 1000
> 2. Thread-1 retrieves the next ID from ZK UUID service, which is 1001
> 3. Thread-2 retrieves the next ID from ZK UUID service, which is 1002
> 4. Thread-2 runs faster than Thread-1 and tries to call {{list.add(1002, 
> true)}} which fails. 
> The following error was caught during a test run:
> {code}
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 53.571 sec 
> <<< FAILURE!
> testMultipleIDGeneration_withMultiThread(org.apache.oozie.service.TestZKUUIDService)
>   Time elapsed: 0.02 sec  <<< ERROR!
> java.lang.IndexOutOfBoundsException: Index: 89, Size: 89
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at 
> org.apache.oozie.service.TestZKUUIDService.testMultipleIDGeneration_withMultiThread(TestZKUUIDService.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> 

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/
---

(Updated June 9, 2016, 2 a.m.)


Review request for oozie.


Bugs: OOZIE-2445
https://issues.apache.org/jira/browse/OOZIE-2445


Repository: oozie-git


Description
---

Doc for - Specifying coordinator input datasets in more logical ways 
(OOZIE-1976)


Diffs (updated)
-

  docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
75b7354a12ae8c255b9d713856b44ec939092c95 
  examples/src/main/apps/coord-input-logic/coordinator.xml 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  examples/src/main/apps/coord-input-logic/job.properties 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  examples/src/main/apps/coord-input-logic/workflow.xml 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 

Diff: https://reviews.apache.org/r/48149/diff/


Testing
---


Thanks,

Purshotam Shah

[jira] [Commented] (OOZIE-2541) Resource leak in Hive2Credentials

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321761#comment-15321761
 ] 

Robert Kanter commented on OOZIE-2541:
--

+1

> Resource leak in Hive2Credentials
> -
>
> Key: OOZIE-2541
> URL: https://issues.apache.org/jira/browse/OOZIE-2541
> Project: Oozie
>  Issue Type: Bug
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Trivial
> Attachments: OOZIE-2541-001.patch, OOZIE-2541-002.patch, 
> OOZIE-2541-003.patch
>
>
> Connection to Hive is not closed in case of an SQLException:
> {code}
> Connection con = DriverManager.getConnection(url);
> XLog.getLog(getClass()).debug("Connected successfully to " + url);
> // get delegation token for the given proxy user
> String tokenStr = 
> ((HiveConnection)con).getDelegationToken(jobconf.get(USER_NAME), principal);
> XLog.getLog(getClass()).debug("Got token");
> {code}
> Close the Connection in a finally block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321746#comment-15321746
 ] 

Hadoop QA commented on OOZIE-2547:
--

Testing JIRA OOZIE-2547

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch, 
> yarn-cluster_launcher.txt
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #2967

[Apache Jenkins Server]

See 

Changes:

[rohini] OOZIE-2563 Pass spark-defaults.conf to spark action (satishsaley via

--
[...truncated 2449 lines...]
[INFO] Building Apache Oozie Tools 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ oozie-tools ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-tools ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-tools 
---
[INFO] Compiling 2 source files to 

[WARNING] 
:[156,20]
 [deprecation] getConf() in Services has been deprecated
[WARNING] 
:[158,20]
 [deprecation] getConf() in Services has been deprecated
[WARNING] 
:[166,37]
 [deprecation] getConf() in Services has been deprecated
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Compiling 6 source files to 

[WARNING] 
:[196,20]
 [deprecation] getConf() in Services has been deprecated
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ oozie-mini ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Compiling 1 source file to 

[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie WorkflowGenerator 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ 
oozie-workflowgenerator ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
oozie-workflowgenerator ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-workflowgenerator ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] 

[jira] [Updated] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2547:
-
Attachment: yarn-cluster_launcher.txt

yarn-client and local mode both work for me.  However, yarn-cluster is still 
giving me the {{IllegalArgumentException}}.  I've attached the full logs 
(yarn-cluster_launcher.txt) in case that's helpful.  Any ideas?

Bu the way, I did notice that yarn-client with this patch seems to be about 15 
seconds faster than without it; so that's good.  We've seen some complaints 
about the Spark Action being slow.

As a general question, have you tested this with NN HA?  I'm concerned about 
{{SparkMain#fixFsDefaultUris}} because it does some URI host/port 
parsing/manipulation.

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch, 
> yarn-cluster_launcher.txt
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321707#comment-15321707
 ] 

Hadoop QA commented on OOZIE-2547:
--

Testing JIRA OOZIE-2547

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1786
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2966/

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: oozie-trunk-precommit-build #2966

[Apache Jenkins Server]

See 

--
[...truncated 8402 lines...]
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ 

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Rohini Palaniswamy]

> On June 8, 2016, 9:08 p.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 3173
> > 
> >
> > The conditional logic can be specified using the  tag in 
> > the coordinator.xml using the Oozie Coordinator Schema 0.5.
> > 
> > Make "Oozie Coordinator Schema 0.5" into a link into the newly added 
> > 0.5 schema section.
> 
> Purshotam Shah wrote:
> Should we need to specify that, because it going to be available higher 
> version also.

Yes please. Make it "using the Oozie Coordinator Schema 0.5 and above".


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/#review136699
---


On June 8, 2016, 11:24 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48149/
> ---
> 
> (Updated June 8, 2016, 11:24 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2445
> https://issues.apache.org/jira/browse/OOZIE-2445
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Doc for - Specifying coordinator input datasets in more logical ways 
> (OOZIE-1976)
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
> 75b7354a12ae8c255b9d713856b44ec939092c95 
>   examples/src/main/apps/coord-input-logic/coordinator.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/job.properties 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/workflow.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 
> 
> Diff: https://reviews.apache.org/r/48149/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/
---

(Updated June 8, 2016, 11:24 p.m.)


Review request for oozie.


Bugs: OOZIE-2445
https://issues.apache.org/jira/browse/OOZIE-2445


Repository: oozie-git


Description
---

Doc for - Specifying coordinator input datasets in more logical ways 
(OOZIE-1976)


Diffs (updated)
-

  docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
75b7354a12ae8c255b9d713856b44ec939092c95 
  examples/src/main/apps/coord-input-logic/coordinator.xml 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  examples/src/main/apps/coord-input-logic/job.properties 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  examples/src/main/apps/coord-input-logic/workflow.xml 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 

Diff: https://reviews.apache.org/r/48149/diff/


Testing
---


Thanks,

Purshotam Shah

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Purshotam Shah]

> On June 8, 2016, 9:08 p.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 3173
> > 
> >
> > The conditional logic can be specified using the  tag in 
> > the coordinator.xml using the Oozie Coordinator Schema 0.5.
> > 
> > Make "Oozie Coordinator Schema 0.5" into a link into the newly added 
> > 0.5 schema section.

Should we need to specify that, because it going to be available higher version 
also.


> On June 8, 2016, 9:08 p.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 3159
> > 
> >
> > (since Oozie 4.3)

I thought of this, then  realized that we don't need this because the document 
is only going to be available in 4.3, which means that input logic is going to 
be available in 4.3 and above.


- Purshotam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/#review136699
---


On June 8, 2016, 7:39 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48149/
> ---
> 
> (Updated June 8, 2016, 7:39 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2445
> https://issues.apache.org/jira/browse/OOZIE-2445
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Doc for - Specifying coordinator input datasets in more logical ways 
> (OOZIE-1976)
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
> 75b7354a12ae8c255b9d713856b44ec939092c95 
>   examples/src/main/apps/coord-input-logic/coordinator.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/job.properties 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/workflow.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 
> 
> Diff: https://reviews.apache.org/r/48149/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

[jira] [Commented] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321630#comment-15321630
 ] 

Robert Kanter commented on OOZIE-2547:
--

I'll try to take a look later today.  Otherwise, I'll get to it tomorrow 
afternoon.

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2554) Spark action tests on yarn-cluster, yarn-client mode

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321565#comment-15321565
 ] 

Rohini Palaniswamy commented on OOZIE-2554:
---

We need to add tests for OOZIE-2563 in yarn cluster mode.

> Spark action tests on yarn-cluster, yarn-client mode
> 
>
> Key: OOZIE-2554
> URL: https://issues.apache.org/jira/browse/OOZIE-2554
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>
> Currently, we have unit test cases for Spark Action in local mode. To verify 
> other functionality, we should have tests in yarn-cluster and yarn-client 
> mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321553#comment-15321553
 ] 

Rohini Palaniswamy commented on OOZIE-2547:
---

I will wait for Robert's comments before checking in.

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321550#comment-15321550
 ] 

Rohini Palaniswamy commented on OOZIE-2547:
---

+1 for OOZIE-2547-4.patch (https://reviews.apache.org/r/48298/diff/5/). When we 
do OOZIE-2554, Robert's case would be covered as a test. It will be hard to add 
test for spark-assembly jar though as it is not available in maven.

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2547:
---
Attachment: OOZIE-2547-4.patch

We need to include the files in the current working directory in driver and 
executor classpath. Hello [~rkanter], I have tested it locally without 
spark-assembly.jar in lib/ , could you please confirm if the failures you 
observed goes away with this patch?

> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch, OOZIE-2547-4.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2554) Spark action tests on yarn-cluster, yarn-client mode

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2554:
---
Issue Type: New Feature  (was: Bug)

> Spark action tests on yarn-cluster, yarn-client mode
> 
>
> Key: OOZIE-2554
> URL: https://issues.apache.org/jira/browse/OOZIE-2554
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>
> Currently, we have unit test cases for Spark Action in local mode. To verify 
> other functionality, we should have tests in yarn-cluster and yarn-client 
> mode.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/#review136715
---


Ship it!




Ship It!

- Rohini Palaniswamy


On June 8, 2016, 10:07 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48298/
> ---
> 
> (Updated June 8, 2016, 10:07 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2547
> https://issues.apache.org/jira/browse/OOZIE-2547
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> 
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
> 0e6e271 
> 
> Diff: https://reviews.apache.org/r/48298/diff/
> 
> 
> Testing
> ---
> 
> I have tested it in local, yarn-cluster and yarn-client.
> 
> 
> File Attachments
> 
> 
> OOZIE-2547-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
> Making changes suggested by Rohini
>   
> https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/
---

(Updated June 8, 2016, 3:07 p.m.)


Review request for oozie.


Changes
---

considering the case where user has also provided extra classpaths for executor 
and driver


Bugs: OOZIE-2547
https://issues.apache.org/jira/browse/OOZIE-2547


Repository: oozie-git


Description
---

Currently, we pass jars using --jars option while submitting spark job. Also, 
we add spark.yarn.dist.files option in case of yarn-client mode. 
Instead of that, we can have only --files option and pass on the files which 
are present in mapreduce.job.cache.files. While doing so, we make sure that 
spark won't make another copy of the files if files exist on the hdfs. We saw 
the issues when files are getting copied multiple times and causing exceptions 
such as :

Diagnostics: Resource 
hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
 changed on src filesystem


Diffs (updated)
-

  docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 
  sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
0e6e271 

Diff: https://reviews.apache.org/r/48298/diff/


Testing
---

I have tested it in local, yarn-cluster and yarn-client.


File Attachments


OOZIE-2547-1.patch
  
https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
Making changes suggested by Rohini
  
https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch


Thanks,

Satish Saley

Build failed in Jenkins: oozie-trunk-precommit-build #2965

[Apache Jenkins Server]

See 

--
[...truncated 8380 lines...]
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-tools ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ 

[jira] [Commented] (OOZIE-2563) pass spark-defaults.conf to spark action

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321487#comment-15321487
 ] 

Hadoop QA commented on OOZIE-2563:
--

Testing JIRA OOZIE-2563

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1786
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2965/

> pass spark-defaults.conf to spark action
> 
>
> Key: OOZIE-2563
> URL: https://issues.apache.org/jira/browse/OOZIE-2563
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Fix For: trunk
>
> Attachments: OOZIE-2563-1.patch
>
>
> If user has specified spark-defaults.conf file in workflow lib/ directory for 
> spark action, then we will add it pass it on to the spark job. Another way is 
> server side setting using 
> [OOZIE-2170|https://issues.apache.org/jira/browse/OOZIE-2170]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/
---

(Updated June 8, 2016, 2:24 p.m.)


Review request for oozie.


Bugs: OOZIE-2547
https://issues.apache.org/jira/browse/OOZIE-2547


Repository: oozie-git


Description
---

Currently, we pass jars using --jars option while submitting spark job. Also, 
we add spark.yarn.dist.files option in case of yarn-client mode. 
Instead of that, we can have only --files option and pass on the files which 
are present in mapreduce.job.cache.files. While doing so, we make sure that 
spark won't make another copy of the files if files exist on the hdfs. We saw 
the issues when files are getting copied multiple times and causing exceptions 
such as :

Diagnostics: Resource 
hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
 changed on src filesystem


Diffs (updated)
-

  docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 
  sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
0e6e271 

Diff: https://reviews.apache.org/r/48298/diff/


Testing
---

I have tested it in local, yarn-cluster and yarn-client.


File Attachments


OOZIE-2547-1.patch
  
https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
Making changes suggested by Rohini
  
https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch


Thanks,

Satish Saley

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/
---

(Updated June 8, 2016, 2:21 p.m.)


Review request for oozie.


Bugs: OOZIE-2547
https://issues.apache.org/jira/browse/OOZIE-2547


Repository: oozie-git


Description
---

Currently, we pass jars using --jars option while submitting spark job. Also, 
we add spark.yarn.dist.files option in case of yarn-client mode. 
Instead of that, we can have only --files option and pass on the files which 
are present in mapreduce.job.cache.files. While doing so, we make sure that 
spark won't make another copy of the files if files exist on the hdfs. We saw 
the issues when files are getting copied multiple times and causing exceptions 
such as :

Diagnostics: Resource 
hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
 changed on src filesystem


Diffs (updated)
-

  docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 
  sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
0e6e271 

Diff: https://reviews.apache.org/r/48298/diff/


Testing
---

I have tested it in local, yarn-cluster and yarn-client.


File Attachments


OOZIE-2547-1.patch
  
https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
Making changes suggested by Rohini
  
https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch


Thanks,

Satish Saley

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/#review136699
---




docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3159)


(since Oozie 4.3)



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3165)


will evaluate to true



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3168)


Remove "MIN and WAIT with INPUT-LOGIC operator."



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3169)


Minimum number of input instances that should be available



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3173)


The conditional logic can be specified using the  tag in the 
coordinator.xml using the Oozie Coordinator Schema 0.5.

Make "Oozie Coordinator Schema 0.5" into a link into the newly added 0.5 
schema section.



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3174)


There should be new lines between the paragraphs below and above. For eg: 
http://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a4.3._Timezones_and_Daylight-Saving



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (lines 3208 - 3215)


Please move this to the first example



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3276)


typo. propogate -> propagated



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3278)


Move the line after the Example like other ones.



docs/src/site/twiki/CoordinatorFunctionalSpec.twiki (line 3743)


Appendix needs to be updated with new schema coordinator:0.5


- Rohini Palaniswamy


On June 8, 2016, 7:39 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48149/
> ---
> 
> (Updated June 8, 2016, 7:39 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2445
> https://issues.apache.org/jira/browse/OOZIE-2445
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Doc for - Specifying coordinator input datasets in more logical ways 
> (OOZIE-1976)
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
> 75b7354a12ae8c255b9d713856b44ec939092c95 
>   examples/src/main/apps/coord-input-logic/coordinator.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/job.properties 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/workflow.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 
> 
> Diff: https://reviews.apache.org/r/48149/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/#review136691
---




File Attachment: Making changes suggested by Rohini - OOZIE-2547-2.patch


Need to change it. fs.getScheme() not available in hadoop-1


- Satish Saley


On June 8, 2016, 1:08 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48298/
> ---
> 
> (Updated June 8, 2016, 1:08 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2547
> https://issues.apache.org/jira/browse/OOZIE-2547
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> 
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 
>   sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
> 0e6e271 
> 
> Diff: https://reviews.apache.org/r/48298/diff/
> 
> 
> Testing
> ---
> 
> I have tested it in local, yarn-cluster and yarn-client.
> 
> 
> File Attachments
> 
> 
> OOZIE-2547-1.patch
>   
> https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
> Making changes suggested by Rohini
>   
> https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/
---

(Updated June 8, 2016, 1:08 p.m.)


Review request for oozie.


Bugs: OOZIE-2547
https://issues.apache.org/jira/browse/OOZIE-2547


Repository: oozie-git


Description
---

Currently, we pass jars using --jars option while submitting spark job. Also, 
we add spark.yarn.dist.files option in case of yarn-client mode. 
Instead of that, we can have only --files option and pass on the files which 
are present in mapreduce.job.cache.files. While doing so, we make sure that 
spark won't make another copy of the files if files exist on the hdfs. We saw 
the issues when files are getting copied multiple times and causing exceptions 
such as :

Diagnostics: Resource 
hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
 changed on src filesystem


Diffs (updated)
-

  docs/src/site/twiki/DG_SparkActionExtension.twiki d7d75a1 
  sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
0e6e271 

Diff: https://reviews.apache.org/r/48298/diff/


Testing
---

I have tested it in local, yarn-cluster and yarn-client.


File Attachments


OOZIE-2547-1.patch
  
https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
Making changes suggested by Rohini
  
https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch


Thanks,

Satish Saley

Re: Review Request 48298: Add mapreduce.job.cache.files to spark action

[Satish Saley]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48298/
---

(Updated June 8, 2016, 12:59 p.m.)


Review request for oozie.


Bugs: OOZIE-2547
https://issues.apache.org/jira/browse/OOZIE-2547


Repository: oozie-git


Description
---

Currently, we pass jars using --jars option while submitting spark job. Also, 
we add spark.yarn.dist.files option in case of yarn-client mode. 
Instead of that, we can have only --files option and pass on the files which 
are present in mapreduce.job.cache.files. While doing so, we make sure that 
spark won't make another copy of the files if files exist on the hdfs. We saw 
the issues when files are getting copied multiple times and causing exceptions 
such as :

Diagnostics: Resource 
hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
 changed on src filesystem


Diffs
-

  sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 
0e6e271 

Diff: https://reviews.apache.org/r/48298/diff/


Testing
---

I have tested it in local, yarn-cluster and yarn-client.


File Attachments (updated)


OOZIE-2547-1.patch
  
https://reviews.apache.org/media/uploaded/files/2016/06/06/6d718b0f-4533-4e9e-992a-03aea079a6c7__OOZIE-2547-1.patch
Making changes suggested by Rohini
  
https://reviews.apache.org/media/uploaded/files/2016/06/08/e26bb7e3-7011-4549-9026-0b7cb4262f76__OOZIE-2547-2.patch


Thanks,

Satish Saley

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Purshotam Shah]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/
---

(Updated June 8, 2016, 7:39 p.m.)


Review request for oozie.


Bugs: OOZIE-2445
https://issues.apache.org/jira/browse/OOZIE-2445


Repository: oozie-git


Description
---

Doc for - Specifying coordinator input datasets in more logical ways 
(OOZIE-1976)


Diffs (updated)
-

  docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
75b7354a12ae8c255b9d713856b44ec939092c95 
  examples/src/main/apps/coord-input-logic/coordinator.xml 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  examples/src/main/apps/coord-input-logic/job.properties 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  examples/src/main/apps/coord-input-logic/workflow.xml 
e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
  src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 

Diff: https://reviews.apache.org/r/48149/diff/


Testing
---


Thanks,

Purshotam Shah

Re: Review Request 48149: Doc for - Specifying coordinator input datasets in more logical ways (OOZIE-1976)

[Purshotam Shah]

> On June 8, 2016, 12:08 a.m., Rohini Palaniswamy wrote:
> > docs/src/site/twiki/CoordinatorFunctionalSpec.twiki, line 3240
> > 
> >
> > instance -> instances. 
> > 
> > Why wait for all dependant data available if min is specified?

If min and wait are specified then it means that I am ok to start processing if 
my minimum dependencies are met but I want to wait for more time to see if mora 
data arrives.

Ex. min = 2 and wait = 10 min and  data range = 5 and available = 4.
(min=2 and wait = 10) = false.
but after 10 min.
(min=2 and wait = 10) = true.


- Purshotam


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/48149/#review136504
---


On June 8, 2016, 7:39 p.m., Purshotam Shah wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/48149/
> ---
> 
> (Updated June 8, 2016, 7:39 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-2445
> https://issues.apache.org/jira/browse/OOZIE-2445
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Doc for - Specifying coordinator input datasets in more logical ways 
> (OOZIE-1976)
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/CoordinatorFunctionalSpec.twiki 
> 75b7354a12ae8c255b9d713856b44ec939092c95 
>   examples/src/main/apps/coord-input-logic/coordinator.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/job.properties 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   examples/src/main/apps/coord-input-logic/workflow.xml 
> e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 
>   src/main/assemblies/examples.xml d70c280ec6e189089dcacb57c6f6f340e77bec3b 
> 
> Diff: https://reviews.apache.org/r/48149/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>

[jira] [Updated] (OOZIE-2563) pass spark-defaults.conf to spark action

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2563:
---
Attachment: OOZIE-2563-1.patch

> pass spark-defaults.conf to spark action
> 
>
> Key: OOZIE-2563
> URL: https://issues.apache.org/jira/browse/OOZIE-2563
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2563-1.patch
>
>
> If user has specified spark-defaults.conf file in workflow lib/ directory for 
> spark action, then we will add it pass it on to the spark job. Another way is 
> server side setting using 
> [OOZIE-2170|https://issues.apache.org/jira/browse/OOZIE-2170]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2563) pass spark-defaults.conf to spark action

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2563:
---
Priority: Minor  (was: Major)

> pass spark-defaults.conf to spark action
> 
>
> Key: OOZIE-2563
> URL: https://issues.apache.org/jira/browse/OOZIE-2563
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
>
> If user has specified spark-defaults.conf file in workflow lib/ directory for 
> spark action, then we will add it pass it on to the spark job. Another way is 
> server side setting using 
> [OOZIE-2170|https://issues.apache.org/jira/browse/OOZIE-2170]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2563) pass spark-defaults.conf to spark action

[Satish Subhashrao Saley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Subhashrao Saley updated OOZIE-2563:
---
Issue Type: New Feature  (was: Bug)

> pass spark-defaults.conf to spark action
> 
>
> Key: OOZIE-2563
> URL: https://issues.apache.org/jira/browse/OOZIE-2563
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>
> If user has specified spark-defaults.conf file in workflow lib/ directory for 
> spark action, then we will add it pass it on to the spark job. Another way is 
> server side setting using 
> [OOZIE-2170|https://issues.apache.org/jira/browse/OOZIE-2170]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (OOZIE-2563) pass spark-defaults.conf to spark action

[Satish Subhashrao Saley (JIRA)]

Satish Subhashrao Saley created OOZIE-2563:
--

 Summary: pass spark-defaults.conf to spark action
 Key: OOZIE-2563
 URL: https://issues.apache.org/jira/browse/OOZIE-2563
 Project: Oozie
  Issue Type: Bug
Reporter: Satish Subhashrao Saley
Assignee: Satish Subhashrao Saley


If user has specified spark-defaults.conf file in workflow lib/ directory for 
spark action, then we will add it pass it on to the spark job. Another way is 
server side setting using 
[OOZIE-2170|https://issues.apache.org/jira/browse/OOZIE-2170]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Satish Subhashrao Saley (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319852#comment-15319852
 ] 

Satish Subhashrao Saley edited comment on OOZIE-2547 at 6/8/16 4:15 PM:


It is happening because spark-assembly.jar is getting a special treatment by 
Spark. In launch_container.sh, I see - 

{{export 
SPARK_YARN_CACHE_FILES="hdfs://localhost/user/saley/.sparkStaging/application_1234_123/spark-assembly.jar#__spark__.jar}}

When we use spark-assembly.jar, it is getting copied with alias as 
*__spark__.jar* to the current directory of container. And while setting the 
classpath, it is referring to that file 

{{export CLASSPATH="$PWD:$PWD/__spark__.jar:}}

But when we don't use spark-assembly.jar, we need to set the classpath 
explicitly. 



was (Author: satishsaley):
It is happening because spark-assembly.jar is getting a special treatment by 
Spark. In launch_container.sh, I see - 

{{export 
SPARK_YARN_CACHE_FILES="hdfs://clusterx-nodey.yahoo.com/user/saley/.sparkStaging/application_1464113035484_0834/spark-assembly.jar#__spark__.jar}}

When we use spark-assembly.jar, it is getting copied with alias as 
*__spark__.jar* to the current directory of container. And while setting the 
classpath, it is referring to that file 

{{export CLASSPATH="$PWD:$PWD/__spark__.jar:}}

But when we don't use spark-assembly.jar, we need to set the classpath 
explicitly. 


> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2316) Drop support for Hadoop 1 and 0.23

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320575#comment-15320575
 ] 

ASF GitHub Bot commented on OOZIE-2316:
---

GitHub user asasvari opened a pull request:

https://github.com/apache/oozie/pull/16

OOZIE-2316 drop support for Hadoop 1 and 0.23



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/asasvari/oozie oya

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oozie/pull/16.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16


commit b2627c2106ef50ad484dba59cab60540c7e400c6
Author: Attila Sasvari 
Date:   2016-06-07T15:53:14Z

OOZIE-2316 drop support for Hadoop 1 and 0.23

Change-Id: I7be33084148697f236e7d397736bfbbbc2288af6




> Drop support for Hadoop 1 and 0.23
> --
>
> Key: OOZIE-2316
> URL: https://issues.apache.org/jira/browse/OOZIE-2316
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2316-001.patch
>
>
> Given that Hadoop 2.x has had many releases now (2.8 is next), Hadoop 1.x 
> hasn't had a release in a really long time, and Hadoop 3.0 is coming somewhat 
> soonish, it would be good to drop support Hadoop 1.x and Hadoop 0.23.  I 
> believe most users are using Hadoop 2.x anyway.
> This should help simplify the build a lot as we can probably get rid of the 
> hadooplibs stuff and don't need to set Hadoop 1 and Hadoop 2+ properties 
> (e.g. JT and RM addresses), so we can get rid of those cases as well.
> It might also be good to create new versions of the schemas that use 
> {{}} instead of {{}} (the existing schemas 
> would of course still work).  While this is a minor thing, it does bring up 
> questions every so often of "does Oozie support Hadoop 2/YARN?"  And "how do 
> I specify the RM?"  Alternatively, we could update the existing schemas to 
> allow one of {{}} or {{}}.  I'm not sure what 
> we should do about any config properties in Oozie site that mention 
> "job-tracker".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] oozie pull request #16: OOZIE-2316 drop support for Hadoop 1 and 0.23

[asasvari]

GitHub user asasvari opened a pull request:

https://github.com/apache/oozie/pull/16

OOZIE-2316 drop support for Hadoop 1 and 0.23



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/asasvari/oozie oya

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/oozie/pull/16.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16


commit b2627c2106ef50ad484dba59cab60540c7e400c6
Author: Attila Sasvari 
Date:   2016-06-07T15:53:14Z

OOZIE-2316 drop support for Hadoop 1 and 0.23

Change-Id: I7be33084148697f236e7d397736bfbbbc2288af6




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : oozie-trunk-precommit-build #2964

[Apache Jenkins Server]

See 

[jira] [Commented] (OOZIE-2550) Flaky tests in TestZKUUIDService.java

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320548#comment-15320548
 ] 

Hadoop QA commented on OOZIE-2550:
--

Testing JIRA OOZIE-2550

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1786
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2964/

> Flaky tests in TestZKUUIDService.java
> -
>
> Key: OOZIE-2550
> URL: https://issues.apache.org/jira/browse/OOZIE-2550
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2550-001.patch, OOZIE-2550-002.patch, 
> OOZIE-2550-003.patch, OOZIE-2550-004.patch
>
>
> 1. Test case testMultipleIDGeneration_withMultiThread in TestZKUUIDService 
> uses an ArrayList which is written by two threads simultaneously. This is 
> dangerous and the list must be externally synchronized to prevent race 
> conditions.
> Another problem is that you cannot put items at arbitrary indexes in an 
> ArrayList. For example, if the list is empty, the following code throws 
> ArrayIndexOutOfBoundException:
> {code}
> List test = new ArrayList<>(1);
> test.add(22, true);
> {code}
> In an unlucky scheduling event, the following can happen:
> 1. The list has a certain number of elements, the value of "size" inside the 
> list is 1000
> 2. Thread-1 retrieves the next ID from ZK UUID service, which is 1001
> 3. Thread-2 retrieves the next ID from ZK UUID service, which is 1002
> 4. Thread-2 runs faster than Thread-1 and tries to call {{list.add(1002, 
> true)}} which fails. 
> The following error was caught during a test run:
> {code}
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 53.571 sec 
> <<< FAILURE!
> testMultipleIDGeneration_withMultiThread(org.apache.oozie.service.TestZKUUIDService)
>   Time elapsed: 0.02 sec  <<< ERROR!
> java.lang.IndexOutOfBoundsException: Index: 89, Size: 89
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at 
> org.apache.oozie.service.TestZKUUIDService.testMultipleIDGeneration_withMultiThread(TestZKUUIDService.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at 
> 

[jira] [Updated] (OOZIE-2316) Drop support for Hadoop 1 and 0.23

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2316:
--
Attachment: OOZIE-2316-001.patch

> Drop support for Hadoop 1 and 0.23
> --
>
> Key: OOZIE-2316
> URL: https://issues.apache.org/jira/browse/OOZIE-2316
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2316-001.patch
>
>
> Given that Hadoop 2.x has had many releases now (2.8 is next), Hadoop 1.x 
> hasn't had a release in a really long time, and Hadoop 3.0 is coming somewhat 
> soonish, it would be good to drop support Hadoop 1.x and Hadoop 0.23.  I 
> believe most users are using Hadoop 2.x anyway.
> This should help simplify the build a lot as we can probably get rid of the 
> hadooplibs stuff and don't need to set Hadoop 1 and Hadoop 2+ properties 
> (e.g. JT and RM addresses), so we can get rid of those cases as well.
> It might also be good to create new versions of the schemas that use 
> {{}} instead of {{}} (the existing schemas 
> would of course still work).  While this is a minor thing, it does bring up 
> questions every so often of "does Oozie support Hadoop 2/YARN?"  And "how do 
> I specify the RM?"  Alternatively, we could update the existing schemas to 
> allow one of {{}} or {{}}.  I'm not sure what 
> we should do about any config properties in Oozie site that mention 
> "job-tracker".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2316) Drop support for Hadoop 1 and 0.23

[Attila Sasvari (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Sasvari updated OOZIE-2316:
--
Affects Version/s: (was: trunk)
   oya

> Drop support for Hadoop 1 and 0.23
> --
>
> Key: OOZIE-2316
> URL: https://issues.apache.org/jira/browse/OOZIE-2316
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
>
> Given that Hadoop 2.x has had many releases now (2.8 is next), Hadoop 1.x 
> hasn't had a release in a really long time, and Hadoop 3.0 is coming somewhat 
> soonish, it would be good to drop support Hadoop 1.x and Hadoop 0.23.  I 
> believe most users are using Hadoop 2.x anyway.
> This should help simplify the build a lot as we can probably get rid of the 
> hadooplibs stuff and don't need to set Hadoop 1 and Hadoop 2+ properties 
> (e.g. JT and RM addresses), so we can get rid of those cases as well.
> It might also be good to create new versions of the schemas that use 
> {{}} instead of {{}} (the existing schemas 
> would of course still work).  While this is a minor thing, it does bring up 
> questions every so often of "does Oozie support Hadoop 2/YARN?"  And "how do 
> I specify the RM?"  Alternatively, we could update the existing schemas to 
> allow one of {{}} or {{}}.  I'm not sure what 
> we should do about any config properties in Oozie site that mention 
> "job-tracker".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2500) -DtestJarSimple option mentioned in minioozie doc does not work

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320440#comment-15320440
 ] 

Hadoop QA commented on OOZIE-2500:
--

Testing JIRA OOZIE-2500

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1786
.Tests failed: 2
.Tests errors: 0

.The patch failed the following testcases:

.  testSingleRecord(org.apache.oozie.servlet.TestBulkMonitorWebServiceAPI)
.  testRecovery(org.apache.oozie.action.hadoop.TestJavaActionExecutor)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

.   https://builds.apache.org/job/oozie-trunk-precommit-build/2963/

> -DtestJarSimple option mentioned in minioozie doc does not work
> ---
>
> Key: OOZIE-2500
> URL: https://issues.apache.org/jira/browse/OOZIE-2500
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: abhishek bafna
> Attachments: OOZIE-2500-01.patch, out.txt
>
>
> [Minioozie doc|https://oozie.apache.org/docs/4.2.0/ENG_MiniOozie.html] says 
> to run {{$ mvn clean install -DskipTests -DtestJarSimple}} to populate local 
> maven repo with required jars. But the command fails with following message:
> {code}
> [INFO] 100 errors 
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Oozie Main .. SUCCESS [  1.339 
> s]
> [INFO] Apache Oozie Hadoop Utils .. SUCCESS [  0.999 
> s]
> [INFO] Apache Oozie Hadoop Distcp hadoop-1-4.3.0-SNAPSHOT . SUCCESS [  0.138 
> s]
> [INFO] Apache Oozie Hadoop Auth hadoop-1-4.3.0-SNAPSHOT ... SUCCESS [  0.244 
> s]
> [INFO] Apache Oozie Hadoop Libs ... SUCCESS [  0.052 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  3.282 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  2.257 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  2.088 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.693 
> s]
> [INFO] Apache Oozie Core .. SUCCESS [ 20.951 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... FAILURE [  2.688 
> s]
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] 

Build failed in Jenkins: oozie-trunk-precommit-build #2963

[Apache Jenkins Server]

See 

--
[...truncated 8406 lines...]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-tools ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-tools ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-tools ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-tools ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-tools ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/tools.xml
[WARNING] The following patterns were never triggered in this artifact 
exclusion filter:
o  '*:*:pom:*'

[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] 
[INFO] 
[INFO] Building Apache Oozie MiniOozie 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-mini ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-mini 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-mini ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-mini ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-mini ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-mini ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-mini ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-mini ---
[INFO] Reading assembly descriptor: src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.3.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run 

Jenkins build is back to normal : oozie-trunk-find-patches-available #123086

[Apache Jenkins Server]

See 

Build failed in Jenkins: oozie-trunk-find-patches-available #123085

[Apache Jenkins Server]

See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Mapreduce Hadoop Pig Hdfs yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 7779f3184a085a832df3116a0eab33d2d56eed78 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7779f3184a085a832df3116a0eab33d2d56eed78
 > git rev-list 7779f3184a085a832df3116a0eab33d2d56eed78 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson1592648803598781312.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100  166k0  166k0 0   220k  0 --:--:-- --:--:-- --:--:--  220k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Updated] (OOZIE-2550) Flaky tests in TestZKUUIDService.java

[Peter Bacsko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2550:

Attachment: OOZIE-2550-004.patch

> Flaky tests in TestZKUUIDService.java
> -
>
> Key: OOZIE-2550
> URL: https://issues.apache.org/jira/browse/OOZIE-2550
> Project: Oozie
>  Issue Type: Bug
>  Components: tests
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Minor
> Attachments: OOZIE-2550-001.patch, OOZIE-2550-002.patch, 
> OOZIE-2550-003.patch, OOZIE-2550-004.patch
>
>
> 1. Test case testMultipleIDGeneration_withMultiThread in TestZKUUIDService 
> uses an ArrayList which is written by two threads simultaneously. This is 
> dangerous and the list must be externally synchronized to prevent race 
> conditions.
> Another problem is that you cannot put items at arbitrary indexes in an 
> ArrayList. For example, if the list is empty, the following code throws 
> ArrayIndexOutOfBoundException:
> {code}
> List test = new ArrayList<>(1);
> test.add(22, true);
> {code}
> In an unlucky scheduling event, the following can happen:
> 1. The list has a certain number of elements, the value of "size" inside the 
> list is 1000
> 2. Thread-1 retrieves the next ID from ZK UUID service, which is 1001
> 3. Thread-2 retrieves the next ID from ZK UUID service, which is 1002
> 4. Thread-2 runs faster than Thread-1 and tries to call {{list.add(1002, 
> true)}} which fails. 
> The following error was caught during a test run:
> {code}
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 53.571 sec 
> <<< FAILURE!
> testMultipleIDGeneration_withMultiThread(org.apache.oozie.service.TestZKUUIDService)
>   Time elapsed: 0.02 sec  <<< ERROR!
> java.lang.IndexOutOfBoundsException: Index: 89, Size: 89
> at java.util.ArrayList.rangeCheck(ArrayList.java:653)
> at java.util.ArrayList.get(ArrayList.java:429)
> at 
> org.apache.oozie.service.TestZKUUIDService.testMultipleIDGeneration_withMultiThread(TestZKUUIDService.java:134)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at 
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> 2. The order in which the tests are executed is random. The problem is that 
> testResetSequence sets the maximum sequence number to 900. Because this value 
> is stored in a static variable inside ZKUUIDService, it affects 
> testIDGeneration and testMultipleIDGeneration if these tests are run after 
> testResetSequence.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (OOZIE-2500) -DtestJarSimple option mentioned in minioozie doc does not work

[abhishek bafna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

abhishek bafna updated OOZIE-2500:
--
Attachment: OOZIE-2500-01.patch

> -DtestJarSimple option mentioned in minioozie doc does not work
> ---
>
> Key: OOZIE-2500
> URL: https://issues.apache.org/jira/browse/OOZIE-2500
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: abhishek bafna
> Attachments: OOZIE-2500-01.patch, out.txt
>
>
> [Minioozie doc|https://oozie.apache.org/docs/4.2.0/ENG_MiniOozie.html] says 
> to run {{$ mvn clean install -DskipTests -DtestJarSimple}} to populate local 
> maven repo with required jars. But the command fails with following message:
> {code}
> [INFO] 100 errors 
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Oozie Main .. SUCCESS [  1.339 
> s]
> [INFO] Apache Oozie Hadoop Utils .. SUCCESS [  0.999 
> s]
> [INFO] Apache Oozie Hadoop Distcp hadoop-1-4.3.0-SNAPSHOT . SUCCESS [  0.138 
> s]
> [INFO] Apache Oozie Hadoop Auth hadoop-1-4.3.0-SNAPSHOT ... SUCCESS [  0.244 
> s]
> [INFO] Apache Oozie Hadoop Libs ... SUCCESS [  0.052 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  3.282 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  2.257 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  2.088 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.693 
> s]
> [INFO] Apache Oozie Core .. SUCCESS [ 20.951 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... FAILURE [  2.688 
> s]
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 35.474 s
> [INFO] Finished at: 2016-04-04T16:04:41-07:00
> [INFO] Final Memory: 161M/1262M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile 
> (default-testCompile) on project oozie-sharelib-streaming: Compilation 
> failure: Compilation failure:
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestStreamingMain.java:[31,39]
>  error: cannot find symbol
> [ERROR] class MainTestCase
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[76,49]
>  error: cannot find symbol
> [ERROR] class ActionExecutorTestCase
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[342,14]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[365,14]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[389,38]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestStreamingMain.java:[34,24]
>  error: cannot find symbol
> [ERROR] class TestStreamingMain
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (OOZIE-2500) -DtestJarSimple option mentioned in minioozie doc does not work

[abhishek bafna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

abhishek bafna reassigned OOZIE-2500:
-

Assignee: abhishek bafna

> -DtestJarSimple option mentioned in minioozie doc does not work
> ---
>
> Key: OOZIE-2500
> URL: https://issues.apache.org/jira/browse/OOZIE-2500
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: abhishek bafna
> Attachments: out.txt
>
>
> [Minioozie doc|https://oozie.apache.org/docs/4.2.0/ENG_MiniOozie.html] says 
> to run {{$ mvn clean install -DskipTests -DtestJarSimple}} to populate local 
> maven repo with required jars. But the command fails with following message:
> {code}
> [INFO] 100 errors 
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Oozie Main .. SUCCESS [  1.339 
> s]
> [INFO] Apache Oozie Hadoop Utils .. SUCCESS [  0.999 
> s]
> [INFO] Apache Oozie Hadoop Distcp hadoop-1-4.3.0-SNAPSHOT . SUCCESS [  0.138 
> s]
> [INFO] Apache Oozie Hadoop Auth hadoop-1-4.3.0-SNAPSHOT ... SUCCESS [  0.244 
> s]
> [INFO] Apache Oozie Hadoop Libs ... SUCCESS [  0.052 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  3.282 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  2.257 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  2.088 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.693 
> s]
> [INFO] Apache Oozie Core .. SUCCESS [ 20.951 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... FAILURE [  2.688 
> s]
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 35.474 s
> [INFO] Finished at: 2016-04-04T16:04:41-07:00
> [INFO] Final Memory: 161M/1262M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile 
> (default-testCompile) on project oozie-sharelib-streaming: Compilation 
> failure: Compilation failure:
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestStreamingMain.java:[31,39]
>  error: cannot find symbol
> [ERROR] class MainTestCase
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[76,49]
>  error: cannot find symbol
> [ERROR] class ActionExecutorTestCase
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[342,14]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[365,14]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[389,38]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestStreamingMain.java:[34,24]
>  error: cannot find symbol
> [ERROR] class TestStreamingMain
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (OOZIE-2500) -DtestJarSimple option mentioned in minioozie doc does not work

[abhishek bafna (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320311#comment-15320311
 ] 

abhishek bafna commented on OOZIE-2500:
---

[~satishsaley] I am assigning the issue to myself. Will be uploading the patch 
soon. Thanks.

> -DtestJarSimple option mentioned in minioozie doc does not work
> ---
>
> Key: OOZIE-2500
> URL: https://issues.apache.org/jira/browse/OOZIE-2500
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
> Attachments: out.txt
>
>
> [Minioozie doc|https://oozie.apache.org/docs/4.2.0/ENG_MiniOozie.html] says 
> to run {{$ mvn clean install -DskipTests -DtestJarSimple}} to populate local 
> maven repo with required jars. But the command fails with following message:
> {code}
> [INFO] 100 errors 
> [INFO] -
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Apache Oozie Main .. SUCCESS [  1.339 
> s]
> [INFO] Apache Oozie Hadoop Utils .. SUCCESS [  0.999 
> s]
> [INFO] Apache Oozie Hadoop Distcp hadoop-1-4.3.0-SNAPSHOT . SUCCESS [  0.138 
> s]
> [INFO] Apache Oozie Hadoop Auth hadoop-1-4.3.0-SNAPSHOT ... SUCCESS [  0.244 
> s]
> [INFO] Apache Oozie Hadoop Libs ... SUCCESS [  0.052 
> s]
> [INFO] Apache Oozie Client  SUCCESS [  3.282 
> s]
> [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [  2.257 
> s]
> [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [  2.088 
> s]
> [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [  0.693 
> s]
> [INFO] Apache Oozie Core .. SUCCESS [ 20.951 
> s]
> [INFO] Apache Oozie Share Lib Streaming ... FAILURE [  2.688 
> s]
> [INFO] Apache Oozie Share Lib Pig . SKIPPED
> [INFO] Apache Oozie Share Lib Hive  SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> [INFO] Apache Oozie Examples .. SKIPPED
> [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> [INFO] Apache Oozie Share Lib . SKIPPED
> [INFO] Apache Oozie Docs .. SKIPPED
> [INFO] Apache Oozie WebApp  SKIPPED
> [INFO] Apache Oozie Tools . SKIPPED
> [INFO] Apache Oozie MiniOozie . SKIPPED
> [INFO] Apache Oozie Distro  SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 35.474 s
> [INFO] Finished at: 2016-04-04T16:04:41-07:00
> [INFO] Final Memory: 161M/1262M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile 
> (default-testCompile) on project oozie-sharelib-streaming: Compilation 
> failure: Compilation failure:
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestStreamingMain.java:[31,39]
>  error: cannot find symbol
> [ERROR] class MainTestCase
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[76,49]
>  error: cannot find symbol
> [ERROR] class ActionExecutorTestCase
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[342,14]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[365,14]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java:[389,38]
>  error: cannot find symbol
> [ERROR] class TestMapReduceActionExecutor
> [ERROR] 
> /Users/saley/src/oozie/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestStreamingMain.java:[34,24]
>  error: cannot find symbol
> [ERROR] class TestStreamingMain
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (OOZIE-2547) Add mapreduce.job.cache.files to spark action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319852#comment-15319852
 ] 

Rohini Palaniswamy edited comment on OOZIE-2547 at 6/8/16 8:52 AM:
---

It is happening because spark-assembly.jar is getting a special treatment by 
Spark. In launch_container.sh, I see - 

{{export 
SPARK_YARN_CACHE_FILES="hdfs://clusterx-nodey.yahoo.com/user/saley/.sparkStaging/application_1464113035484_0834/spark-assembly.jar#__spark__.jar}}

When we use spark-assembly.jar, it is getting copied with alias as 
*__spark__.jar* to the current directory of container. And while setting the 
classpath, it is referring to that file 

{{export CLASSPATH="$PWD:$PWD/__spark__.jar:}}

But when we don't use spark-assembly.jar, we need to set the classpath 
explicitly. 



was (Author: satishsaley):
It is happening because spark-assembly.jar is getting a special treatment by 
Spark. In launch_container.sh, I see - 

{{export 
SPARK_YARN_CACHE_FILES="hdfs://openqe38blue-n2.blue.ygrid.yahoo.com/user/saley/.sparkStaging/application_1464113035484_0834/spark-assembly.jar#__spark__.jar}}

When we use spark-assembly.jar, it is getting copied with alias as 
*__spark__.jar* to the current directory of container. And while setting the 
classpath, it is referring to that file 

{{export CLASSPATH="$PWD:$PWD/__spark__.jar:}}

But when we don't use spark-assembly.jar, we need to set the classpath 
explicitly. 


> Add mapreduce.job.cache.files to spark action
> -
>
> Key: OOZIE-2547
> URL: https://issues.apache.org/jira/browse/OOZIE-2547
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-2547-1.patch
>
>
> Currently, we pass jars using --jars option while submitting spark job. Also, 
> we add spark.yarn.dist.files option in case of yarn-client mode. 
> Instead of that, we can have only --files option and pass on the files which 
> are present in mapreduce.job.cache.files. While doing so, we make sure that 
> spark won't make another copy of the files if files exist on the hdfs. We saw 
> the issues when files are getting copied multiple times and causing 
> exceptions such as :
> {code}
> Diagnostics: Resource 
> hdfs://localhost/user/saley/.sparkStaging/application_1234_123/oozie-examples.jar
>  changed on src filesystem
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : oozie-trunk-find-patches-available #123056

[Apache Jenkins Server]

See 

Build failed in Jenkins: oozie-trunk-find-patches-available #123055

[Apache Jenkins Server]

See 

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Mapreduce Hadoop Pig Hdfs yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 7779f3184a085a832df3116a0eab33d2d56eed78 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7779f3184a085a832df3116a0eab33d2d56eed78
 > git rev-list 7779f3184a085a832df3116a0eab33d2d56eed78 # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson7351916768665942189.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100  245k0  245k0 0   274k  0 --:--:-- --:--:-- --:--:--  274k
100  693k0  693k0 0   365k  0 --:--:--  0:00:01 --:--:--  365k
100 1079k0 1079k0 0   370k  0 --:--:--  0:00:02 --:--:--  370k
100 1079k0 1079k0 0   370k  0 --:--:--  0:00:02 --:--:--  370k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

[jira] [Updated] (OOZIE-2562) Web application not getting started and Tomcat is shutting down with hadoop-2.7.2

[Prakash K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash K updated OOZIE-2562:
-
Flags: Important

> Web application not getting started and Tomcat is shutting down with 
> hadoop-2.7.2
> -
>
> Key: OOZIE-2562
> URL: https://issues.apache.org/jira/browse/OOZIE-2562
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.2.0
> Environment: Centos7
>Reporter: Prakash K
> Fix For: 4.2.0
>
>   Original Estimate: 53h
>  Remaining Estimate: 53h
>
> Jun 08, 2016 11:40:47 AM org.apache.catalina.core.AprLifecycleListener init
> INFO: The APR based Apache Tomcat Native library which allows optimal 
> performance in production environments was not found on the 
> java.library.path: 
> Jun 08, 2016 11:40:48 AM org.apache.coyote.http11.Http11Protocol init
> SEVERE: Error initializing endpoint
> java.net.BindException: Address already in use :11000
>   at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)
>   at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:182)
>   at 
> org.apache.catalina.connector.Connector.initialize(Connector.java:1123)
>   at 
> org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
>   at 
> org.apache.catalina.core.StandardServer.initialize(StandardServer.java:843)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
>   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Caused by: java.net.BindException: Address already in use
>   at java.net.PlainSocketImpl.socketBind(Native Method)
>   at 
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
>   at java.net.ServerSocket.bind(ServerSocket.java:376)
>   at java.net.ServerSocket.(ServerSocket.java:237)
>   at java.net.ServerSocket.(ServerSocket.java:181)
>   at 
> org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:50)
>   at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
>   ... 12 more
> Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardService initialize
> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-11000]]
> LifecycleException:  Protocol handler initialization failed: 
> java.net.BindException: Address already in use :11000
>   at 
> org.apache.catalina.connector.Connector.initialize(Connector.java:1125)
>   at 
> org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
>   at 
> org.apache.catalina.core.StandardServer.initialize(StandardServer.java:843)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
>   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Jun 08, 2016 11:40:48 AM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 594 ms
> Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.43
> Jun 08, 2016 11:40:48 AM org.apache.catalina.startup.HostConfig 
> deployDescriptor
> INFO: Deploying configuration descriptor oozie.xml
> Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
> validateJarFile
> INFO: 
> validateJarFile(/usr/local/oozie/distro/target/oozie-4.2.0-distro/oozie-4.2.0/oozie-server/webapps/oozie/WEB-INF/lib/hive-jdbc-2.0.0-standalone.jar)
>  - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: 
> javax/servlet/Servlet.class
> Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
> validateJarFile
> INFO: 
> validateJarFile(/usr/local/oozie/distro/target/oozie-4.2.0-distro/oozie-4.2.0/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5.jar)

[jira] [Updated] (OOZIE-2562) Web application not getting started and Tomcat is shutting down with hadoop-2.7.2

[Prakash K (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prakash K updated OOZIE-2562:
-
Summary: Web application not getting started and Tomcat is shutting down 
with hadoop-2.7.2  (was: Web application not getting started and Tomcat is 
shutting down)

> Web application not getting started and Tomcat is shutting down with 
> hadoop-2.7.2
> -
>
> Key: OOZIE-2562
> URL: https://issues.apache.org/jira/browse/OOZIE-2562
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 4.2.0
> Environment: Centos7
>Reporter: Prakash K
> Fix For: 4.2.0
>
>   Original Estimate: 53h
>  Remaining Estimate: 53h
>
> Jun 08, 2016 11:40:47 AM org.apache.catalina.core.AprLifecycleListener init
> INFO: The APR based Apache Tomcat Native library which allows optimal 
> performance in production environments was not found on the 
> java.library.path: 
> Jun 08, 2016 11:40:48 AM org.apache.coyote.http11.Http11Protocol init
> SEVERE: Error initializing endpoint
> java.net.BindException: Address already in use :11000
>   at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)
>   at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:182)
>   at 
> org.apache.catalina.connector.Connector.initialize(Connector.java:1123)
>   at 
> org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
>   at 
> org.apache.catalina.core.StandardServer.initialize(StandardServer.java:843)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
>   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Caused by: java.net.BindException: Address already in use
>   at java.net.PlainSocketImpl.socketBind(Native Method)
>   at 
> java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
>   at java.net.ServerSocket.bind(ServerSocket.java:376)
>   at java.net.ServerSocket.(ServerSocket.java:237)
>   at java.net.ServerSocket.(ServerSocket.java:181)
>   at 
> org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:50)
>   at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
>   ... 12 more
> Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardService initialize
> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-11000]]
> LifecycleException:  Protocol handler initialization failed: 
> java.net.BindException: Address already in use :11000
>   at 
> org.apache.catalina.connector.Connector.initialize(Connector.java:1125)
>   at 
> org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
>   at 
> org.apache.catalina.core.StandardServer.initialize(StandardServer.java:843)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
>   at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
>   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
> Jun 08, 2016 11:40:48 AM org.apache.catalina.startup.Catalina load
> INFO: Initialization processed in 594 ms
> Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardService start
> INFO: Starting service Catalina
> Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardEngine start
> INFO: Starting Servlet Engine: Apache Tomcat/6.0.43
> Jun 08, 2016 11:40:48 AM org.apache.catalina.startup.HostConfig 
> deployDescriptor
> INFO: Deploying configuration descriptor oozie.xml
> Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
> validateJarFile
> INFO: 
> validateJarFile(/usr/local/oozie/distro/target/oozie-4.2.0-distro/oozie-4.2.0/oozie-server/webapps/oozie/WEB-INF/lib/hive-jdbc-2.0.0-standalone.jar)
>  - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: 
> javax/servlet/Servlet.class
> Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
> validateJarFile

[jira] [Created] (OOZIE-2562) Web application not getting started and Tomcat is shutting down

[Prakash K (JIRA)]

Prakash K created OOZIE-2562:


 Summary: Web application not getting started and Tomcat is 
shutting down
 Key: OOZIE-2562
 URL: https://issues.apache.org/jira/browse/OOZIE-2562
 Project: Oozie
  Issue Type: Bug
Affects Versions: 4.2.0
 Environment: Centos7
Reporter: Prakash K
 Fix For: 4.2.0


Jun 08, 2016 11:40:47 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal 
performance in production environments was not found on the java.library.path: 
Jun 08, 2016 11:40:48 AM org.apache.coyote.http11.Http11Protocol init
SEVERE: Error initializing endpoint
java.net.BindException: Address already in use :11000
at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:549)
at org.apache.coyote.http11.Http11Protocol.init(Http11Protocol.java:182)
at 
org.apache.catalina.connector.Connector.initialize(Connector.java:1123)
at 
org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
at 
org.apache.catalina.core.StandardServer.initialize(StandardServer.java:843)
at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)
Caused by: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at 
java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.(ServerSocket.java:237)
at java.net.ServerSocket.(ServerSocket.java:181)
at 
org.apache.tomcat.util.net.DefaultServerSocketFactory.createSocket(DefaultServerSocketFactory.java:50)
at org.apache.tomcat.util.net.JIoEndpoint.init(JIoEndpoint.java:538)
... 12 more

Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardService initialize
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-11000]]
LifecycleException:  Protocol handler initialization failed: 
java.net.BindException: Address already in use :11000
at 
org.apache.catalina.connector.Connector.initialize(Connector.java:1125)
at 
org.apache.catalina.core.StandardService.initialize(StandardService.java:703)
at 
org.apache.catalina.core.StandardServer.initialize(StandardServer.java:843)
at org.apache.catalina.startup.Catalina.load(Catalina.java:538)
at org.apache.catalina.startup.Catalina.load(Catalina.java:562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:261)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

Jun 08, 2016 11:40:48 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 594 ms
Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jun 08, 2016 11:40:48 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.43
Jun 08, 2016 11:40:48 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
validateJarFile
INFO: 
validateJarFile(/usr/local/oozie/distro/target/oozie-4.2.0-distro/oozie-4.2.0/oozie-server/webapps/oozie/WEB-INF/lib/hive-jdbc-2.0.0-standalone.jar)
 - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: 
javax/servlet/Servlet.class
Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
validateJarFile
INFO: 
validateJarFile(/usr/local/oozie/distro/target/oozie-4.2.0-distro/oozie-4.2.0/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api-2.5.jar)
 - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: 
javax/servlet/Servlet.class
Jun 08, 2016 11:40:52 AM org.apache.catalina.loader.WebappClassLoader 
validateJarFile
INFO: 
validateJarFile(/usr/local/oozie/distro/target/oozie-4.2.0-distro/oozie-4.2.0/oozie-server/webapps/oozie/WEB-INF/lib/servlet-api.jar)
 - jar not loaded. See Servlet Spec 2.3, section 

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (88 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-2555  Oozie SSL enable setup does not return port for admin -servers
https://issues.apache.org/jira/browse/OOZIE-2555
OOZIE-2552  Update ActiveMQ version for security and other fixes
https://issues.apache.org/jira/browse/OOZIE-2552
OOZIE-2550  Flaky tests in TestZKUUIDService.java
https://issues.apache.org/jira/browse/OOZIE-2550
OOZIE-2541  Resource leak in Hive2Credentials
https://issues.apache.org/jira/browse/OOZIE-2541
OOZIE-2539  Incorrect property key is used for 'hive log4j configuration file 
for execution mode'
https://issues.apache.org/jira/browse/OOZIE-2539
OOZIE-2538  Update HttpClient versions to close security vulnerabilities
https://issues.apache.org/jira/browse/OOZIE-2538
OOZIE-2534  Versioned action libs (similar to how SharedLibs works)
https://issues.apache.org/jira/browse/OOZIE-2534
OOZIE-2530  Remove jline from parent and sharelib pig pom
https://issues.apache.org/jira/browse/OOZIE-2530
OOZIE-2528  Print out environment variables in LauncherMapper
https://issues.apache.org/jira/browse/OOZIE-2528
OOZIE-2521  Filter options are case sensitive for jobtye=bundle
https://issues.apache.org/jira/browse/OOZIE-2521
OOZIE-2520  SortBy filter for ordering the jobs query results
https://issues.apache.org/jira/browse/OOZIE-2520
OOZIE-2518  Oozie is creating many files and directories in /tmp per day
https://issues.apache.org/jira/browse/OOZIE-2518
OOZIE-2517  Add support for startCreatedTime and endCreatedTime filters for 
coord and bundles
https://issues.apache.org/jira/browse/OOZIE-2517
OOZIE-2516  Update web service documentation for jobs API
https://issues.apache.org/jira/browse/OOZIE-2516
OOZIE-2515  Duplicate information for "Changing endtime/pausetime of a Bundle 
Job" in CommandLineTool wiki
https://issues.apache.org/jira/browse/OOZIE-2515
OOZIE-2513  log.scan.duration should not be used for error and audit logs
https://issues.apache.org/jira/browse/OOZIE-2513
OOZIE-2508  Documentation change for Coord action rerun [OOZIE-1735]
https://issues.apache.org/jira/browse/OOZIE-2508
OOZIE-2507  Expose monitoring via JMX beans in Oozie
https://issues.apache.org/jira/browse/OOZIE-2507
OOZIE-2506  Add logs into RecoverService for logging information about queued 
commnads
https://issues.apache.org/jira/browse/OOZIE-2506
OOZIE-2504  Create a log4j.properties under HADOOP_CONF_DIR in Shell Action
https://issues.apache.org/jira/browse/OOZIE-2504
OOZIE-2499  map-reduce launcher does not need distributed files, archives 
except  jar of input/outputformat class
https://issues.apache.org/jira/browse/OOZIE-2499
OOZIE-2498  Oozie CallerId configuration for downstream components 
https://issues.apache.org/jira/browse/OOZIE-2498
OOZIE-2497  Testcase fails on windows with hard coded URIs
https://issues.apache.org/jira/browse/OOZIE-2497
OOZIE-2496  Testcase failure due to environment specific line.separator value
https://issues.apache.org/jira/browse/OOZIE-2496
OOZIE-2495  change action status from  ErrorType.NON_TRANSIENT to TRANSIENT 
when SSH action occurs AUTH_FAILED occasionally
https://issues.apache.org/jira/browse/OOZIE-2495
OOZIE-2494  Cron syntax not handling DST properly
https://issues.apache.org/jira/browse/OOZIE-2494
OOZIE-2493  Remove action.xml file from local file system instead of hadoop 
file system
https://issues.apache.org/jira/browse/OOZIE-2493
OOZIE-2491  oozie acl cannot specify group,it does`t work
https://issues.apache.org/jira/browse/OOZIE-2491
OOZIE-2487  Unable to build distro on Java 8 (on Mac)
https://issues.apache.org/jira/browse/OOZIE-2487
OOZIE-2473  Connection pool for SMTP connection
https://issues.apache.org/jira/browse/OOZIE-2473
OOZIE-2462  When calling ParamChecker.notNull() in 
CoordActionsIgnoreXCommand.java, "Action" should be passed instead of "Action 
cannot be null".
https://issues.apache.org/jira/browse/OOZIE-2462
OOZIE-2461  Workflow, Coordinator and Bundle job querying should have last 
modified filter
https://issues.apache.org/jira/browse/OOZIE-2461
OOZIE-2458  'oozie-setup.sh sharelib create' should ensure uploaded jars are 
world readable
https://issues.apache.org/jira/browse/OOZIE-2458
OOZIE-2457  Oozie log parsing regex consume more than 90% cpu
https://issues.apache.org/jira/browse/OOZIE-2457
OOZIE-2445  Doc for -  Specifying coordinator input datasets in more logical 
ways (OOZIE-1976)
https://issues.apache.org/jira/browse/OOZIE-2445
OOZIE-2440  Exponential re-try policy for workflow action