[jira] [Commented] (OOZIE-3300) Hadoop 3 Pig (with Tez) action shall not use given distributed cache

[Daniel Dai (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532035#comment-16532035
 ] 

Daniel Dai commented on OOZIE-3300:
---

Thanks Rohini, that will work if we have to fix in Pig side. Let me know if you 
can upload the patch by tomorrow. Otherwise I can take care.

> Hadoop 3 Pig (with Tez) action shall not use given distributed cache
> 
>
> Key: OOZIE-3300
> URL: https://issues.apache.org/jira/browse/OOZIE-3300
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1
> Environment: Oozie-4.3.1
> Hadoop-3
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: OOZIE-3300_001.patch
>
>
> When I run my Pig action and set -x Tez to use Tez as the execution engine, I 
> got the following error:
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "mycluster" port: -1 file: 
> "/tmp/temp257132233/hive-exec-3.0.0.3.0.0.0-1469.jar" conflicts with cache 
> file (mapreduce.job.cache.files)
>  
> hdfs://mycluster/user/oozie/share/lib/lib_20180611041054/pig/hive-exec-3.0.0.3.0.0.0-1469.jar
> {code}
> or
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "ns1" port: -1 file: 
> "/tmp/temp36150863/joda-time-2.9.6.jar" conflicts with cache file 
> (mapreduce.job.cache.files) 
> hdfs://ns1/user/oozie/share/lib/lib_20180612003013/pig/joda-time-2.9.6.jar
> {code}
> When I exclude the conflicting jar, then Pig cannot start due to class not 
> found exception.
> I think a workaround or solution? could be if we exclude the 
> *mapreduce.job.cache.files* and *mapreduce.job.cache.archives* from 
> pig.properties file.
>  
> It is a bit scary change because I am not sure if the user's job shall use 
> specific driver or implementation which needs to be on distributed cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531828#comment-16531828
 ] 

Julia Kinga Marton edited comment on OOZIE-2791 at 7/3/18 7:26 PM:
---

The test errors seems unrelated. [~andras.piros], [~pbacsko] or anybody else 
who has the proper rights, could please start another pre commit build?


was (Author: kmarton):
The test errors seems unrelated. [~andras.piros] or [~pbacsko] could you please 
start another pre commit build?

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-010.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This 

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531828#comment-16531828
 ] 

Julia Kinga Marton commented on OOZIE-2791:
---

The test errors seems unrelated. [~andras.piros] or [~pbacsko] could you please 
start another pre commit build?

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-010.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3300) Hadoop 3 Pig (with Tez) action shall not use given distributed cache

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531810#comment-16531810
 ] 

Rohini Palaniswamy commented on OOZIE-3300:
---

bq.  Is this direction right?
 This is not right. For lot of pig jobs, users specify their own archives and 
files. It will break them. Fixing PIG-4373 should make it work. I will try to 
upload the patch for it tomorrow and you can try that.

> Hadoop 3 Pig (with Tez) action shall not use given distributed cache
> 
>
> Key: OOZIE-3300
> URL: https://issues.apache.org/jira/browse/OOZIE-3300
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1
> Environment: Oozie-4.3.1
> Hadoop-3
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: OOZIE-3300_001.patch
>
>
> When I run my Pig action and set -x Tez to use Tez as the execution engine, I 
> got the following error:
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "mycluster" port: -1 file: 
> "/tmp/temp257132233/hive-exec-3.0.0.3.0.0.0-1469.jar" conflicts with cache 
> file (mapreduce.job.cache.files)
>  
> hdfs://mycluster/user/oozie/share/lib/lib_20180611041054/pig/hive-exec-3.0.0.3.0.0.0-1469.jar
> {code}
> or
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "ns1" port: -1 file: 
> "/tmp/temp36150863/joda-time-2.9.6.jar" conflicts with cache file 
> (mapreduce.job.cache.files) 
> hdfs://ns1/user/oozie/share/lib/lib_20180612003013/pig/joda-time-2.9.6.jar
> {code}
> When I exclude the conflicting jar, then Pig cannot start due to class not 
> found exception.
> I think a workaround or solution? could be if we exclude the 
> *mapreduce.job.cache.files* and *mapreduce.job.cache.archives* from 
> pig.properties file.
>  
> It is a bit scary change because I am not sure if the user's job shall use 
> specific driver or implementation which needs to be on distributed cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531715#comment-16531715
 ] 

Robert Kanter commented on OOZIE-3160:
--

+1 on having a design document.  Not only is this a critical part of Oozie, but 
we also haven't made any _real_ changes in at least 4 years, and even that was 
relatively minimal.  Prior to that, it hadn't been touched since Oozie 2.2 back 
in 2011!
https://github.com/apache/oozie/commits/master/core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java



> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-POC01.patch
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531699#comment-16531699
 ] 

Hadoop QA commented on OOZIE-2791:
--


Testing JIRA OOZIE-2791

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2908
.Tests failed: 0
.Tests errors: 488

.The patch failed the following testcases:



.Tests failing with errors:
testShellScriptHadoopConfDir(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testShellScriptError(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testSetupMethods(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testShellScript(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testEnvVar(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testShellScriptHadoopConfDirWithNoL4J(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testPerlScript(org.apache.oozie.action.hadoop.TestShellActionExecutor)
testUpdateSLA(org.apache.oozie.sla.TestSLAService)
testEndMissDBConfirm(org.apache.oozie.sla.TestSLAService)
testSLAOperations(org.apache.oozie.sla.TestSLAService)
testBasicService(org.apache.oozie.sla.TestSLAService)
testJobs(org.apache.oozie.servlet.TestV1JobsServlet)
testSubmit(org.apache.oozie.servlet.TestV1JobsServlet)
testDefaultConfigurationInActionConf(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testParseJobXmlAndConfiguration(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCannotKillActionWhenACLSpecified(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testOutputSubmitOK(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testSubmitLauncherConfigurationOverridesLauncherMapperProperties(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testJobXmlWithOozieLauncher(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testCredentialsSkip(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testACLDefaults_noFalseChange(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testJobSubmissionWithoutYarnKill(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testFilesystemScheme(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testExceptionSubmitException(org.apache.oozie.action.hadoop.TestJavaActionExecutor)
testKill(org.apache.oozie.action.hadoop.TestJavaActionExecutor)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531568#comment-16531568
 ] 

Hadoop QA commented on OOZIE-2791:
--

PreCommit-OOZIE-Build started


> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-010.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Julia Kinga Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2791:
--
Attachment: OOZIE-2791-010.patch

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-010.patch, OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[Kinga Marton via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/
---

(Updated July 3, 2018, 3:24 p.m.)


Review request for oozie, András Piros and Peter Cseh.


Repository: oozie-git


Description
---

On a busy Hadoop cluster it can happen that users cannot install properly 
Oozie ShareLib.


Diffs (updated)
-

  tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
  tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java f53d987c 


Diff: https://reviews.apache.org/r/66930/diff/8/

Changes: https://reviews.apache.org/r/66930/diff/7-8/


Testing
---

Tested manually


Thanks,

Kinga Marton

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[Peter Bacsko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review205672
---




tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Line 226 (original), 245 (patched)


Remove comment



tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
Lines 251 (patched)


Remove comment



tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java
Lines 129 (patched)


Almost - I think the try block should include more code, just to be safe. 
At least from the for loop where you call listFiles(). At that point, the pool 
already exists.


- Peter Bacsko


On júl. 3, 2018, 8:12 de, Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated júl. 3, 2018, 8:12 de)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> f53d987c 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/7/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

Failed: OOZIE-3301 PreCommit Build #654

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-3301
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/654/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.74 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/hive].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/spark].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [examples].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2451 bytes
[TRACE] Full summary file size is 1399 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-3301

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
-1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
-1 the patch does not add/modify any testcase
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2898
Tests failed at first run:
TestCoordActionsKillXCommand#testActionKillCommandDate
For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/654/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0{"self":"https://issues.apache.org/jira/rest/api/2/issue/13169760/comment/16531522","id":"16531522","author":{"self":"https://issues.apache.org/jira/rest/api/2/user?username=hadoopqa","name":"hadoopqa","key":"hadoopqa","avatarUrls":{"48x48":"https://issues.apache.org/jira/secure/useravatar?ownerId=hadoopqa=10393","24x24":"https://issues.apache.org/jira/secure/useravatar?size=small=hadoopqa=10393","16x16":"https://issues.apache.org/jira/secure/useravatar?size=xsmall=hadoopqa=10393","32x32":"https://issues.apache.org/jira/secure/useravatar?size=medium=hadoopqa=10393"},"displayName":"Hadoop
 QA","active":true,"timeZone":"Etc/UTC"},"body":"\nTesting JIRA 

Re: Review Request 66656: Exclusion pattern for sharelib.

[András Piros via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66656/#review205669
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1119-1120 (patched)


Do a multi-catch instead.

Would also re-throw some kind of `IllegalStateException` with appropriate 
error message, but by no means a `NullPointerException` down the code path 
without any meaning.


- András Piros


On July 3, 2018, 1:30 p.m., Mate Juhasz wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66656/
> ---
> 
> (Updated July 3, 2018, 1:30 p.m.)
> 
> 
> Review request for oozie, András Piros and Denes Bodo.
> 
> 
> Bugs: OOZIE-1624
> https://issues.apache.org/jira/browse/OOZIE-1624
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-1624 Exclusion pattern for sharelib.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> ed809ef0 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShareLibExcluder.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/service/ShareLibService.java a901567d 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
> f39bba2c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a31079a4 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutorLibAddition.java
>  PRE-CREATION 
>   core/src/test/java/org/apache/oozie/action/hadoop/TestShareLibExcluder.java 
> PRE-CREATION 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 76cbe21e 
> 
> 
> Diff: https://reviews.apache.org/r/66656/diff/4/
> 
> 
> Testing
> ---
> 
> Tested on a cluster
> 
> 
> Thanks,
> 
> Mate Juhasz
> 
>

[jira] [Comment Edited] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531441#comment-16531441
 ] 

Julia Kinga Marton edited comment on OOZIE-3301 at 7/3/18 2:00 PM:
---

Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

According to the linked Redistribution part 
([http://www.apache.org/licenses/LICENSE-2.0.html#redistribution]) is enough to 
have it in one place.

 


was (Author: kmarton):
Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

According to the [linked Redistribution part| 
[http://www.apache.org/licenses/LICENSE-2.0.html#redistribution]] is enough to 
have it in one place.

 

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531441#comment-16531441
 ] 

Julia Kinga Marton edited comment on OOZIE-3301 at 7/3/18 1:59 PM:
---

Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

According to the [linked Redistribution part| 
[http://www.apache.org/licenses/LICENSE-2.0.html#redistribution]] is enough to 
have it in one place.

 


was (Author: kmarton):
Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

According to the linked Redistribution part is enough to have it in one place.

 

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531441#comment-16531441
 ] 

Julia Kinga Marton edited comment on OOZIE-3301 at 7/3/18 1:57 PM:
---

Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

According to the linked Redistribution part is enough to have it in one place.

 


was (Author: kmarton):
Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

 

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531441#comment-16531441
 ] 

Julia Kinga Marton commented on OOZIE-3301:
---

Hi [~asasvari], in the documentation there is a reference to the dependencies 
report and their licences (which actually is broken, but I have opened a Jira 
for it: OOZIE-3302). Do you think that we should include more information in 
the NOTICE.txt, or it would be enough to have a reference to the documentation?

 

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3301) Update NOTICE file

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531427#comment-16531427
 ] 

Attila Sasvari commented on OOZIE-3301:
---

Hi [~kmarton] thanks for working on it, we should ideally review all the 
licenses of included Oozie dependencies (extjs etc).  Maybe you could use a 
maven plugin or some tool to automate this.

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66656: Exclusion pattern for sharelib.

[Mate Juhasz]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66656/
---

(Updated July 3, 2018, 1:30 p.m.)


Review request for oozie, András Piros and Denes Bodo.


Bugs: OOZIE-1624
https://issues.apache.org/jira/browse/OOZIE-1624


Repository: oozie-git


Description
---

OOZIE-1624 Exclusion pattern for sharelib.


Diffs (updated)
-

  core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
ed809ef0 
  core/src/main/java/org/apache/oozie/action/hadoop/ShareLibExcluder.java 
PRE-CREATION 
  core/src/main/java/org/apache/oozie/service/ShareLibService.java a901567d 
  core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 
f39bba2c 
  core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
a31079a4 
  
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutorLibAddition.java
 PRE-CREATION 
  core/src/test/java/org/apache/oozie/action/hadoop/TestShareLibExcluder.java 
PRE-CREATION 
  docs/src/site/twiki/WorkflowFunctionalSpec.twiki 76cbe21e 


Diff: https://reviews.apache.org/r/66656/diff/4/

Changes: https://reviews.apache.org/r/66656/diff/3-4/


Testing
---

Tested on a cluster


Thanks,

Mate Juhasz

[jira] [Commented] (OOZIE-3301) Update NOTICE file

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531378#comment-16531378
 ] 

Hadoop QA commented on OOZIE-3301:
--

PreCommit-OOZIE-Build started


> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2392) Add Kafka support for notifications

[Attila Sasvari (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531375#comment-16531375
 ] 

Attila Sasvari commented on OOZIE-2392:
---

It is an interesting idea that would help to integrate Oozie jobs with external 
systems. I am wondering if you are still working on it, [~rupalph]? 

> Add Kafka support for notifications
> ---
>
> Key: OOZIE-2392
> URL: https://issues.apache.org/jira/browse/OOZIE-2392
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Robert Kanter
>Assignee: Rupal H
>Priority: Major
>
> Oozie currently lets you send JMS and email notifications.  It could be 
> useful to add support for Kafka as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3302) Broken link in documentation: Dependencies Report, Licenses section

[Julia Kinga Marton (JIRA)]

Julia Kinga Marton created OOZIE-3302:
-

 Summary:  Broken link in documentation: Dependencies Report, 
Licenses section 
 Key: OOZIE-3302
 URL: https://issues.apache.org/jira/browse/OOZIE-3302
 Project: Oozie
  Issue Type: Bug
  Components: docs
Affects Versions: 5.0.0
Reporter: Julia Kinga Marton


In the documentation there is a reference to Dependencies report page, which 
page does not exist:

[https://oozie.apache.org/docs/5.0.0/index.html#Licensing_Information]

[https://oozie.apache.org/docs/5.0.0/dependencies.html#Licenses]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-3301:
--
Attachment: OOZIE-3301-001.patch

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-3301-001.patch
>
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2942) [examples] Fix Findbugs warnings

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531200#comment-16531200
 ] 

Hadoop QA commented on OOZIE-2942:
--


Testing JIRA OOZIE-2942

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:red}-1{color} the patch contains 10 line(s) with tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2898
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/653/



> [examples] Fix Findbugs warnings
> 
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
>  Issue Type: Sub-task
>  Components: build
>Reporter: Jan Hentschel
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2942-003.patch, OOZIE-2942-004.patch, 
> OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the 
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close 
> stream on exception [org.apache.oozie.example.DateList] At 
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in 
> org.apache.oozie.example.DateList.main(String[]) 
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At 
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method 
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called 
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> 

[jira] [Commented] (OOZIE-3301) Update NOTICE file

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531188#comment-16531188
 ] 

Andras Piros commented on OOZIE-3301:
-

Thanks [~kmarton]!

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton reassigned OOZIE-3301:
-

Assignee: Julia Kinga Marton

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3301) Update NOTICE file

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531180#comment-16531180
 ] 

Julia Kinga Marton commented on OOZIE-3301:
---

[~andras.piros], yes, I can work on it

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531175#comment-16531175
 ] 

Hadoop QA commented on OOZIE-2791:
--


Testing JIRA OOZIE-2791

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch adds/modifies 5 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warning(s)
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
error(s)
.{color:red}ERROR{color}: the current HEAD has 2 Javadoc error(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [webapp].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in 
[fluent-job/fluent-job-api].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2908
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/652/



> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> 

Success: OOZIE-2791 PreCommit Build #652

[Apache Jenkins Server]

Jira: https://issues.apache.org/jira/browse/OOZIE-2791
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/652/

###
## LAST 100 LINES OF THE CONSOLE 
###
[...truncated 1.77 MB...]
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/oozie].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [sharelib/streaming].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [webapp].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [tools].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [docs].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [server].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [fluent-job/fluent-job-api].
[TRACE] New XMLLib present, calling 'xmllint --xpath' to get bug instance counts
[DEBUG] There are no new bugs found in [client].
[INFO] There are no new bugs found totally].
[TRACE] FindBugs diffs checked and reports created
[TRACE] Summary file size is 2450 bytes
[TRACE] Full summary file size is 1399 bytes
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar]
 removed
[TRACE] File 
[/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build/test-patch/tmp/FINDBUGS_DIFF/diff/findbugs-diff-0.1.0-all.jar.md5sum]
 removed
  Running test-patch task BACKWARDS_COMPATIBILITY
  Running test-patch task TESTS
  Running test-patch task DISTRO


Testing JIRA OOZIE-2791

Cleaning local git workspace



+1 PATCH_APPLIES
+1 CLEAN
+1 RAW_PATCH_ANALYSIS
+1 the patch does not introduce any @author tags
+1 the patch does not introduce any tabs
+1 the patch does not introduce any trailing spaces
+1 the patch does not introduce any line longer than 132
+1 the patch adds/modifies 5 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warning(s)
+1 the patch does not seem to introduce new Javadoc error(s)
ERROR: the current HEAD has 2 Javadoc error(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [webapp].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [fluent-job/fluent-job-api].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2908
+1 DISTRO
+1 distro tarball builds with the patch 


+1 Overall result, good!, no -1s


The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/652/

Adding comment to JIRA
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100  30880 0  100  3088  0   4215 --:--:-- --:--:-- --:--:--  

[jira] [Commented] (OOZIE-3301) Update NOTICE file

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531154#comment-16531154
 ] 

Andras Piros commented on OOZIE-3301:
-

[~kmarton] do you have free cycles to work on this one?

> Update NOTICE file
> --
>
> Key: OOZIE-3301
> URL: https://issues.apache.org/jira/browse/OOZIE-3301
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>Reporter: Attila Sasvari
>Priority: Major
>
> Oozie NOTICE file is not up to date. Please overview and update it as per 
> http://www.apache.org/licenses/LICENSE-2.0.html#redistribution
> For example: Oozie uses embedded jetty, however Eclipse license 
> (http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is 
> not listed in 
> [NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3301) Update NOTICE file

[Attila Sasvari (JIRA)]

Attila Sasvari created OOZIE-3301:
-

 Summary: Update NOTICE file
 Key: OOZIE-3301
 URL: https://issues.apache.org/jira/browse/OOZIE-3301
 Project: Oozie
  Issue Type: Bug
Affects Versions: 5.0.0
Reporter: Attila Sasvari


Oozie NOTICE file is not up to date. Please overview and update it as per 
http://www.apache.org/licenses/LICENSE-2.0.html#redistribution

For example: Oozie uses embedded jetty, however Eclipse license 
(http://www.eclipse.org/jetty/documentation/9.3.x/embedding-jetty.html) is not 
listed in 
[NOTICE.txt|https://github.com/apache/oozie/blob/a299d4a6d435a2c92cd1d0ffce7f35a2ef8d639b/NOTICE.txt#L9].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (OOZIE-2942) [examples] Fix Findbugs warnings

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531079#comment-16531079
 ] 

Julia Kinga Marton edited comment on OOZIE-2942 at 7/3/18 9:34 AM:
---

The whole files (DateList.java and Repeatable.java) are indented using tabs 
instead of whitespaces. I am not sure if I should change it, because of the 
conflicts, it may cause.

Other opinions?


was (Author: kmarton):
The whole files (DateList.java and Repeatable.java) is indented using tabs 
instead of whitespaces. I am not sure if I should change it, because of the 
conflicts, it may cause.

Other opinions?

> [examples] Fix Findbugs warnings
> 
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
>  Issue Type: Sub-task
>  Components: build
>Reporter: Jan Hentschel
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2942-003.patch, OOZIE-2942-004.patch, 
> OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the 
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close 
> stream on exception [org.apache.oozie.example.DateList] At 
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in 
> org.apache.oozie.example.DateList.main(String[]) 
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At 
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method 
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called 
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource 
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues 
> at LocalOozieExample.java:[line 76]Path continues at 
> LocalOozieExample.java:[line 77]Path continues at 
> LocalOozieExample.java:[line 78]Path continues at 
> LocalOozieExample.java:[line 81]Path continues at 
> LocalOozieExample.java:[line 88]Path continues at 
> LocalOozieExample.java:[line 89]Path continues at 
> LocalOozieExample.java:[line 91]Path continues at 
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to close stream [org.apache.oozie.example.LocalOozieExample] At 
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal 
> representation by returning Repeatable.baseline 
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose 
> internal representation by storing an externally mutable object into 
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At 
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2942) [examples] Fix Findbugs warnings

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531096#comment-16531096
 ] 

Hadoop QA commented on OOZIE-2942:
--

PreCommit-OOZIE-Build started


> [examples] Fix Findbugs warnings
> 
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
>  Issue Type: Sub-task
>  Components: build
>Reporter: Jan Hentschel
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2942-003.patch, OOZIE-2942-004.patch, 
> OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the 
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close 
> stream on exception [org.apache.oozie.example.DateList] At 
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in 
> org.apache.oozie.example.DateList.main(String[]) 
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At 
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method 
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called 
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource 
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues 
> at LocalOozieExample.java:[line 76]Path continues at 
> LocalOozieExample.java:[line 77]Path continues at 
> LocalOozieExample.java:[line 78]Path continues at 
> LocalOozieExample.java:[line 81]Path continues at 
> LocalOozieExample.java:[line 88]Path continues at 
> LocalOozieExample.java:[line 89]Path continues at 
> LocalOozieExample.java:[line 91]Path continues at 
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to close stream [org.apache.oozie.example.LocalOozieExample] At 
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal 
> representation by returning Repeatable.baseline 
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose 
> internal representation by storing an externally mutable object into 
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At 
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2942) [examples] Fix Findbugs warnings

[Andras Piros (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531095#comment-16531095
 ] 

Andras Piros commented on OOZIE-2942:
-

[~kmarton] I'm fine addressing only the relevant parts of those classes. You 
can file a separate Jira to clean up all the tabs to whitespaces in different 
modules, though.

> [examples] Fix Findbugs warnings
> 
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
>  Issue Type: Sub-task
>  Components: build
>Reporter: Jan Hentschel
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2942-003.patch, OOZIE-2942-004.patch, 
> OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the 
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close 
> stream on exception [org.apache.oozie.example.DateList] At 
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in 
> org.apache.oozie.example.DateList.main(String[]) 
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At 
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method 
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called 
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource 
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues 
> at LocalOozieExample.java:[line 76]Path continues at 
> LocalOozieExample.java:[line 77]Path continues at 
> LocalOozieExample.java:[line 78]Path continues at 
> LocalOozieExample.java:[line 81]Path continues at 
> LocalOozieExample.java:[line 88]Path continues at 
> LocalOozieExample.java:[line 89]Path continues at 
> LocalOozieExample.java:[line 91]Path continues at 
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to close stream [org.apache.oozie.example.LocalOozieExample] At 
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal 
> representation by returning Repeatable.baseline 
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose 
> internal representation by storing an externally mutable object into 
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At 
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66656: Exclusion pattern for sharelib.

[András Piros via Review Board]

> On June 20, 2018, 10:56 a.m., András Piros wrote:
> > core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutorLibAddition.java
> > Lines 433-441 (patched)
> > 
> >
> > Extract method `checkFilesInClassPath(final Path[] baseClasspath, final 
> > Path[] filesToCheck, final Predicate assertion)`
> 
> Mate Juhasz wrote:
> Could you help me with an example how to use 'Predicate 
> assertion' in this case?
> 
> András Piros wrote:
> It's even simpler than that :) a `boolean` would suffice.
> 
> Here is the complete refactor:
> ```
> 
> private void verifyFilesInDistributedCache(Map libs, 
> Configuration jobConf) throws IOException {
> assertTrue(DistributedCache.getSymlink(jobConf));
> 
> Path[] baseClassPath = {new Path(getAppPath(), TEST_JAR), 
> libs.get(TEST_ROOT_JAR)};
> Path[] filesToCheck = DistributedCache.getFileClassPaths(jobConf);
> boolean shouldBePresent = true;
> checkFilesInClasspath(baseClassPath, filesToCheck, 
> shouldBePresent);
> 
> baseClassPath = new Path[]{new Path(getAppPath(), TEST_FILE), 
> libs.get(TEST_ROOT_FILE),
> new Path(getAppPath(), TEST_FILE_SO), 
> libs.get(TEST_ROOT_SO),
> new Path(getAppPath(), TEST_FILE_SO_1), 
> libs.get(TEST_ROOT_SO_1)};
> shouldBePresent = false;
> checkFilesInClasspath(baseClassPath, filesToCheck, 
> shouldBePresent);
> 
> baseClassPath = new Path[]{new Path(getAppPath(), TEST_JAR), 
> libs.get(TEST_ROOT_JAR),
> new Path(getAppPath(), TEST_FILE), 
> libs.get(TEST_ROOT_FILE),
> new Path(getAppPath(), TEST_FILE_SO), 
> libs.get(TEST_ROOT_SO),
> new Path(getAppPath(), TEST_FILE_SO_1), 
> libs.get(TEST_ROOT_SO_1)};
> filesToCheck = 
> urisToPaths(DistributedCache.getCacheFiles(jobConf));
> shouldBePresent = true;
> checkFilesInClasspath(baseClassPath, filesToCheck, 
> shouldBePresent);
> 
> baseClassPath = new Path[]{new Path(getAppPath(), TEST_ARCHIVE), 
> libs.get(TEST_ROOT_ARCHIVE)};
> filesToCheck = 
> urisToPaths(DistributedCache.getCacheArchives(jobConf));
> checkFilesInClasspath(baseClassPath, filesToCheck, 
> shouldBePresent);
> }
> 
> private void checkFilesInClasspath(final Path[] baseClasspath, final 
> Path[] filesToCheck, final boolean shouldBePresent) {
> for (final Path basePath : baseClasspath) {
> boolean found = false;
> for (final Path fileToCheck : filesToCheck) {
> if (!found && 
> basePath.toUri().getPath().equals(fileToCheck.toUri().getPath())) {
> found = true;
> }
> }
> if (shouldBePresent) {
> assertTrue("file " + basePath.toUri().getPath() + " not 
> found in classpath", found);
> }
> else {
> assertFalse("file " + basePath.toUri().getPath() + " 
> found in classpath", found);
> }
> }
> 
> }
> ```
> 
> Isn't it way nicer? ;)

Oh sorry, the complete refactor again:

```
private void verifyFilesInDistributedCache(Map libs, 
Configuration jobConf) throws IOException {
assertTrue(DistributedCache.getSymlink(jobConf));

Path[] baseClassPath = {new Path(getAppPath(), TEST_JAR), 
libs.get(TEST_ROOT_JAR)};
Path[] filesToCheck = DistributedCache.getFileClassPaths(jobConf);
boolean shouldBePresent = true;
checkFilesInClasspath(baseClassPath, filesToCheck, shouldBePresent);

baseClassPath = new Path[]{new Path(getAppPath(), TEST_FILE), 
libs.get(TEST_ROOT_FILE),
new Path(getAppPath(), TEST_FILE_SO), libs.get(TEST_ROOT_SO),
new Path(getAppPath(), TEST_FILE_SO_1), 
libs.get(TEST_ROOT_SO_1)};
shouldBePresent = false;
checkFilesInClasspath(baseClassPath, filesToCheck, shouldBePresent);

baseClassPath = new Path[]{new Path(getAppPath(), TEST_JAR), 
libs.get(TEST_ROOT_JAR),
new Path(getAppPath(), TEST_FILE), libs.get(TEST_ROOT_FILE),
new Path(getAppPath(), TEST_FILE_SO), libs.get(TEST_ROOT_SO),
new Path(getAppPath(), TEST_FILE_SO_1), 
libs.get(TEST_ROOT_SO_1)};
filesToCheck = urisToPaths(DistributedCache.getCacheFiles(jobConf));
shouldBePresent = true;
checkFilesInClasspath(baseClassPath, filesToCheck, shouldBePresent);

baseClassPath = new Path[]{new Path(getAppPath(), TEST_ARCHIVE), 
libs.get(TEST_ROOT_ARCHIVE)};
filesToCheck = urisToPaths(DistributedCache.getCacheArchives(jobConf));
checkFilesInClasspath(baseClassPath, filesToCheck, 

[jira] [Commented] (OOZIE-2942) [examples] Fix Findbugs warnings

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531079#comment-16531079
 ] 

Julia Kinga Marton commented on OOZIE-2942:
---

The whole files (DateList.java and Repeatable.java) is indented using tabs 
instead of whitespaces. I am not sure if I should change it, because of the 
conflicts, it may cause.

Other opinions?

> [examples] Fix Findbugs warnings
> 
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
>  Issue Type: Sub-task
>  Components: build
>Reporter: Jan Hentschel
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2942-003.patch, OOZIE-2942-004.patch, 
> OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the 
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close 
> stream on exception [org.apache.oozie.example.DateList] At 
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in 
> org.apache.oozie.example.DateList.main(String[]) 
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At 
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method 
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called 
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource 
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues 
> at LocalOozieExample.java:[line 76]Path continues at 
> LocalOozieExample.java:[line 77]Path continues at 
> LocalOozieExample.java:[line 78]Path continues at 
> LocalOozieExample.java:[line 81]Path continues at 
> LocalOozieExample.java:[line 88]Path continues at 
> LocalOozieExample.java:[line 89]Path continues at 
> LocalOozieExample.java:[line 91]Path continues at 
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to close stream [org.apache.oozie.example.LocalOozieExample] At 
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal 
> representation by returning Repeatable.baseline 
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose 
> internal representation by storing an externally mutable object into 
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At 
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-2942) [examples] Fix Findbugs warnings

[Julia Kinga Marton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Julia Kinga Marton updated OOZIE-2942:
--
Attachment: OOZIE-2942-004.patch

> [examples] Fix Findbugs warnings
> 
>
> Key: OOZIE-2942
> URL: https://issues.apache.org/jira/browse/OOZIE-2942
> Project: Oozie
>  Issue Type: Sub-task
>  Components: build
>Reporter: Jan Hentschel
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2942-003.patch, OOZIE-2942-004.patch, 
> OOZIE-2942-1.patch, OOZIE-2942-2.patch
>
>
> Currently Findbugs complains about the following warnings in the 
> *oozie-examples* module:
> {code}
> [INFO] org.apache.oozie.example.DateList.main(String[]) may fail to close 
> stream on exception [org.apache.oozie.example.DateList] At 
> DateList.java:[line 69]
> [INFO] Nullcheck of date at line 55 of value previously dereferenced in 
> org.apache.oozie.example.DateList.main(String[]) 
> [org.apache.oozie.example.DateList, org.apache.oozie.example.DateList] At 
> DateList.java:[line 55]Redundant null check at DateList.java:[line 62]
> [INFO] Private method 
> org.apache.oozie.example.DateList.formatDateUTC(Calendar) is never called 
> [org.apache.oozie.example.DateList] At DateList.java:[line 97]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to clean up java.io.InputStream [org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample, 
> org.apache.oozie.example.LocalOozieExample] Obligation to clean up resource 
> created at LocalOozieExample.java:[line 72] is not dischargedPath continues 
> at LocalOozieExample.java:[line 76]Path continues at 
> LocalOozieExample.java:[line 77]Path continues at 
> LocalOozieExample.java:[line 78]Path continues at 
> LocalOozieExample.java:[line 81]Path continues at 
> LocalOozieExample.java:[line 88]Path continues at 
> LocalOozieExample.java:[line 89]Path continues at 
> LocalOozieExample.java:[line 91]Path continues at 
> LocalOozieExample.java:[line 100]
> [INFO] org.apache.oozie.example.LocalOozieExample.execute(String[]) may fail 
> to close stream [org.apache.oozie.example.LocalOozieExample] At 
> LocalOozieExample.java:[line 72]
> [INFO] org.apache.oozie.example.Repeatable.getBaseline() may expose internal 
> representation by returning Repeatable.baseline 
> [org.apache.oozie.example.Repeatable] At Repeatable.java:[line 168]
> [INFO] org.apache.oozie.example.Repeatable.setBaseline(Date) may expose 
> internal representation by storing an externally mutable object into 
> Repeatable.baseline [org.apache.oozie.example.Repeatable] At 
> Repeatable.java:[line 172]
> {code}
> They should be fixed to get the code more reliable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531035#comment-16531035
 ] 

Hadoop QA commented on OOZIE-2791:
--

PreCommit-OOZIE-Build started


> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2791) ShareLib installation may fail on busy Hadoop clusters

[Julia Kinga Marton (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530991#comment-16530991
 ] 

Julia Kinga Marton commented on OOZIE-2791:
---

[~pbacsko], I have made the suggested changes. Can you please check the new 
patch?

> ShareLib installation may fail on busy Hadoop clusters
> --
>
> Key: OOZIE-2791
> URL: https://issues.apache.org/jira/browse/OOZIE-2791
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Attachments: OOZIE-2791-003.patch, OOZIE-2791-004.patch, 
> OOZIE-2791-005.patch, OOZIE-2791-006.patch, OOZIE-2791-007.patch, 
> OOZIE-2791-008.patch, OOZIE-2791-009.patch, OOZIE-2791-01.patch, 
> OOZIE-2791-02.patch
>
>
> On a busy Hadoop cluster it can happen that users cannot install properly  
> Oozie ShareLib.
> Example on a Hadoop 2.4.0 pseudo cluster sharelib installion with a  
> concurrency number set high (to simulate a busy cluster):
> {code}
> oozie-setup.sh sharelib create -fs hdfs://localhost:9000 -locallib 
> oozie-sharelib-*.tar.gz -concurrency 150
> {code}
> You can see a lot of errors (failed copy tasks) on the output:
> {code}
> Running 464 copy tasks on 150 threads
> Error: Copy task failed with exception
> Stack trace for the error was (for debug purposes):
> --
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File 
> /user/asasvari/share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar 
> could only be replicated to 0 nodes instead of minReplication (=1).  There 
> are 1 datanode(s) running and no node(s) are excluded in this operation.
>   at 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1430)
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2684)
>   at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:584)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:440)
>   at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>   at org.apache.hadoop.ipc.Client.call(Client.java:1363)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
>   at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:361)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1439)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1261)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:525)
> --
> ...
> {code}
> You can see file is created but it's size is 0.
> {code}
> -rw-r--r--   3 asasvari supergroup  0 2017-02-07 10:59 
> share/lib/lib_20170207105926/distcp/hadoop-distcp-2.4.0.jar
> {code}
> This behaviour is clearly wrong. 
> In case of such an exception, we should retry copying or rollback changes. We 
> should also consider throttling HDFS requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[Kinga Marton via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/
---

(Updated July 3, 2018, 8:12 a.m.)


Review request for oozie, András Piros and Peter Cseh.


Changes
---

review fix


Repository: oozie-git


Description
---

On a busy Hadoop cluster it can happen that users cannot install properly 
Oozie ShareLib.


Diffs (updated)
-

  tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
  tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
PRE-CREATION 
  tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java f53d987c 


Diff: https://reviews.apache.org/r/66930/diff/7/

Changes: https://reviews.apache.org/r/66930/diff/6-7/


Testing
---

Tested manually


Thanks,

Kinga Marton

Re: Review Request 66930: OOZIE-2791 ShareLib installation may fail on busy Hadoop clusters

[Kinga Marton via Review Board]

> On July 2, 2018, 2:01 p.m., Peter Bacsko wrote:
> > tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
> > Lines 256-259 (patched)
> > 
> >
> > To me this looks weird. The class is named "configuration", which is 
> > telling me that it holds data. But it doesn't just hold data, it also 
> > performs an FS operation.

You are right, I have moved it to ConcurrentCopyFromLocal


- Kinga


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66930/#review205638
---


On June 18, 2018, 10:03 a.m., Kinga Marton wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66930/
> ---
> 
> (Updated June 18, 2018, 10:03 a.m.)
> 
> 
> Review request for oozie, András Piros and Peter Cseh.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> On a busy Hadoop cluster it can happen that users cannot install properly 
> Oozie ShareLib.
> 
> 
> Diffs
> -
> 
>   tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java dce1c559 
>   tools/src/test/java/org/apache/oozie/tools/OozieSharelibFileOperations.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestBlockSizeCalculator.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestConcurrentCopyFromLocal.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestCopyTaskCallable.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 
> ccad273b 
> 
> 
> Diff: https://reviews.apache.org/r/66930/diff/6/
> 
> 
> Testing
> ---
> 
> Tested manually
> 
> 
> Thanks,
> 
> Kinga Marton
> 
>

[jira] Subscription: Oozie Patch Available

[jira]

Issue Subscription
Filter: Oozie Patch Available (93 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3297  Retry logic does not handle the exception from BulkJPAExecutor 
properly
https://issues.apache.org/jira/browse/OOZIE-3297
OOZIE-3274  Remove slf4j
https://issues.apache.org/jira/browse/OOZIE-3274
OOZIE-3265  properties RERUN_FAIL_NODES and RERUN_SKIP_NODES should be able to 
appear together
https://issues.apache.org/jira/browse/OOZIE-3265
OOZIE-3256  refactor OozieCLI class
https://issues.apache.org/jira/browse/OOZIE-3256
OOZIE-3249  [tools] Instrumentation log parser
https://issues.apache.org/jira/browse/OOZIE-3249
OOZIE-3218  Oozie Sqoop action with command splits the select clause into 
multiple parts due to delimiter being space
https://issues.apache.org/jira/browse/OOZIE-3218
OOZIE-3208  "It should never happen" error messages should be more specific to 
root cause
https://issues.apache.org/jira/browse/OOZIE-3208
OOZIE-3199  Let system property restriction configurable
https://issues.apache.org/jira/browse/OOZIE-3199
OOZIE-3196  Authorization: restrict world readability by user
https://issues.apache.org/jira/browse/OOZIE-3196
OOZIE-3194  Oozie should set proper permissions to sharelib after upload
https://issues.apache.org/jira/browse/OOZIE-3194
OOZIE-3193  Applications are not killed when submitted via subworkflow
https://issues.apache.org/jira/browse/OOZIE-3193
OOZIE-3186  Oozie is unable to use configuration linked using jceks://file/...
https://issues.apache.org/jira/browse/OOZIE-3186
OOZIE-3179  Adding a configurable config-default.xml location to a workflow
https://issues.apache.org/jira/browse/OOZIE-3179
OOZIE-3170  Oozie Diagnostic Bundle tool fails with NPE due to missing service 
class
https://issues.apache.org/jira/browse/OOZIE-3170
OOZIE-3160  PriorityDelayQueue put()/take() can cause significant CPU load due 
to busy waiting
https://issues.apache.org/jira/browse/OOZIE-3160
OOZIE-3135  Configure log4j2 in SqoopMain
https://issues.apache.org/jira/browse/OOZIE-3135
OOZIE-3091  Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
org/apache/avro/mapred/AvroWrapper"
https://issues.apache.org/jira/browse/OOZIE-3091
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3061  Kill only those child jobs which are not already killed
https://issues.apache.org/jira/browse/OOZIE-3061
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  [examples] Fix Findbugs warnings
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2812  SparkConfigurationService should support loading configurations 
from multiple Spark versions
https://issues.apache.org/jira/browse/OOZIE-2812
OOZIE-2795  Create lib directory or symlink for Oozie CLI during packaging
https://issues.apache.org/jira/browse/OOZIE-2795
OOZIE-2791  ShareLib installation may fail on busy Hadoop clusters
https://issues.apache.org/jira/browse/OOZIE-2791
OOZIE-2784  Include WEEK as a parameter in the Coordinator Expression Language 
Evaulator
https://issues.apache.org/jira/browse/OOZIE-2784
OOZIE-2779  Mask Hive2 action Beeline JDBC password
https://issues.apache.org/jira/browse/OOZIE-2779
OOZIE-2736  Reduce the number of threads during test execution
https://issues.apache.org/jira/browse/OOZIE-2736
OOZIE-2718  Improve -dryrun for bundles
https://issues.apache.org/jira/browse/OOZIE-2718
OOZIE-2714  Detect conflicting resources during class loading