[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x

[Rohini Palaniswamy (Jira)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579253#comment-17579253
 ] 

Rohini Palaniswamy commented on OOZIE-3136:
---

Is anyone working on this? I did migrate Oozie completely to 2.x (just deployed 
internally and seems to be good) and can put up the patch for review. Local and 
mini oozie are yet to be done but I can get that done soon. The Oozie client 
and server will be on 2.x but launched jobs will still be on hadoop 1.x till 
Hadoop migrates. 

> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Attila Sasvári
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee 
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end 
> of life (EOL) and is no longer officially supported.}} 
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (OOZIE-3564) Upgrade to JSP EL 3.0

[Rohini Palaniswamy (Jira)]

Rohini Palaniswamy created OOZIE-3564:
-

 Summary: Upgrade to JSP EL 3.0
 Key: OOZIE-3564
 URL: https://issues.apache.org/jira/browse/OOZIE-3564
 Project: Oozie
  Issue Type: Improvement
Reporter: Rohini Palaniswamy


 Currently Oozie EL functions are implemented with JSP EL 2.0. Should consider 
upgrading to EL 3.0 which has more features

https://www.baeldung.com/jsf-expression-language-el-3
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/cwlp_el30.html
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (OOZIE-3548) FS EL action to compute size of glob

[Rohini Palaniswamy (Jira)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962547#comment-16962547
 ] 

Rohini Palaniswamy commented on OOZIE-3548:
---

Would suggest 
https://github.com/apache/pig/blob/trunk/src/org/apache/pig/LoadFunc.java#L230-L250
 instead

> FS EL action to compute size of glob
> 
>
> Key: OOZIE-3548
> URL: https://issues.apache.org/jira/browse/OOZIE-3548
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Jacob Tolar
>Priority: Major
>
> There is an FS el function (dirSize) that computes the size of a single 
> directory. 
>  
> It would be helpful if oozie could give the size of a list or glob of 
> directories (for example, to get the total size of an input dependency with 
> multiple paths).
>  
> I'd like to use this to compute a parallelism value in Pig. Pig sort of 
> supports this with the bytesPerReducer setting but that settings is applied 
> to the entire job; I'd like more flexibility.
>  
> Unfortunately, correctly supporting both a list of filese (as provided by 
> oozie) and globs requires copying in this method: 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L498-L538]
>  
> [~rohini] [~puru] I am happy to submit a patch. Any suggestions before I do 
> so?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (OOZIE-3530) Avoid creating Credential class for every type for all actions

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy resolved OOZIE-3530.
---
Resolution: Invalid

Closing it as this does not apply to master.

> Avoid creating Credential class for every type for all actions
> --
>
> Key: OOZIE-3530
> URL: https://issues.apache.org/jira/browse/OOZIE-3530
> Project: Oozie
>  Issue Type: Bug
>Reporter: Manasa Gogineni
>Assignee: Manasa Gogineni
>Priority: Major
>
> We want to only create the relevant credential classes for each action type. 
> The current Oozie functioning can be seen below:



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (OOZIE-3530) Avoid creating Credential class for every type for all actions

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated OOZIE-3530:
--
Description: 
We want to only create the relevant credential classes for each action type. 
The current Oozie functioning can be seen below:



  was:
We want to only create the relevant credential classes for each action type. 
The current Oozie functioning can be seen below:

2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] About to relogin from keytab
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Relogin from keytab 
successful
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Credentials Provider is 
created for Type: hcat
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential class 
for : yca=com.yahoo.oozie.action.hadoop.YCAV2Credentials
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential type : 
'yca', class Name : 'com.yahoo.oozie.action.hadoop.YCAV2Credentials'
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential class 
for : athens=com.yahoo.oozie.action.hadoop.AthensCredentials
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential type : 
'athens', class Name : 'com.yahoo.oozie.action.hadoop.AthensCredentials'
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential class 
for : ykeykey=com.yahoo.oozie.action.hadoop.YKeyKeyCredentials
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential type : 
'ykeykey', class Name : 'com.yahoo.oozie.action.hadoop.YKeyKeyCredentials'
2019-07-05 22:46:22,267 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential class 
for : aws=com.yahoo.oozie.action.hadoop.AWSCredentials
2019-07-05 22:46:22,268 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential type : 
'aws', class Name : 'com.yahoo.oozie.action.hadoop.AWSCredentials'
2019-07-05 22:46:22,268 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential class 
for : hcat=org.apache.oozie.action.hadoop.HCatCredentials
2019-07-05 22:46:22,268 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential type : 
'hcat', class Name : 'org.apache.oozie.action.hadoop.HCatCredentials'
2019-07-05 22:46:22,268 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf-allactions] 
JOB[0131754-190702130507683-oozie_KR-W] 
ACTION[0131754-190702130507683-oozie_KR-W@hcat_pig] Creating Credential class 
for : hbase=org.apache.oozie.action.hadoop.HbaseCredentials
2019-07-05 22:46:22,268 DEBUG CredentialsProvider:526 [pool-13-thread-196] - 
SERVER[wf451] USER[gmon] GROUP[-] TOKEN[] APP[wf

[jira] [Comment Edited] (OOZIE-3526) Global job-xml not being overwritten by job-xml specified for an action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887271#comment-16887271
 ] 

Rohini Palaniswamy edited comment on OOZIE-3526 at 7/17/19 5:34 PM:


The test failures are unrelated to the patch.

+1. Committed to master. Thanks for the contribution [~mgogineni]


was (Author: rohini):
+1. Committed to master. Thanks for the contribution [~mgogineni]

> Global job-xml not being overwritten by job-xml specified for an action
> ---
>
> Key: OOZIE-3526
> URL: https://issues.apache.org/jira/browse/OOZIE-3526
> Project: Oozie
>  Issue Type: Bug
>Reporter: Manasa Gogineni
>Assignee: Manasa Gogineni
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3526-1.patch, OOZIE-3526-2.patch, 
> OOZIE-3526-3.patch, OOZIE-3526-4.patch
>
>
> Configurations specified within the configuration tag of the action-xml will 
> take precedence over the global configuration, but any configurations 
> specified in the job-xml will not override the global configurations.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Updated] (OOZIE-3526) Global job-xml not being overwritten by job-xml specified for an action

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated OOZIE-3526:
--
Summary: Global job-xml not being overwritten by job-xml specified for an 
action  (was: Global-configuration not being overwritten by configuration 
specified for an action in oozie workflow.xml)

> Global job-xml not being overwritten by job-xml specified for an action
> ---
>
> Key: OOZIE-3526
> URL: https://issues.apache.org/jira/browse/OOZIE-3526
> Project: Oozie
>  Issue Type: Bug
>Reporter: Manasa Gogineni
>Assignee: Manasa Gogineni
>Priority: Major
> Attachments: OOZIE-3526-1.patch, OOZIE-3526-2.patch, 
> OOZIE-3526-3.patch, OOZIE-3526-4.patch
>
>
> Configurations specified within the configuration tag of the action-xml will 
> take precedence over the global configuration, but any configurations 
> specified in the job-xml will not override the global configurations.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Commented] (OOZIE-3526) Global-configuration not being overwritten by configuration specified for an action in oozie workflow.xml

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886525#comment-16886525
 ] 

Rohini Palaniswamy commented on OOZIE-3526:
---

There is newline missing before createJobXml method. Can you add it?

> Global-configuration not being overwritten by configuration specified for an 
> action in oozie workflow.xml
> -
>
> Key: OOZIE-3526
> URL: https://issues.apache.org/jira/browse/OOZIE-3526
> Project: Oozie
>  Issue Type: Bug
>Reporter: Manasa Gogineni
>Assignee: Manasa Gogineni
>Priority: Major
> Attachments: OOZIE-3526-1.patch, OOZIE-3526-2.patch, 
> OOZIE-3526-3.patch
>
>
> Configurations specified within the configuration tag of the action-xml will 
> take precedence over the global configuration, but any configurations 
> specified in the job-xml will not override the global configurations.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (OOZIE-3527) Oozie stuck in waiting state if CoordPushDependencyCheckXCommand is not requeued

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created OOZIE-3527:
-

 Summary: Oozie stuck in waiting state if 
CoordPushDependencyCheckXCommand is not requeued
 Key: OOZIE-3527
 URL: https://issues.apache.org/jira/browse/OOZIE-3527
 Project: Oozie
  Issue Type: Bug
Reporter: Rohini Palaniswamy
Assignee: Manasa Gogineni


We had a case where CoordPushDependencyCheckXCommand failed during loadState 
during a DB service failover to another node. Failure in loadState does not 
requeue. Ideally RecoveryService should pick it up, but since the 
CoordActionInputCheckXCommand was also running due to hdfs dependencies and 
constantly updating the lastModifiedTime, the RecoveryService was also not 
picking it up. Action was stuck in WAITING forever as the hcat dependencies 
were never discovered.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (OOZIE-3524) fs:fileSize() does not work correctly for files with extra slash in path

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created OOZIE-3524:
-

 Summary: fs:fileSize() does not work correctly for files with 
extra slash in path
 Key: OOZIE-3524
 URL: https://issues.apache.org/jira/browse/OOZIE-3524
 Project: Oozie
  Issue Type: Bug
Reporter: Rohini Palaniswamy
Assignee: Manasa Gogineni


${fs:fileSize('hdfs://mynamendoe:8020//user/rohini/myfile.gz') gt 0} will 
return false even though the file size is > 0 due to //user not being 
normalized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3523) First missing dependency is shown incorrectly

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created OOZIE-3523:
-

 Summary: First missing dependency is shown incorrectly
 Key: OOZIE-3523
 URL: https://issues.apache.org/jira/browse/OOZIE-3523
 Project: Oozie
  Issue Type: Bug
Reporter: Rohini Palaniswamy
Assignee: Manasa Gogineni


When there are two missing dependencies, and the first one is available and the 
check for second one fails with AccessControlException: Permission denied or 
some other exception, we still show the first missing dependency as the first 
one. This is very misleading for the user and they usually waste time trying to 
see why it is showing as missing when it is available. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3514) Investigate the future of the Pig action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862258#comment-16862258
 ] 

Rohini Palaniswamy commented on OOZIE-3514:
---

[~asalamon74],
We cannot drop Pig action support as there are still folks like us who use 
it heavily. On the Pig side, we will plan for a release soon with Hadoop 3 
support.

> Investigate the future of the Pig action
> 
>
> Key: OOZIE-3514
> URL: https://issues.apache.org/jira/browse/OOZIE-3514
> Project: Oozie
>  Issue Type: Bug
>Reporter: Andras Salamon
>Priority: Blocker
> Fix For: 6.0
>
>
> Pig is one of the oldest actions in Oozie, it was part of the original open 
> source release.
> Right now one important Oozie task ( OOZIE-2973 - Make sure Oozie works with 
> Hadoop 3 ) is blocked by a PIG issue ( PIG-5253 ) for a very long time and I 
> don't really think this issue will be solved soon:
>  * The latest Pig version (0.17) was released almost 2 years ago.
>  * There are only [6 commits|https://github.com/apache/pig/commits/trunk] in 
> the project in 2019.
> I think sooner or later we have to choose if we want to support Hadoop 3 or 
> Pig, and Hadoop3 is much more important for Oozie. I think we may consider 
> dropping Pig support. Of course such a big change should only targeted to 
> 6.0, but we could already make it deprecated in the next minor (5.2) version.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (OOZIE-1122) Move to TestNG from junit

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy resolved OOZIE-1122.
---
Resolution: Won't Fix

Not worth spending time on now. Closing it as wont fix

> Move to TestNG from junit
> -
>
> Key: OOZIE-1122
> URL: https://issues.apache.org/jira/browse/OOZIE-1122
> Project: Oozie
>  Issue Type: Improvement
>    Reporter: Rohini Palaniswamy
>Priority: Major
>
>   It would be good to move to TestNG to make good use of BeforeClass and 
> AfterClass. Service initialization is done before every method and teardown 
> after every mothod which is not necessary. Junit also has the BeforeClass and 
> AfterClass but they need to be static making all the variables to be declared 
> as static which is ugly and also doesn't gel with the current model of 
> inheritance in the test classes. Will cut down on the time taken for unit 
> tests by a lot.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3300) Hadoop 3 Pig (with Tez) action shall not use given distributed cache

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531810#comment-16531810
 ] 

Rohini Palaniswamy commented on OOZIE-3300:
---

bq.  Is this direction right?
 This is not right. For lot of pig jobs, users specify their own archives and 
files. It will break them. Fixing PIG-4373 should make it work. I will try to 
upload the patch for it tomorrow and you can try that.

> Hadoop 3 Pig (with Tez) action shall not use given distributed cache
> 
>
> Key: OOZIE-3300
> URL: https://issues.apache.org/jira/browse/OOZIE-3300
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1
> Environment: Oozie-4.3.1
> Hadoop-3
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: OOZIE-3300_001.patch
>
>
> When I run my Pig action and set -x Tez to use Tez as the execution engine, I 
> got the following error:
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "mycluster" port: -1 file: 
> "/tmp/temp257132233/hive-exec-3.0.0.3.0.0.0-1469.jar" conflicts with cache 
> file (mapreduce.job.cache.files)
>  
> hdfs://mycluster/user/oozie/share/lib/lib_20180611041054/pig/hive-exec-3.0.0.3.0.0.0-1469.jar
> {code}
> or
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "ns1" port: -1 file: 
> "/tmp/temp36150863/joda-time-2.9.6.jar" conflicts with cache file 
> (mapreduce.job.cache.files) 
> hdfs://ns1/user/oozie/share/lib/lib_20180612003013/pig/joda-time-2.9.6.jar
> {code}
> When I exclude the conflicting jar, then Pig cannot start due to class not 
> found exception.
> I think a workaround or solution? could be if we exclude the 
> *mapreduce.job.cache.files* and *mapreduce.job.cache.archives* from 
> pig.properties file.
>  
> It is a bit scary change because I am not sure if the user's job shall use 
> specific driver or implementation which needs to be on distributed cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3300) Hadoop 3 Pig (with Tez) action shall not use given distributed cache

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530744#comment-16530744
 ] 

Rohini Palaniswamy commented on OOZIE-3300:
---

Will talk to the Hadoop folks. This was done once before and reverted. Doing 
this would break a lot of other user hadoop jobs, not just Pig. 

> Hadoop 3 Pig (with Tez) action shall not use given distributed cache
> 
>
> Key: OOZIE-3300
> URL: https://issues.apache.org/jira/browse/OOZIE-3300
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.1
> Environment: Oozie-4.3.1
> Hadoop-3
>Reporter: Denes Bodo
>Assignee: Denes Bodo
>Priority: Critical
>  Labels: usability
> Attachments: OOZIE-3300_001.patch
>
>
> When I run my Pig action and set -x Tez to use Tez as the execution engine, I 
> got the following error:
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "mycluster" port: -1 file: 
> "/tmp/temp257132233/hive-exec-3.0.0.3.0.0.0-1469.jar" conflicts with cache 
> file (mapreduce.job.cache.files)
>  
> hdfs://mycluster/user/oozie/share/lib/lib_20180611041054/pig/hive-exec-3.0.0.3.0.0.0-1469.jar
> {code}
> or
> {code:java}
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.mapred.InvalidJobConfException: cache file 
> (mapreduce.job.cache.files) scheme: "hdfs" host: "ns1" port: -1 file: 
> "/tmp/temp36150863/joda-time-2.9.6.jar" conflicts with cache file 
> (mapreduce.job.cache.files) 
> hdfs://ns1/user/oozie/share/lib/lib_20180612003013/pig/joda-time-2.9.6.jar
> {code}
> When I exclude the conflicting jar, then Pig cannot start due to class not 
> found exception.
> I think a workaround or solution? could be if we exclude the 
> *mapreduce.job.cache.files* and *mapreduce.job.cache.archives* from 
> pig.properties file.
>  
> It is a bit scary change because I am not sure if the user's job shall use 
> specific driver or implementation which needs to be on distributed cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 66656: Exclusion pattern for sharelib.

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66656/#review201707
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1912 (patched)
<https://reviews.apache.org/r/66656/#comment283259>

Just a minor thing. Call it exclude instead of exclusion which is more 
standard.


- Rohini Palaniswamy


On April 17, 2018, 8:29 a.m., Mate Juhasz wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66656/
> ---
> 
> (Updated April 17, 2018, 8:29 a.m.)
> 
> 
> Review request for oozie, András Piros and Denes Bodo.
> 
> 
> Bugs: OOZIE-1624
> https://issues.apache.org/jira/browse/OOZIE-1624
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-1624 Exclusion pattern for sharelib.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> bc0f405 
>   core/src/test/java/org/apache/oozie/service/TestShareLibService.java 
> d244166 
> 
> 
> Diff: https://reviews.apache.org/r/66656/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on a cluster
> 
> 
> Thanks,
> 
> Mate Juhasz
> 
>

[CVE-2017-15712] Apache Oozie Server vulnerability

[Rohini Palaniswamy]

Apache Oozie is a workflow scheduler system to manage Apache Hadoop jobs.

Severity: Severe

Vendor:
The Apache Software Foundation

Versions Affected:
Oozie 3.1.3-incubating to Oozie 4.3.0
Oozie 5.0.0-beta1

Description:
Vulnerability allows a user of Oozie to expose private files on the Oozie
server process.  The malicious user can construct a workflow XML file
containing XML directives and configuration that reference sensitive files
on the Oozie server host.

Mitigation:
Users should upgrade to Apache Oozie 4.3.1 release from
http://oozie.apache.org/ .
Users should use 5.0.0-beta1 release only for testing purposes and wait for
the 5.0.0 GA which will have the fix.

Credit:
The issues were discovered by Daryn Sharp and Jason Lowe of Oath (formerly
Yahoo! Inc).

Re: 4.3.0 release is missing from Apache

[Rohini Palaniswamy]

That does not sound right. From what I have seen in other projects current
release and one or more older releases are available.
For eg:
http://apache.claz.org/hive/
http://apache.claz.org/pig/

Regards,
Rohini

On Thu, Feb 15, 2018 at 8:56 AM, Peter Cseh  wrote:

> Thanks for reporting! I have to update the links as only the latest release
> will be available on the mirrors.
>
>
> On Thu, Feb 15, 2018 at 5:13 PM, Artem Ervits 
> wrote:
>
> > I can't locate 4.3.0 release, 4.3.1 is only available as a src release.
> >
> > Not Found
> >
> > The requested URL /apache/oozie/4.3.0 was not found on this server.
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> 
>

Re: [VOTE][RESULT] Oozie 4.3.1(candidate 4)

[Rohini Palaniswamy]

Thanks Artem for reporting. But it should be fine. We mostly consider
important or critical bugs and some nice to haves for patch releases.
OOZIE-1213 is a minor build issue and can be ignored.

Regards,
Rohini

On Thu, Feb 15, 2018 at 8:05 AM, Artem Ervits  wrote:

> it's late but seems I'm hitting the following jira on 4.3.1
> https://issues.apache.org/jira/browse/OOZIE-1213, I see the fix in master
> but in 4.3 branch clover.version is missing.
>
> https://github.com/apache/oozie/blob/branch-4.3/pom.xml#L1687-#L1700
>
>
>
> On Tue, Feb 13, 2018 at 7:57 AM, Gézapeti  wrote:
>
> > Hi,
> >
> > Oozie 4.3.1 RC4 has passed the vote.
> >
> > The vote tally is:
> > +1 --> 3 votes from Oozie PMC members (binding), 1 vote from
> > Contributors (non-binding)
> > -1 --> 0
> > 0 --> 0
> >
> >
> > On Tue, Feb 6, 2018 at 1:42 PM, Gézapeti Cseh 
> wrote:
> > > Hi,
> > >
> > > I have created a build for Oozie 4.3.1, candidate 4.
> > >
> > > It includes all the changes in rc3 plus OOZIE-2888 to fix sharelib
> > > creation with Hadoop 2.7.5. and possibly other newer versions.
> > >
> > > Keys to verify the signature of the release artifact are available at
> > >   http://www.apache.org/dist/oozie/KEYS
> > >
> > > Please download, test, and try it out:
> > >
> > > http://people.apache.org/~gezapeti/oozie-4.3.1-rc4/
> > >
> > > The release, md5 signature, gpg signature, and rat report can be found
> at
> > > the above address.
> > >
> > > Vote closes on 02/12/2018 11:59 PM CET.
> >
>

Re: [VOTE] Release Oozie 4.3.1(candidate 4)

[Rohini Palaniswamy]

+1 (binding)

- Verified md5, signature, release log and rat report
- Ran all the unit tests and they are good.
- Created distro with bin/mkdistro.sh -DskipTests -Puber -Phadoop-2
- Checked the UI and executed a simple workflow
- Verified that the tomcat is now 6.0.53

Looks like the vote ends on Feb 12 which is 7 days and is very long. Isn't
it usually 72 hours excluding weekends and holidays?

Regards,
Rohini

On Wed, Feb 7, 2018 at 1:35 PM, Robert Kanter  wrote:

> Here's what I did:
> - Verified md5
> - Verified signature (gpg)
> - Looked at rat report
> - Looked at release log
> - Verified Tomcat is version 6.0.53
> - Built Oozie against Hadoop 2.7.2, using an empty local maven repo
> - Ran a few of the examples, clicked around the UI, tried various CLI
> commands, looked at the REST output, launcher log output, etc
>
>
> +1
>
> On Tue, Feb 6, 2018 at 7:24 PM, Artem Ervits 
> wrote:
>
> > Attila, AFAIK, these properties are in my core-site.xml except for last
> > one, mapreduce.framework.name is in mapred-site.xml. I'll have to double
> > check on the proxyuser.root.hosts as I believe my service runs with user
> > oozie not root and get back to you.
> >
> > On Tue, Feb 6, 2018 at 3:49 PM, Attila Sasvari 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > - Verified MD5 hash, signature, rat_report, release log are OK
> > > - Built Oozie against Hadoop 2.7.5 (did not build against other Hadoop
> > > versions this time)
> > > - Installed Oozie sharelib
> > > - Some example workflows I submitted to a 1-node Hadoop 2.7.5 (running
> in
> > > pseudo distributed mode for tests) succeeded: ( aggregator bundle
> > > coord-input-logic cron cron-schedule custom-main datelist-java-main
> demo
> > > distcp hadoop-el hcatalog java-main map-reduce no-op pig shell sla
> sqoop
> > > sqoop-freeform ssh streaming subwf )
> > > - Verified basic functionality of the Web UI (version, Job DAG, Job
> logs)
> > >
> > >
> > > Artem, have you specified conf/hadoop-conf/core-site.xml with all the
> > > details Oozie needs to communicate with a Hadoop cluster?
> > >
> > > In my local environment, I use a pseudo Hadoop 2.7.5. My
> > > conf/hadoop-conf/core-site.xml is as follows:
> > > 
> > > 
> > > 
> > > 
> > > 
> > >   
> > > fs.defaultFS
> > > hdfs://localhost:9000
> > >   
> > >   
> > > hadoop.proxyuser.root.hosts
> > > *
> > >   
> > >   
> > > hadoop.proxyuser.root.groups
> > > *
> > >   
> > > 
> > > mapreduce.framework.name
> > > yarn
> > > 
> > > 
> > >
> > >
> > > On Tue, Feb 6, 2018 at 8:10 PM, Artem Ervits 
> > > wrote:
> > >
> > > > jdk: 1.8.0_161
> > > > hadoop: 2.7.5
> > > > md5: OK
> > > > curl http://localhost:11000/oozie/v1/admin/status
> > > > {"systemMode":"NORMAL"}
> > > >
> > > > curl http://localhost:11000/oozie/v2/admin/status
> > > > {"systemMode":"NORMAL"}
> > > >
> > > >
> > > > I also got in oozie-ops.log
> > > >
> > > >   STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [oozie] on
> > > > [2018.02.06-16:30:43GMT]
> > > >   STARTUP MSG:   revision [unavailable]@[unavailable]
> > > >
> > > > so perhaps we may want to add but not critical OOZIE-2004,
> OOZIE-3083,
> > > > OOZIE-3152.
> > > >
> > > >  submitted a job and got but that is most likely my environment.
> > > Otherwise
> > > > +1
> > > >
> > > > 2018-02-06 18:09:37,635  WARN ActionStartXCommand:523 - SERVER[
> > > > hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
> > > > JOB[000-180206180622504-oozie-oozi-W]
> > > > ACTION[000-180206180622504-oozie-oozi-W@python-node] Error
> > starting
> > > > action [python-node]. ErrorType [ERROR], ErrorCode
> > > > [UnsupportedOperationException], Message [
> > UnsupportedOperationException:
> > > > Accessing local file system is not allowed]
> > > > org.apache.oozie.action.ActionExecutorException:
> > > > UnsupportedOperationException: Accessing local file system is not
> > allowed
> > > > at
> > > > org.apache.oozie.action.ActionExecutor.convertException(Acti
> > > > onExecutor.java:446)
> > > > at
> > > > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLaun
> > > > cher(JavaActionExecutor.java:1204)
> > > > at
> > > > org.apache.oozie.action.hadoop.JavaActionExecutor.start(Java
> > > > ActionExecutor.java:1386)
> > > > at
> > > > org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> > > > onStartXCommand.java:234)
> > > > at
> > > > org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> > > > onStartXCommand.java:65)
> > > > at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> > > > at
> > > > org.apache.oozie.service.CallableQueueService$CompositeCalla
> > > > ble.call(CallableQueueService.java:332)
> > > > at
> > > > org.apache.oozie.service.CallableQueueService$CompositeCalla
> > > > ble.call(CallableQueueService.java:261)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354633#comment-16354633
 ] 

Rohini Palaniswamy commented on OOZIE-3134:
---

bq. This issue was originally targeted to fix only scenario 2)
 Just looked at the patch. All it does is a minor change of adding the entry to 
the map after the database operation. This should be fine to go in. 1) can be 
addressed in OOZIE-1980. We started with that jira and patch when he had a 
database issue. But never got around to wrapping that up and it did not became 
a priority as we have not run into database issues after that for long (Touch 
wood). But it is something that has to be fixed.

Was not aware of OOZIE-2854 and database update retries. Will take a look at 
that sometime.

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3134) Potential inconsistency between the in-memory SLA map and the Oozie database

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354017#comment-16354017
 ] 

Rohini Palaniswamy commented on OOZIE-3134:
---

There is more to this issue. Please refer OOZIE-1980. 

There are two issues to be addressed.

1) Include the SLA inserts as part of the coord action table inserts and make 
them one transaction which JPA can rollback on database failure. Currently they 
are two separate JPA transactions.
2) Remove the SLA entry from the in memory map on JPA rollback.

> Potential inconsistency between the in-memory SLA map and the Oozie database
> 
>
> Key: OOZIE-3134
> URL: https://issues.apache.org/jira/browse/OOZIE-3134
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3134-001.patch, OOZIE-3134-002.patch
>
>
> Upon {{SLACalculatorMemory.addRegistration}}, Oozie puts an entry into an 
> in-memory concurrent hashmap ("slaMap"), and later [persists 
> it|https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L380]
>  in the Oozie database. 
> However, if there is a failure during the database operation, a 
> {{JPAExecutorException}} is thrown, and the entry [is not removed from the 
> SLA map| 
> https://github.com/apache/oozie/blob/50f4b5984832941f1341586e43fd832c293b3275/core/src/main/java/org/apache/oozie/sla/SLACalculatorMemory.java#L393].
> It may introduce inconsistency between the Oozie database and the SLA map.
> To prevent this, a rollback mechanism (with proper logging) should be 
> implemented.  It would also make sense to do more sanity/consistency check in 
> the Oozie server.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1 (candidate 3)

[Rohini Palaniswamy]

I think we should include OOZIE-2888 in the 4.3.1 release as well so that
sharelib creation works with newer versions of hadoop. Satish is on
vacation for next two weeks. Peter Cseh volunteered to take over as the
release manager and do the next RC.

Regards,
Rohini

On Wed, Jan 31, 2018 at 10:14 AM, Satish Saley <satishsa...@apache.org>
wrote:

> The vote for rc-3 has been extended till 02/02/2018 11:59 PST as some devs
> have asked for more time.
>
> On Wed, Jan 31, 2018 at 8:16 AM, Artem Ervits <artemerv...@gmail.com>
> wrote:
>
> > @Rohini, yes Oozie-2888 fixed the issue what I meant was to replace
> commons
> > lib with java.nio just like the Hadoop jira he had referenced. The
> > Oozie-2888 bumps commons lib to latest only, it doesn't use built-in
> libs.
> >
> > On Jan 31, 2018 9:41 AM, "Rohini Palaniswamy" <roh...@apache.org> wrote:
> >
> > > >for 4.3.1 it's probably a non-issue but for 5.0 perhaps we'd like to
> > > tackle
> > > it
> > > Attila mentioned that OOZIE-2888 already fixed the problem in 5.0.
> > >
> > >
> > > On Wed, Jan 31, 2018 at 6:34 AM, Artem Ervits <artemerv...@gmail.com>
> > > wrote:
> > >
> > > > @Attila, would you like me to open jira for:
> > > >
> > > > -  HADOOP-13444 replaced org.apache.commons.io.Charsets with
> > > > java.nio.charset.StandardCharsets so it shall not be a problem on
> > hadoop
> > > > 2.8+
> > > >
> > > > for 4.3.1 it's probably a non-issue but for 5.0 perhaps we'd like to
> > > tackle
> > > > it? I'm happy to dedicate some cycles to it.
> > > >
> > > > On Tue, Jan 30, 2018 at 11:52 PM, Attila Sasvari <
> > asasv...@cloudera.com>
> > > > wrote:
> > > >
> > > > > - I could not reproduce the NoClassDefFoundError Artem reported
> with
> > > > hadoop
> > > > > 2.6.0. I changed  bin/oozie-setup.sh so that it prints loaded
> classes
> > > > > (passed -verbose:class to the JVM). Interestingly,
> > > org/apache/commons/io/
> > > > > Charsets was not even loaded.
> > > > >
> > > > > However, when I tried again with hadoop.version *2.7.5* , it
> resulted
> > > in
> > > > > the error mentioned:
> > > > >
> > > > > java.lang.NoClassDefFoundError: org/apache/commons/io/Charsets
> > > > > at org.apache.hadoop.security.Credentials.(
> > > Credentials.java:222)
> > > > > at org.apache.hadoop.mapred.JobConf.(JobConf.java:334)
> > > > > at org.apache.oozie.service.HadoopAccessorService.createJobConf(
> > > > > HadoopAccessorService.java:349)
> > > > > at org.apache.oozie.tools.OozieSharelibCLI.run(
> > > > OozieSharelibCLI.java:171)
> > > > > at org.apache.oozie.tools.OozieSharelibCLI.main(
> > > > OozieSharelibCLI.java:67)
> > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > org.apache.commons.io.Charsets
> > > > >
> > > > > It looks like https://issues.apache.org/jira/browse/OOZIE-2396
> > > > >
> > > > > Some more info:
> > > > > - ./lib/commons-io-2.1.jar does not contain org/apache/commons/io/
> > > > > Charsets.
> > > > >
> > > > > - related hadoop 2.7.0 jira:
> > > > > https://issues.apache.org/jira/browse/HADOOP-11389
> > > > > - related hadoop code:
> > > > > https://github.com/apache/hadoop/blame/branch-2.7.0/
> > > > > hadoop-common-project/hadoop-common/src/main/java/org/
> > > > > apache/hadoop/security/Credentials.java#L36
> > > > > -  HADOOP-13444 replaced org.apache.commons.io.Charsets with
> > > > > java.nio.charset.StandardCharsets so it shall not be a problem on
> > > hadoop
> > > > > 2.8+
> > > > >
> > > > > Note: Oozie 5.0.0-beta1 includes OOZIE-2888 that add updated
> > commons-io
> > > > to
> > > > > 2.4. That is why it is not a problem there.
> > > > >
> > > > > On Wed, Jan 31, 2018 at 1:18 AM, Satish Saley <
> > satishsa...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > 1. However, json2.js is referenced by webapp/index.html and it
> > > > > > is not found (not a blocker in my opinion).
> > > > > >
> > > > > > -> I

Re: [VOTE] Release Oozie 4.3.1 (candidate 3)

[Rohini Palaniswamy]

> buildProject(
> > > > LifecycleModuleBuilder.java:84)
> > > > at
> > > > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> > buildProject(
> > > > LifecycleModuleBuilder.java:59)
> > > > at
> > > > org.apache.maven.lifecycle.internal.LifecycleStarter.
> > > singleThreadedBuild(
> > > > LifecycleStarter.java:183)
> > > > at
> > > > org.apache.maven.lifecycle.internal.LifecycleStarter.
> > > > execute(LifecycleStarter.java:161)
> > > > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> > > > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> > > > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> > > > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> > > > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:
> > > > 62)
> > > > at
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > at
> > > > org.codehaus.plexus.classworlds.launcher.Launcher.
> > > > launchEnhanced(Launcher.java:290)
> > > > at
> > > > org.codehaus.plexus.classworlds.launcher.Launcher.
> > > > launch(Launcher.java:230)
> > > > at
> > > > org.codehaus.plexus.classworlds.launcher.Launcher.
> > > > mainWithExitCode(Launcher.java:414)
> > > > at org.codehaus.plexus.classworlds.launcher.Launcher.
> > > > main(Launcher.java:357)
> > > >
> > > > Then after a successful build getting the following version string
> > > >
> > > > Oozie distro created, DATE[2018.01.30-16:22:57GMT]
> VC-REV[unavailable],
> > > > available at [/tmp/oozie-4.3.1/distro/target]
> > > >
> > > > we can now pull VC-REV with https://issues.apache.org/
> > > > jira/browse/OOZIE-3152
> > > >
> > > > finally saw the following
> > > >
> > > > [oozie@hadoop oozie-4.3.1]$ bin/oozie-setup.sh sharelib create -fs
> > > > hdfs://localhost:9000 -concurrency 8 -locallib
> > > oozie-sharelib-4.3.1.tar.gz
> > > >   setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.apache.hadoop.util.Shell).
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > for
> > > > more info.
> > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > SLF4J: Found binding in
> > > > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1-
> > > > distro/oozie-4.3.1/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/
> > > > impl/StaticLoggerBinder.class]
> > > > SLF4J: Found binding in
> > > > [jar:file:/opt/oozie/oozie-4.3.1/distro/target/oozie-4.3.1-
> > > > distro/oozie-4.3.1/lib/slf4j-simple-1.6.6.jar!/org/slf4j/
> > > > impl/StaticLoggerBinder.class]
> > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > > > explanation.
> > > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > > > the destination path for sharelib is:
> > > > /user/oozie/share/lib/lib_20180130164518
> > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > org/apache/commons/io/Charsets
> > > > at org.apache.hadoop.security.Credentials.(
> > Credentials.java:222)
> > > > at org.apache.hadoop.mapred.JobConf.(JobConf.java:334)
> > > > at
> > > > org.apache.oozie.service.HadoopAccessorService.createJobConf(
> > > > HadoopAccessorService.java:349)
> > > > at org.apache.oozie.tools.OozieSharelibCLI.run(
> > > OozieSharelibCLI.java:171)
> > > > at org.apache.oozie.tools.OozieSharelibCLI.main(
> > > OozieSharelibCLI.java:67)
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.commons.io.Charsets
> > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > > at java.lang.ClassLoader.loadClass(ClassLoade

Re: [VOTE] Release Oozie 4.3.1 (candidate 3)

[Rohini Palaniswamy]

> the server directory is missing altogether now.
 That is as expected. It is supposed to be only in 5.0

On Mon, Jan 29, 2018 at 12:47 PM, Artem Ervits 
wrote:

> the server directory is missing altogether now.
>
> On Mon, Jan 29, 2018 at 6:59 AM, Attila Sasvari 
> wrote:
>
> > +1 (non-binding)
> >
> > - Verified hash, signature files, rat report
> > - Built Oozie distro with hadoop.version set to 2.6.0 using the uber,
> > hadoop-2, spark-1 profiles
> > - Submitted example workflows/coordinators/bundles (except hive related
> > ones) on a hadoop 2.6.0 in pseudo distributed mode
> > - Verified Tomcat version was 6.0.53
> > - Clicked through the web UI, log retrieval worked, job DAG-s were
> > rendered, version was correct.
> > - Verified instrumentation, SLA pages; they are now HTML based as
> expected
> > (OOZIE-2533). However, json2.js is referenced by webapp/index.html and it
> > is not found (not a blocker in my opinion).
> > - Enabled SLA, re-submitted sla coordinator example, and retrieved SLA
> > details for the "one-op-wf" application, graph was displayed properly.
> >
> > I used JDK 1.8.0_121
> >
> > On Sun, Jan 28, 2018 at 7:09 AM, Satish Saley 
> > wrote:
> >
> > > Hi,
> > >
> > > I have created a build for Oozie 4.3.1, candidate 3.
> > >
> > > It includes all the changes decided earlier except OOZIE-3018,
> OOZIE-2771
> > > and
> > > OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService which we
> are
> > > not picking.
> > > OOZIE-2771 makes changes to file created in OOZIE-2666.
> > > OOZIE-3072 makes changes to files created in OOZIE-1770.
> > >
> > >
> > > Keys to verify the signature of the release artifact are available at
> > >   http://www.apache.org/dist/oozie/KEYS
> > >
> > > Please download, test, and try it out:
> > >
> > > http://people.apache.org/~satishsaley/oozie-4.3.1-rc3/
> > >
> > >
> > > The release, md5 signature, gpg signature, and rat report can be found
> at
> > > the above address.
> > >
> > > Vote closes on 01/31/2018 11:59 PM PT.
> > >
> >
> >
> >
> > --
> > --
> > Attila Sasvari
> > Software Engineer
> > 
> >
>

[jira] [Commented] (OOZIE-3171) Revert OOZIE-2771 commit on branch-4.3

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342194#comment-16342194
 ] 

Rohini Palaniswamy commented on OOZIE-3171:
---

+1. Please do remember to remove OOZIE-2771 from release-log and also remove 
fixversion 4.3.1 from OOZIE-2771

> Revert OOZIE-2771 commit on branch-4.3
> --
>
> Key: OOZIE-3171
> URL: https://issues.apache.org/jira/browse/OOZIE-3171
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 4.3.1
>
> Attachments: OOZIE-3171-1.patch
>
>
> File - 
> server/src/main/java/org/apache/oozie/server/SSLServerConnectorFactory.java 
> was picked by  
> [https://github.com/apache/oozie/commit/6e5dccf75c8898320850e8386f45a169dacb9ad7]as
>  OOZIE-2771 made changes to this file. But origin of this file is in 
> OOZIE-2666 
> [https://github.com/apache/oozie/commit/ca01c28362ede63bac17e32af42809b1c90c5ce9#diff-2b2f9e59c833f4dc95f84499e5baeb4d].
>  
> OOZIE-2771 has to be excluded form release 4.3.1
>  
> Thank you [~dbist13] for finding the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1 (candidate 2)

[Rohini Palaniswamy]

+1 (binding)

- Verified md5, signature, release log and rat report
- Ran the unit tests and they are good.
- Executed a simple workflow
- Verified that the tomcat is now 6.0.53

Regards,
Rohini

On Tue, Jan 23, 2018 at 4:27 PM, Satish Saley 
wrote:

> Hi,
>
> I have created a build for Oozie 4.3.1, candidate 2.
>
> It includes all the changes decided earlier except OOZIE-3018 and
> OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService which we are
> not picking. OOZIE-3072 makes changes to files created in OOZIE-1770.
>
> Keys to verify the signature of the release artifact are available at
>   http://www.apache.org/dist/oozie/KEYS
>
> Please download, test, and try it out:
>
> http://people.apache.org/~satishsaley/oozie-4.3.1-rc-2/
>
>
> The release, md5 signature, gpg signature, and rat report can be found at
> the above address.
>
> Vote closes on 01/26/2018 11:59 PM PT.
>

[jira] [Commented] (OOZIE-3167) Upgrade tomcat version on Oozie 4.3 branch

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336592#comment-16336592
 ] 

Rohini Palaniswamy commented on OOZIE-3167:
---

+1

> Upgrade tomcat version on Oozie 4.3 branch
> --
>
> Key: OOZIE-3167
> URL: https://issues.apache.org/jira/browse/OOZIE-3167
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 4.3.1
>
> Attachments: OOZIE-3167-1.patch
>
>
> Currently its 6.0.47. Upgrade to 6.0.53



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1 (candidate 1)

[Rohini Palaniswamy]

+1 (binding)

- Verified md5, signature, release log and rat report
- Ran the unit tests and they are all good. There was one known flaky test
which was fine on rerun.
- Executed a simple workflow

Regards,
Rohini

On Wed, Jan 17, 2018 at 5:57 AM, Attila Sasvari 
wrote:

> Many thanks Satish.
>
> +1 (non-binding)
>
> Testing done:
> - Verified md5 and signature, release-log.txt is up to date
> - Created distro with the mkdistro script using -Puber -Phadoop-2
> -Dhadoop.version-2.6.0
> - Executed example workflows (all except Hive related ones) / coordinators
> / bundles on a pseudo Hadoop 2.6.0 - each succeeded
> - Submitted Spark example workflow no only in local mode, but also on YARN
> (master=yarn, mode=client/cluster)
> - Tested FS action
> - Clicked through the Web UI. Job logs can be retrieved, DAG is displayed,
> version is correct
>
>
>
> On Wed, Jan 17, 2018 at 6:18 AM, satish saley 
> wrote:
>
> > Hi,
> > I have created a build for Oozie 4.3.1, candidate 1.
> > It includes all the changes decided earlier except OOZIE-3018 and
> > OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService which we are
> > not picking. OOZIE-3072 makes changes to files created in OOZIE-1770.
> > Keys to verify the signature of the release artifact are available at
> >   http://www.apache.org/dist/oozie/KEYS
> > Please download, test, and try it out:
> >   http://people.apache.org/~satishsaley/oozie-4.3.1-rc1
> > The release, md5 signature, gpg signature, and rat report can allbe found
> > at the above address.
> > Vote closes on 01/19/2018 11:59 PM PT.
> > -Satish Saley
>
>
>
>
> --
> --
> Attila Sasvari
> Software Engineer
> 
>

[jira] [Commented] (OOZIE-3161) Spark job failing because of servlet-api-2.5 jar

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327922#comment-16327922
 ] 

Rohini Palaniswamy commented on OOZIE-3161:
---

+1

> Spark job failing because of servlet-api-2.5 jar
> 
>
> Key: OOZIE-3161
> URL: https://issues.apache.org/jira/browse/OOZIE-3161
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Blocker
> Fix For: 4.3.1
>
> Attachments: OOZIE-3161-1.patch
>
>
> Spark Job fails with
> {code}
> 18/01/15 15:08:38 ERROR SparkContext: Error initializing SparkContext.
> java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s 
> signer information does not match signer information of other classes in the 
> same package
>   at java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
>   at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
>   at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
>   at 
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>   at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
>   at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>   at 
> org.spark-project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:136)
>   at 
> org.spark-project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:129)
>   at 
> org.spark-project.jetty.servlet.ServletContextHandler.(ServletContextHandler.java:98)
>   at 
> org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:126)
>   at 
> org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:113)
>   at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:78)
>   at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62)
>   at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>   at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:62)
>   at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:63)
>   at org.apache.spark.ui.SparkUI.(SparkUI.scala:76)
>   at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195)
>   at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146)
>   at org.apache.spark.SparkContext.(SparkContext.scala:473)
>   at 
> org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:59)
>   at org.apache.oozie.example.SparkFileCopy.main(SparkFileCopy.java:35)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
> {code}
> spark-streaming-flume-sink_2.10 is fetching jetty:servlet-api-2.5:jar. We 
> need to exclude this.
> Here is dependency tree for sharelib/spark
> {code}
> [INFO] +- org.apache.spark:spark-streaming-flume_2.10:jar:1.6.1:compile
> [INFO] | +- org.apache.spark:spark-streaming-flume-sink_2.10:jar:1.6.1:compile
> [INFO] | | +- (org.apache.flume:flume-ng-sdk:jar:1.6.0:compile - omitted for 
> duplicate)
> [INFO] | | +- (org.apache.flume:flume-ng-core:jar:1.6.0:compile - omitted for 
> duplicate)
> [INFO] | | +- (org.scala-lang:scala-library:jar:2.10.5:compile - omitted for 
> duplicate)
> [INFO] | | \- (org.spark-project.spark:unused:jar:1.0.0:compile - omitted for 
> duplicate)
> [INFO] | +- org.apache.flume:flume-ng-core:jar:1.6.0:compile
> [INFO] | | +- (org.apache.flume:flume-ng-sdk:jar:1.6.0:compile - omitted for 
> duplicate)
> [INFO] | | +- org.apache.flume:flume-ng-configuration:jar:1.6.0:compile
> [INFO] | | | +- (org.slf4j:slf

[jira] [Commented] (OOZIE-3113) Retry for ZK lock release

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304834#comment-16304834
 ] 

Rohini Palaniswamy commented on OOZIE-3113:
---

+1

> Retry for ZK lock release
> -
>
> Key: OOZIE-3113
> URL: https://issues.apache.org/jira/browse/OOZIE-3113
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3113-1.patch, OOZIE-3113-2.patch, 
> OOZIE-3113-3.patch, OOZIE-3113-4.patch
>
>
> ZK Lock might not get released if release call fails. Oozie should do 
> exponential retry for some time before giving up.
> {code}
> 2017-10-25 03:07:45,787  WARN ZKLocksService:523 [pool-12-thread-74] - 
> SERVER[localhost] USER[-] GROUP[-] TOKEN[-] APP[-] 
> JOB[123-123--oozie_saley-C] ACTION[123-123--oozie_saley-C@67] Could not 
> release lock: KeeperErrorCode = ConnectionLoss for 
> /oozie/localhost-oozie/locks/123-123--oozie_saley-C/1234__WRIT__000111
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode 
> = ConnectionLoss for 
> /oozie/localhost-oozie/locks/123-123--oozie_saley-C/1234__WRIT__000111
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at 
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at 
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:239)
> at 
> org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:234)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
> at 
> org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:230)
> at 
> org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:215)
> at 
> org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:42)
> at 
> org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:345)
> at 
> org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:123)
> at 
> org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:142)
> at 
> org.apache.oozie.service.ZKLocksService$ZKLockToken.release(ZKLocksService.java:206)
> at org.apache.oozie.command.XCommand.releaseLock(XCommand.java:233)
> at org.apache.oozie.command.XCommand.call(XCommand.java:305)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63739: OOZIE-3113 Retry for ZK lock release

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63739/#review194554
---


Ship it!




Please commit after you get +1 from András as well

- Rohini Palaniswamy


On Dec. 27, 2017, 10:13 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63739/
> ---
> 
> (Updated Dec. 27, 2017, 10:13 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3113
> https://issues.apache.org/jira/browse/OOZIE-3113
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3113 Retry for ZK lock release
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/ZKLocksService.java 2c71c001f 
>   core/src/main/resources/oozie-default.xml 8285df0a7 
>   core/src/test/java/org/apache/oozie/service/TestZKLocksService.java 
> d04f04e80 
> 
> 
> Diff: https://reviews.apache.org/r/63739/diff/4/
> 
> 
> Testing
> ---
> 
> Tested locally
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

Re: Review Request 63739: OOZIE-3113 Retry for ZK lock release

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63739/#review194551
---




core/src/test/java/org/apache/oozie/service/TestZKLocksService.java
Lines 603 (patched)
<https://reviews.apache.org/r/63739/#comment273382>

Why is this needed?


- Rohini Palaniswamy


On Dec. 23, 2017, 12:43 a.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63739/
> ---
> 
> (Updated Dec. 23, 2017, 12:43 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3113
> https://issues.apache.org/jira/browse/OOZIE-3113
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3113 Retry for ZK lock release
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/ZKLocksService.java 2c71c001f 
>   core/src/main/resources/oozie-default.xml 8285df0a7 
>   core/src/test/java/org/apache/oozie/service/TestZKLocksService.java 
> d04f04e80 
> 
> 
> Diff: https://reviews.apache.org/r/63739/diff/3/
> 
> 
> Testing
> ---
> 
> Tested locally
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300449#comment-16300449
 ] 

Rohini Palaniswamy commented on OOZIE-2601:
---

Thanks [~asasvari]

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 63739: OOZIE-3113 Retry for ZK lock release

[Rohini Palaniswamy]

> On Nov. 30, 2017, 5:33 p.m., András Piros wrote:
> > core/src/main/java/org/apache/oozie/service/ZKLocksService.java
> > Lines 239 (patched)
> > 
> >
> > Yep, blocking a thread for 30-ish minutes doesn't seem like a good 
> > idea. E.g. ZooKeeper outage lasts for only a few seconds under heavy load - 
> > it can happen that Oozie JVM ends up blocking lots of `Thread`s in 
> > `TIMED_WAITING` state.
> > 
> > Can you think of some, more elaborate solution? Maybe having an 
> > `ExecutorService` just to try to unlock, and `Callable` instances that 
> > would try unlocking. In that case, the number of `Thread`s blocked in 
> > `TIMED_WAITING` state would be:
> > * configurable
> > * limited to a manageable extent

There are couple of cases where ZK outages require Ops team restarting the 
quorum. From finding the issue to having it restarted, 30 mins would easily 
pass. Instead of exponential retry, the retry should be every 10 seconds. 
Retrying after 2,4,8 and 16 minutes is waste of time and blocking thread 
unnecessarily as you mentioned if ZK is back up. Otherwise it does not matter 
if retry continues for 30 mins or 1 hour (NN, RM retry time limit is also about 
the same). Everything in Oozie is going to be totally blocked if ZK is down as 
you can't acquire locks for any command. Having higher retry time avoids having 
to restart Oozie.


> On Nov. 30, 2017, 5:33 p.m., András Piros wrote:
> > core/src/test/java/org/apache/oozie/service/TestZKLocksService.java
> > Lines 579-598 (patched)
> > 
> >
> > Extract nested class w/ a catchy name.

Prefer to disagree on this one. I think it is a overkill to extract into one 
method and one class for 20 lines of test code. Unless the method is being 
reused or it is a big chunk of code, it does not make sense to extract into 
methods and that too in a test class.

More methods + more classes != Clean code. It kills readability as well when 
one has to unnecessarily jump to multiple places to finish reading logic of one 
method.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63739/#review192309
---


On Nov. 27, 2017, 7:22 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63739/
> ---
> 
> (Updated Nov. 27, 2017, 7:22 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3113
> https://issues.apache.org/jira/browse/OOZIE-3113
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3113 Retry for ZK lock release
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/ZKLocksService.java 2c71c001f 
>   core/src/main/resources/oozie-default.xml 8285df0a7 
>   core/src/test/java/org/apache/oozie/service/TestZKLocksService.java 
> d04f04e80 
> 
> 
> Diff: https://reviews.apache.org/r/63739/diff/2/
> 
> 
> Testing
> ---
> 
> Tested locally
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

Re: Review Request 63739: OOZIE-3113 Retry for ZK lock release

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63739/#review194265
---




core/src/main/java/org/apache/oozie/service/ZKLocksService.java
Lines 240 (patched)
<https://reviews.apache.org/r/63739/#comment272993>

You should log the retry number as well



core/src/test/java/org/apache/oozie/service/TestZKLocksService.java
Lines 584 (patched)
<https://reviews.apache.org/r/63739/#comment272994>

Try to keep runtime of every test under 30 sec - 1 min.



core/src/test/java/org/apache/oozie/service/TestZKLocksService.java
Lines 601 (patched)
<https://reviews.apache.org/r/63739/#comment272995>

You should have a catch block and fail the test if there is any exception


- Rohini Palaniswamy


On Nov. 27, 2017, 7:22 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63739/
> ---
> 
> (Updated Nov. 27, 2017, 7:22 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3113
> https://issues.apache.org/jira/browse/OOZIE-3113
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3113 Retry for ZK lock release
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/service/ZKLocksService.java 2c71c001f 
>   core/src/main/resources/oozie-default.xml 8285df0a7 
>   core/src/test/java/org/apache/oozie/service/TestZKLocksService.java 
> d04f04e80 
> 
> 
> Diff: https://reviews.apache.org/r/63739/diff/2/
> 
> 
> Testing
> ---
> 
> Tested locally
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295801#comment-16295801
 ] 

Rohini Palaniswamy commented on OOZIE-2601:
---

bq. We should take care of the order the user specifies their dependencies
  I am not sure you can do much there. Yarn just does $PWD/* to include all the 
localized files. You can either add the local file directories to the end or to 
the beginning. I would suggest end so that any user jars in $PWD take 
precedence.

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295798#comment-16295798
 ] 

Rohini Palaniswamy commented on OOZIE-2601:
---

Separate jira as it is an enhancement

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295475#comment-16295475
 ] 

Rohini Palaniswamy edited comment on OOZIE-2601 at 12/18/17 7:00 PM:
-

bq. to set the classpath via JavaActionExecutor prior to application submission 
in a different way (now we are adding them via "yarn.application.classpath") 
and the improvement of ShareLibService
  I am not suggesting changing setting of yarn.application.classpath

bq. Right now, ShareLibService checks the existence of files either on HDFS or 
local filesystem (symlinks followed too). It could help users (i.e. warning 
them if specified dependencies are not present).
   I am actually asking to skip this step. What it is doing is actually 
checking the files or resolving symlinks in the Oozie server, while the actual 
files are in the task nodes. Instead of letting it go through regular code 
path, I would just suggest keep a separate datastructure for local files and 
adding a new getShareLibJarsLocal(String actiontype) method, which will return 
the local files path as is (if it is a file with extension) or return both the 
path and /* appended to the path (/* works only for jars. The original 
directory path is required if there are config files under the directory). That 
can be then added directly to yarn.application.classpath setting.


was (Author: rohini):
bq. to set the classpath via JavaActionExecutor prior to application submission 
in a different way (now we are adding them via "yarn.application.classpath") 
and the improvement of ShareLibService
  I am not suggesting changing setting of yarn.application.classpath

bq. Right now, ShareLibService checks the existence of files either on HDFS or 
local filesystem (symlinks followed too). It could help users (i.e. warning 
them if specified dependencies are not present).
   I am actually asking to skip this step. What it is doing is actually 
checking the files or resolving symlinks in the Oozie server, while the actual 
files are in the task nodes. Instead of letting it go through regular code 
path, I would just suggest adding a new getShareLibJarsLocal(String actiontype) 
method, which will return the local files path as is (if it is a file with 
extension) or return both the path and /* appended to the path (/* works only 
for jars. The original directory path is required if there are config files 
under the directory). That can be then added directly to 
yarn.application.classpath setting.

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295475#comment-16295475
 ] 

Rohini Palaniswamy commented on OOZIE-2601:
---

bq. to set the classpath via JavaActionExecutor prior to application submission 
in a different way (now we are adding them via "yarn.application.classpath") 
and the improvement of ShareLibService
  I am not suggesting changing setting of yarn.application.classpath

bq. Right now, ShareLibService checks the existence of files either on HDFS or 
local filesystem (symlinks followed too). It could help users (i.e. warning 
them if specified dependencies are not present).
   I am actually asking to skip this step. What it is doing is actually 
checking the files or resolving symlinks in the Oozie server, while the actual 
files are in the task nodes. Instead of letting it go through regular code 
path, I would just suggest adding a new getShareLibJarsLocal(String actiontype) 
method, which will return the local files path as is (if it is a file with 
extension) or return both the path and /* appended to the path (/* works only 
for jars. The original directory path is required if there are config files 
under the directory). That can be then added directly to 
yarn.application.classpath setting.

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291532#comment-16291532
 ] 

Rohini Palaniswamy edited comment on OOZIE-2601 at 12/14/17 8:13 PM:
-

I think this can be simplified a lot by just adding /usr/lib/hive/lib/* 
directly to yarn classpath setting. Listing all the jars in that directory from 
the Oozie node and adding that to classpath seems wasteful.


was (Author: rohini):
I think this can be simplified a lot by just adding file:///usr/lib/hive/lib/* 
directly to yarn classpath setting. Listing all the jars in that directory from 
the Oozie node and adding that to classpath seems wasteful.

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2601) Ability to use local paths for the sharelib

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291532#comment-16291532
 ] 

Rohini Palaniswamy commented on OOZIE-2601:
---

I think this can be simplified a lot by just adding file:///usr/lib/hive/lib/* 
directly to yarn classpath setting. Listing all the jars in that directory from 
the Oozie node and adding that to classpath seems wasteful.

> Ability to use local paths for the sharelib
> ---
>
> Key: OOZIE-2601
> URL: https://issues.apache.org/jira/browse/OOZIE-2601
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: oya
>Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Fix For: oya, 5.0.0b1
>
> Attachments: OOZIE-2601-01.patch, OOZIE-2601-03.patch, 
> OOZIE-2601-05.patch, OOZIE-2601-06.patch, OOZIE-2601-07.patch, 
> OOZIE-2601-09.patch, OOZIE-2601-10.patch, OOZIE-2601-11.patch
>
>
> With OOZIE-2590, as part of OOZIE-1770 Oozie on Yarn work, Oozie now has full 
> control over the classpath given to the Launcher AM.  In a cluster where all 
> nodes have everything installed locally (in the same paths), it should be 
> possible to have the Launcher AM reference the local jars instead of having 
> to localize them from HDFS.
> For example, if you have Hive installed on all nodes at {{/usr/lib/hive/}} 
> and all Hive jars under {{/usr/lib/hive/lib/}}, we could have the Launcher AM 
> add {{/usr/lib/hive/lib}} to its classpath.  This saves on the overhead of 
> localizing the same jars from the hive sharelib in HDFS.  
> I think the best way to implement this is to augment the [Sharelib Mapping 
> File|https://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Share_Lib] 
> feature to accept {{file:///}} paths.
> If we had this also work with the "oozie" sharelib and the Oozie jars in the 
> individual sharelibs (e.g. have the Mapping file take comma-separated 
> dirs/jars), then in a cluster with everything installed on all of the nodes, 
> you wouldn't need to bother with the sharelib at all!



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3127) Remove redundant check for user

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274791#comment-16274791
 ] 

Rohini Palaniswamy commented on OOZIE-3127:
---

+1

> Remove redundant check for user
> ---
>
> Key: OOZIE-3127
> URL: https://issues.apache.org/jira/browse/OOZIE-3127
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3127-1.patch
>
>
> [validateJobConfiguration(conf);|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L95]
>  is redundant because we are checking it anyways [inside 
> here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java#L100]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3027) Convert Graph to JAXB objects

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269307#comment-16269307
 ] 

Rohini Palaniswamy commented on OOZIE-3027:
---

[~andras.piros],
Please update with details before closing jiras. This jira has no 
description and no update on why it was closed as Fixed.

> Convert Graph to JAXB objects
> -
>
> Key: OOZIE-3027
> URL: https://issues.apache.org/jira/browse/OOZIE-3027
> Project: Oozie
>  Issue Type: Sub-task
>  Components: client
>Reporter: Andras Piros
>Assignee: Daniel Becker
>   Original Estimate: 32h
>  Time Spent: 16h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3126) Add option to allow list of users to access system config

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255958#comment-16255958
 ] 

Rohini Palaniswamy commented on OOZIE-3126:
---

+1

> Add option to allow list of users to access system config
> -
>
> Key: OOZIE-3126
> URL: https://issues.apache.org/jira/browse/OOZIE-3126
> Project: Oozie
>  Issue Type: Improvement
>    Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3126-1.patch, OOZIE-3126-2.patch, 
> OOZIE-3126-3.patch, OOZIE-3126-4.patch, OOZIE-3126-5.patch
>
>
>Currently we allow access to system config for all users. Though we mask 
> any sensitive information it is unnecessary to show it to all users. Can 
> restrict the access to admins, but services like Hue use the server side 
> config values to display UI accordingly (For eg: Server Timezone, whether SLA 
> is configured, what kind of credentials are supported, etc). So it would be 
> good to support restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3126) Add option to allow list of users to access system config

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254049#comment-16254049
 ] 

Rohini Palaniswamy commented on OOZIE-3126:
---

Can we don conf.unset instead if setSystemInfoAuthUsers is false? 
useAdminUsersFile should be just in a if block and not else if.

> Add option to allow list of users to access system config
> -
>
> Key: OOZIE-3126
> URL: https://issues.apache.org/jira/browse/OOZIE-3126
> Project: Oozie
>  Issue Type: Improvement
>    Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3126-1.patch, OOZIE-3126-2.patch
>
>
>Currently we allow access to system config for all users. Though we mask 
> any sensitive information it is unnecessary to show it to all users. Can 
> restrict the access to admins, but services like Hue use the server side 
> config values to display UI accordingly (For eg: Server Timezone, whether SLA 
> is configured, what kind of credentials are supported, etc). So it would be 
> good to support restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3126) Add option to allow list of users to access system config

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252741#comment-16252741
 ] 

Rohini Palaniswamy commented on OOZIE-3126:
---

+1

> Add option to allow list of users to access system config
> -
>
> Key: OOZIE-3126
> URL: https://issues.apache.org/jira/browse/OOZIE-3126
> Project: Oozie
>  Issue Type: Improvement
>    Reporter: Rohini Palaniswamy
>Assignee: Satish Subhashrao Saley
> Fix For: 4.3.1
>
> Attachments: OOZIE-3126-1.patch
>
>
>Currently we allow access to system config for all users. Though we mask 
> any sensitive information it is unnecessary to show it to all users. Can 
> restrict the access to admins, but services like Hue use the server side 
> config values to display UI accordingly (For eg: Server Timezone, whether SLA 
> is configured, what kind of credentials are supported, etc). So it would be 
> good to support restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Oozie 4.3.1. release

[Rohini Palaniswamy]

Peter,
  OOZIE-3004 Forked action retry info is not working in your list is a
bug fix for the feature OOZIE-2691 which is new in 5.0. So that cannot be
included. But others are good to be included.

Regards,
Rohini

On Tue, Nov 14, 2017 at 2:58 PM, Robert Kanter <rkan...@cloudera.com> wrote:

> Sounds good to me.
>
> On Tue, Nov 14, 2017 at 10:09 AM, Peter Bacsko <pbac...@cloudera.com>
> wrote:
>
> > Good idea.
> >
> > If I'm not mistaken, there's also a pending review/patch submitted by
> > someone (apologies, but I forgot how it was) that cannot be applied to
> the
> > master anymore, at least not easily. It's worth checking that too
> (perhaps
> > there are more than one patch).
> >
> > Peter
> >
> > On Tue, Nov 14, 2017 at 5:07 PM, Peter Cseh <gezap...@cloudera.com>
> wrote:
> >
> > > Hey!
> > >
> > > I agree, users wanting stability would appreciate a more stable 4.3.1
> or
> > > even 4.4.0 to be released with improvements.
> > > I think we could include the following issues as well:
> > > OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
> > > OOZIE-3077 SLA documentation fixes (satishsaley)
> > > OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
> > > (dbist13 via rkanter)
> > > OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)
> > > OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
> > > Hentschel via rkanter)
> > > OOZIE-3004 Forked action retry info is not working
> > > Most of these are minor improvements or user-facing fixes which would
> be
> > > nice to get through the door.
> > >
> > > BRs,
> > > gp
> > >
> > > On Tue, Nov 14, 2017 at 12:56 AM, Rohini Palaniswamy <
> > > rohini.adi...@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >I think this would be a good time to do a patch release on 4.3
> > branch.
> > > > The 5.0 release is going to be a major change with the move of
> launcher
> > > > from Mapreduce AM to Oozie AM and might take time to stabilize. So it
> > > would
> > > > be good to do one final patch release on 4.3 branch pulling in some
> > > > important fixes and Spark action related fixes (Spark 2.x support)
> that
> > > > have gone in master so far.  Satish has volunteered to be the Release
> > > > Manager.
> > > >
> > > > Below is a initial list of jiras that I identified.
> > > >
> > > >
> > > > OOZIE-3095 Job submit command is not logged in audit log (puru)
> > > > OOZIE-3031 Coord job with only unresolved dependencies doesn't
> timeout
> > > > (puru via satishsaley)
> > > > OOZIE-3079 Filtering coordinators returns bundle id as null
> > (satishsaley)
> > > > OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> > > > OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
> > > >  should overwrite default values set in Hadoop's configuration files
> > > > (gezapeti)
> > > > OOZIE-2885 Running Spark actions should not need Hive on the
> classpath
> > > > (satishsaley)
> > > > OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
> > > > (satishsaley)
> > > > OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation
> > > tokens
> > > > (andras.piros via gezapeti)
> > > > OOZIE-3028 Oozie Pig Action fails with no python dependencies
> (dbist13
> > > via
> > > > rohini)
> > > > OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
> > > > SchemaCheckerService (andras.piros via gezapeti)
> > > > OOZIE-2984 Parse spark-defaults.conf values with spaces without
> needing
> > > the
> > > > quotes (andras.piros via gezapeti)
> > > > OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
> > > > OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server
> > > start
> > > > (puru via satishsaley)
> > > > OOZIE-2862 Coord change command doesn't change job to running if job
> > was
> > > > killed without creating any actions (puru)
> > > > OOZIE-2807 Oozie gets RM delegation token even for checking job
> status
> > > > (satishsaley)
> > > > OOZIE-2811 Add support for filtering out properties from
> > > > SparkConfiguratio

[jira] [Created] (OOZIE-3126) Add option to allow list of users to access system config

[Rohini Palaniswamy (JIRA)]

Rohini Palaniswamy created OOZIE-3126:
-

 Summary: Add option to allow list of users to access system config
 Key: OOZIE-3126
 URL: https://issues.apache.org/jira/browse/OOZIE-3126
 Project: Oozie
  Issue Type: Improvement
Reporter: Rohini Palaniswamy
Assignee: Satish Subhashrao Saley


   Currently we allow access to system config for all users. Though we mask any 
sensitive information it is unnecessary to show it to all users. Can restrict 
the access to admins, but services like Hue use the server side config values 
to display UI accordingly (For eg: Server Timezone, whether SLA is configured, 
what kind of credentials are supported, etc). So it would be good to support 
restricting it to admins and a list of users.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3122) Oozie spark action unable to handle some options in spark-defaults.conf

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251571#comment-16251571
 ] 

Rohini Palaniswamy commented on OOZIE-3122:
---

Just started a discussion on dev mailing list on doing a Apache 4.3.1 release. 
Hopefully we will have a release in another month.

> Oozie spark action unable to handle some options in spark-defaults.conf
> ---
>
> Key: OOZIE-3122
> URL: https://issues.apache.org/jira/browse/OOZIE-3122
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.3.0
> Environment: Operating on EMR cluster (5.4.0) which includes Oozie 
> 4.3.0
>Reporter: Simon Whitelaw
>  Labels: usability
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> When using the setting 
> oozie.service.SparkConfigurationService.spark.configurations *=etc/spark/conf 
> to specify a spark-defaults.conf file for spark to use, a few options are not 
> handled properly by the oozie spark action and cause the job to fail. These 
> include the following:
> * spark.driver.extraClassPath - causes a blank --conf tag to be sent to the 
> spark submit and the spark submit will fail
> * spark.executor.extraClassPath - causes a blank --conf tag to be sent to the 
> spark submit and the spark submit will fail
> * spark.executor.extraJavaOptions with multiple options set (e.g. 
> spark.executor.extraJavaOptions  -verbose:gc -XX:+PrintGCDetails 
> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 
> -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p') - causes 
> all options except for the first to be passed as singletons rather than 
> extraJavaOptions
> * spark.driver.extraJavaOptions with multiple option set - causes all options 
> except for the first to be passed as singletons rather than extraJavaOptions
> * Within the spark.driver.extraJavaOptions and 
> spark.executor.extraJavaOptions, the option -XX:OnOutOfMemoryError='kill -9 
> %p' is not parsed correctly even from the spark-opts tag and requires 
> additional double quotations within the single quote marks: '"kill -9 %p"'
> These issues were noticed running the oozie spark action on an amazon EMR 
> cluster. Workaround involves sending the options as part of the spark-opts 
> tag, but is not ideal because in EMR, the spark-defaults.conf file is 
> autogenerated for the specifications of the cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Oozie 4.3.1. release

[Rohini Palaniswamy]

Hi folks,
   I think this would be a good time to do a patch release on 4.3 branch.
The 5.0 release is going to be a major change with the move of launcher
from Mapreduce AM to Oozie AM and might take time to stabilize. So it would
be good to do one final patch release on 4.3 branch pulling in some
important fixes and Spark action related fixes (Spark 2.x support) that
have gone in master so far.  Satish has volunteered to be the Release
Manager.

Below is a initial list of jiras that I identified.


OOZIE-3095 Job submit command is not logged in audit log (puru)
OOZIE-3031 Coord job with only unresolved dependencies doesn't timeout
(puru via satishsaley)
OOZIE-3079 Filtering coordinators returns bundle id as null (satishsaley)
OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
 should overwrite default values set in Hadoop's configuration files
(gezapeti)
OOZIE-2885 Running Spark actions should not need Hive on the classpath
(satishsaley)
OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
(satishsaley)
OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation tokens
(andras.piros via gezapeti)
OOZIE-3028 Oozie Pig Action fails with no python dependencies (dbist13 via
rohini)
OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
SchemaCheckerService (andras.piros via gezapeti)
OOZIE-2984 Parse spark-defaults.conf values with spaces without needing the
quotes (andras.piros via gezapeti)
OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server start
(puru via satishsaley)
OOZIE-2862 Coord change command doesn't change job to running if job was
killed without creating any actions (puru)
OOZIE-2807 Oozie gets RM delegation token even for checking job status
(satishsaley)
OOZIE-2811 Add support for filtering out properties from
SparkConfigurationService (gezapeti via rkanter)
OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate sharelibs
(gezapeti via rkanter)
OOZIE-2803 Mask passwords when printing out configs/args in MapReduceMain
and SparkMain (pbacsko via rkanter)
OOZIE-2799 Setting log location for spark sql on hive (satishsaley)
OOZIE-2792 Hive2 action is not parsing Spark application ID from log file
properly when Hive is on Spark (zhengxb2005 via rkanter)
OOZIE-2786 Pass Oozie workflow ID and settings to Spark application
configuration (zhengxb2005 via rkanter)
OOZIE-2790 log4j configuration is not passed to spark executors
(satishsaley)
OOZIE-2787 Oozie distributes application jar twice making the spark job
fail (satishsaley)
OOZIE-2777 Config-default.xml longer than 64k results in
java.io.UTFDataFormatException (gezapeti via harsh)
OOZIE-2771 Allow retrieving keystore and truststore passwords from Hadoop
Credential Provider (asasvari via abhishekbafna)
OOZIE-2748 NPE in LauncherMapper.printArgs() (pbacsko via rkanter)
OOZIE-2654 Zookeeper dependent services should not depend on
Connectionstate to be valid before cleaning up (venkatnrangan via
abhishekbafna)
OOZIE-2690 OOZIE NPE while executing kill() (abhishekbafna via
jaydeepvishwakarma)


Please reply to this mail thread if you find other jiras that you think are
important and should be part of 4.3.1.


Regards,
Rohini

[jira] [Commented] (OOZIE-3101) Oozie should add error message to the response body

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242497#comment-16242497
 ] 

Rohini Palaniswamy commented on OOZIE-3101:
---

+1

> Oozie should add error message to the response body
> ---
>
> Key: OOZIE-3101
> URL: https://issues.apache.org/jira/browse/OOZIE-3101
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3101-1.patch, OOZIE-3101-2.patch, 
> OOZIE-3101-3.patch, OOZIE-3101-4.patch, OOZIE-3101-5.patch
>
>
> For web service APIs, currently Oozie adds error message in a custom response 
> header which is parsed out by the OozieClient. Oozie should also put error 
> message in the response body so that clients like curl can display the error 
> message to users without having to turn on verbose output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3101) Oozie should add error message to the response body

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241224#comment-16241224
 ] 

Rohini Palaniswamy commented on OOZIE-3101:
---

+1

> Oozie should add error message to the response body
> ---
>
> Key: OOZIE-3101
> URL: https://issues.apache.org/jira/browse/OOZIE-3101
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3101-1.patch, OOZIE-3101-2.patch, 
> OOZIE-3101-3.patch, OOZIE-3101-4.patch
>
>
> For web service APIs, currently Oozie adds error message in a custom response 
> header which is parsed out by the OozieClient. Oozie should also put error 
> message in the response body so that clients like curl can display the error 
> message to users without having to turn on verbose output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3101) Oozie should add error message to the response body

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240990#comment-16240990
 ] 

Rohini Palaniswamy commented on OOZIE-3101:
---

1) Do we need all three escaping - xml, javascript and html to get rid of the 
findbugs XSS vulnerability warning? OOZIE-3109 only does html escaping. If it 
is better to do all three, then we should do that in OOZIE-3109 as well.
2) The new constants should be in JsonTags instead of RestConstants and camel 
casing should be used. RestConstants is only used for query params.

> Oozie should add error message to the response body
> ---
>
> Key: OOZIE-3101
> URL: https://issues.apache.org/jira/browse/OOZIE-3101
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3101-1.patch, OOZIE-3101-2.patch, 
> OOZIE-3101-3.patch
>
>
> For web service APIs, currently Oozie adds error message in a custom response 
> header which is parsed out by the OozieClient. Oozie should also put error 
> message in the response body so that clients like curl can display the error 
> message to users without having to turn on verbose output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3101) Oozie should add error message to the response body

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238094#comment-16238094
 ] 

Rohini Palaniswamy commented on OOZIE-3101:
---

Looks good. Couple of minor comments

1) oozie-error-code and oozie-error-message to httpcode and error-message
2) ErrorPageErrorHandler setup should be part of EmbeddedServletContainer by 
default. Should not be in test.
3) Can use IOUtils api to read connection error stream in testcase

> Oozie should add error message to the response body
> ---
>
> Key: OOZIE-3101
> URL: https://issues.apache.org/jira/browse/OOZIE-3101
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Major
> Attachments: OOZIE-3101-1.patch, OOZIE-3101-2.patch
>
>
> For web service APIs, currently Oozie adds error message in a custom response 
> header which is parsed out by the OozieClient. Oozie should also put error 
> message in the response body so that clients like curl can display the error 
> message to users without having to turn on verbose output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3101) Oozie should add error message to the response body

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225585#comment-16225585
 ] 

Rohini Palaniswamy commented on OOZIE-3101:
---

https://docs.oracle.com/javaee/6/api/javax/servlet/http/HttpServletResponse.html#sendError(int,
 java.lang.String) always sends it as a text/html. We should send a Json message

> Oozie should add error message to the response body
> ---
>
> Key: OOZIE-3101
> URL: https://issues.apache.org/jira/browse/OOZIE-3101
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3101-1.patch
>
>
> For web service APIs, currently Oozie adds error message in a custom response 
> header which is parsed out by the OozieClient. Oozie should also put error 
> message in the response body so that clients like curl can display the error 
> message to users without having to turn on verbose output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3101) Oozie should add error message to the response body

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated OOZIE-3101:
--
Description: For web service APIs, currently Oozie adds error message in a 
custom response header which is parsed out by the OozieClient. Oozie should 
also put error message in the response body so that clients like curl can 
display the error message to users without having to turn on verbose output  
(was: For web service APIs, currently Oozie adds error message in the response 
header. Oozie should also put error message in the response body.)

> Oozie should add error message to the response body
> ---
>
> Key: OOZIE-3101
> URL: https://issues.apache.org/jira/browse/OOZIE-3101
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3101-1.patch
>
>
> For web service APIs, currently Oozie adds error message in a custom response 
> header which is parsed out by the OozieClient. Oozie should also put error 
> message in the response body so that clients like curl can display the error 
> message to users without having to turn on verbose output



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3095) Job submit command is not logged in audit log

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219112#comment-16219112
 ] 

Rohini Palaniswamy commented on OOZIE-3095:
---

+1

> Job submit command is not logged in audit log
> -
>
> Key: OOZIE-3095
> URL: https://issues.apache.org/jira/browse/OOZIE-3095
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-3095-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2897) LauncherAM should support ACLs

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216072#comment-16216072
 ] 

Rohini Palaniswamy commented on OOZIE-2897:
---

bq. but eventually I would hope we could drop those
  That never happens in practice as it would involve everyone fixing and 
retesting their workflow.xmls. Especially in this case, it will be a total 
waste of time for users as it is not going to make any difference. We had the 
same situation in mapreduce->tez transition. We support all mapreduce settings 
(even though there are equivalent tez settings) and have no plans of ever 
dropping support for the mapreduce settings.

bq. Doesn't it seem weird to have users set 
oozie.launcher.mapreduce. instead of oozie.?
  Even if we add new Oozie AM specific properties later, for consistency we 
should name it as oozie.launcher. as it is still for the 
launcher AM.

> LauncherAM should support ACLs
> --
>
> Key: OOZIE-2897
> URL: https://issues.apache.org/jira/browse/OOZIE-2897
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Blocker
>
> In MapReduce, you can define ACL-related properties:
> {noformat}
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modify-job
> {noformat}
> {{acl-view-job}} defines a list of users/groups who can retrieve the job 
> statistics. {{acl-modify-job}} defines a list of users/groups who can kill a 
> job or adjust the priority of it.
> Docs: 
> https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization
> In YARN, we can provide backward compatibility for these properties. Example 
> code:
> {code}
> Map<ApplicationAccessType, String> acls = new HashMap<ApplicationAccessType, 
> String>();
> acls.put(ApplicationAccessType.MODIFY_APP, "*");
> acls.put(ApplicationAccessType.VIEW_APP, "*");
> amContainer.setApplicationACLs(acls);
> {code}
> This has to be done before application submission. We have to do what 
> YARNRunner.java does: 
> https://github.com/apache/hadoop/blob/3721cfe1fbd98c5b6aa46aefdfcf62276c28c4a4/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L502-L507
>  
> We should also ensure the compatibility with the properties of:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3095) Job submit command is not logged in audit log

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216039#comment-16216039
 ] 

Rohini Palaniswamy commented on OOZIE-3095:
---

What does the audit message look like in this case (no job id) after the fix? 
Can you paste that as well?

> Job submit command is not logged in audit log
> -
>
> Key: OOZIE-3095
> URL: https://issues.apache.org/jira/browse/OOZIE-3095
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-3095-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3077) SLA documentation fixes

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197771#comment-16197771
 ] 

Rohini Palaniswamy commented on OOZIE-3077:
---

+1

> SLA documentation fixes
> ---
>
> Key: OOZIE-3077
> URL: https://issues.apache.org/jira/browse/OOZIE-3077
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-3077-1.patch, OOZIE-3077-2.patch
>
>
> Following properties can be specified in job.xml -> Following properties can 
> be specified in job.xml, job.properties
> Default is 24 hours -> Default is 48 hours
> oozie.sla.disable.alerts -> oozie.sla.enable.alerts for property name in the 
> example configuration having 2009-01-01T01:00Z::2009-05-31T23:59Z as value



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3077) SLA documentation fixes

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197677#comment-16197677
 ] 

Rohini Palaniswamy commented on OOZIE-3077:
---

Couple of minor rewordings.
  1) Following are ways to enable/disable SLA alerts. -> Following are ways to 
enable/disable SLA alerts for coordinator actions.
  2) 1. Following properties can be specified in bundle xml as properties for 
coordinator. -> 1. Specify in Bundle XML during submission \n Following 
properties can be specified in bundle xml as properties for coordinator.
  3) Change using job command -> Specify during coordinator job submission or 
update

> SLA documentation fixes
> ---
>
> Key: OOZIE-3077
> URL: https://issues.apache.org/jira/browse/OOZIE-3077
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
>Priority: Minor
> Attachments: OOZIE-3077-1.patch
>
>
> Following properties can be specified in job.xml -> Following properties can 
> be specified in job.xml, job.properties
> Default is 24 hours -> Default is 48 hours
> oozie.sla.disable.alerts -> oozie.sla.enable.alerts for property name in the 
> example configuration having 2009-01-01T01:00Z::2009-05-31T23:59Z as value



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3079) Filtering coordinators returns bundle id as null

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197281#comment-16197281
 ] 

Rohini Palaniswamy commented on OOZIE-3079:
---

+1

> Filtering coordinators returns bundle id as null
> 
>
> Key: OOZIE-3079
> URL: https://issues.apache.org/jira/browse/OOZIE-3079
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3079-1.patch, OOZIE-3079-2.patch, 
> OOZIE-3079-3.patch
>
>
> If we filter coordinator jobs, the json reply shows (parent) bundle id as 
> null even though coordinator was started from a bundle.
> Example query - 
> http://localhost:11000/oozie/v1/jobs?filter=user=saley=coordinator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (OOZIE-1723) Setting 'use.new.api=true' gives compatibility error even with using new 'mapreduce.*' hadoop api

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy resolved OOZIE-1723.
---
Resolution: Not A Problem

https://cwiki.apache.org/confluence/display/OOZIE/Map+Reduce+Cookbook

To run MapReduce jobs written using the new API in Oozie, below are the changes 
you need to make in workflow.xml.

#   change mapred.mapper.class to mapreduce.map.class
#   change mapred.reducer.class to mapreduce.reduce.class
#add mapred.output.key.class
#add mapred.output.value.class
#and, include the following property into MR action configuration
{code}

mapred.reducer.new-api
true
  
  
mapred.mapper.new-api
true

{code}

> Setting 'use.new.api=true' gives compatibility error even with using new 
> 'mapreduce.*' hadoop api
> -
>
> Key: OOZIE-1723
> URL: https://issues.apache.org/jira/browse/OOZIE-1723
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: trunk, 4.0.0
>Reporter: Mona Chitnis
>  Labels: api
>
> Error encountered - JA009: mapred.mapper.class is incompatible with new map 
> API mode.
> complete stack trace:
> {code}
> org.apache.oozie.action.ActionExecutorException: JA009: mapred.mapper.class 
> is incompatible with new map API mode.
>   at 
> org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:412)
>   at 
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:396)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:871)
>   at 
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1025)
>   at 
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:217)
>   at 
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:62)
>   at org.apache.oozie.command.XCommand.call(XCommand.java:280)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:326)
>   at 
> org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:255)
>   at 
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException: mapred.mapper.class is incompatible with new 
> map API mode.
>   at org.apache.hadoop.mapreduce.Job.ensureNotSet(Job.java:1122)
>   at org.apache.hadoop.mapreduce.Job.setUseNewAPI(Job.java:1149)
>   at org.apache.hadoop.mapreduce.Job.submit(Job.java:1211)
>   at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3079) Filtering coordinators returns bundle id as null

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193402#comment-16193402
 ] 

Rohini Palaniswamy commented on OOZIE-3079:
---

+1

> Filtering coordinators returns bundle id as null
> 
>
> Key: OOZIE-3079
> URL: https://issues.apache.org/jira/browse/OOZIE-3079
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3079-1.patch, OOZIE-3079-2.patch
>
>
> If we filter coordinator jobs, the json reply shows (parent) bundle id as 
> null even though coordinator was started from a bundle.
> Example query - 
> http://localhost:11000/oozie/v1/jobs?filter=user=saley=coordinator



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/#review187014
---




core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
Lines 149-151 (original)
<https://reviews.apache.org/r/62470/#comment263966>

I missed this earlier in the review. If we have this, then we should set 
the env here (will require method signature change) and remove the earlier code 
for setting SPARK_HOME that was done in setupLauncherConf as well.


- Rohini Palaniswamy


On Oct. 3, 2017, 9:52 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62470/
> ---
> 
> (Updated Oct. 3, 2017, 9:52 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3062
> https://issues.apache.org/jira/browse/OOZIE-3062
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3062 Set HADOOP_CONF_DIR for spark action
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec8 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
>  d97f1f06 
> 
> 
> Diff: https://reviews.apache.org/r/62470/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

Re: Review Request 62470: OOZIE-3062 Set HADOOP_CONF_DIR for spark action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62470/#review186997
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 331 (original), 331 (patched)
<https://reviews.apache.org/r/62470/#comment263948>

Would like to disagree with András Piros comment on creating yet another 
method in JavaActionExecutor. It does not make it clean. We should not be 
renaming method to include every step done inside the method. Adding default 
child env is still part of setting up the launcher config and it does not 
require special mention in the method name and a new method. I would suggest 
reverting back to old method name and overriding it as in the initial patch.



sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
Lines 136 (patched)
<https://reviews.apache.org/r/62470/#comment263949>

You can get rid of this and just use ArrayUtils.contains method from 
commons-lang


- Rohini Palaniswamy


On Oct. 3, 2017, 7:35 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62470/
> ---
> 
> (Updated Oct. 3, 2017, 7:35 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3062
> https://issues.apache.org/jira/browse/OOZIE-3062
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3062 Set HADOOP_CONF_DIR for spark action
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> be056038 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 80d64ec8 
>   
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java
>  d97f1f06 
> 
> 
> Diff: https://reviews.apache.org/r/62470/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189035#comment-16189035
 ] 

Rohini Palaniswamy commented on OOZIE-2406:
---

bq. I also considered the Oozie UI to have GraphGenerator render in svg format, 
it should take less effort on Oozie server side, as for SVGs we don't need 
BufferedImage instances
  Big +1 for switching to svg. We had to turn off graph rendering for large 
dags as the huge image objects caused the Oozie servers to go out of memory.

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png, OOZIE-2406.004.patch, 
> OOZIE-2406.004.patch, OOZIE-2406.005.patch, OOZIE-2406.006.patch, 
> OOZIE-2406.007.patch, OOZIE-2406.008.patch, OOZIE-2406.009.patch, Screen Shot 
> 2017-10-02 at 1.55.58 PM.png, Screen Shot 2017-10-02 at 1.56.33 PM.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189031#comment-16189031
 ] 

Rohini Palaniswamy commented on OOZIE-2896:
---

bq. for properties within the same group, where the user has defined at least 
two different properties, emit an ERROR log stating that the user has inserted 
contradictory parameter values, and die. 
   Should not error out for these cases. Just need to keep current behavior as 
is which does not fail for any of the scenarios mentioned.
-  Map/Reduce specific settings have priority over generic child ones. For 
eg: mapred.map.child.env has precedence over mapred.child.env.
- If both new and deprecated setting is specified last one takes precedence
- admin settings are prepended to regular settings. For eg:  
yarn.app.mapreduce.am.admin-command-opts and yarn.app.mapreduce.am.command-opts 
both are applied when starting jvm. If there is a conflict, 
yarn.app.mapreduce.am.command-opts will take precedence because it comes last.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3068) Environmental variables should be separated by comma (,)

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated OOZIE-3068:
--
Issue Type: Sub-task  (was: Bug)
Parent: OOZIE-2889

> Environmental variables should be separated by comma (,)
> 
>
> Key: OOZIE-3068
> URL: https://issues.apache.org/jira/browse/OOZIE-3068
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Satish Subhashrao Saley
>
> Hadoop expects comma separated list of env variables.
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L289
> It will be confusing for users to deal with different conventions. 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1310-L1311
> Being a workflow scheduler system to manage Apache Hadoop jobs, Oozie should 
> avoid this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3068) Environmental variables should be separated by comma (,)

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189016#comment-16189016
 ] 

Rohini Palaniswamy commented on OOZIE-3068:
---

Another reason , is used as separator is that the path and file separators vary 
between Windows and Linux. So in general , is used in configuration and during 
the actual env variable construction appropriate separator is chosen.

> Environmental variables should be separated by comma (,)
> 
>
> Key: OOZIE-3068
> URL: https://issues.apache.org/jira/browse/OOZIE-3068
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>
> Hadoop expects comma separated list of env variables.
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L289
> It will be confusing for users to deal with different conventions. 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L1310-L1311
> Being a workflow scheduler system to manage Apache Hadoop jobs, Oozie should 
> avoid this.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2885) Running Spark actions should not need Hive on the classpath

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177438#comment-16177438
 ] 

Rohini Palaniswamy commented on OOZIE-2885:
---

+1

> Running Spark actions should not need Hive on the classpath
> ---
>
> Key: OOZIE-2885
> URL: https://issues.apache.org/jira/browse/OOZIE-2885
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Satish Subhashrao Saley
>Priority: Critical
> Attachments: OOZIE-2885-1.patch, OOZIE-2885-2.patch
>
>
> SparkMain references HiveConf so we need hive-common jar on the classpath 
> even if the Spark job has nothing to do with Hive. 
> This shouldn't be the case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3062) Set HADOOP_CONF_DIR for spark action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175127#comment-16175127
 ] 

Rohini Palaniswamy commented on OOZIE-3062:
---

env setting is usually separated by , in hadoop configs and not 
File.pathSeparator. Can you verify?

> Set HADOOP_CONF_DIR for spark action
> 
>
> Key: OOZIE-3062
> URL: https://issues.apache.org/jira/browse/OOZIE-3062
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3062-1.patch
>
>
> OOZIE-2569 created core-site.xml, hdfs-site.xml, mapred-site.xml, 
> yarn-site.xml for spark action. But for spark to consider these files ([spark 
> code 
> ref|https://github.com/apache/spark/blob/83fe3b5e10f8dc62245ea37143abb96be0f39805/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L284]),
>  we need to set HADOOP_CONF_DIR environmental variable, otherwise it will 
> point to default HADOOP_CONF_DIR configured on the node. Therefore, Oozie 
> should set HADOOP_CONF_DIR.
> Due to a bug YARN-4727, yarn was not considering user specified value for 
> whiltelisted environment variables including HADOOP_CONF_DIR. So, this will 
> work on specified hadoop fix versions in YARN-4727.
> To verify this on one node machine. 
> 1. In hadoop mapred-site.xml, set 
> {code}
> 
>mapreduce.fileoutputcommitter.marksuccessfuljobs
>false
> 
> {code}
> 2. Run any spark file copy example from Oozie. Check output directory, 
> _SUCCESS flag is missing. It should be there because we explicitly set 
> conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "true"); in 
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L293



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2885) Running Spark actions should not need Hive on the classpath

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175120#comment-16175120
 ] 

Rohini Palaniswamy commented on OOZIE-2885:
---

bq. This depends on OOZIE-2554
  [~satishsaley], can you create a separate jira for adding a Spark SQL 
testcase making it dependent on OOZIE-2554 and relating to this jira. Testing 
spark sql should automatically cover using hive-site.xml which is the actual 
use case we have in production. 

> Running Spark actions should not need Hive on the classpath
> ---
>
> Key: OOZIE-2885
> URL: https://issues.apache.org/jira/browse/OOZIE-2885
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Satish Subhashrao Saley
>Priority: Critical
> Attachments: OOZIE-2885-1.patch
>
>
> SparkMain references HiveConf so we need hive-common jar on the classpath 
> even if the Spark job has nothing to do with Hive. 
> This shouldn't be the case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2885) Running Spark actions should not need Hive on the classpath

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175096#comment-16175096
 ] 

Rohini Palaniswamy commented on OOZIE-2885:
---

bq. Please extract the code in the try-catch block to a method w/ a describing 
name
  It is already in a setupHiveSite method. Please do not add another method for 
this. 

[~satishsaley],
   I think you can actually remove
{code}
HiveConf.setHiveSiteLocation(HiveConf.class.getClassLoader().getResource("hive-site.xml"));
{code}

and it should still work fine.

bq. and add a unit test to TestSparkMain 
  This depends on OOZIE-2554. Any normal test in cluster mode would have caught 
this. We need to fix OOZIE-2554 soon.

> Running Spark actions should not need Hive on the classpath
> ---
>
> Key: OOZIE-2885
> URL: https://issues.apache.org/jira/browse/OOZIE-2885
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Satish Subhashrao Saley
>Priority: Critical
> Attachments: OOZIE-2885-1.patch
>
>
> SparkMain references HiveConf so we need hive-common jar on the classpath 
> even if the Spark job has nothing to do with Hive. 
> This shouldn't be the case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3048) Check El Functions for the coordinator action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173454#comment-16173454
 ] 

Rohini Palaniswamy commented on OOZIE-3048:
---

+1. Test failure is unrelated

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3048-1.patch, OOZIE-3408-2.patch, 
> OOZIE-3408-3.patch
>
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3048) Check El Functions for the coordinator action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172079#comment-16172079
 ] 

Rohini Palaniswamy commented on OOZIE-3048:
---

[~rkanter],
   Thanks. But I actually did not mean to get you to look into that failure. 
Was asking Satish to look into it. Sorry for not being clear.

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3048-1.patch, OOZIE-3408-2.patch
>
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3048) Check El Functions for the coordinator action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172020#comment-16172020
 ] 

Rohini Palaniswamy commented on OOZIE-3048:
---

[~satishsaley],
   Reattaching patch might not be of much help. Looks like [~rkanter] has 
changed the builds to a new job -OOZIE-3051. 
https://builds.apache.org/blue/organizations/jenkins/PreCommit-OOZIE-Build/activity
 - Looks like it ran for the patch, but failed. Can you look into the reason 
for the failure?

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3048-1.patch, OOZIE-3408-2.patch
>
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3048) Check El Functions for the coordinator action

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171074#comment-16171074
 ] 

Rohini Palaniswamy commented on OOZIE-3048:
---

+1

> Check El Functions for the coordinator action
> -
>
> Key: OOZIE-3048
> URL: https://issues.apache.org/jira/browse/OOZIE-3048
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3048-1.patch
>
>
> In OOZIE-2873, we tried introduce check for EL functions before submitting 
> the coordinator. But this not possible as described 
> [here|https://issues.apache.org/jira/browse/OOZIE-2873?focusedCommentId=15999124=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15999124].
>  Instead, we can do this check in CoordActionInputCheckXCommand when we 
> resolve coordinator configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62387: OOZIE-3048 Check El Functions for the coordinator action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62387/#review185642
---


Ship it!




- Rohini Palaniswamy


On Sept. 19, 2017, 1 a.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62387/
> ---
> 
> (Updated Sept. 19, 2017, 1 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3048
> https://issues.apache.org/jira/browse/OOZIE-3048
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3048 Check El Functions for the coordinator action
> 
> 
> Diffs
> -
> 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  650e7f8c 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordPushDependencyCheckXCommand.java
>  0ad39703 
>   core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java fd2f92dc 
>   core/src/main/java/org/apache/oozie/coord/ElException.java PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java
>  bbe9ea7a 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordPushDependencyCheckXCommand.java
>  25f250ce 
>   core/src/test/resources/coord-hcatinput-invalid-elfunction.xml PRE-CREATION 
>   core/src/test/resources/coord-hdfsinput-invalid-elfunction.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62387/diff/2/
> 
> 
> Testing
> ---
> 
> tested locally
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

Re: Review Request 62387: OOZIE-3048 Check El Functions for the coordinator action

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62387/#review185572
---




core/src/test/java/org/apache/oozie/command/coord/TestCoordPushDependencyCheckXCommand.java
Lines 443 (patched)
<https://reviews.apache.org/r/62387/#comment261886>

Can you rename the files as coord-hcatinput-invalid-elfunction.xml and 
coord-hdfsinput-invalid-elfunction.xml?


- Rohini Palaniswamy


On Sept. 18, 2017, 5:05 p.m., Satish Saley wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62387/
> ---
> 
> (Updated Sept. 18, 2017, 5:05 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-3048
> https://issues.apache.org/jira/browse/OOZIE-3048
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-3048 Check El Functions for the coordinator action
> 
> 
> Diffs
> -
> 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java
>  650e7f8c 
>   
> core/src/main/java/org/apache/oozie/command/coord/CoordPushDependencyCheckXCommand.java
>  0ad39703 
>   core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java fd2f92dc 
>   core/src/main/java/org/apache/oozie/coord/ElException.java PRE-CREATION 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java
>  bbe9ea7a 
>   
> core/src/test/java/org/apache/oozie/command/coord/TestCoordPushDependencyCheckXCommand.java
>  25f250ce 
>   core/src/test/resources/coord-action-for-invalid-elfunction.xml 
> PRE-CREATION 
>   core/src/test/resources/coord-job-for-invalid-elfunction.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62387/diff/1/
> 
> 
> Testing
> ---
> 
> tested locally
> 
> 
> Thanks,
> 
> Satish Saley
> 
>

[jira] [Commented] (OOZIE-3058) nocleanup option is missing in oozie-coordinator-0.5.xsd

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170330#comment-16170330
 ] 

Rohini Palaniswamy commented on OOZIE-3058:
---

+1

> nocleanup option is missing in oozie-coordinator-0.5.xsd
> 
>
> Key: OOZIE-3058
> URL: https://issues.apache.org/jira/browse/OOZIE-3058
> Project: Oozie
>  Issue Type: Bug
>Reporter: Satish Subhashrao Saley
>Assignee: Satish Subhashrao Saley
> Attachments: OOZIE-3058-1.patch
>
>
> nocleanup option is missing in oozie-coordinator-0.5.xsd which is there in 
> [oozie-coordinator-0.4.xsd|https://github.com/apache/oozie/blob/master/client/src/main/resources/oozie-coordinator-0.4.xsd#L119].
>  And it is missing in documentation for versions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2687) Create XML schema for launcher configurations

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159438#comment-16159438
 ] 

Rohini Palaniswamy commented on OOZIE-2687:
---

Haven't gotten time to look into this jira in detail. I just have one feedback. 
Keep in mind backward compatibility. Users should be able to upgrade to the new 
schema without their existing code breaking and having to fix it. 

> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, 
> OOZIE-2687-003.patch, OOZIE-2687-008.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through 
> properties like:
> # mapreduce.map.memory.mb 
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts  
> # mapred.job.queue.name  - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper 
> and make PySpark work. 
> Fixing OOZIE-2596 added a hack. We should decide how we support or break 
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in 
> sub-workflows as well. It did not use to work before and was only fixed in 
> OOZIE-2030. It would be good to have that testcase 
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new 
> launcher section as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3028) Oozie Pig Action fails with no python dependencies

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122356#comment-16122356
 ] 

Rohini Palaniswamy commented on OOZIE-3028:
---

+1. Will commit once pre-commit build finishes.

> Oozie Pig Action fails with no python dependencies
> --
>
> Key: OOZIE-3028
> URL: https://issues.apache.org/jira/browse/OOZIE-3028
> Project: Oozie
>  Issue Type: Bug
>  Components: build
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Artem Ervits
>  Labels: pig
> Fix For: 5.0.0
>
> Attachments: OOZIE-3028-0.patch, OOZIE-3028-1.patch
>
>
> Oozie Pig Action with Python script fails with below as the pig sharelib 
> jython-2.5.3.jar does not have python dependencies. We should include 
> jython-standalone jar into pig sharelib which has the right dependencies.
> {code}
> Caused by: java.lang.NoSuchFieldError: __code__
> at 
> org.apache.pig.scripting.jython.JythonFunction.(JythonFunction.java:57)
> ... 51 more
> {code}
> # /usr/hdp/current/oozie-server/bin/oozie admin -shareliblist pig |grep 
> jython 
>   
> hdfs://mycluster/user/oozie/share/lib/lib_20170106054154/pig/jython-2.5.3.jar



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3028) Oozie Pig Action fails with no python dependencies

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122310#comment-16122310
 ] 

Rohini Palaniswamy commented on OOZIE-3028:
---

bq. Rohini Palaniswamy do you think it makes sense to up Jython to 2.7.1 in Pig 
and here?
  Sure. Looks like there are both bug fixes and enhancements in it - 
http://fwierzbicki.blogspot.ru/2017/07/jython-271-final-released.html

> Oozie Pig Action fails with no python dependencies
> --
>
> Key: OOZIE-3028
> URL: https://issues.apache.org/jira/browse/OOZIE-3028
> Project: Oozie
>  Issue Type: Bug
>  Components: build
>Affects Versions: 4.3.0
>Reporter: Prabhu Joseph
>Assignee: Artem Ervits
>  Labels: pig
> Fix For: 5.0.0
>
> Attachments: OOZIE-3028-0.patch
>
>
> Oozie Pig Action with Python script fails with below as the pig sharelib 
> jython-2.5.3.jar does not have python dependencies. We should include 
> jython-standalone jar into pig sharelib which has the right dependencies.
> {code}
> Caused by: java.lang.NoSuchFieldError: __code__
> at 
> org.apache.pig.scripting.jython.JythonFunction.(JythonFunction.java:57)
> ... 51 more
> {code}
> # /usr/hdp/current/oozie-server/bin/oozie admin -shareliblist pig |grep 
> jython 
>   
> hdfs://mycluster/user/oozie/share/lib/lib_20170106054154/pig/jython-2.5.3.jar



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2826) Falcon feed fails to aws s3; Oozie joda time version does not meet required jar version 2.2 or later

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035400#comment-16035400
 ] 

Rohini Palaniswamy commented on OOZIE-2826:
---

bq. should I file a jira against Pig to bump it to 2.9.9 or change Oozie dep to 
2.9.3?
   Increase version in Pig. Always better to have latest joda-time as it ships 
updated timezone data.

> Falcon feed fails to aws s3; Oozie joda time version does not meet required 
> jar version 2.2 or later
> 
>
> Key: OOZIE-2826
> URL: https://issues.apache.org/jira/browse/OOZIE-2826
> Project: Oozie
>  Issue Type: Bug
>  Components: workflow
>Affects Versions: 4.3.0
>Reporter: Péter Gergő Barna
>Assignee: Artem Ervits
>Priority: Minor
> Fix For: 5.0.0
>
> Attachments: OOZIE-2826-1.patch, OOZIE-2826-2.patch, 
> OOZIE-2826.patch, oozie-joda.example-feed.xml, oozie-joda.log.txt
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Symptom:
> 2016-11-04 17:34:50,283 WARN [main] 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler: Unable to parse 
> last modified date: Wed, 02 Nov 2016 23:05:38 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is 
> required, but found version: 2.1
>   at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
>   at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
>   at 
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
>   at 
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
>   at 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:32)
>   at 
> com.amazonaws.services.s3.internal.S3MetadataResponseHandler.handle(S3MetadataResponseHandler.java:25)
>   at 
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
>   at 
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
>   at 
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
>   at 
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1050)
>   at 
> com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1027)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:850)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1403)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1343)
>   at 
> org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1308)
>   at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1898)
>   at 
> org.apache.falcon.util.DistCPOptionsUtil.getDistCpOptions(DistCPOptionsUtil.java:90)
>   at 
> org.apache.falcon.replication.FeedReplicator.getDistCpOptions(FeedReplicator.java:229)
>   at 
> org.apache.falcon.replication.FeedReplicator.run(FeedReplicator.java:70)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>   at 
> org.apache.falcon.replication.FeedReplicator.main(FeedReplicator.java:64)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
>   at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51)
>   at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242)
>   at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>   at org.ap

[jira] [Commented] (OOZIE-2897) LauncherAM should support ACLs

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033822#comment-16033822
 ] 

Rohini Palaniswamy commented on OOZIE-2897:
---

bq. I think we should have Oozie properties (or part of the schema) for this. I 
think we should do that in general with other things as well.
  Creating equivalent Oozie properties for all the MR properties is unnecessary 
in my opinion and just another confusing thing for user to know and learn. 
Making the common launcher properties as part of xml schema already takes care 
of simplifying them and new users can start using them.

> LauncherAM should support ACLs
> --
>
> Key: OOZIE-2897
> URL: https://issues.apache.org/jira/browse/OOZIE-2897
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>
> In MapReduce, you can define ACL-related properties:
> {noformat}
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modify-job
> {noformat}
> {{acl-view-job}} defines a list of users/groups who can retrieve the job 
> statistics. {{acl-modify-job}} defines a list of users/groups who can kill a 
> job or adjust the priority of it.
> Docs: 
> https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization
> In YARN, we can provide backward compatibility for these properties. Example 
> code:
> {code}
> Map<ApplicationAccessType, String> acls = new HashMap<ApplicationAccessType, 
> String>();
> acls.put(ApplicationAccessType.MODIFY_APP, "*");
> acls.put(ApplicationAccessType.VIEW_APP, "*");
> amContainer.setApplicationACLs(acls);
> {code}
> This has to be done before application submission. We have to open 
> {{mapred-site.xml}} and check if {{mapred.acls.enabled}} is true. If so, then 
> read the values of view-job/modify-job and configure the launcher context as 
> described above.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2897) LauncherAM should support ACLs

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033534#comment-16033534
 ] 

Rohini Palaniswamy commented on OOZIE-2897:
---

You need exactly this - 
https://github.com/apache/hadoop/blob/3721cfe1fbd98c5b6aa46aefdfcf62276c28c4a4/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L502-L507
 on the application submission side.

Checking for acls.enabled and the acls is done in the AM webservices and UI 
calls in mapreduce. We will have to do that with Oozie AM as well.

> LauncherAM should support ACLs
> --
>
> Key: OOZIE-2897
> URL: https://issues.apache.org/jira/browse/OOZIE-2897
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>
> In MapReduce, you can define ACL-related properties:
> {noformat}
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modify-job
> {noformat}
> {{acl-view-job}} defines a list of users/groups who can retrieve the job 
> statistics. {{acl-modify-job}} defines a list of users/groups who can kill a 
> job or adjust the priority of it.
> Docs: 
> https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization
> In YARN, we can provide backward compatibility for these properties. Example 
> code:
> {code}
> Map<ApplicationAccessType, String> acls = new HashMap<ApplicationAccessType, 
> String>();
> acls.put(ApplicationAccessType.MODIFY_APP, "*");
> acls.put(ApplicationAccessType.VIEW_APP, "*");
> amContainer.setApplicationACLs(acls);
> {code}
> This has to be done before application submission. We have to open 
> {{mapred-site.xml}} and check if {{mapred.acls.enabled}} is true. If so, then 
> read the values of view-job/modify-job and configure the launcher context as 
> described above.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (OOZIE-1770) Create Oozie Application Master for YARN

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027992#comment-16027992
 ] 

Rohini Palaniswamy edited comment on OOZIE-1770 at 5/29/17 1:43 AM:


bq. Shall we delete the oya branch now?
  We don't delete branches after the merge is done. They are kept for history.


was (Author: rohini):
bq. Shall we delete the oya branch now?
  We don't delete branches after the merge is done.

> Create Oozie Application Master for YARN
> 
>
> Key: OOZIE-1770
> URL: https://issues.apache.org/jira/browse/OOZIE-1770
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Peter Cseh
> Fix For: 5.0.0
>
> Attachments: OOZIE-1770-001.patch, OOZIE-1770-002.patch, 
> OOZIE-1770-003.patch, OOZIE-1770-004.patch, OOZIE-1770-005.patch, 
> OOZIE-1770-006.patch, OOZIE-1770-007.patch, OOZIE-1770-008.patch, 
> OOZIE-1770-009.patch, OOZIE-1770-010.patch, OOZIE-1770-011.patch, 
> OOZIE-1770-012.patch, OOZIE-1770-013.patch, OOZIE-1770-014.patch, 
> OOZIE-1770-015.patch, OOZIE-1770-016.patch, OozieYarnAM.pdf, oya.patch, 
> oya-rm-screenshot.jpg, OYA Scoping Doc 001.pdf, Prelim OYA Scoping Doc 
> 001.pdf, Prelim OYA Scoping Doc 002.pdf, Prelim OYA Scoping Doc 003.pdf
>
>
> After the first release of oozie on hadoop 2, it will be good if users can 
> set execution engine in oozie conf, be it YARN AM or traditional MR. We can 
> target this for post oozie 4.1 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-1770) Create Oozie Application Master for YARN

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027992#comment-16027992
 ] 

Rohini Palaniswamy commented on OOZIE-1770:
---

bq. Shall we delete the oya branch now?
  We don't delete branches after the merge is done.

> Create Oozie Application Master for YARN
> 
>
> Key: OOZIE-1770
> URL: https://issues.apache.org/jira/browse/OOZIE-1770
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Peter Cseh
> Fix For: 5.0.0
>
> Attachments: OOZIE-1770-001.patch, OOZIE-1770-002.patch, 
> OOZIE-1770-003.patch, OOZIE-1770-004.patch, OOZIE-1770-005.patch, 
> OOZIE-1770-006.patch, OOZIE-1770-007.patch, OOZIE-1770-008.patch, 
> OOZIE-1770-009.patch, OOZIE-1770-010.patch, OOZIE-1770-011.patch, 
> OOZIE-1770-012.patch, OOZIE-1770-013.patch, OOZIE-1770-014.patch, 
> OOZIE-1770-015.patch, OOZIE-1770-016.patch, OozieYarnAM.pdf, oya.patch, 
> oya-rm-screenshot.jpg, OYA Scoping Doc 001.pdf, Prelim OYA Scoping Doc 
> 001.pdf, Prelim OYA Scoping Doc 002.pdf, Prelim OYA Scoping Doc 003.pdf
>
>
> After the first release of oozie on hadoop 2, it will be good if users can 
> set execution engine in oozie conf, be it YARN AM or traditional MR. We can 
> target this for post oozie 4.1 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2919) make Pig run with Tez execution launcher easier

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16027988#comment-16027988
 ] 

Rohini Palaniswamy commented on OOZIE-2919:
---

bq. There's a workaround that requires Tez libs added to oozie sharelib rather 
having its own directory and also requires some properties in the oozie wf xml.
   At Yahoo, we deploy tez libs and tez-site.xml part of pig sharelib itself so 
that users don't have to specify an additional sharelib for tez. Probably we 
can add some kind of support for that for Pig and Hive sharelibs in Oozie 
sharelib deployment through configuration. What is the oozie wf xml property 
that you are talking about?

> make Pig run with Tez execution launcher easier
> ---
>
> Key: OOZIE-2919
> URL: https://issues.apache.org/jira/browse/OOZIE-2919
> Project: Oozie
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>
> I've been trying to make Pig run with Tez via Oozie and was not able to do so 
> easily. There's a workaround that requires Tez libs added to oozie sharelib 
> rather having its own directory and also requires some properties in the 
> oozie wf xml. I propose to have Tez treated as first class citizen as per 
> Yahoo talk at Hadoop Summit, they moved off of MR to TEZ for all of their Pig 
> workloads.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (OOZIE-2920) Extend FS action to do HDFS copy

[Rohini Palaniswamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy resolved OOZIE-2920.
---
Resolution: Won't Fix

Users should be using distcp action to do file copy.

> Extend FS action to do HDFS copy
> 
>
> Key: OOZIE-2920
> URL: https://issues.apache.org/jira/browse/OOZIE-2920
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Jan Hentschel
>
> Today FS action has a limited number of actions, it would benefit community 
> if FS copy would be available.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026930#comment-16026930
 ] 

Rohini Palaniswamy commented on OOZIE-2896:
---

bq. mapred.cache.archives
  We do not have to support this. Users are supposed to use  tag.

bq. java.library.path and sun.boot.library.path
Do not have to support this either as mapreduce itself discourages this and 
asks users to use LD_LIBRARY_PATH in mapreduce.map.env. 

bq. yarn.app.mapreduce.am.log.level
  Should support this and mapreduce.map.log.level for debugging purpose.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026650#comment-16026650
 ] 

Rohini Palaniswamy commented on OOZIE-2896:
---

Most common ones specified for launcher (prefixed with oozie.launcher. ) are

mapreduce.job.queuename
mapreduce.map.memory.mb
mapreduce.map.java.opts
mapreduce.map.env
mapreduce.job.acl-view-job
mapreduce.job.acl-modfy-job
mapred.job.priority
mapreduce.task.classpath.user.precedence
mapreduce.job.classloader and mapreduce.job.classloader.system.classes
mapreduce.job.hdfs-servers
fs.permissions.umask-mode
dfs.umask (different way of specifying than fs.permissions.umask-mode)

// Since it is running in uber mode, many do configure AM memory settings 
directly
yarn.app.mapreduce.am.resource.mb
yarn.app.mapreduce.am.command-opts
yarn.app.mapreduce.am.env

I am just specifying one version of the setting. Deprecated mapred and dfs 
equivalent settings are also used. Usually it is taken care by the config 
deprecation code and you could just refer by one setting.

Have asked [~satishsaley] to do a git search and find if there are others.

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-2890) Implement the equivalent of mapreduce.task.classpath.user.precedence

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023168#comment-16023168
 ] 

Rohini Palaniswamy commented on OOZIE-2890:
---

bq. It's very hacky and encourages the user to load in incompatible/conflicting 
jars and rely on classpath ordering, which is bad.
  Problem is there are lot of users who are already using this feature. 
Removing support means lot of running jobs will break. When we do the Oozie on 
YARN upgrade, my expectation is all coordinators continue to run without issues 
and there is no migration effort for the user or behavior change except for the 
fact that the launcher job UI will look different.

> Implement the equivalent of mapreduce.task.classpath.user.precedence
> 
>
> Key: OOZIE-2890
> URL: https://issues.apache.org/jira/browse/OOZIE-2890
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (OOZIE-1770) Create Oozie Application Master for YARN

[Rohini Palaniswamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023141#comment-16023141
 ] 

Rohini Palaniswamy commented on OOZIE-1770:
---

+1 for https://reviews.apache.org/r/54705/diff/11/

> Create Oozie Application Master for YARN
> 
>
> Key: OOZIE-1770
> URL: https://issues.apache.org/jira/browse/OOZIE-1770
> Project: Oozie
>  Issue Type: New Feature
>Reporter: Bowen Zhang
>Assignee: Peter Cseh
> Attachments: OOZIE-1770-001.patch, OOZIE-1770-002.patch, 
> OOZIE-1770-003.patch, OOZIE-1770-004.patch, OOZIE-1770-005.patch, 
> OOZIE-1770-006.patch, OOZIE-1770-007.patch, OOZIE-1770-008.patch, 
> OOZIE-1770-009.patch, OOZIE-1770-010.patch, OOZIE-1770-011.patch, 
> OOZIE-1770-012.patch, OOZIE-1770-013.patch, OOZIE-1770-014.patch, 
> OOZIE-1770-015.patch, OOZIE-1770-016.patch, OozieYarnAM.pdf, oya.patch, 
> oya-rm-screenshot.jpg, OYA Scoping Doc 001.pdf, Prelim OYA Scoping Doc 
> 001.pdf, Prelim OYA Scoping Doc 002.pdf, Prelim OYA Scoping Doc 003.pdf
>
>
> After the first release of oozie on hadoop 2, it will be good if users can 
> set execution engine in oozie conf, be it YARN AM or traditional MR. We can 
> target this for post oozie 4.1 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175946
---


Ship it!




Ship It!

- Rohini Palaniswamy


On May 24, 2017, 12:05 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 24, 2017, 12:05 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f759d820b20 
>   core/src/main/java/org/apache/oozie/client/rest/BulkResponseImpl.java 
> fb021bd022d199f93fc5674843514abf45ae12a5 
>   core/src/main/java/org/apache/oozie/command/XCommand.java 
> bdf13f65ea25e8e0f5125e2798568a2c03fffe81 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitHttpXCommand.java 
> d2a2742ae418070a1cd0a889f14194224bc944a6 
>   core/src/main/java/org/apache/oozie/command/wf/SubmitMRXCommand.java 
> cc61d3de813c75c7f1cf4d405fc043253e3ca4ea 
>   
> core/src/main/java/org/apache/oozie/command/wf/SubmitScrip

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
> > Line 49 (original), 50 (patched)
> > <https://reviews.apache.org/r/54705/diff/8-9/?file=1714632#file1714632line50>
> >
> > Why would this require a ConcurrentHashMap?
> 
> Peter Cseh wrote:
> It a precaution to avoid issues when getInstance() is called from 
> multiple threads and the map is filled up from two threads. It shouldn't 
> cause any issues with performance or threads this way.

Using a ConcurrentHashMap does not help in anyway. That variable is local to 
the new CredentialsProviderFactory being created. Two threads cannot fill the 
same map as the constructors will create two different instances. If two 
threads call getInstance at same time and if instance is null, both will create 
a instance and the last one will win. Ideally we should avoid that by 
synchronizing, but in this case based on the nature of data it is not a 
problem. Please switch it back to HashMap.


> On May 22, 2017, 11:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
> > Line 145 (original), 148 (patched)
> > <https://reviews.apache.org/r/54705/diff/8-9/?file=1714649#file1714649line148>
> >
> > This method can be removed now.

It is still not removed.


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---


On May 23, 2017, 4:43 p.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 23, 2017, 4:43 p.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceAction

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review175722
---




core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
Line 49 (original), 50 (patched)
<https://reviews.apache.org/r/54705/#comment249074>

Why would this require a ConcurrentHashMap?



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 48 (original), 48 (patched)
<https://reviews.apache.org/r/54705/#comment249075>

to credentials



core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java
Line 66 (original), 67 (patched)
<https://reviews.apache.org/r/54705/#comment249076>

to credentials



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 959 (patched)
<https://reviews.apache.org/r/54705/#comment249077>

Fix typo. Creating YarnClient



core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java
Line 145 (original), 148 (patched)
<https://reviews.apache.org/r/54705/#comment249079>

This method can be removed now.



sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMapper.java
Line 45 (original), 44 (patched)
<https://reviews.apache.org/r/54705/#comment249080>

Why are they methods in this class renamed to start with testLauncherMapper 
when the test class itself is TestLauncherMapper?  

Are these tests still relevant as LauncherMapper is not used anymore? 
Shouldn't they be changed to adapt to Oozie AM instead? Please revert current 
changes and create a jira for it if that is the case.


- Rohini Palaniswamy


On May 20, 2017, 11:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 20, 2017, 11:55 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
&g

Re: Review Request 54705: OOZIE-1770: Create Oozie Application Master for YARN

[Rohini Palaniswamy]

> On May 14, 2017, 4:16 a.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Line 423 (original), 321 (patched)
> > <https://reviews.apache.org/r/54705/diff/8/?file=1714643#file1714643line463>
> >
> > This should not be required anymore. The Oozie AM code should be doing 
> > the same thing as done in MAPREDUCE-6472.
> 
> Peter Bacsko wrote:
> Right - deleted

Code to be added to Oozie AM is still not done. Refer 
https://issues.apache.org/jira/secure/attachment/12755788/MAPREDUCE-6472.20150914-2.patch


- Rohini


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54705/#review174883
---


On May 20, 2017, 11:55 a.m., Peter Cseh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54705/
> ---
> 
> (Updated May 20, 2017, 11:55 a.m.)
> 
> 
> Review request for oozie, Purshotam Shah, Robert Kanter, and Rohini 
> Palaniswamy.
> 
> 
> Bugs: OOZIE-1770
> https://issues.apache.org/jira/browse/OOZIE-1770
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Replacing the launcher MR job with a launcher AM
> 
> 
> Diffs
> -
> 
>   client/pom.xml f99366c90887a5a45f3942291c90e1cdbfd84b3f 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 38fb84e8a8918aee6ae57d19a29cdf181aba1955 
>   client/src/main/java/org/apache/oozie/client/XOozieClient.java 
> 8538ec7c742cc013f7c18eaff3ecf61c2835cc8a 
>   core/pom.xml 14aa03483e107db326628ba627cc3ee9f5fd3f3d 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> 55d79a5d5e5e4efab9eb832e9341a16bcf8959cc 
>   core/src/main/java/org/apache/oozie/action/ActionExecutor.java 
> 1d6456b236bf9e27c0f55d9f9e3b4bd94517c797 
>   core/src/main/java/org/apache/oozie/action/hadoop/Credentials.java 
> eadb47b07d157f45f086517ab41c4b9826f003f0 
>   core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProvider.java 
> 6fe22fbd2fca5d873d58ece60f20d0f131371475 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/CredentialsProviderFactory.java
>  PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 
> 78cd257a836707c1efeac13566d6285d42970545 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java 
> 27654741101d2056b25626fbbe345183eafd612d 
>   core/src/main/java/org/apache/oozie/action/hadoop/FsELFunctions.java 
> 801bfe6eb212b549888f0e9fb7df7b65bcf59393 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentialHelper.java 
> 36ad1df727e869b8051481ad969771e91bc6ae07 
>   core/src/main/java/org/apache/oozie/action/hadoop/HCatCredentials.java 
> d7689a924059e553abaeed374eed4072933292cb 
>   core/src/main/java/org/apache/oozie/action/hadoop/HadoopELFunctions.java 
> ad2a71d641caa71efc098ad00bfbab3cec4d9506 
>   core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java 
> 307f565786ad77f5a35b5790bd5b052b04c6eac8 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java 
> 2aed936af75da38c433be5d202bdc01c2f612c76 
>   core/src/main/java/org/apache/oozie/action/hadoop/Hive2Credentials.java 
> bd2f1f768c5b04bef4a33613baede9202144c747 
>   core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 
> 8e2453ef4f78f4a5f1f95b1ca2793f5d7e07ad81 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 06ae5fd9a52099c5bc63b038d336ce96103c8819 
>   core/src/main/java/org/apache/oozie/action/hadoop/LauncherMapperHelper.java 
> 72ed2f150be73f0a5e8b71b0fb6cc3594dc23078 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
>  55c937249ac35e14bd63dd0c1fa962ea636afa3a 
>   core/src/main/java/org/apache/oozie/action/hadoop/OozieJobInfo.java 
> 581d3b3219eff882f961c8e495611e873c6a5a52 
>   core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 
> 8b2dc16db80ad602911f1f64e60cfad41b198922 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java
>  92e149d5cac6509893c8507b776263a38312a53b 
>   core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 
> b9ffa7ab7ec87db8a76a3653c74974d296c8cd17 
>   core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 
> 1a3197abb52f80b545dcc8634a8cac7a281a9eac 
>   core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 
> 22e28746a9a39ae6c1e22326cc4f8f