[jira] [Updated] (OOZIE-2929) Coordinator SLA should support configuration variable

2017-10-25 Thread Chi Wen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chi Wen updated OOZIE-2929:
---
Attachment: OOZIE-2929-002.patch

> Coordinator SLA should support configuration variable
> -
>
> Key: OOZIE-2929
> URL: https://issues.apache.org/jira/browse/OOZIE-2929
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 4.3.0
>Reporter: Chi Wen
> Attachments: OOZIE-2929-001.patch, OOZIE-2929-002.patch
>
>
> if coordinator.xml define SLA like this:
>   
> ${nominal_time}
> ${ 1 * HOURS}
> ${1 * MINUTES}
>   
> In job.properties define: nominal_time=${coord:nominalTime()}
> Then coordinator will run failed for "Coord Job Materialization Error: Could 
> not parse [${coord:nominalTime()}] using [-MM-dd'T'HH:mm+0800] mask";
> Oozie should support use define a variable for SLA;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-2231:

Attachment: OOZIE-2231-05.patch

[~rkanter] thanks for the tip, I made the suggested changes and build is 
successful, please review.

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231-04.patch, OOZIE-2231-05.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3097) Change version in pom in master to 5.0.0-beta1

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-3097:

Attachment: OOZIE-3097-00.patch

command issued
{code}
mvn versions:set -DnewVersion=5.0.0-beta1 -DgenerateBackupPoms=false
{code}

> Change version in pom in master to 5.0.0-beta1
> --
>
> Key: OOZIE-3097
> URL: https://issues.apache.org/jira/browse/OOZIE-3097
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Peter Cseh
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3097-00.patch
>
>
> In preparation to the 5b1 release we should modify the version in the POM 
> files to 5.0.0-beta1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-3097) Change version in pom in master to 5.0.0-beta1

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits reassigned OOZIE-3097:
---

Assignee: Artem Ervits

> Change version in pom in master to 5.0.0-beta1
> --
>
> Key: OOZIE-3097
> URL: https://issues.apache.org/jira/browse/OOZIE-3097
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0b1
>Reporter: Peter Cseh
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> In preparation to the 5b1 release we should modify the version in the POM 
> files to 5.0.0-beta1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Robert Kanter (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219752#comment-16219752
 ] 

Robert Kanter commented on OOZIE-2231:
--

We unfortunately have to be careful about upgrading guava due to Hadoop.  If 
{{Closeables.closeQuietly(yarnClient);}} is the only issue, then I'd suggest 
copying that into Oozie.  In fact, we already have our own version in 
{{IOUtils}} 
[here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/IOUtils.java#L270]
 :)

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231-04.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219741#comment-16219741
 ] 

Artem Ervits edited comment on OOZIE-2231 at 10/25/17 11:43 PM:


[~abhishekbafna] hope you don't mind if I take over here. I'm uploading a patch 
w/out your guava changes. Guava 16 breaks JavaActionExecutor
{code}
Closeables.closeQuietly(yarnClient);
{code}
closeQuietly is missing from Guava 16 but is available in Guava 17.x - 19x. 
Please advise

[~gezapeti]  [~rkanter] please review

should I open a jira to bump guava to 17+?


was (Author: dbist13):
[~abhishekbafna] hope you don't mind if I take over here. I'm uploading a patch 
w/out your guava changes. Guava 16 breaks JavaActionExecutor
{code}
Closeables.closeQuietly(yarnClient);
{code}
closeQuietly is missing from Guava 16 but is available in Guava 17.x - 19x. 
Please advise

[~gezapeti]  [~rkanter] please review

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231-04.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-2231:

Attachment: OOZIE-2231-04.patch

Abhishek Bafna hope you don't mind if I take over here. I'm uploading a patch 
w/out your guava changes. Guava 16 breaks JavaActionExecutor
Closeables.closeQuietly(yarnClient);
closeQuietly is missing from Guava 16 but is available in Guava 17.x - 19x. 
Please advise
Peter Cseh Robert Kanter please review

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231-04.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Issue Comment Deleted] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-2231:

Comment: was deleted

(was: Abhishek Bafna hope you don't mind if I take over here. I'm uploading a 
patch w/out your guava changes. Guava 16 breaks JavaActionExecutor
Closeables.closeQuietly(yarnClient);
closeQuietly is missing from Guava 16 but is available in Guava 17.x - 19x. 
Please advise
Peter Cseh Robert Kanter please review)

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231-04.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-2231:

Attachment: (was: OOZIE-2231.04.patch)

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-2231:

Fix Version/s: (was: 5.0.0)
   5.0.0b1

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits updated OOZIE-2231:

Attachment: OOZIE-2231.04.patch

[~abhishekbafna] hope you don't mind if I take over here. I'm uploading a patch 
w/out your guava changes. Guava 16 breaks JavaActionExecutor
{code}
Closeables.closeQuietly(yarnClient);
{code}
closeQuietly is missing from Guava 16 but is available in Guava 17.x - 19x. 
Please advise

[~gezapeti]  [~rkanter] please review

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231.04.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Artem Ervits (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Artem Ervits reassigned OOZIE-2231:
---

Assignee: Artem Ervits  (was: Abhishek Bafna)

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3095) Job submit command is not logged in audit log

2017-10-25 Thread Purshotam Shah (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Shah updated OOZIE-3095:
--
Fix Version/s: 5.0.0

> Job submit command is not logged in audit log
> -
>
> Key: OOZIE-3095
> URL: https://issues.apache.org/jira/browse/OOZIE-3095
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-3095-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3095) Job submit command is not logged in audit log

2017-10-25 Thread Purshotam Shah (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219712#comment-16219712
 ] 

Purshotam Shah commented on OOZIE-3095:
---

Thanks Rohini for review. Committed to trunk.

> Job submit command is not logged in audit log
> -
>
> Key: OOZIE-3095
> URL: https://issues.apache.org/jira/browse/OOZIE-3095
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Fix For: 5.0.0
>
> Attachments: OOZIE-3095-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3095) Job submit command is not logged in audit log

2017-10-25 Thread Rohini Palaniswamy (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219112#comment-16219112
 ] 

Rohini Palaniswamy commented on OOZIE-3095:
---

+1

> Job submit command is not logged in audit log
> -
>
> Key: OOZIE-3095
> URL: https://issues.apache.org/jira/browse/OOZIE-3095
> Project: Oozie
>  Issue Type: Bug
>Reporter: Purshotam Shah
>Assignee: Purshotam Shah
> Attachments: OOZIE-3095-V1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2896 PreCommit Build #135

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2896
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/135/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.38 MB...]
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-sharelib-streaming ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ 
oozie-sharelib-streaming ---
[INFO] Surefire report directory: 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build@2/sharelib/streaming/target/surefire-reports

---
 T E S T S
---
Concurrency config is parallel='classes', perCoreThreadCount=false, 
threadCount=1, useUnlimitedThreads=false
Running org.apache.oozie.action.hadoop.TestStreamingMain
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.113 sec
FATAL: command execution failed
hudson.remoting.ProxyException: java.util.concurrent.TimeoutException: Ping 
started at 1508950394026 hasn't completed by 1508950634028
Caused: Command close created at
at hudson.remoting.Command.(Command.java:60)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1132)
at hudson.remoting.Channel$CloseCommand.(Channel.java:1130)
at hudson.remoting.Channel.close(Channel.java:1290)
at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:180)
at hudson.remoting.PingThread.ping(PingThread.java:130)
at hudson.remoting.PingThread.run(PingThread.java:86)
Caused: hudson.remoting.Channel$OrderlyShutdown: 
hudson.remoting.ProxyException: java.util.concurrent.TimeoutException: Ping 
started at 1508950394026 hasn't completed by 1508950634028
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1138)
at hudson.remoting.Channel$1.handle(Channel.java:535)
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused: java.io.IOException: Backing channel 'H3' is disconnected.
at 
hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:192)
at 
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy129.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
at hudson.model.Run.execute(Run.java:1737)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:419)
Build step 'Execute shell' marked build as failure
ERROR: H3 is offline; cannot locate JDK 1.8 (latest)
[description-setter] Description set: OOZIE-2896
ERROR: Step ?Archive the artifacts? failed: no workspace for 
PreCommit-OOZIE-Build #135
ERROR: Step ?Publish JUnit test result report? failed: no workspace for 
PreCommit-OOZIE-Build #135
ERROR: H3 is offline; cannot locate JDK 1.8 (latest)
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
ERROR: H3 is offline; cannot locate JDK 1.8 (latest)
ERROR: H3 is offline; cannot locate JDK 1.8 (latest)
ERROR: H3 is offline; cannot locate JDK 1.8 (latest)
ERROR: H3 is offline; cannot locate JDK 1.8 (latest)




###
## FAILED TESTS (if any) 
##
No tests ran.

Failed: OOZIE-2929 PreCommit Build #134

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2929
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/134/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.53 MB...]
+1 the patch does not introduce any line longer than 132
+1 the patch does adds/modifies 1 testcase(s)
+1 RAT
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/pig].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS - patch does not compile, cannot run testcases
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/134/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  3720k  0 --:--:-- --:--:-- --:--:-- 3720k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2929
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 48.6% relative to #122
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-2929) Coordinator SLA should support configuration variable

2017-10-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218946#comment-16218946
 ] 

Hadoop QA commented on OOZIE-2929:
--

Testing JIRA OOZIE-2929

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/134/

> Coordinator SLA should support configuration variable
> -
>
> Key: OOZIE-2929
> URL: https://issues.apache.org/jira/browse/OOZIE-2929
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 4.3.0
>Reporter: Chi Wen
> Attachments: OOZIE-2929-001.patch
>
>
> if coordinator.xml define SLA like this:
>   
> ${nominal_time}
> ${ 1 * HOURS}
> ${1 * MINUTES}
>   
> In job.properties define: nominal_time=${coord:nominalTime()}
> Then coordinator will run failed for "Coord Job Materialization Error: Could 
> not parse [${coord:nominalTime()}] using [-MM-dd'T'HH:mm+0800] mask";
> Oozie should support use define a variable for SLA;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner

2017-10-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218844#comment-16218844
 ] 

Peter Bacsko commented on OOZIE-2985:
-

[~dionusos] that's certainly possible and could help to a certain extent. 

With the callback mechanism, we can avoid some initial polling after submitting 
the application, so we just wait until we get a response.

I can imagine creating a new state - when the application is successfully 
submitted, it goes into SUBMITTED state and not RUNNING. And when we receive 
the initial callback from LauncherAM it's updated to RUNNING. That could also 
indicate that something is wrong.

But again, the problem is that we can't wait indefinitely for the callback, so 
we have to let it go after a while. Not to mention that it also adds further 
complexity to the execution model. I'm not sure it's worth it.




> If LauncherAM fails, Oozie is not notified in a timely manner
> -
>
> Key: OOZIE-2985
> URL: https://issues.apache.org/jira/browse/OOZIE-2985
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> I've noticed if LauncherAM fails, Oozie is notified about the launcher's 
> failure with a lot of delay. It gives the impression that the workflow is 
> running.
> {{oozie job -oozie http://localhost:11000/oozie -config 
> examples/apps/datelist-java-main/job.properties  -info  
> 000-170712153835057-oozie-asas-W}}
> {code}
> 000-170712153835057-oozie-asas-W@java1
> RUNNING   application_1499866588585_0001RUNNING- 
> {code}
> I've looked at yarn logs for the launcher and seen that the launcher failed. 
> For example, in my case , during development, oozie-sharelib launcher was not 
> found:  
> {code}
> Error: Could not find or load main class 
> org.apache.oozie.action.hadoop.LauncherAM
> {code}
> The problem is only after the specified timeout (by default 10 minutes) we 
> see that the workflow has actually failed /errored.
> {code}
> Created   : 2017-07-12 13:38 GMT
> Started   : 2017-07-12 13:38 GMT
> Last Modified : 2017-07-12 13:49 GMT
> ...
> 000-170712153835057-oozie-asas-W@java1
> ERROR application_1499866588585_0001FAILED/KILLED- 
> {code} 
> The problem might be that in {{JavaActionExecutor}} in the {{start()}} method 
> the check is too fast.
> {code}
> LOG.debug("Starting action " + action.getId() + " getting Action File 
> System");
> FileSystem actionFs = context.getAppFileSystem();
> LOG.debug("Preparing action Dir through copying " + 
> context.getActionDir());
> prepareActionDir(actionFs, context);
> LOG.debug("Action Dir is ready. Submitting the action ");
> submitLauncher(actionFs, context, action);
> LOG.debug("Action submit completed. Performing check ");
> check(context, action);
> LOG.debug("Action check is done after submission
> {code}
> There should be some delay after {{submitLauncher()}} before {{check()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner

2017-10-25 Thread Denes Bodo (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218807#comment-16218807
 ] 

Denes Bodo commented on OOZIE-2985:
---

[~pbacsko] Am I right that Oozie can provide a notification URL to the job that 
can call back to Oozie? Can we use the same method in LauncherAM to notify when 
the main() starts running?

> If LauncherAM fails, Oozie is not notified in a timely manner
> -
>
> Key: OOZIE-2985
> URL: https://issues.apache.org/jira/browse/OOZIE-2985
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> I've noticed if LauncherAM fails, Oozie is notified about the launcher's 
> failure with a lot of delay. It gives the impression that the workflow is 
> running.
> {{oozie job -oozie http://localhost:11000/oozie -config 
> examples/apps/datelist-java-main/job.properties  -info  
> 000-170712153835057-oozie-asas-W}}
> {code}
> 000-170712153835057-oozie-asas-W@java1
> RUNNING   application_1499866588585_0001RUNNING- 
> {code}
> I've looked at yarn logs for the launcher and seen that the launcher failed. 
> For example, in my case , during development, oozie-sharelib launcher was not 
> found:  
> {code}
> Error: Could not find or load main class 
> org.apache.oozie.action.hadoop.LauncherAM
> {code}
> The problem is only after the specified timeout (by default 10 minutes) we 
> see that the workflow has actually failed /errored.
> {code}
> Created   : 2017-07-12 13:38 GMT
> Started   : 2017-07-12 13:38 GMT
> Last Modified : 2017-07-12 13:49 GMT
> ...
> 000-170712153835057-oozie-asas-W@java1
> ERROR application_1499866588585_0001FAILED/KILLED- 
> {code} 
> The problem might be that in {{JavaActionExecutor}} in the {{start()}} method 
> the check is too fast.
> {code}
> LOG.debug("Starting action " + action.getId() + " getting Action File 
> System");
> FileSystem actionFs = context.getAppFileSystem();
> LOG.debug("Preparing action Dir through copying " + 
> context.getActionDir());
> prepareActionDir(actionFs, context);
> LOG.debug("Action Dir is ready. Submitting the action ");
> submitLauncher(actionFs, context, action);
> LOG.debug("Action submit completed. Performing check ");
> check(context, action);
> LOG.debug("Action check is done after submission
> {code}
> There should be some delay after {{submitLauncher()}} before {{check()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3096) Pre-commit job is timing out regularly

2017-10-25 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218790#comment-16218790
 ] 

Peter Cseh commented on OOZIE-3096:
---

The second job has a failing test, but it did not time out so I'm committing 
the patch.
[~pbacsko], [~andras.piros], [~asasvari], [~satishsaley], [~dbist13], please 
let me know if there are open issues recently where I should retrigger the 
pre-commit job manually.


> Pre-commit job is timing out regularly
> --
>
> Key: OOZIE-3096
> URL: https://issues.apache.org/jira/browse/OOZIE-3096
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3096.001.diff
>
>
> Many of the recent pre-commit runs are running to a timeout.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3096) Pre-commit job is timing out regularly

2017-10-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218739#comment-16218739
 ] 

Hadoop QA commented on OOZIE-3096:
--

Testing JIRA OOZIE-3096

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 2041
.Tests failed: 0
.Tests errors: 1

.The patch failed the following testcases:

.  

.Tests failing with errors:
.  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/132/

> Pre-commit job is timing out regularly
> --
>
> Key: OOZIE-3096
> URL: https://issues.apache.org/jira/browse/OOZIE-3096
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3096.001.diff
>
>
> Many of the recent pre-commit runs are running to a timeout.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3096 PreCommit Build #132

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3096
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/132/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [examples].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
-1 TESTS
Tests run: 2041
Tests failed: 0
Tests errors: 1

The patch failed the following testcases:

  

Tests failing with errors:
  testJMXInstrumentation(org.apache.oozie.util.TestMetricsInstrumentation)

+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/132/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  3209k  0  0:00:01  0:00:01 --:--:-- 3209k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3096
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 64.3% relative to #122
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
1 tests failed.
FAILED:  org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation

Error Message:
Unable to open socket file: target process not responding or HotSpot VM not 
loaded

Stack Trace:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: 
target process not responding or HotSpot VM not loaded
at 
sun.tools.attach.LinuxVirtualMachine.(LinuxVirtualMachine.java:106)
at 
sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:78)
at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:250)
at 
org.apache.oozie.util.TestMetricsInstrumentation.testJMXInstrumentation(TestMetricsInstrumentation.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:255)
at junit.framework.TestSuite.run(TestSuite.java:250)
at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at

[jira] [Commented] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner

2017-10-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218677#comment-16218677
 ] 

Peter Bacsko commented on OOZIE-2985:
-

The problem is you don't always know how much time you have to wait. If it's a 
busy cluster, it could take seconds for the application to start running or 
even more. Until that happens, the application is stuck in ACCEPTED state.

I can imagine a way to work this problem around: on a best effort basis, we 
wait a couple of seconds until it reaches RUNNING state, but we can't block for 
too long. If it does not get scheduled in, let's say, 5 seconds, we give up and 
move on. 

But this is not the kind of solution I'd really implement. Best would be some 
sort of callback mechanism from YARN, but I don't think it's supported.

> If LauncherAM fails, Oozie is not notified in a timely manner
> -
>
> Key: OOZIE-2985
> URL: https://issues.apache.org/jira/browse/OOZIE-2985
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> I've noticed if LauncherAM fails, Oozie is notified about the launcher's 
> failure with a lot of delay. It gives the impression that the workflow is 
> running.
> {{oozie job -oozie http://localhost:11000/oozie -config 
> examples/apps/datelist-java-main/job.properties  -info  
> 000-170712153835057-oozie-asas-W}}
> {code}
> 000-170712153835057-oozie-asas-W@java1
> RUNNING   application_1499866588585_0001RUNNING- 
> {code}
> I've looked at yarn logs for the launcher and seen that the launcher failed. 
> For example, in my case , during development, oozie-sharelib launcher was not 
> found:  
> {code}
> Error: Could not find or load main class 
> org.apache.oozie.action.hadoop.LauncherAM
> {code}
> The problem is only after the specified timeout (by default 10 minutes) we 
> see that the workflow has actually failed /errored.
> {code}
> Created   : 2017-07-12 13:38 GMT
> Started   : 2017-07-12 13:38 GMT
> Last Modified : 2017-07-12 13:49 GMT
> ...
> 000-170712153835057-oozie-asas-W@java1
> ERROR application_1499866588585_0001FAILED/KILLED- 
> {code} 
> The problem might be that in {{JavaActionExecutor}} in the {{start()}} method 
> the check is too fast.
> {code}
> LOG.debug("Starting action " + action.getId() + " getting Action File 
> System");
> FileSystem actionFs = context.getAppFileSystem();
> LOG.debug("Preparing action Dir through copying " + 
> context.getActionDir());
> prepareActionDir(actionFs, context);
> LOG.debug("Action Dir is ready. Submitting the action ");
> submitLauncher(actionFs, context, action);
> LOG.debug("Action submit completed. Performing check ");
> check(context, action);
> LOG.debug("Action check is done after submission
> {code}
> There should be some delay after {{submitLauncher()}} before {{check()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2985) If LauncherAM fails, Oozie is not notified in a timely manner

2017-10-25 Thread Denes Bodo (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218630#comment-16218630
 ] 

Denes Bodo commented on OOZIE-2985:
---

I have experienced the same issue when tried to run the example distcp acrtion. 
Both the ClassNotFoundException for LauncherAM and the long wait.

> If LauncherAM fails, Oozie is not notified in a timely manner
> -
>
> Key: OOZIE-2985
> URL: https://issues.apache.org/jira/browse/OOZIE-2985
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>
> I've noticed if LauncherAM fails, Oozie is notified about the launcher's 
> failure with a lot of delay. It gives the impression that the workflow is 
> running.
> {{oozie job -oozie http://localhost:11000/oozie -config 
> examples/apps/datelist-java-main/job.properties  -info  
> 000-170712153835057-oozie-asas-W}}
> {code}
> 000-170712153835057-oozie-asas-W@java1
> RUNNING   application_1499866588585_0001RUNNING- 
> {code}
> I've looked at yarn logs for the launcher and seen that the launcher failed. 
> For example, in my case , during development, oozie-sharelib launcher was not 
> found:  
> {code}
> Error: Could not find or load main class 
> org.apache.oozie.action.hadoop.LauncherAM
> {code}
> The problem is only after the specified timeout (by default 10 minutes) we 
> see that the workflow has actually failed /errored.
> {code}
> Created   : 2017-07-12 13:38 GMT
> Started   : 2017-07-12 13:38 GMT
> Last Modified : 2017-07-12 13:49 GMT
> ...
> 000-170712153835057-oozie-asas-W@java1
> ERROR application_1499866588585_0001FAILED/KILLED- 
> {code} 
> The problem might be that in {{JavaActionExecutor}} in the {{start()}} method 
> the check is too fast.
> {code}
> LOG.debug("Starting action " + action.getId() + " getting Action File 
> System");
> FileSystem actionFs = context.getAppFileSystem();
> LOG.debug("Preparing action Dir through copying " + 
> context.getActionDir());
> prepareActionDir(actionFs, context);
> LOG.debug("Action Dir is ready. Submitting the action ");
> submitLauncher(actionFs, context, action);
> LOG.debug("Action submit completed. Performing check ");
> check(context, action);
> LOG.debug("Action check is done after submission
> {code}
> There should be some delay after {{submitLauncher()}} before {{check()}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3098) Allow to use the rest api to get the creation time of the Coordinator

2017-10-25 Thread Chi Wen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chi Wen updated OOZIE-3098:
---
Description: 
It seems that the rest api cannot get the creation time of the Coordinator;

RB link: https://reviews.apache.org/r/63296/

  was:
It seems that the rest api cannot get the creation time of the Coordinator;

PR link: https://reviews.apache.org/r/63296/


> Allow to use the rest api to get the creation time of the Coordinator
> -
>
> Key: OOZIE-3098
> URL: https://issues.apache.org/jira/browse/OOZIE-3098
> Project: Oozie
>  Issue Type: Improvement
>  Components: coordinator
>Reporter: Chi Wen
>
> It seems that the rest api cannot get the creation time of the Coordinator;
> RB link: https://reviews.apache.org/r/63296/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 63296: Allow rest api to get the creation time of the Coordinator

2017-10-25 Thread Wen Chi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63296/
---

Review request for oozie.


Bugs: OOZIE-3098
https://issues.apache.org/jira/browse/OOZIE-3098


Repository: oozie-git


Description
---

Allow to use the rest api to get the creation time of the Coordinator


Diffs
-

  client/src/main/java/org/apache/oozie/client/CoordinatorJob.java 
d98a6acc9efb0d787257143eabd4ef95b49cb759 
  client/src/main/java/org/apache/oozie/client/rest/JsonTags.java 
3ec4cca9d4212c9e101cf97b1f2c952bc2a6342c 
  client/src/main/java/org/apache/oozie/client/rest/JsonToBean.java 
4c653920fe234599e05d35fb09f2051c949342ac 
  client/src/test/java/org/apache/oozie/client/rest/TestJsonToBean.java 
a102699253a6a1e5952f830e3704bde3b5df83a8 
  core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 
7dda508d7bf7ad48aee045c370eca2fe06c091fa 


Diff: https://reviews.apache.org/r/63296/diff/1/


Testing
---


Thanks,

Wen Chi

[jira] [Created] (OOZIE-3098) Allow to use the rest api to get the creation time of the Coordinator

2017-10-25 Thread Chi Wen (JIRA) 
Chi Wen created OOZIE-3098:
--

 Summary: Allow to use the rest api to get the creation time of the 
Coordinator
 Key: OOZIE-3098
 URL: https://issues.apache.org/jira/browse/OOZIE-3098
 Project: Oozie
  Issue Type: Improvement
  Components: coordinator
Reporter: Chi Wen


It seems that the rest api cannot get the creation time of the Coordinator;

PR link: https://reviews.apache.org/r/63296/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2644) Skip queuing Notification Commands when there's nothing to notify

2017-10-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218544#comment-16218544
 ] 

Hadoop QA commented on OOZIE-2644:
--

Testing JIRA OOZIE-2644

Cleaning local git workspace



{color:red}-1{color} Patch failed to apply to head of branch



> Skip queuing Notification Commands when there's nothing to notify
> -
>
> Key: OOZIE-2644
> URL: https://issues.apache.org/jira/browse/OOZIE-2644
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 3.1.3
>Reporter: Robert Kanter
>Assignee: Azrael Seoeun
> Fix For: 5.0.0
>
> Attachments: OOZIE-2644.1.patch, OOZIE-2644.2.patch
>
>
> When you use the 
> [Workflow|https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a5_Workflow_Notifications]
>  or 
> [Coordinator|https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a15._Coordinator_Notifications]
>  Notification features, Oozie can end up queuing up a lot of 
> {{WorkflowNotificationXCommand}} and {{CoordActionNotificationXCommand}}.  
> This happens even if there's no notification configured on the job (which I 
> imagine is most of the time); in this case, the {{execute}} method simply 
> does nothing.  This is wasteful and clogs the queue up.  
> We should change the code so that it doesn't queue up one of these Commands 
> unless there's actually a URL to notify.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-2644 PreCommit Build #133

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-2644
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/133/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 680.92 KB...]
Checking patch 
src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java...
error: src/main/java/org/apache/oozie/command/coord/CoordRerunXCommand.java: No 
such file or directory
Checking patch 
src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ActionCheckXCommand.java: No 
such file or directory
Checking patch 
src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ActionEndXCommand.java: No 
such file or directory
Checking patch 
src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ActionKillXCommand.java: No 
such file or directory
Checking patch 
src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ActionStartXCommand.java: No 
such file or directory
Checking patch src/main/java/org/apache/oozie/command/wf/ActionXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ActionXCommand.java: No such 
file or directory
Checking patch src/main/java/org/apache/oozie/command/wf/KillXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/KillXCommand.java: No such 
file or directory
Checking patch src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/ResumeXCommand.java: No such 
file or directory
Checking patch src/main/java/org/apache/oozie/command/wf/SignalXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/SignalXCommand.java: No such 
file or directory
Checking patch src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/SuspendXCommand.java: No such 
file or directory
Checking patch 
src/main/java/org/apache/oozie/command/wf/WorkflowXCommand.java...
error: src/main/java/org/apache/oozie/command/wf/WorkflowXCommand.java: No such 
file or directory
Checking patch 
src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java...
error: 
src/main/java/org/apache/oozie/executor/jpa/CoordActionQueryExecutor.java: No 
such file or directory
Checking patch 
src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java...
error: src/main/java/org/apache/oozie/executor/jpa/CoordJobQueryExecutor.java: 
No such file or directory
Checking patch src/main/java/org/apache/oozie/local/LocalOozie.java...
error: src/main/java/org/apache/oozie/local/LocalOozie.java: No such file or 
directory
Checking patch src/main/resources/oozie-default.xml...
error: src/main/resources/oozie-default.xml: No such file or directory
Checking patch 
src/test/java/org/apache/oozie/command/coord/TestCoordActionNotificationXCommand.java...
error: 
src/test/java/org/apache/oozie/command/coord/TestCoordActionNotificationXCommand.java:
 No such file or directory
Checking patch 
src/test/java/org/apache/oozie/command/coord/TestCoordActionsKillXCommand.java...
error: 
src/test/java/org/apache/oozie/command/coord/TestCoordActionsKillXCommand.java: 
No such file or directory
Checking patch 
src/test/java/org/apache/oozie/command/wf/TestWorkflowNotificationXCommand.java...
error: 
src/test/java/org/apache/oozie/command/wf/TestWorkflowNotificationXCommand.java:
 No such file or directory
Checking patch 
src/test/java/org/apache/oozie/executor/jpa/TestCoordJobQueryExecutor.java...
error: 
src/test/java/org/apache/oozie/executor/jpa/TestCoordJobQueryExecutor.java: No 
such file or directory
Checking patch src/test/java/org/apache/oozie/test/XDataTestCase.java...
error: src/test/java/org/apache/oozie/test/XDataTestCase.java: No such file or 
directory
Patch failed to apply to head of branch

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  0 
00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
100 3706k  100 3706k0 0  2641k  0  0:00:01  0:00:01 --:--:-- 2641k
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-2644
Archiving artifacts
[Fast Archiver] Compressed 839.42 KB of artifacts by 68.6% relative to #122
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for

[jira] [Commented] (OOZIE-3096) Pre-commit job is timing out regularly

2017-10-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218543#comment-16218543
 ] 

Peter Bacsko commented on OOZIE-3096:
-

+1

> Pre-commit job is timing out regularly
> --
>
> Key: OOZIE-3096
> URL: https://issues.apache.org/jira/browse/OOZIE-3096
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3096.001.diff
>
>
> Many of the recent pre-commit runs are running to a timeout.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3096) Pre-commit job is timing out regularly

2017-10-25 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218506#comment-16218506
 ] 

Peter Cseh commented on OOZIE-3096:
---

The first run looks fine.
I'll retrigger the job to have an other run with this config.


> Pre-commit job is timing out regularly
> --
>
> Key: OOZIE-3096
> URL: https://issues.apache.org/jira/browse/OOZIE-3096
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3096.001.diff
>
>
> Many of the recent pre-commit runs are running to a timeout.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3096 PreCommit Build #130

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3096
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/130/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 1.59 MB...]
+1 the patch does not seem to introduce new RAT warnings
+1 JAVADOC
+1 the patch does not seem to introduce new Javadoc warnings
WARNING: the current HEAD has 77 Javadoc warning(s)
+1 COMPILE
+1 HEAD compiles
+1 patch compiles
+1 the patch does not seem to introduce new javac warnings
+1 There are no new bugs found in total.
 +1 There are no new bugs found in [examples].
 +1 There are no new bugs found in [core].
 +1 There are no new bugs found in [sharelib/distcp].
 +1 There are no new bugs found in [sharelib/hive].
 +1 There are no new bugs found in [sharelib/pig].
 +1 There are no new bugs found in [sharelib/spark].
 +1 There are no new bugs found in [sharelib/hive2].
 +1 There are no new bugs found in [sharelib/hcatalog].
 +1 There are no new bugs found in [sharelib/sqoop].
 +1 There are no new bugs found in [sharelib/oozie].
 +1 There are no new bugs found in [sharelib/streaming].
 +1 There are no new bugs found in [tools].
 +1 There are no new bugs found in [docs].
 +1 There are no new bugs found in [server].
 +1 There are no new bugs found in [client].
+1 BACKWARDS_COMPATIBILITY
+1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient 
annotations
+1 the patch does not modify JPA files
+1 TESTS
Tests run: 2041
Tests rerun: 96
Tests failed at first run: 
org.apache.oozie.command.TestPurgeXCommand,org.apache.oozie.service.TestPartitionDependencyManagerEhcache,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
+1 DISTRO
+1 distro tarball builds with the patch 


-1 Overall result, please check the reported -1(s)

 There is at least one warning, please check

The full output of the test-patch run is available at

 https://builds.apache.org/job/PreCommit-OOZIE-Build/130/

  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 3706k0 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0100 3706k  100 3706k0 0  2755k  0  0:00:01  0:00:01 --:--:--  9.9M
Adding comment to JIRA
Comment added.

test-patch exit code: 1

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3096
Archiving artifacts
[Fast Archiver] Compressed 1.80 MB of artifacts by 64.3% relative to #122
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
All tests passed

[jira] [Commented] (OOZIE-3096) Pre-commit job is timing out regularly

2017-10-25 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218502#comment-16218502
 ] 

Hadoop QA commented on OOZIE-3096:
--

Testing JIRA OOZIE-3096

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
.{color:red}WARNING{color}: the current HEAD has 77 Javadoc warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:green}+1{color} There are no new bugs found in total.
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 2041
.Tests rerun: 96
.Tests failed at first run: 
org.apache.oozie.command.TestPurgeXCommand,org.apache.oozie.service.TestPartitionDependencyManagerEhcache,org.apache.oozie.action.hadoop.TestJavaActionExecutor,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}

{color:red}. There is at least one warning, please check{color}

The full output of the test-patch run is available at

. https://builds.apache.org/job/PreCommit-OOZIE-Build/130/

> Pre-commit job is timing out regularly
> --
>
> Key: OOZIE-3096
> URL: https://issues.apache.org/jira/browse/OOZIE-3096
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3096.001.diff
>
>
> Many of the recent pre-commit runs are running to a timeout.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2644) Skip queuing Notification Commands when there's nothing to notify

2017-10-25 Thread Attila Sasvari (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218463#comment-16218463
 ] 

Attila Sasvari commented on OOZIE-2644:
---

Adding a lot of {{CoordActionNotificationXCommand}} or 
{{WorkflowNotificationXCommand}} to the callable queue might also result in 
delay in marking an action completed in the Oozie database. For example: after 
the CallbackServlet receives a GET request with action status=SUCCEEDED (i.e. 
Oozie launcher job has ended and notified the Oozie server), it might take a 
while to persist this information if there are a lot of XCommand-s doing 
basically nothing. In the worst case, some might even fail to meet their SLA-s 
because of this (e.g. next action is not started on time).

> Skip queuing Notification Commands when there's nothing to notify
> -
>
> Key: OOZIE-2644
> URL: https://issues.apache.org/jira/browse/OOZIE-2644
> Project: Oozie
>  Issue Type: Improvement
>Affects Versions: 3.1.3
>Reporter: Robert Kanter
>Assignee: Azrael Seoeun
> Fix For: 5.0.0
>
> Attachments: OOZIE-2644.1.patch, OOZIE-2644.2.patch
>
>
> When you use the 
> [Workflow|https://oozie.apache.org/docs/4.2.0/WorkflowFunctionalSpec.html#a5_Workflow_Notifications]
>  or 
> [Coordinator|https://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a15._Coordinator_Notifications]
>  Notification features, Oozie can end up queuing up a lot of 
> {{WorkflowNotificationXCommand}} and {{CoordActionNotificationXCommand}}.  
> This happens even if there's no notification configured on the job (which I 
> imagine is most of the time); in this case, the {{execute}} method simply 
> does nothing.  This is wasteful and clogs the queue up.  
> We should change the code so that it doesn't queue up one of these Commands 
> unless there's actually a URL to notify.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2897) LauncherAM should support ACLs

2017-10-25 Thread Peter Bacsko (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218455#comment-16218455
 ] 

Peter Bacsko commented on OOZIE-2897:
-

I attached the first version of the change.

Note that in order to test the ACL feature, I had to do the following:
* set Fair Scheduler explicitly in {{XTestCase}}
* define queue allocation policies so that only "test" has admin privileges

The second point is important - if this is not set, all users are considered 
admin and application ACLs have no real effect because it's possible for anyone 
to kill a running applications in the cluster. 

When checking permissions, both queue and application ACLs are taken into 
account: 
https://github.com/apache/hadoop/blob/921338cd86e7215b0c4b1efdf2daf9449fb12c7b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java#L326-L330

I think it's important to mention this in the new documentation.

> LauncherAM should support ACLs
> --
>
> Key: OOZIE-2897
> URL: https://issues.apache.org/jira/browse/OOZIE-2897
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2897-001.patch
>
>
> In MapReduce, you can define ACL-related properties:
> {noformat}
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modify-job
> {noformat}
> {{acl-view-job}} defines a list of users/groups who can retrieve the job 
> statistics. {{acl-modify-job}} defines a list of users/groups who can kill a 
> job or adjust the priority of it.
> Docs: 
> https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization
> In YARN, we can provide backward compatibility for these properties. Example 
> code:
> {code}
> Map acls = new HashMap String>();
> acls.put(ApplicationAccessType.MODIFY_APP, "*");
> acls.put(ApplicationAccessType.VIEW_APP, "*");
> amContainer.setApplicationACLs(acls);
> {code}
> This has to be done before application submission. We have to do what 
> YARNRunner.java does: 
> https://github.com/apache/hadoop/blob/3721cfe1fbd98c5b6aa46aefdfcf62276c28c4a4/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L502-L507
>  
> We should also ensure the compatibility with the properties of:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2897) LauncherAM should support ACLs

2017-10-25 Thread Peter Bacsko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Bacsko updated OOZIE-2897:

Attachment: OOZIE-2897-001.patch

> LauncherAM should support ACLs
> --
>
> Key: OOZIE-2897
> URL: https://issues.apache.org/jira/browse/OOZIE-2897
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2897-001.patch
>
>
> In MapReduce, you can define ACL-related properties:
> {noformat}
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modify-job
> {noformat}
> {{acl-view-job}} defines a list of users/groups who can retrieve the job 
> statistics. {{acl-modify-job}} defines a list of users/groups who can kill a 
> job or adjust the priority of it.
> Docs: 
> https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html#Job+Authorization
> In YARN, we can provide backward compatibility for these properties. Example 
> code:
> {code}
> Map acls = new HashMap String>();
> acls.put(ApplicationAccessType.MODIFY_APP, "*");
> acls.put(ApplicationAccessType.VIEW_APP, "*");
> amContainer.setApplicationACLs(acls);
> {code}
> This has to be done before application submission. We have to do what 
> YARNRunner.java does: 
> https://github.com/apache/hadoop/blob/3721cfe1fbd98c5b6aa46aefdfcf62276c28c4a4/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L502-L507
>  
> We should also ensure the compatibility with the properties of:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3097) Change version in pom in master to 5.0.0-beta1

2017-10-25 Thread Peter Cseh (JIRA) 
Peter Cseh created OOZIE-3097:
-

 Summary: Change version in pom in master to 5.0.0-beta1
 Key: OOZIE-3097
 URL: https://issues.apache.org/jira/browse/OOZIE-3097
 Project: Oozie
  Issue Type: Bug
Affects Versions: 5.0.0b1
Reporter: Peter Cseh
Priority: Blocker
 Fix For: 5.0.0b1


In preparation to the 5b1 release we should modify the version in the POM files 
to 5.0.0-beta1.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-${ENV, var="ISSUE_NUM"} PreCommit Build #${BUILD_NUMBER}

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-${ENV, var="ISSUE_NUM"}
Build: ${BUILD_URL}

###
## LAST 60 LINES OF THE CONSOLE 
###
${BUILD_LOG, maxLines=60}


###
## FAILED TESTS (if any) 
##
${FAILED_TESTS}

[jira] [Updated] (OOZIE-2975) code clean up in pig sharelib, replace Exception with more explicit, add try with resources, StringBuilder instead of StringBuffer

2017-10-25 Thread Peter Cseh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2975:
--
Fix Version/s: (was: 5.0.0)
   trunk

> code clean up in pig sharelib, replace Exception with more explicit, add try 
> with resources, StringBuilder instead of StringBuffer
> --
>
> Key: OOZIE-2975
> URL: https://issues.apache.org/jira/browse/OOZIE-2975
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
>  Labels: newbie
> Fix For: trunk
>
> Attachments: OOZIE-2975-0.patch
>
>
> 1. Replaced StringBuffer with StringBuilder
> 2. Replaced Exception with IOException and ExecException
> 3. Replaced old Hashmap notation with diamond notation
> 4. Added try-with-resources notation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.12.0

2017-10-25 Thread Peter Cseh (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218370#comment-16218370
 ] 

Peter Cseh commented on OOZIE-2231:
---

Curator 2.12.0 is 
[released|https://cwiki.apache.org/confluence/display/CURATOR/Releases#Releases-March7,2017,Releases2.12.0and3.3.0available]
 [~abhishekbafna], can you update your patch or can somebody take over?

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Abhishek Bafna
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2966) OozieCLI and TestOozieCLI test class are in different packages

2017-10-25 Thread Peter Cseh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-2966:
--
Fix Version/s: (was: 5.0.0)

> OozieCLI and TestOozieCLI test class are in different packages
> --
>
> Key: OOZIE-2966
> URL: https://issues.apache.org/jira/browse/OOZIE-2966
> Project: Oozie
>  Issue Type: Task
>  Components: tests
>Affects Versions: 4.3.0
>Reporter: Artem Ervits
>Assignee: Artem Ervits
>Priority: Trivial
> Fix For: trunk
>
>
> OozieCLI class is in client/org/apache/oozie/cli whereas the associated test 
> class resides in core/org/apache/oozie/client shouldn't they belong to same 
> project?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Oozie 5 beta

2017-10-25 Thread Peter Cseh 
Hi everyone!

I would like to release an Oozie 5 beta 1 in early November. I hope it will
get some testing and bugfixes in before we release Oozie 5.0.
There are still some open tasks I would like to wait for like OOZIE-2896
 for backward
compatibility and OOZIE-2973
 for Hadoop3 support.
Can I create a 5.0.0b1 fixversion in Jira and assing some tasks accordingly?
Any ideas what issues are must-dos before the beta?

Thanks in advance
gp

[jira] [Updated] (OOZIE-3096) Pre-commit job is timing out regularly

2017-10-25 Thread Peter Cseh (JIRA) 

 [ 
https://issues.apache.org/jira/browse/OOZIE-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Cseh updated OOZIE-3096:
--
Attachment: OOZIE-3096.001.diff

I'll try to increase the memory in the surefire runs. 1G might be not enough

> Pre-commit job is timing out regularly
> --
>
> Key: OOZIE-3096
> URL: https://issues.apache.org/jira/browse/OOZIE-3096
> Project: Oozie
>  Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Peter Cseh
> Attachments: OOZIE-3096.001.diff
>
>
> Many of the recent pre-commit runs are running to a timeout.
> We should fix that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Failed: OOZIE-3095 PreCommit Build #129

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-3095
Build: https://builds.apache.org/job/PreCommit-OOZIE-Build/129/

###
## LAST 60 LINES OF THE CONSOLE 
###
Started by remote host 127.0.0.1
[EnvInject] - Loading node environment variables.
Building remotely on H11 (Hadoop xenial) in workspace 
/home/jenkins/jenkins-slave/workspace/PreCommit-OOZIE-Build
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 883c7556d2f302261eae5aec6e323a5b1de04f74 
(refs/remotes/origin/master)
Commit message: "OOZIE-2406 Completely rewrite GraphGenerator code 
(andras.piros via gezapeti)"
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 883c7556d2f302261eae5aec6e323a5b1de04f74
 > git rev-list 883c7556d2f302261eae5aec6e323a5b1de04f74 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
No emails were triggered.
[PreCommit-OOZIE-Build] $ /bin/bash /tmp/jenkins6262202861611281450.sh
OOZIE-3095
rm: cannot remove 'distro/downloads/*': No such file or directory
rm: invalid option -- '.'
Try 'rm --help' for more information.

Invalid option

Usage: bin/test-patch 
  (--jira= | --patch=)
  (--reset-scm | --dirty-scm)
  [--tasks=]
  [--skip-tasks=]
  [--jira-cli=]
  [--jira-user=]
  [--jira-password=]
  [-D...]
  [-P...]
  [--list-tasks]
  [--verbose]

Build step 'Execute shell' marked build as failure
[description-setter] Description set: OOZIE-3095
Archiving artifacts
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###
## FAILED TESTS (if any) 
##
No tests ran.

Failed: OOZIE-${ENV, var="ISSUE_NUM"} PreCommit Build #${BUILD_NUMBER}

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-${ENV, var="ISSUE_NUM"}
Build: ${BUILD_URL}

###
## LAST 60 LINES OF THE CONSOLE 
###
${BUILD_LOG, maxLines=60}


###
## FAILED TESTS (if any) 
##
${FAILED_TESTS}

Failed: OOZIE-${ENV, var="ISSUE_NUM"} PreCommit Build #${BUILD_NUMBER}

2017-10-25 Thread Apache Jenkins Server 
Jira: https://issues.apache.org/jira/browse/OOZIE-${ENV, var="ISSUE_NUM"}
Build: ${BUILD_URL}

###
## LAST 60 LINES OF THE CONSOLE 
###
${BUILD_LOG, maxLines=60}


###
## FAILED TESTS (if any) 
##
${FAILED_TESTS}

[jira] Subscription: Oozie Patch Available

2017-10-25 Thread jira 
Issue Subscription
Filter: Oozie Patch Available (109 issues)

Subscriber: ooziedaily

Key Summary
OOZIE-3095  Job submit command is not logged in audit log
https://issues.apache.org/jira/browse/OOZIE-3095
OOZIE-3094  fix for grammar mistake
https://issues.apache.org/jira/browse/OOZIE-3094
OOZIE-3084  Add missing JVM properties to Jetty startup script
https://issues.apache.org/jira/browse/OOZIE-3084
OOZIE-3083  Make improved version Info backward compatible
https://issues.apache.org/jira/browse/OOZIE-3083
OOZIE-3071  Oozie 4.3 Spark sharelib ueses a different version of commons-lang3 
than Spark 2.2.0
https://issues.apache.org/jira/browse/OOZIE-3071
OOZIE-3063  Sanitizing variables that are part of openjpa.ConnectionProperties
https://issues.apache.org/jira/browse/OOZIE-3063
OOZIE-3062  Set HADOOP_CONF_DIR for spark action
https://issues.apache.org/jira/browse/OOZIE-3062
OOZIE-3022  fix for warning has no file and won't be listed in dependency files 
details
https://issues.apache.org/jira/browse/OOZIE-3022
OOZIE-3002  address findbugs errors in client lib
https://issues.apache.org/jira/browse/OOZIE-3002
OOZIE-3001  core library has many instances of warnings with trailing spaces 
and lines longer than 132 chars
https://issues.apache.org/jira/browse/OOZIE-3001
OOZIE-2997  files contain trailing white spaces in client lib
https://issues.apache.org/jira/browse/OOZIE-2997
OOZIE-2996  add option for -UseGCOverheadLimit to maven opts as sometimes local 
testing fails
https://issues.apache.org/jira/browse/OOZIE-2996
OOZIE-2975  code clean up in pig sharelib, replace Exception with more 
explicit, add try with resources, StringBuilder instead of StringBuffer
https://issues.apache.org/jira/browse/OOZIE-2975
OOZIE-2969  Drop support for Java 1.7
https://issues.apache.org/jira/browse/OOZIE-2969
OOZIE-2964  Add -Xdoclint:none to javadoc opts to avoid warnings
https://issues.apache.org/jira/browse/OOZIE-2964
OOZIE-2962  bump maven-javadoc-plugin to 2.10.4
https://issues.apache.org/jira/browse/OOZIE-2962
OOZIE-2957  Documentation states that starting a coordinator is possible
https://issues.apache.org/jira/browse/OOZIE-2957
OOZIE-2956  Fix Findbugs warnings related to reliance on default encoding in 
oozie-core
https://issues.apache.org/jira/browse/OOZIE-2956
OOZIE-2955  Fix Findbugs warnings related to reliance on default encoding in 
oozie-client
https://issues.apache.org/jira/browse/OOZIE-2955
OOZIE-2954  Fix Checkstyle issues in oozie-client
https://issues.apache.org/jira/browse/OOZIE-2954
OOZIE-2953  Fix Checkstyle issues in oozie-tools
https://issues.apache.org/jira/browse/OOZIE-2953
OOZIE-2952  Fix Findbugs warnings in oozie-sharelib-oozie
https://issues.apache.org/jira/browse/OOZIE-2952
OOZIE-2949  Escape quotes whitespaces in Sqoop  field
https://issues.apache.org/jira/browse/OOZIE-2949
OOZIE-2942  Fix Findbugs warnings in oozie-examples
https://issues.apache.org/jira/browse/OOZIE-2942
OOZIE-2937  Remove redundant groupId from the child pom's
https://issues.apache.org/jira/browse/OOZIE-2937
OOZIE-2934  Fix "Exceptional return value of java.io.File.mkdirs() ignored" 
Findbugs error in oozie-sharelib-spark
https://issues.apache.org/jira/browse/OOZIE-2934
OOZIE-2927  Append new line character for Hive2 query using query tag
https://issues.apache.org/jira/browse/OOZIE-2927
OOZIE-2914  Consolidate Trim 
https://issues.apache.org/jira/browse/OOZIE-2914
OOZIE-2896  Ensure compatibility for existing LauncherMapper settings
https://issues.apache.org/jira/browse/OOZIE-2896
OOZIE-2883  OOZIE throw the error "Missing 
[oozie.service.ProxyUserService.proxyuser.oozie.service.ProxyUserService.proxyuser.mr.groups]
 property"
https://issues.apache.org/jira/browse/OOZIE-2883
OOZIE-2877  Oozie Git Action
https://issues.apache.org/jira/browse/OOZIE-2877
OOZIE-2867  Timezone handling for Coordinators: emphasize "Continent/City" 
format
https://issues.apache.org/jira/browse/OOZIE-2867
OOZIE-2834  ParameterVerifier logging non-useful warning for workflow definition
https://issues.apache.org/jira/browse/OOZIE-2834
OOZIE-2833  when using uber mode the regex pattern used in the 
extractHeapSizeMB method does not allow heap sizes specified in bytes.
https://issues.apache.org/jira/browse/OOZIE-2833
OOZIE-2829  Improve sharelib upload to accept multiple source folders
https://issues.apache.org/jira/browse/OOZIE-2829
OOZIE-2826  Falcon feed fails to aws s3; Oozie joda time version does not meet 
required jar version 2.2 or later
https://issues.apache.org/jira/browse/OOZIE-2826
OOZIE-2812