Re: Questions about next Oozie release

2017-04-26 Thread Dongying Jiao 
Hi Peter:
Thanks very much for your detail information.

Best Regards,
Dongying Jiao

2017-04-26 16:21 GMT+08:00 Peter Bacsko :

> Hi Dongying,
>
> Yes, the next release will be 5.0.
>
> We don't have definite dates yet, in fact, I don't think we eve have rough
> dates. But the release should happen this year, sometime around sept-oct.
> IMO it's feasible.
> OYA will definitely be included in 5.0 and it's currently being reviewed.
> I'd say it's on track.
>
> Another thing that we have to agree on inside the Oozie community is the
> support of 4.3.0 and 5.0. Since OYA introduces a significant change to
> Oozie and certain things will be removed, we probably need to support both
> but there's no consensus at the moment about how we'll accomplish this,
> perhaps we have to follow the Hadoop-way. But it's not yet decided.
>
> Peter
>
> On Wed, Apr 26, 2017 at 9:50 AM, Dongying Jiao 
> wrote:
>
> > Hi:
> > Will the next Oozie release be 5.0.0? If yes, whether the OYA feature
> will
> > also be included in the release. And is there any plan on the release
> date?
> > Thank you very much.
> >
> > Best Regards,
> > Dongying Jiao
> >
>

[jira] [Commented] (OOZIE-2829) Improve sharelib upload to accept multiple source folders

2017-04-26 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985330#comment-15985330
 ] 

Hadoop QA commented on OOZIE-2829:
--

Testing JIRA OOZIE-2829

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 2 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:red}-1{color} There are [1] new bugs found below threshold in total that 
must be fixed.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [core].
. {color:red}-1{color} There are [1] new bugs found below threshold in [tools] 
that must be fixed.
. You can find the FindBugs diff here (look for the red and orange ones): 
tools/findbugs-new.html
. The most important FindBugs errors are:
. At OozieSharelibCLI.java:[line 169]: 
org.apache.oozie.tools.OozieSharelibCLI.run(String[]) invokes System.exit(...), 
which shuts down the entire virtual machine
. {color:green}+1{color} There are no new bugs found in [examples].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1933
.Tests failed: 1
.Tests errors: 2

.The patch failed the following testcases:

.  testNofindLogs(org.apache.oozie.util.TestTimestampedMessageParser)

.Tests failing with errors:
.  
testProcessRemainingLog(org.apache.oozie.util.TestTimestampedMessageParser)
.  
testProcessRemainingCoordinatorLogForActions(org.apache.oozie.util.TestTimestampedMessageParser)

{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:red}*-1 Overall result, please check the reported -1(s)*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3794/

> Improve sharelib upload to accept multiple source folders
> -
>
> Key: OOZIE-2829
> URL: https://issues.apache.org/jira/browse/OOZIE-2829
> Project: Oozie
>  Issue Type: Bug
>  Components: tools
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
> Attachments: OOZIE-2829-01.patch, OOZIE-2829-02.patch
>
>
> Right now sharelib can be created via {{sharelib create -fs FS_URI -locallib 
> SHARED_LIBRARY}} where the SHARED_LIBRARY can be a tarbal or a folder. 
> It would be nice to have the possibility to define additional folders to be 
> uploaded into the sharelib, so the users don't have to copy or link the files 
> together on their machine.
> The syntax could be something like -additional-lib 
> sharelibName=/path/to/source/;/path/to/some/file,sharelibName2=/path/to/some/folder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: oozie-trunk-precommit-build #3794

2017-04-26 Thread Apache Jenkins Server 
See 


Changes:

[satishsaley] OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in 
server

[gezapeti] OOZIE-2851 spelling mistakes in examples (Artem Ervits via gezapeti)

--
[...truncated 969.56 KB...]
[INFO] Copying jersey-server-1.9.jar to 

[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-server ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/empty.xml
[INFO] 
[INFO] 
[INFO] Building Apache Oozie Distro 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ oozie-distro 
---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ oozie-distro 
---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-distro ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-distro ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ oozie-distro ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ oozie-distro ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:2.0-beta-6:attach-descriptor (attach-descriptor) @ 
oozie-distro ---
[INFO] 
[INFO] --- maven-assembly-plugin:2.2.1:single (default-cli) @ oozie-distro ---
[INFO] Reading assembly descriptor: ../src/main/assemblies/distro.xml
[INFO] Copying files to 

[WARNING] Assembly file: 

 is not a regular file (it may be a directory). It cannot be attached to the 
project build for installation or deployment.
[INFO] Building tar : 

[INFO] 
[INFO] 
[INFO] Building Apache Oozie ZooKeeper Security Tests 4.4.0-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
oozie-zookeeper-security-tests ---
[INFO] 
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ 
oozie-zookeeper-security-tests ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) @ 
oozie-zookeeper-security-tests ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 

[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ 
oozie-zookeeper-security-tests ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.2:test (default-test) @ 
oozie-zookeeper-security-tests ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ 
oozie-zookeeper-security-tests ---
[INFO] Building jar: 

[INFO]

[jira] [Commented] (OOZIE-2751) LocalOozieClient is missing methods from OozieClient

2017-04-26 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/OOZIE-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985315#comment-15985315
 ] 

Hadoop QA commented on OOZIE-2751:
--

Testing JIRA OOZIE-2751

Cleaning local git workspace



{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than 
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT 
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc 
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac 
warnings
{color:orange}0{color} There are [6] new bugs found in total that would be nice 
to have fixed.
. {color:green}+1{color} There are no new bugs found in [server].
. {color:green}+1{color} There are no new bugs found in [client].
. {color:orange}0{color} There are [6] new bugs found in [core] that would be 
nice to have fixed.
. You can find the FindBugs diff here: core/findbugs-new.html
. {color:green}+1{color} There are no new bugs found in [docs].
. {color:green}+1{color} There are no new bugs found in 
[hadooplibs/hadoop-utils-2].
. {color:green}+1{color} There are no new bugs found in [tools].
. {color:green}+1{color} There are no new bugs found in [examples].
. {color:green}+1{color} There are no new bugs found in [sharelib/streaming].
. {color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
. {color:green}+1{color} There are no new bugs found in [sharelib/distcp].
. {color:green}+1{color} There are no new bugs found in [sharelib/oozie].
. {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive].
. {color:green}+1{color} There are no new bugs found in [sharelib/hive2].
. {color:green}+1{color} There are no new bugs found in [sharelib/pig].
. {color:green}+1{color} There are no new bugs found in [sharelib/spark].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA 
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 1933
.Tests rerun: 15
.Tests failed at first run: org.apache.oozie.jms.TestJMSJobEventListener,
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch 


{color:green}*+1 Overall result, good!, no -1s*{color}


The full output of the test-patch run is available at

. https://builds.apache.org/job/oozie-trunk-precommit-build/3793/

> LocalOozieClient is missing methods from OozieClient
> 
>
> Key: OOZIE-2751
> URL: https://issues.apache.org/jira/browse/OOZIE-2751
> Project: Oozie
>  Issue Type: Bug
>Reporter: Robert Kanter
>Assignee: Abhishek Bafna
> Fix For: 5.0.0
>
> Attachments: OOZIE-2751-00.patch, OOZIE-2751-01.patch, 
> OOZIE-2751-02.patch, OOZIE-2751-03.patch, OOZIE-2751-04.patch, 
> OOZIE-2751-05.patch, OOZIE-2751-06.patch, OOZIE-2751-07.patch, 
> OOZIE-2751-08.patch
>
>
> {{LocalOozieClient}} works by wrapping direct calls to {{DagEngine}} in the 
> normal {{OozieClient}} methods instead of making REST calls to an Oozie 
> Server.  However, it currently doesn't override all methods, which means you 
> hit the {{OozieClient}} implementation, which tries to make a REST call and 
> will fail (because it's not setup for that).  For example, {{getStatus(String 
> jobId)}} isn't overridden, and doesn't work when used from 
> {{LocalOozieClient}}.
> We should also create a test that verifies that all methods are overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : oozie-trunk-precommit-build #3793

2017-04-26 Thread Apache Jenkins Server 
See

Jenkins build is back to normal : oozie-trunk-find-patches-available #156832

2017-04-26 Thread Apache Jenkins Server 
See

[jira] [Created] (OOZIE-2870) non working examples in oozie documentation coordinator spec

2017-04-26 Thread Clemens Valiente (JIRA) 
Clemens Valiente created OOZIE-2870:
---

 Summary: non working examples in oozie documentation coordinator 
spec
 Key: OOZIE-2870
 URL: https://issues.apache.org/jira/browse/OOZIE-2870
 Project: Oozie
  Issue Type: Bug
  Components: docs
Affects Versions: 4.3.0
Reporter: Clemens Valiente


in the chapters:
4.4.1.2. The coord:endOfDays(int n) EL function
https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.html#a4.4.1.2._The_coord:endOfDaysint_n_EL_function
4.4.2.2. The coord:endOfMonths(int n) EL function
https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.html#a4.4.2.2._The_coord:endOfMonthsint_n_EL_function
there is a hello world coordinator example:
{code:xml}

  
10
${concurrency_level}
${execution_order}
${materialization_throttle}
  
...{code}

The throttle element does not exist for the oozie:coordinator specification 0.1 
so the version should probably be updated here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [Coordinators] Timezone handling during job materialization

2017-04-26 Thread Peter Bacsko 
Valid point Andras.

It would be great to understand the rationale behind all this. To me it's
extremely counter-intuitive why the timezone doesn't work as it should (eg.
use the actual TZ defined there, not take it from a so-called "processing
timezone").
I'm sure it has an explanation though.

I can imagine an improvement there - we can add an extra attribute to the
XML like ignoreProcessingTimeZone="true" to preserve backward
compatibility. This would actually solve DST related problem described in
OOZIE-2494. I've created a POC for that JIRA but I don't really like that.
Handling DST changes like that feels very hacky to me.

Robert Kanter, Abhishek, Puru, Rohini? What do you guys think?

Peter

On Wed, Apr 12, 2017 at 4:21 PM, Andras Piros 
wrote:

> Hi there,
>
> as we all know the coordinator jobs  are materialized, when frequency is
> given w/ a Cron expression, regardless of timezone attribute
>  main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCom
> mand.java#L471>,
> considering only oozie.processing.timezone
>  main/java/org/apache/oozie/command/coord/CoordCommandUtils.java#L801>
> configuration entry. This is often very confusing for users.
>
> I think following can be done:
>
>- in 4.4.0 timeframe: extend the Coordinator Functional Specification by
>mentioning this fact
>- in 5.0.0 timeframe:
>   - alter functionality that timezone attribute is also considered when
>   frequency is Cron based
>   - make this behavior configurable
>   - make this behavior switched off by default, in order all
> previous/> instances continue working
>
> What are your thoughts?
>
> Andras
>

Re: Questions about next Oozie release

2017-04-26 Thread Peter Bacsko 
Hi Dongying,

Yes, the next release will be 5.0.

We don't have definite dates yet, in fact, I don't think we eve have rough
dates. But the release should happen this year, sometime around sept-oct.
IMO it's feasible.
OYA will definitely be included in 5.0 and it's currently being reviewed.
I'd say it's on track.

Another thing that we have to agree on inside the Oozie community is the
support of 4.3.0 and 5.0. Since OYA introduces a significant change to
Oozie and certain things will be removed, we probably need to support both
but there's no consensus at the moment about how we'll accomplish this,
perhaps we have to follow the Hadoop-way. But it's not yet decided.

Peter

On Wed, Apr 26, 2017 at 9:50 AM, Dongying Jiao 
wrote:

> Hi:
> Will the next Oozie release be 5.0.0? If yes, whether the OYA feature will
> also be included in the release. And is there any plan on the release date?
> Thank you very much.
>
> Best Regards,
> Dongying Jiao
>

Build failed in Jenkins: oozie-trunk-find-patches-available #156831

2017-04-26 Thread Apache Jenkins Server 
See 


Changes:

[pbacsko] OOZIE-2864 Maven artifacts for package com.codahale.metrics have

[pbacsko] OOZIE-2827 More directly view of the coordinator’s history from

[pbacsko] OOZIE-2818 Can't overwrite oozie.action.max.output.data on a

[pbacsko] OOZIE-2843 Enhance logging inside ZKLocksService and 
MemoryLocksService

[satishsaley] OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in 
server

[gezapeti] OOZIE-2851 spelling mistakes in examples (Artem Ervits via gezapeti)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on H1 (Hadoop) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/oozie.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/oozie.git
 > git --version # timeout=10
 > git fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/oozie.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision 59a8b407da93784f38266515e2e1e06d249e8a40 
(refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 59a8b407da93784f38266515e2e1e06d249e8a40
 > git rev-list cf3b400a2a8e66f8689856bfb16b2547526a012a # timeout=10
[oozie-trunk-find-patches-available] $ /bin/bash 
/tmp/hudson4437223013645206547.sh
mkdir: cannot create directory 
':
 File exists
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0  
0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0100 
 671k0  671k0 0   428k  0 --:--:--  0:00:01 --:--:--  428k100 
1335k0 1335k0 0   532k  0 --:--:--  0:00:02 --:--:--  532k100 
1703k0 1703k0 0   519k  0 --:--:--  0:00:03 --:--:--  519k
curl: (18) transfer closed with outstanding read data remaining
Could not retrieve available patches from JIRA
Build step 'Execute shell' marked build as failure

Questions about next Oozie release

2017-04-26 Thread Dongying Jiao 
Hi:
Will the next Oozie release be 5.0.0? If yes, whether the OYA feature will
also be included in the release. And is there any plan on the release date?
Thank you very much.

Best Regards,
Dongying Jiao