Re: Why does Oozie need a Loadbalancer for High Availability?

[Robert Kanter]

There's really only the REST API - the CLI actually just makes REST calls.
The REST API is fully documented in the Oozie docs (except for the callback
because it's not a user-facing API).

I'm not actually that familiar with virtual IP - I put that in there
because someone else mentioned it could work as an alternative to the load
balancer.  My understanding is that it's essentially like using a load
balancer but without extra hardware because the DNS server handles the
routing.  Basically, (at least the way things work now), we need a single
address for the Oozie server, and anything that can accomplish that task
will work.

That said, I believe you can technically get away with not using a load
balancer (or equivalent) and simply having all callbacks and API calls go
to a single Oozie server.  The other Oozie servers would still process
workflows and do things correctly, but the "main" one would be doing most
of the work so it would be very unbalanced.  Though you could mitigate some
of that by telling different users to use different Oozie server
addresses.  You'd also lose the benefit of High Availability because if the
"main" Oozie server goes down, then nobody will know how to contact the
other Oozie servers.

- Robert

On Mon, Nov 4, 2019 at 10:11 AM Poepping, Thomas 
wrote:

> The callouts about Oozie launcher and REST API calls are good ones -- I
> only ever interact with Oozie through the Oozie CLI. Is there any one place
> where all supported ways of interacting with Oozie server are documented?
>
> Robert, does virtual IP (as suggested in the doc) solve the issue you
> speak of? The proxy in Oozie client could round robin between three Virtual
> IPs, and additional Oozie servers could be behind those Virtual IPs? That
> becomes an issue of integration then, rather than implementation on Oozie's
> part.
>
> On 11/4/19, 8:42 AM, "Robert Kanter"  wrote:
>
> In addition to the reasons both Andras's mentioned, another reason is
> that
> the client doesn't need to know all of the Oozie server addresses.
> While
> you can update the Oozie client config on your laptop if it worked that
> way, once the Oozie Launcher has started, you can't update the Oozie
> servers it knows about.  For example, suppose you had an Oozie launcher
> that ran for 3 days - you may have added/removed some Oozie servers in
> that
> time and now the Oozie Launcher's list of Oozie servers would be out of
> date.
>
> - Robert
>
> On Mon, Nov 4, 2019 at 4:01 AM Andras Piros  wrote:
>
> > Another point to add is that there are lots of users accessing Oozie
> not
> > via OozieCLI but via direct REST calls. As to my understanding the
> proxied
> > client story could work for OozieCLI only. In any case, it could
> make sense
> > to implement it that way.
> >
> > Regards,
> >
> > Andras
> >
> > On Mon, Nov 4, 2019 at 12:56 PM Andras Salamon
> >  wrote:
> >
> > > Hi,
> > >
> > > HA was added a long time ago, back in 2013. You can find the jira
> here:
> > > https://issues.apache.org/jira/browse/OOZIE-615. There is a
> design docs
> > > attached to the jira, which could be a good starting point. There
> is a
> > > section int it about the Load Balancer:
> > >
> > > "A loadbalancer, virtualIP, or DNSroundrobin: This would go in
> front of
> > the
> > > Oozie servers to (a) provide a single entry point for users so
> they don’t
> > > have to choose between, or even beaware of, multiple Oozie
> servers; and
> > (b)
> > > for callbacks from the JobTracker when a hadoop job is done (which
> can
> > only
> > > take a single address and simply choosing an arbitrary Oozie
> server could
> > > be a problem if that server goes down."
> > >
> > > Best,
> > > Sala
> > >
> > >
> > > On Sat, Nov 2, 2019 at 12:13 AM Poepping, Thomas
> >  > > >
> > > wrote:
> > >
> > > > Hi Oozie development community!
> > > >
> > > > I am looking through documentation for the Oozie High
> Availability
> > > feature
> > > > (
> > >
> https://oozie.apache.org/docs/5.1.0/AG_Install.html#High_Availability_HA
> > > > ) and I am wondering why we need to set up virtual IP or load
> balancing
> > > for
> > > > callbacks from Resource Manager to Oozie? YARN follows a
> different
> > > > convention – including a proxied client that round robins
> between DNS
> > > names
> > > > configured in a list. Is there something blocking Oozie from
> doing the
> > > > same, or was this decision made because it also provides users
> with a
> > > > single endpoint to hit any of the oozie servers running?
> > > >
> > > > If there aren’t strong arguments against, I would like to open a
> JIRA
> > to
> > > > implement this. But first, please give me your comments!
> > > >
> > > > Thanks,
> > > > Tom
> > > >
> > >
> >
>
>
>

Re: Why does Oozie need a Loadbalancer for High Availability?

[Robert Kanter]

In addition to the reasons both Andras's mentioned, another reason is that
the client doesn't need to know all of the Oozie server addresses.  While
you can update the Oozie client config on your laptop if it worked that
way, once the Oozie Launcher has started, you can't update the Oozie
servers it knows about.  For example, suppose you had an Oozie launcher
that ran for 3 days - you may have added/removed some Oozie servers in that
time and now the Oozie Launcher's list of Oozie servers would be out of
date.

- Robert

On Mon, Nov 4, 2019 at 4:01 AM Andras Piros  wrote:

> Another point to add is that there are lots of users accessing Oozie not
> via OozieCLI but via direct REST calls. As to my understanding the proxied
> client story could work for OozieCLI only. In any case, it could make sense
> to implement it that way.
>
> Regards,
>
> Andras
>
> On Mon, Nov 4, 2019 at 12:56 PM Andras Salamon
>  wrote:
>
> > Hi,
> >
> > HA was added a long time ago, back in 2013. You can find the jira here:
> > https://issues.apache.org/jira/browse/OOZIE-615. There is a design docs
> > attached to the jira, which could be a good starting point. There is a
> > section int it about the Load Balancer:
> >
> > "A loadbalancer, virtualIP, or DNSroundrobin: This would go in front of
> the
> > Oozie servers to (a) provide a single entry point for users so they don’t
> > have to choose between, or even beaware of, multiple Oozie servers; and
> (b)
> > for callbacks from the JobTracker when a hadoop job is done (which can
> only
> > take a single address and simply choosing an arbitrary Oozie server could
> > be a problem if that server goes down."
> >
> > Best,
> > Sala
> >
> >
> > On Sat, Nov 2, 2019 at 12:13 AM Poepping, Thomas
>  > >
> > wrote:
> >
> > > Hi Oozie development community!
> > >
> > > I am looking through documentation for the Oozie High Availability
> > feature
> > > (
> > https://oozie.apache.org/docs/5.1.0/AG_Install.html#High_Availability_HA
> > > ) and I am wondering why we need to set up virtual IP or load balancing
> > for
> > > callbacks from Resource Manager to Oozie? YARN follows a different
> > > convention – including a proxied client that round robins between DNS
> > names
> > > configured in a list. Is there something blocking Oozie from doing the
> > > same, or was this decision made because it also provides users with a
> > > single endpoint to hit any of the oozie servers running?
> > >
> > > If there aren’t strong arguments against, I would like to open a JIRA
> to
> > > implement this. But first, please give me your comments!
> > >
> > > Thanks,
> > > Tom
> > >
> >
>

Re: [VOTE] Release Oozie 5.1.0 (candidate 0)

[Robert Kanter]

Thanks for rolling an RC Andras!

Here's what I did:
- Verified sha512
- Verified signature (gpg)
- Looked at rat report
- Looked at release log
- Built Oozie against Hadoop 2.7.2, using an empty local maven repo
- Ran a few of the examples, clicked around the UI, tried various CLI
commands, looked at the REST output, launcher log output, etc
- Looked at the new docs
- Tried out the Fluent Job API from one of the examples

I did find one issue:
- The Fluent Job API docs page (docs/DG_FluentJobAPI.html) isn't linked to
from the index page.  I only found it at the bottom of the Oozie Examples
page (docs/DG_Examples.html).  Given that this is supposed to be a big
feature for 5.1.0, I suggest we add it to the index page and do an RC1.


On Thu, Oct 25, 2018 at 7:32 AM Andras Piros  wrote:

> Hi there,
>
> I have created a build for Oozie 5.1.0, release candidate 0.
>
> This release introduces following major changes and fixes:
> - Fluent Job API
> - Git action
> - reduced heap waste
> - switched documentation from TWiki to Markdown
> - callable queue improvements
> - Spark action fixes
> - SLA improvements
> - third party dependency revamp
> - fixed CVE-2018-11799
>
> Keys to verify the signature of the release artifact are available at
>
>  http://www.apache.org/dist/oozie/KEYS
>
> Please download, test, and try it out:
>
> http://people.apache.org/~andras/oozie-5.1.0-rc0/
>
> The release candidate, sha512 signature, gpg signature, and rat report can
> all be found
> at the above address.
>
> The vote is open until Tuesday, October 30, 2018 at 12:00am PDT.
>
> Thanks,
>
> Andras Piros
>

Re: Kerberos and delegation token

[Robert Kanter]

Oops: ignore this - wrong list.

On Tue, Oct 16, 2018 at 1:18 PM Robert Kanter  wrote:

> Hi Bala,
>
> It sounds like you'll want to use an HBase Delegation Token within the
> Hive UDF to connect to HBase.  The user will need to obtain an HBase
> Delegation Token with their Kerberos Credentials and either give the token
> to Hive or Hive will need to get the token on behalf of the user (like
> Oozie).  The code Oozie uses for getting an HBase token is here:
> https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java
>
> Let me know if you need more details.  I've also added some oozie lists,
> for people who have used this more recently in case they have any
> additional info.
>
> - Robert
>
> On Tue, Oct 16, 2018 at 1:04 PM Bala Chander 
> wrote:
>
>> Hello Robert,
>>
>> I have a customer who is writing a UDF (user defined function) in java
>> for usage in hive/beeline.
>> The UDF contacts HBASE to pull in some data and returns back a single
>> value. The USER of hive/beeline can issue a statement like the following:
>>
>> select a, UDF("xyz") from table_a;
>>
>> Hive/beeline and HBase operate on kerberos secure cluster.
>>
>> Michael Yoder (cc'ed) mentioned something similar in Oozie (Delegation
>> token based connection for Oozie (OOZIE-1457
>> <https://issues.apache.org/jira/browse/OOZIE-1457>)
>>
>> Do you have a snippet of this code that I can use in above scenario.
>> Thanks in advance for reading through this email.
>>
>> --
>> Thanks and best regards,
>> *Bala Chander * |  Solutions Architect, Services | *Cloudera*
>> Phone: +1 408-896-1334
>> bchan...@cloudera.com
>>
>>

Re: Kerberos and delegation token

[Robert Kanter]

Hi Bala,

It sounds like you'll want to use an HBase Delegation Token within the Hive
UDF to connect to HBase.  The user will need to obtain an HBase Delegation
Token with their Kerberos Credentials and either give the token to Hive or
Hive will need to get the token on behalf of the user (like Oozie).  The
code Oozie uses for getting an HBase token is here:
https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java

Let me know if you need more details.  I've also added some oozie lists,
for people who have used this more recently in case they have any
additional info.

- Robert

On Tue, Oct 16, 2018 at 1:04 PM Bala Chander  wrote:

> Hello Robert,
>
> I have a customer who is writing a UDF (user defined function) in java for
> usage in hive/beeline.
> The UDF contacts HBASE to pull in some data and returns back a single
> value. The USER of hive/beeline can issue a statement like the following:
>
> select a, UDF("xyz") from table_a;
>
> Hive/beeline and HBase operate on kerberos secure cluster.
>
> Michael Yoder (cc'ed) mentioned something similar in Oozie (Delegation
> token based connection for Oozie (OOZIE-1457
> )
>
> Do you have a snippet of this code that I can use in above scenario.
> Thanks in advance for reading through this email.
>
> --
> Thanks and best regards,
> *Bala Chander * |  Solutions Architect, Services | *Cloudera*
> Phone: +1 408-896-1334
> bchan...@cloudera.com
>
>

[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646713#comment-16646713
 ] 

Robert Kanter commented on OOZIE-3136:
--

You probably don't need to create two full LocalOozie/MiniOozie instances; just 
two (or more) of the XLogService and maybe some others - that'll help cut down 
on any interference we might get from running two Oozies on the same machine.  
There may be other options two; for instance, TestZKXLogStreamingService 
actually tests log streaming from another server by making a dummy servlet.

> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee 
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end 
> of life (EOL) and is no longer officially supported.}} 
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3136) Upgrade from Log4j 1.x to 2.x

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643738#comment-16643738
 ] 

Robert Kanter commented on OOZIE-3136:
--

+1 on not going to log4j2 until Oozie 6. Cleaning up our logging story and 
using the bridge in the meantime is probably a good idea.

Our logging is somewhat brittle because of the log streaming. IIRC, we require 
log files to be rolled every hour because the log streaming makes some 
assumptions about that to improve performance. Similarly, logs can be 
automatically gzipped, but we also make assumptions about that and the 
filenames. That's part of what the {{OozieRollingPolicy}} was for (it used to 
be messier before I added it :)). So if we need to get rid of 
{{OozieRollingPolicy}}, then we'll need to either replace it with something 
equivalent, or refactor the log streaming code to be more flexible.

> Upgrade from Log4j 1.x to 2.x
> -
>
> Key: OOZIE-3136
> URL: https://issues.apache.org/jira/browse/OOZIE-3136
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Attila Sasvari
>Assignee: Julia Kinga Marton
>Priority: Major
>
> {{5 August 2015 --The Apache Logging Services™ Project Management Committee 
> (PMC) has announced that the Log4j™ 1.x logging framework has reached its end 
> of life (EOL) and is no longer officially supported.}} 
> https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
> We should upgrade from Log4j 1.x to 2.x . Perhaps we could use slf4j .
> Related tickets: MAPREDUCE-6983, HADOOP-12956, OOZIE-3135



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2734) Switch docs from twiki to markdown

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578657#comment-16578657
 ] 

Robert Kanter commented on OOZIE-2734:
--

Oh yeah - that's a limitation we run into every so often.  It always tries to 
grab the newest attachement, and doesn't check the type.  So it's best to 
attach any other files first, and then the patch file last.

> Switch docs from twiki to markdown
> --
>
> Key: OOZIE-2734
> URL: https://issues.apache.org/jira/browse/OOZIE-2734
> Project: Oozie
>  Issue Type: Improvement
>  Components: docs
>Affects Versions: trunk
>    Reporter: Robert Kanter
>Assignee: Andras Salamon
>Priority: Major
> Attachments: OOZIE-2734-1.patch, OOZIE-2734-2.patch, 
> OOZIE-2734-3.patch, OOZIE-2734-4.patch, oozie_generated_html_20180731.zip, 
> oozie_generated_html_markdown_20180802.zip, twiki2md_20180730.zip
>
>
> This won't be a fun JIRA, but it would be great if we could switch to 
> something easier to work with and better supported like markdown, and get rid 
> of the super old twiki we're using for docs.
> I found two tools that I haven't tried out that claim to be able to convert 
> this for us:
> # http://pandoc.org/
> # https://github.com/jcodagnone/twiki2markdown
> The former supports a lot of different formats, and according to [this blog 
> post|http://goer.org/Journal/2013/02/converting_twiki_foswiki_to_markdown_restructuredtext.html]
>  it's better to convert from the rendered html instead of the twiki.  But who 
> knows which is better, or if the second tool is better than either.
> I think once we switch to a better format, it will be easier to keep the docs 
> in better shape, as they could really use some reorganizing.  Though we can 
> keep that for a separate task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3160) PriorityDelayQueue put()/take() can cause significant CPU load due to busy waiting

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531715#comment-16531715
 ] 

Robert Kanter commented on OOZIE-3160:
--

+1 on having a design document.  Not only is this a critical part of Oozie, but 
we also haven't made any _real_ changes in at least 4 years, and even that was 
relatively minimal.  Prior to that, it hadn't been touched since Oozie 2.2 back 
in 2011!
https://github.com/apache/oozie/commits/master/core/src/main/java/org/apache/oozie/util/PriorityDelayQueue.java



> PriorityDelayQueue put()/take() can cause significant CPU load due to busy 
> waiting
> --
>
> Key: OOZIE-3160
> URL: https://issues.apache.org/jira/browse/OOZIE-3160
> Project: Oozie
>  Issue Type: Bug
>  Components: core
> Environment: all platforms
>Reporter: jj
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: 11.png, 22.png, 
> OOZIE-3160-POC01.patch
>
>
> oozie process always  consume  high cpu. in my mechine,around 10%. 
> I check the source code，find take() method in PriorityDelayQueue class。
> code:
> {code:java}
> public QueueElement take() throws InterruptedException {
> QueueElement e = poll();
> while (e == null) {
> Thread.sleep(10);
> e = poll();
> }
> return e;
> }
> {code}
> i think it's the reason of this problem. it's keep while, not await.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512962#comment-16512962
 ] 

Robert Kanter commented on OOZIE-2339:
--

+1 on not using {{ImmutableMap}}, etc.  We should be careful to only expose 
standard Java objects and not any third-party library objects (especially 
Guava) to make future compatibility easier.  
+1 on the {{getAsXml()}} method.  That should cover what I was trying to get at 
in my previous comment about a way to get the XML without having to build a 
JAR.  This can be done as a followup though.


> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>    Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3283) Prototyping: create React workflow list page

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512792#comment-16512792
 ] 

Robert Kanter commented on OOZIE-3283:
--

Wow!  
I don't have time to try it out myself at the moment, but just from the 
screenshots, that's so much nicer than the extjs UI.

> Prototyping: create React workflow list page
> 
>
> Key: OOZIE-3283
> URL: https://issues.apache.org/jira/browse/OOZIE-3283
> Project: Oozie
>  Issue Type: Sub-task
>Reporter: Andras Piros
>Assignee: Thai Bui
>Priority: Major
> Attachments: Screen Shot 2018-06-14 at 9.24.46 AM.png, Screen Shot 
> 2018-06-14 at 9.25.42 AM.png
>
>
> Create a [*React*|https://redux.js.org/basics/usage-with-react] / 
> [*Redux*|https://redux.js.org/basics/usage-with-react] prototype of the 
> workflow list page. Don't have to handle authentication / authorization for 
> now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) [fluent-job] Minimum Viable Fluent Job API

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511650#comment-16511650
 ] 

Robert Kanter commented on OOZIE-2339:
--

+1 LGTM.  Looks like the remaining comments [~pbacsko] is working on are 
largely cleanup.  So I think we're essentially done once that's all taken care 
of.  Great work!

I think it would be helpful to have some easy way to generate the XML from a 
Fluent Workflow as part of developing/debugging it.  For instance, maybe some 
kind of {{main}} class that takes a {{WorkflowFactory}} and will output the XML 
to a file or stdout.  Something that can easily be run when writing a 
{{WorkflowFactory}} in and IDE without having to assemble a JAR.  That would be 
helpful for users more familiar with the XML or are trying out the Fluent API 
and want to get an idea of what it looks like before submitting it.  This could 
be a followup JIRA if you agree.

> [fluent-job] Minimum Viable Fluent Job API
> --
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>  Components: client, core, examples, fluent-job, tests
>Affects Versions: 4.3.0
>    Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch, OOZIE-2339.012.patch, OOZIE-2339.013.patch, 
> OOZIE-2339.014.patch, OOZIE-2339.015.patch, OOZIE-2339.016.patch, 
> OOZIE-2339.017.patch, OOZIE-2339.018.patch, OOZIE-2339.019.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside a {{fluent-job-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython / Py4J REPL to make it easy to experiment with also 
> for data engineers / data scientists
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502088#comment-16502088
 ] 

Robert Kanter commented on OOZIE-2877:
--

I also took a quick look at the patch and left a few minor comments on 
ReviewBoard.

Overall, though; this is really good!

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 59620: This review board request is for an action to provide a Git action for Oozie

[Robert Kanter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59620/#review204323
---




examples/src/main/apps/git/job.properties
Lines 9 (patched)
<https://reviews.apache.org/r/59620/#comment286760>

trailing whitespace; there's some other cases here and in other files too



sharelib/git/pom.xml
Lines 107-118 (patched)
<https://reviews.apache.org/r/59620/#comment286761>

This can be removed (It was needed some time ago in each sharelib pom file, 
but not anymore).



sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java
Lines 216 (patched)
<https://reviews.apache.org/r/59620/#comment286762>

Is this necessary?  Why not simply throw whatever exception (IOException, 
IllegalArgumentException, etc)?  We don't seem to be doing any special handling 
or anything for GitMainExceptions.



sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
Lines 33 (patched)
<https://reviews.apache.org/r/59620/#comment286763>

"is" --> "in"



sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
Lines 36 (patched)
<https://reviews.apache.org/r/59620/#comment286764>

As Peter Bacsko pointed out on the JIRA, we shouldn't be fetching a remote 
repo during a unit test. Perhaps JGit has some easy way of mocking this or 
doing it locally?


- Robert Kanter


On June 4, 2018, 5:44 a.m., Clay B. wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59620/
> ---
> 
> (Updated June 4, 2018, 5:44 a.m.)
> 
> 
> Review request for oozie and András Piros.
> 
> 
> Bugs: OOZIE-2877
> https://issues.apache.org/jira/browse/OOZIE-2877
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2877 - Oozie Git Action
> 
> 
> Diffs
> -
> 
>   client/src/main/resources/git-action-1.0.xsd PRE-CREATION 
>   core/src/main/java/org/apache/oozie/action/hadoop/GitActionExecutor.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml c54db34 
>   docs/src/site/twiki/WorkflowFunctionalSpec.twiki 76cbe21 
>   examples/src/main/apps/git/job.properties PRE-CREATION 
>   examples/src/main/apps/git/workflow.xml PRE-CREATION 
>   pom.xml 6f35868 
>   sharelib/git/pom.xml PRE-CREATION 
>   sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitMain.java 
> PRE-CREATION 
>   
> sharelib/git/src/main/java/org/apache/oozie/action/hadoop/GitOperations.java 
> PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitActionExecutor.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestGitMainGetKey.java
>  PRE-CREATION 
>   
> sharelib/git/src/test/java/org/apache/oozie/action/hadoop/TestIntegrationGitActionExecutor.java
>  PRE-CREATION 
>   sharelib/pom.xml 6a0864d 
>   src/main/assemblies/sharelib.xml 07dc69c 
>   webapp/pom.xml 67526d9 
> 
> 
> Diff: https://reviews.apache.org/r/59620/diff/6/
> 
> 
> Testing
> ---
> 
> Tested using unit and integration tests. Still need to:
> * Test on a cluster
> * Test with an authenticated SSH hosted Git repo
> 
> Sumitted a request to the JGit community as their branch pulling code seems 
> to have an 
> [issue](https://dev.eclipse.org/mhonarc/lists/jgit-dev/msg03343.html).
> 
> Andras has tested against his patch 011 with results at 
> https://issues.apache.org/jira/browse/OOZIE-2877?focusedCommentId=16459523=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16459523
> 
> 
> File Attachments
> 
> 
> 0001-OOZIE-2877-Oozie-Git-Action.patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/24f90a78-3dc1-49fe-bf29-5927a3cd5e72__0001-OOZIE-2877-Oozie-Git-Action.patch
> Patch
>   
> https://reviews.apache.org/media/uploaded/files/2017/05/29/dd23dd72-67e0-456f-9b52-e566d8d17d16__0001-OOZIE-2877-Oozie-Git-Action.patch
> 
> 
> Thanks,
> 
> Clay B.
> 
>

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502065#comment-16502065
 ] 

Robert Kanter commented on OOZIE-2877:
--

Makes sense to me then.  Let's stick with the original credential handling.

[~pbacsko] makes a good point.  We shouldn't be fetching a remote repo during a 
unit test.  Perhaps JGit has some easy way of mocking this or doing it locally?

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2877) Oozie Git Action

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500648#comment-16500648
 ] 

Robert Kanter commented on OOZIE-2877:
--

I haven't been able to find time to look at the patch, but I saw mention of 
"credential files" and just wanted to suggest something (I could be completely 
off base here because I haven't looked at the code, so feel free to ignore me 
if this doesn't make sense :)).  Now that Oozie's directly on YARN, we have 
access to the {{Credentials}} object in the {{ApplicationSubmissionContext}}.  
You can add arbitrary secrets into a hashmap ({{}}) in the 
{{Credentials}}.  YARN will securely provide that to the Container (IIRC you 
can retrieve it via {{Configuration}}).  That might be a better alternative to 
putting the credential file in HDFS.

> Oozie Git Action
> 
>
> Key: OOZIE-2877
> URL: https://issues.apache.org/jira/browse/OOZIE-2877
> Project: Oozie
>  Issue Type: Sub-task
>  Components: action
>Reporter: Clay B.
>Assignee: Clay B.
>Priority: Major
>  Labels: action
> Fix For: trunk
>
> Attachments: 0001-OOZIE-2877-Oozie-Git-Action.patch, 
> 0002-OOZIE-2877-Oozie-Git-Action.patch, 
> 0003-OOZIE-2877-Oozie-Git-Action.patch, 
> 0004-OOZIE-2877-Oozie-Git-Action.patch, 
> 0005-OOZIE-2877-Oozie-Git-Action.patch, 
> 0006-OOZIE-2877-Oozie-Git-Action.patch, 
> 0007-OOZIE-2877-Oozie-Git-Action.patch, 
> 0008-OOZIE-2877-Oozie-Git-Action.patch, 
> 0009-OOZIE-2877-Oozie-Git-Action.patch, OOZIE-2877.010.patch, 
> OOZIE-2877.011.patch, OOZIE-2877.012.patch
>
>
> To aide in deploying ASCII artifacts to clusters, let's provide a tie-in for 
> a source-code management system. Git would be my preferred choice.
> Ideally, this could handle a user's key material e.g. for an ssh key to pull 
> down from a secured repository. This would free users from handling their own 
> key staging and clean-up on YARN nodes and only require them to store the key 
> secured in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2105) Make version of submodules configurable with parent version

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489847#comment-16489847
 ] 

Robert Kanter commented on OOZIE-2105:
--

Yeah, I think we can close this as won't fix.  This was specifically about the 
repackaging of Hadoop stuff where we had hybrid versions (e.g. 
{{1.1.1.oozie-4.1.0}}: that is, Hadoop 1.1.1 in Oozie 4.1.0), which we're not 
doing anymore.  Those versions didn't work with the maven version plugin 
command.

> Make version of submodules configurable with parent version 
> 
>
> Key: OOZIE-2105
> URL: https://issues.apache.org/jira/browse/OOZIE-2105
> Project: Oozie
>  Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2105-02.patch, OOZIE-2105-03.patch, 
> OOZIE-6252.1.patch
>
>
> Currently the versions of the Oozie sub-modules are hard-coded with parent 
> version. If someone changes the parent version, all sub-modules versions will 
> need to be explicitly updated. For example, in hadooplibs/hadoop-1/pom.xml we 
> use
> org.apache.oozie
>  oozie-hadoop
>  1.1.1.oozie-4.1.0
>  ...
> If you want to modify the Oozie version to 4.1.1 (say), you need to go to all 
> pom files and manually change replace 4.1.0 to 4.1.1.
> This JIRA is to use "parent.version" instead of hard-coding. For example, use 
> this :
> 1.1.1.oozie-${parent.version}
>  
> This will allow to change only the parent version in root pom.xml.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2105) Make version of submodules configurable with parent version

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489826#comment-16489826
 ] 

Robert Kanter commented on OOZIE-2105:
--

Can't you just use the maven versions plugin to update the versions as-is?
e.g.
{code}
mvn versions:set -DnewVersion=5.1.0
{code}

As for the warning when using an expression, that's been a long-standing issue 
that Maven doesn't like non-constant versions in poms (other than for 
dependencies).

> Make version of submodules configurable with parent version 
> 
>
> Key: OOZIE-2105
> URL: https://issues.apache.org/jira/browse/OOZIE-2105
> Project: Oozie
>  Issue Type: Bug
>Reporter: Mohammad Kamrul Islam
>Assignee: Artem Ervits
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2105-02.patch, OOZIE-2105-03.patch, 
> OOZIE-6252.1.patch
>
>
> Currently the versions of the Oozie sub-modules are hard-coded with parent 
> version. If someone changes the parent version, all sub-modules versions will 
> need to be explicitly updated. For example, in hadooplibs/hadoop-1/pom.xml we 
> use
> org.apache.oozie
>  oozie-hadoop
>  1.1.1.oozie-4.1.0
>  ...
> If you want to modify the Oozie version to 4.1.1 (say), you need to go to all 
> pom files and manually change replace 4.1.0 to 4.1.1.
> This JIRA is to use "parent.version" instead of hard-coding. For example, use 
> this :
> 1.1.1.oozie-${parent.version}
>  
> This will allow to change only the parent version in root pom.xml.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Robert Kanter via Review Board]

> On May 20, 2018, 10:14 p.m., Robert Kanter wrote:
> > A bunch on minor things.

Looks good though!


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64029/#review203465
---


On May 14, 2018, 8:36 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated May 14, 2018, 8:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 795a92d4af1a6d96603c99506cd70e8cf8e2c347 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 6f8925828090cee29a818de30a7c31db0617d816 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml 2bfc7c8b3fc3de4876c62d70f25bed35b886ece8 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Credentials.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Builder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBase.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ChFSBaseBuilder.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/Chgrp.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/o

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Robert Kanter via Review Board]

1>

Missing Javadoc



fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
Lines 229-243 (patched)
<https://reviews.apache.org/r/64029/#comment285723>

Missing Javadocs



fluent-job/fluent-job-api/src/test/java/org/apache/oozie/fluentjob/api/factory/MultipleShellNodesWorkflowFactory.java
Lines 35 (patched)
<https://reviews.apache.org/r/64029/#comment285726>

This seems like it belongs in the examples?



fluent-job/fluent-job-api/src/test/java/org/apache/oozie/fluentjob/api/factory/SimpleWorkflowFactory.java
Lines 33 (patched)
<https://reviews.apache.org/r/64029/#comment285727>

This seems like it belongs in the examples?



minitest/src/test/java/org/apache/oozie/test/TestWorkflow.java
Line 31 (original), 32 (patched)
<https://reviews.apache.org/r/64029/#comment285728>

* import


- Robert Kanter


On May 14, 2018, 8:36 p.m., András Piros wrote:
> 
> -------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated May 14, 2018, 8:36 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   client/pom.xml 795a92d4af1a6d96603c99506cd70e8cf8e2c347 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/java/org/apache/oozie/client/ApiJarFactory.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 4c8175628ba5ee346d97ad984532662032b88acc 
>   core/pom.xml 45880a619e70a1643f4eb5ceeb61abb2090551f1 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 3eb32d50d07687c136fcc0e467a055bdc1e7ec60 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> 1d800949cbff34a95f2ae111364eb8cbc6a5f5fb 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> b86fa6ace43aead115302aad3ae15368b15f1923 
>   core/src/main/resources/oozie-default.xml 
> 6f8925828090cee29a818de30a7c31db0617d816 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> 3395cc25d27805444ba0b6463357c4ecff977ac1 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobsServlet.java 
> fb481e367653187a94199069199f1d1b06d1cecf 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 0a030d660f9c545b520afcce458a27ba2a367658 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 9a174596dd78fb15989daf4e88936a9a2b09ef13 
>   docs/src/site/twiki/DG_FluentJobAPI.twiki PRE-CREATION 
>   examples/pom.xml 2bfc7c8b3fc3de4876c62d70f25bed35b886ece8 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Credentials.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Global.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/JavaMain.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Parameters.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Shell.java 
> PRE-CREATION 
>   examples/src/main/java/org/apache/oozie/example/fluentjob/Spark.java 
> PRE-CREATION 
>   fluent-job/fluent-job-api/pom.xml PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/Condition.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/GraphVisualization.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/ModifyOnce.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributes.java
>  PRE-CREATION 
>   
> fluent-job/fluent-job-api/src/main/java/org/apache/oozie/fluentjob/api/action/ActionAttributesBuilder.java
>  PRE-CREATION 
>   
> fluen

Re: Moving Oozie to pull requests

[Robert Kanter]

By the way, I was talking with Pat Hunt at lunch and he mentioned
https://gitbox.apache.org/ which does a bunch of integration between github
and apache so you can do pull requests and stuff.

On Sat, May 12, 2018 at 10:44 PM, Andras Piros <andras.pi...@cloudera.com>
wrote:

> Yes, you're right Robert, those steps are needed for pull requests.
>
> However, for most, no-oneliner tasks I prefer doing steps 1-3) and 5)
> anyways, so for me not a big deal to perform additionally 4).
>
> Repo can be the same but another branch like for Oozie On YARN changes, or
> *can
> be different <https://help.github.com/articles/creating-a-pull-request/>*.
>
> Most common things like rebases, reviews, merging are covered by *GitHub
> infrastructure
> <https://help.github.com/categories/collaborating-with-
> issues-and-pull-requests/>*
> .
>
> Andras
>
> On Fri, May 11, 2018 at 9:18 PM, Robert Kanter <robert.kan...@gmail.com>
> wrote:
>
> > Maybe I'm doing it wrong, but I've found pull requests to require more
> > steps.  I have to:
> >
> >1. Fork the repo in GitHub
> >2. Create a branch with the changes (I usually do this step anyway
> >though)
> >3. Push the branch to my forked repo in GitHub
> >4. Go to GitHub and click through a few things to create the pull
> > request
> >5. (Ongoing) Ensure that my fork on GitHub is kept up-to-date
> >
> >
> >
> > On Fri, May 11, 2018 at 11:23 AM, Peter Bacsko <pbac...@cloudera.com>
> > wrote:
> >
> > > Robert, is there anything particular that you don't like about pull
> > > requests? Any clear disadvantages?
> > >
> > > On Thu, May 10, 2018 at 6:08 PM, Robert Kanter <rkan...@cloudera.com>
> > > wrote:
> > >
> > > > In that case, +0 for pull requests.
> > > > (I'm not the biggest fan of pull requests, but I've also not been
> > doing a
> > > > lot of contributing lately, so I won't stand in the way if that's
> what
> > > > everyone wants)
> > > >
> > > >
> > > > - Robert
> > > >
> > > > On Thu, May 10, 2018 at 8:55 AM, Peter Cseh <gezap...@cloudera.com>
> > > wrote:
> > > >
> > > > > I've checked the Oozie pre-commit job config and there is a
> > > pull-request
> > > > > based trigger available.
> > > > > I think it will be more reliable than the current way. There is no
> > need
> > > > for
> > > > > the PreCommit-Admin job to grab all the attachments and coordinate
> on
> > > > > what's new.
> > > > >
> > > > >
> > > > > On Wed, May 9, 2018 at 11:11 PM, Peter Bacsko <
> pbac...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > I'm not sure how we can trigger a Jenkins build for pull
> requests,
> > > but
> > > > > > looking at this, it's possible:
> > > > > >
> > > > > > https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> > > > > request+builder+plugin
> > > > > >
> > > > > > We have to ask Jenkins admins to either install this (or similar)
> > > > plugin,
> > > > > > or configure existing ones so it deals with Oozie too.
> > > > > >
> > > > > > Anyway, +1 for pull requests. Definitely more convenient,
> rebasing
> > is
> > > > > > simple because merge conflicts can be resolved inside GitHub
> (also
> > in
> > > > > > command line). Retrieving the changes as a patch file is also
> super
> > > > easy.
> > > > > >
> > > > > > Peter
> > > > > >
> > > > > > On Tue, May 8, 2018 at 7:55 PM, Robert Kanter <
> rkan...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Is there a way to trigger
> > > > > > > https://builds.apache.org/job/PreCommit-OOZIE-Build/ (
> > > > > > > https://builds.apache.org/job/PreCommit-Admin/) from a pull
> > > request?
> > > > > We
> > > > > > > wouldn't want to lose the pre-commit Jenkins run.
> > > > > > >
> > > > > > > - Robert
> > > > > > >
> > > > > > > On Tue, May 8, 2018 at 7:25 AM, Andras Piros <
> and...@apache.org>
> > > > > wrote:
> > > > > >

Re: Moving Oozie to pull requests

[Robert Kanter]

Maybe I'm doing it wrong, but I've found pull requests to require more
steps.  I have to:

   1. Fork the repo in GitHub
   2. Create a branch with the changes (I usually do this step anyway
   though)
   3. Push the branch to my forked repo in GitHub
   4. Go to GitHub and click through a few things to create the pull request
   5. (Ongoing) Ensure that my fork on GitHub is kept up-to-date



On Fri, May 11, 2018 at 11:23 AM, Peter Bacsko <pbac...@cloudera.com> wrote:

> Robert, is there anything particular that you don't like about pull
> requests? Any clear disadvantages?
>
> On Thu, May 10, 2018 at 6:08 PM, Robert Kanter <rkan...@cloudera.com>
> wrote:
>
> > In that case, +0 for pull requests.
> > (I'm not the biggest fan of pull requests, but I've also not been doing a
> > lot of contributing lately, so I won't stand in the way if that's what
> > everyone wants)
> >
> >
> > - Robert
> >
> > On Thu, May 10, 2018 at 8:55 AM, Peter Cseh <gezap...@cloudera.com>
> wrote:
> >
> > > I've checked the Oozie pre-commit job config and there is a
> pull-request
> > > based trigger available.
> > > I think it will be more reliable than the current way. There is no need
> > for
> > > the PreCommit-Admin job to grab all the attachments and coordinate on
> > > what's new.
> > >
> > >
> > > On Wed, May 9, 2018 at 11:11 PM, Peter Bacsko <pbac...@cloudera.com>
> > > wrote:
> > >
> > > > I'm not sure how we can trigger a Jenkins build for pull requests,
> but
> > > > looking at this, it's possible:
> > > >
> > > > https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> > > request+builder+plugin
> > > >
> > > > We have to ask Jenkins admins to either install this (or similar)
> > plugin,
> > > > or configure existing ones so it deals with Oozie too.
> > > >
> > > > Anyway, +1 for pull requests. Definitely more convenient, rebasing is
> > > > simple because merge conflicts can be resolved inside GitHub (also in
> > > > command line). Retrieving the changes as a patch file is also super
> > easy.
> > > >
> > > > Peter
> > > >
> > > > On Tue, May 8, 2018 at 7:55 PM, Robert Kanter <rkan...@apache.org>
> > > wrote:
> > > >
> > > > > Is there a way to trigger
> > > > > https://builds.apache.org/job/PreCommit-OOZIE-Build/ (
> > > > > https://builds.apache.org/job/PreCommit-Admin/) from a pull
> request?
> > > We
> > > > > wouldn't want to lose the pre-commit Jenkins run.
> > > > >
> > > > > - Robert
> > > > >
> > > > > On Tue, May 8, 2018 at 7:25 AM, Andras Piros <and...@apache.org>
> > > wrote:
> > > > >
> > > > > > Hi Geza,
> > > > > >
> > > > > > +1 on using GitHub pull requests instead of patches uploaded to
> > > > > ReviewBoard
> > > > > > / JIRA. PRs are the modern way of doing code reviews, integrating
> > > > > changes,
> > > > > > doing source code management in general.
> > > > > >
> > > > > > This step will lower the barrier newcomers face when trying to
> > > > contribute
> > > > > > for the first time, too.
> > > > > >
> > > > > > Andras
> > > > > >
> > > > > > On Tue, May 8, 2018 at 4:01 PM, Peter Cseh <
> gezap...@cloudera.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi team Oozie!
> > > > > > >
> > > > > > > Many projects are using pull requests on Github instead of
> > > uploading
> > > > > > > patches to Jira.
> > > > > > > Based on my experience with those projects, pull requests are
> > > easier
> > > > to
> > > > > > > manage and less error prone than the current way of doing
> commits
> > > in
> > > > > > Oozie.
> > > > > > >
> > > > > > > I would like to migrate Oozie to use pull requests instead of
> > patch
> > > > > files
> > > > > > > in the next quarter if the community is up for that change.
> > > > > > > I don't really see any downsides other than we'll have to
> > 

Re: Moving Oozie to pull requests

[Robert Kanter]

In that case, +0 for pull requests.
(I'm not the biggest fan of pull requests, but I've also not been doing a
lot of contributing lately, so I won't stand in the way if that's what
everyone wants)


- Robert

On Thu, May 10, 2018 at 8:55 AM, Peter Cseh <gezap...@cloudera.com> wrote:

> I've checked the Oozie pre-commit job config and there is a pull-request
> based trigger available.
> I think it will be more reliable than the current way. There is no need for
> the PreCommit-Admin job to grab all the attachments and coordinate on
> what's new.
>
>
> On Wed, May 9, 2018 at 11:11 PM, Peter Bacsko <pbac...@cloudera.com>
> wrote:
>
> > I'm not sure how we can trigger a Jenkins build for pull requests, but
> > looking at this, it's possible:
> >
> > https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> request+builder+plugin
> >
> > We have to ask Jenkins admins to either install this (or similar) plugin,
> > or configure existing ones so it deals with Oozie too.
> >
> > Anyway, +1 for pull requests. Definitely more convenient, rebasing is
> > simple because merge conflicts can be resolved inside GitHub (also in
> > command line). Retrieving the changes as a patch file is also super easy.
> >
> > Peter
> >
> > On Tue, May 8, 2018 at 7:55 PM, Robert Kanter <rkan...@apache.org>
> wrote:
> >
> > > Is there a way to trigger
> > > https://builds.apache.org/job/PreCommit-OOZIE-Build/ (
> > > https://builds.apache.org/job/PreCommit-Admin/) from a pull request?
> We
> > > wouldn't want to lose the pre-commit Jenkins run.
> > >
> > > - Robert
> > >
> > > On Tue, May 8, 2018 at 7:25 AM, Andras Piros <and...@apache.org>
> wrote:
> > >
> > > > Hi Geza,
> > > >
> > > > +1 on using GitHub pull requests instead of patches uploaded to
> > > ReviewBoard
> > > > / JIRA. PRs are the modern way of doing code reviews, integrating
> > > changes,
> > > > doing source code management in general.
> > > >
> > > > This step will lower the barrier newcomers face when trying to
> > contribute
> > > > for the first time, too.
> > > >
> > > > Andras
> > > >
> > > > On Tue, May 8, 2018 at 4:01 PM, Peter Cseh <gezap...@cloudera.com>
> > > wrote:
> > > >
> > > > > Hi team Oozie!
> > > > >
> > > > > Many projects are using pull requests on Github instead of
> uploading
> > > > > patches to Jira.
> > > > > Based on my experience with those projects, pull requests are
> easier
> > to
> > > > > manage and less error prone than the current way of doing commits
> in
> > > > Oozie.
> > > > >
> > > > > I would like to migrate Oozie to use pull requests instead of patch
> > > files
> > > > > in the next quarter if the community is up for that change.
> > > > > I don't really see any downsides other than we'll have to resubmit
> > some
> > > > > patches for old jiras, which we'll have to do anyways due to
> possible
> > > > merge
> > > > > conflicts.
> > > > > The tooling and automation possibilities around pull requests will
> > > > probably
> > > > > meet all our needs.
> > > > >
> > > > > I don't know if we'd need an official vote for a change like this
> or
> > > > > regular Jira is enough to track all the necessary changes. Let's
> see
> > if
> > > > > there are any objections against the general idea before going too
> > much
> > > > > into details. :)
> > > > >
> > > > > So... Any objections?
> > > > > gp
> > > > >
> > > > >
> > > > > --
> > > > > *Peter Cseh *| Software Engineer
> > > > > cloudera.com <https://www.cloudera.com>
> > > > >
> > > > > [image: Cloudera] <https://www.cloudera.com/>
> > > > >
> > > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera>
> [image:
> > > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > > Cloudera
> > > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > > > --
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com <https://www.cloudera.com>
>
> [image: Cloudera] <https://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> --
>

Re: Moving Oozie to pull requests

[Robert Kanter]

Is there a way to trigger
https://builds.apache.org/job/PreCommit-OOZIE-Build/ (
https://builds.apache.org/job/PreCommit-Admin/) from a pull request?  We
wouldn't want to lose the pre-commit Jenkins run.

- Robert

On Tue, May 8, 2018 at 7:25 AM, Andras Piros  wrote:

> Hi Geza,
>
> +1 on using GitHub pull requests instead of patches uploaded to ReviewBoard
> / JIRA. PRs are the modern way of doing code reviews, integrating changes,
> doing source code management in general.
>
> This step will lower the barrier newcomers face when trying to contribute
> for the first time, too.
>
> Andras
>
> On Tue, May 8, 2018 at 4:01 PM, Peter Cseh  wrote:
>
> > Hi team Oozie!
> >
> > Many projects are using pull requests on Github instead of uploading
> > patches to Jira.
> > Based on my experience with those projects, pull requests are easier to
> > manage and less error prone than the current way of doing commits in
> Oozie.
> >
> > I would like to migrate Oozie to use pull requests instead of patch files
> > in the next quarter if the community is up for that change.
> > I don't really see any downsides other than we'll have to resubmit some
> > patches for old jiras, which we'll have to do anyways due to possible
> merge
> > conflicts.
> > The tooling and automation possibilities around pull requests will
> probably
> > meet all our needs.
> >
> > I don't know if we'd need an official vote for a change like this or
> > regular Jira is enough to track all the necessary changes. Let's see if
> > there are any objections against the general idea before going too much
> > into details. :)
> >
> > So... Any objections?
> > gp
> >
> >
> > --
> > *Peter Cseh *| Software Engineer
> > cloudera.com 
> >
> > [image: Cloudera] 
> >
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> Cloudera
> > on LinkedIn] 
> > --
> >
>

[jira] [Updated] (OOZIE-3214) Allow configurable timezone for coordinators

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3214:
-
Issue Type: New Feature  (was: Improvement)

> Allow configurable timezone for coordinators
> 
>
> Key: OOZIE-3214
> URL: https://issues.apache.org/jira/browse/OOZIE-3214
> Project: Oozie
>  Issue Type: New Feature
>  Components: coordinator
>Affects Versions: trunk
>Reporter: Julia Kinga Marton
>Assignee: Julia Kinga Marton
>Priority: Major
>
> Right now in case of coordinators only daylight saving time corrections are 
> applied, the processing timezone for the coordinators is always the Oozie 
> processing timezone. 
> It would be more transparent to have an option for changing the {{timezone}} 
> attribute itself, such as an additional attribute in the {{coordinator.xml}} 
> (meaning a new version of {{coordinator.xsd}}). I would make this option 
> switched off by default(to have the actual behavior).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Oozie 5.0.0 released

[Robert Kanter]

Yay!  This is a great milestone for Oozie!  Thanks everyone who worked on
it.

On Mon, Apr 9, 2018 at 8:01 AM, Andras Piros  wrote:

> Hello Oozie community,
>
>
> Apache Oozie is a workflow scheduler system to manage Apache Hadoop jobs.
>
>
> The Apache Oozie team is pleased to announce the release of Oozie 5.0.0.
>
> Oozie 5.0.0 includes new features, bug fixes and minor improvements.
>
>
> The details of the release and download location can be found at
> http://oozie.apache.org.
>
>
>
> The full release notes can be found at the download location, but some
> highlights of this release include:
>
>
> - moved launcher from MapReduce AM to YARN AM,
> - switched from Tomcat to embedded Jetty,
> - updated third party libraries,
> - completely rewritten workflow graph generator,
> - JDK 8 support,
> - deprecated Instrumentation in favor of Metrics,
> - added indexes to speed up DB queries,
> - fixed CVE-2017-15712
>
> Thanks to all developers, QA's, users, and committers for making this
> release happen.
>
> Regards,
>
> Andras
>

[jira] [Comment Edited] (OOZIE-2683) Rewrite the Oozie Web UI

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226001#comment-16226001
 ] 

Robert Kanter edited comment on OOZIE-2683 at 4/9/18 4:45 PM:
--

I think people will be happy with pretty much anything :)

Just pick something that's (a) easy to develop with, (b) easy to deploy 
(ideally, automatically), and (c) license compatible.

 MIT license is fine. See 
[https://www.apache.org/legal/resolved.html#category-a] for a list of allowed 
licenses, and [https://www.apache.org/legal/resolved.html#category-x] for a 
list of disallowed licenses.
  


was (Author: rkanter):
I think people will be happy with pretty much anything :)

Just pick something that's (a) easy to develop with, (b) easy to deploy 
(ideally, automatically), and (c) license compatible.

> Rewrite the Oozie Web UI
> 
>
> Key: OOZIE-2683
> URL: https://issues.apache.org/jira/browse/OOZIE-2683
> Project: Oozie
>  Issue Type: New Feature
>    Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
>
> We're currently relying on a version of ExtJS that's so old, it's not linked 
> on their official website, and is often temporarily deleted.  Looks like it's 
> now been deleted for good (OOZIE-2622).  It also has a GPL license, so we 
> can't include it with Oozie, and users are forced to go and download it 
> themselves.  And finally, it's a really outdated UI that's not very good.
> We should invest in a new Web UI that's more modern and compatibly licensed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2494) Cron syntax not handling DST properly

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426163#comment-16426163
 ] 

Robert Kanter commented on OOZIE-2494:
--

I'm fine with adding a way to tell Oozie to actually use a different timezone 
if we can do that correctly (and I'm sure a lot of users would like this), but 
it's also important to remain consistent so we don't break anything.  So I'd 
suggest that we fix the original issue here (i.e. the cron syntax not handling 
DST as expected) and then we can file a "New Feature" Jira to add a way to use 
other timezones.

> Cron syntax not handling DST properly
> -
>
> Key: OOZIE-2494
> URL: https://issues.apache.org/jira/browse/OOZIE-2494
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 4.2.0
>Reporter: Dennis Pallett
>Assignee: Julia Kinga Marton
>Priority: Blocker
> Attachments: CronExpressionPOC.java, OOZIE-2494-001.patch, 
> OOZIE-2494-002.patch, OOZIE-2494-003.patch, OOZIE-2494-004.patch, 
> testActionMaterWithDST3.patch
>
>
> When specifying a coordinator frequency, you can also specify a "timezone".  
> While the frequency is always calculated in UTC, the timezone’s DST rules are 
> still applied.  We can see this in the following two Coordinators, which ran 
> across the DST shift (March 13 2016 at 2am) for the America/Los_Angeles 
> timezone.  The "el-UTC" job has "UTC" as the timezone, while the "el-LA" job 
> has “America/Los_Angeles” as the timezone.  Both jobs have a frequency of 
> {{$\{coord:days(1)\}}}.
> {noformat}
> Job Name: el-UTC
> Start Time  : 2016-03-13 01:10 GMT | 2016-03-12 17:10 PST
> -
> ID Nominal Time (UTC)   Nominal Time (LA)
> @1 2016-03-13 01:10 GMT 2016-03-12 17:10 PST
> @2 2016-03-14 01:10 GMT 2016-03-13 18:10 PDT
> {noformat}
> {noformat}
> Job Name: el-LA
> Start Time  : 2016-03-13 01:10 GMT | 2016-03-12 17:10 PST
> -
> ID Nominal Time (UTC)   Nominal Time (LA)
> @1 2016-03-13 01:10 GMT 2016-03-12 17:10 PST
> @2 2016-03-14 00:10 GMT 2016-03-13 17:10 PDT
> {noformat}
> As you can see, @2’s nominal time is adjusted to an hour earlier in the 
> "el-LA" job, but not in the "el-UTC" job.
> However, when running a similar set of jobs, but using cron syntax [({{10 1 
> 1/1 * *}}|http://crontab.guru/#10_1_1/1_*_*], which indicates 1:10 every day 
> of every month), this isn’t the case:
> {noformat}
> Job Name: cron-UTC
> Start Time  : 2016-03-13 01:08 GMT | 2016-03-12 17:08 PST
> -
> ID Nominal Time (UTC)   Nominal Time (LA)
> @1 2016-03-13 01:10 GMT 2016-03-12 17:10 PST
> @2 2016-03-14 01:10 GMT 2016-03-13 18:10 PDT
> {noformat}
> {noformat}
> Job Name: cron-LA
> Start Time  : 2016-03-13 01:08 GMT | 2016-03-12 17:08 PST
> -
> ID Nominal Time (UTC)   Nominal Time (LA)
> @1 2016-03-13 01:10 GMT 2016-03-12 17:10 PST
> @2 2016-03-14 01:10 GMT 2016-03-13 18:10 PDT
> {noformat}
> As you can see, @2’s nominal time are the same in both the "cron-UTC" and 
> "cron-LA" jobs.  The "cron-LA" job should have the same nominal time as the 
> "el-LA" job from earlier.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 5.0.0 (candidate 0)

[Robert Kanter]

Did you set oozie.service.HadoopAccessorService.hadoop.configurations and
point it at your Hadoop configs?

On Fri, Mar 30, 2018 at 4:12 AM, Artem Ervits <artemerv...@gmail.com> wrote:

> I ran into a similar issue to Oozie-3208 except I did install sharelib. I
> can see it in hdfs and yet I get /user/share/lib does not exist. That
> message prevented me from running successful workflows except for no-op.
>
> On Thu, Mar 29, 2018, 5:52 PM Robert Kanter <rkan...@apache.org> wrote:
>
> > Here's what I did:
> > - Verified sha512
> > - Verified signature (gpg)
> > - Looked at rat report
> > - Looked at release log
> > - Built Oozie against Hadoop 2.7.2, using an empty local maven repo
> > - Ran a few of the examples, clicked around the UI, tried various CLI
> > commands, looked at the REST output, launcher log output, etc
> >
> > +1 LGTM
> >
> > Non-blocker issues which I've filed JIRAs for
> > - OOZIE-3206 <https://issues.apache.org/jira/browse/OOZIE-3206>: The
> > README.txt links to the 4.3.0 version of the quickstart page:
> > http://oozie.apache.org/docs/4.3.0/DG_QuickStart.html
> > - OOZIE-3207 <https://issues.apache.org/jira/browse/OOZIE-3207>: We're
> > using version 17 of the ASF root pom, and the latest is 19.  version 17
> is
> > from 2015!
> > - OOZIE-3208 <https://issues.apache.org/jira/browse/OOZIE-3208>: If the
> > sharelib isn't installed, you get a "It should never happen" error code
> > when submitting an action
> >
> > If a blocking issue is found, OOZIE-3206 and OOZIE-3208 would be
> > nice-to-have's in an RC1 (OOZIE-3207 is too risky).  Otherwise, we can
> save
> > them for Oozie 5.1.0.
> >
> >
> > thanks
> > - Robert
> >
> >
> > On Thu, Mar 29, 2018 at 11:00 AM, Attila Sasvari <asasv...@cloudera.com>
> > wrote:
> >
> > > Thanks Andras for driving the release!
> > >
> > > +1
> > >
> > > Here is what I did:
> > > - Verified signature, hash
> > > - Created Oozie 5.0.0 distro using hadoop-2.6
> > > - On a pseudo Hadoop 2.6.0, I executed  SparkFileCopy (both in
> > > yarn-cluster, yarn-client mode),  custom-main-wf,
> datelist-java-main-wf,
> > > demo-wf, distcp-wf, hadoop-el-wf, java-main-wf, map-reduce-wf,
> no-op-wf,
> > > pig-wf, shell-wf, sqoop-freeform-wf, sqoop-wf, ssh-wf, streaming-wf,
> > > subwf-wf  - all the workflows have succeeded
> > > - Verified Web UI: version, getting logs and Job DAG
> > >
> > >
> > > On Wed, Mar 28, 2018 at 4:30 PM, Artem Ervits <artemerv...@gmail.com>
> > > wrote:
> > > >
> > > > additionally, I've ran the mvn verify and after some time it failed
> due
> > > to
> > > > timeout, still OK in my opinion.
> > > >
> > > > INFO] Results:
> > > > [INFO]
> > > > [INFO] Tests run: 738, Failures: 0, Errors: 0, Skipped: 0
> > > > [INFO]
> > > > [INFO]
> > > >
> > 
> > > > [INFO] Reactor Summary:
> > > > [INFO]
> > > > [INFO] Apache Oozie Main 5.0.0  SUCCESS [
> > > > 3.859 s]
> > > > [INFO] Apache Oozie Client  SUCCESS
> [
> > > > 34.826 s]
> > > > [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [
> > > > 25.740 s]
> > > > [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [
> > > > 11.842 s]
> > > > [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [
> > > > 7.465 s]
> > > > [INFO] Apache Oozie Core .. FAILURE
> [
> > > > 01:30 h]
> > > > [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> > > > [INFO] Apache Oozie Share Lib Pig . SKIPPED
> > > > [INFO] Apache Oozie Share Lib Hive  SKIPPED
> > > > [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> > > > [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> > > > [INFO] Apache Oozie Examples .. SKIPPED
> > > > [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> > > > [INFO] Apache Oozie Share Lib . SKIPPED
> > &

Re: [VOTE] Release Oozie 5.0.0 (candidate 0)

[Robert Kanter]

Here's what I did:
- Verified sha512
- Verified signature (gpg)
- Looked at rat report
- Looked at release log
- Built Oozie against Hadoop 2.7.2, using an empty local maven repo
- Ran a few of the examples, clicked around the UI, tried various CLI
commands, looked at the REST output, launcher log output, etc

+1 LGTM

Non-blocker issues which I've filed JIRAs for
- OOZIE-3206 : The
README.txt links to the 4.3.0 version of the quickstart page:
http://oozie.apache.org/docs/4.3.0/DG_QuickStart.html
- OOZIE-3207 : We're
using version 17 of the ASF root pom, and the latest is 19.  version 17 is
from 2015!
- OOZIE-3208 : If the
sharelib isn't installed, you get a "It should never happen" error code
when submitting an action

If a blocking issue is found, OOZIE-3206 and OOZIE-3208 would be
nice-to-have's in an RC1 (OOZIE-3207 is too risky).  Otherwise, we can save
them for Oozie 5.1.0.


thanks
- Robert


On Thu, Mar 29, 2018 at 11:00 AM, Attila Sasvari 
wrote:

> Thanks Andras for driving the release!
>
> +1
>
> Here is what I did:
> - Verified signature, hash
> - Created Oozie 5.0.0 distro using hadoop-2.6
> - On a pseudo Hadoop 2.6.0, I executed  SparkFileCopy (both in
> yarn-cluster, yarn-client mode),  custom-main-wf, datelist-java-main-wf,
> demo-wf, distcp-wf, hadoop-el-wf, java-main-wf, map-reduce-wf, no-op-wf,
> pig-wf, shell-wf, sqoop-freeform-wf, sqoop-wf, ssh-wf, streaming-wf,
> subwf-wf  - all the workflows have succeeded
> - Verified Web UI: version, getting logs and Job DAG
>
>
> On Wed, Mar 28, 2018 at 4:30 PM, Artem Ervits 
> wrote:
> >
> > additionally, I've ran the mvn verify and after some time it failed due
> to
> > timeout, still OK in my opinion.
> >
> > INFO] Results:
> > [INFO]
> > [INFO] Tests run: 738, Failures: 0, Errors: 0, Skipped: 0
> > [INFO]
> > [INFO]
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Apache Oozie Main 5.0.0  SUCCESS [
> > 3.859 s]
> > [INFO] Apache Oozie Client  SUCCESS [
> > 34.826 s]
> > [INFO] Apache Oozie Share Lib Oozie ... SUCCESS [
> > 25.740 s]
> > [INFO] Apache Oozie Share Lib HCatalog  SUCCESS [
> > 11.842 s]
> > [INFO] Apache Oozie Share Lib Distcp .. SUCCESS [
> > 7.465 s]
> > [INFO] Apache Oozie Core .. FAILURE [
> > 01:30 h]
> > [INFO] Apache Oozie Share Lib Streaming ... SKIPPED
> > [INFO] Apache Oozie Share Lib Pig . SKIPPED
> > [INFO] Apache Oozie Share Lib Hive  SKIPPED
> > [INFO] Apache Oozie Share Lib Hive 2 .. SKIPPED
> > [INFO] Apache Oozie Share Lib Sqoop ... SKIPPED
> > [INFO] Apache Oozie Examples .. SKIPPED
> > [INFO] Apache Oozie Share Lib Spark ... SKIPPED
> > [INFO] Apache Oozie Share Lib . SKIPPED
> > [INFO] Apache Oozie Docs .. SKIPPED
> > [INFO] Apache Oozie WebApp  SKIPPED
> > [INFO] Apache Oozie Tools . SKIPPED
> > [INFO] Apache Oozie MiniOozie . SKIPPED
> > [INFO] Apache Oozie Server  SKIPPED
> > [INFO] Apache Oozie Distro  SKIPPED
> > [INFO] Apache Oozie ZooKeeper Security Tests 5.0.0  SKIPPED
> > [INFO]
> > 
> > [INFO] BUILD FAILURE
> > [INFO]
> > 
> > [INFO] Total time: 01:32 h
> > [INFO] Finished at: 2018-03-28T14:21:30Z
> > [INFO]
> > 
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test
> (default-test)
> > on project oozie-core: There was a timeout or other error in the fork ->
> > [Help 1]
> > [ERROR]
> > [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e
> > switch.
> > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > [ERROR]
> > [ERROR] For more information about the errors and possible solutions,
> > please read the following articles:
> > [ERROR] [Help 1]
> > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> > [ERROR]
> > [ERROR] After correcting the problems, you can resume the build with the
> > command
> > [ERROR]   mvn  -rf :oozie-core
> >
> > On Wed, Mar 28, 2018 at 8:50 AM, Artem Ervits 
> wrote:
> >
> > > +1 non-binding
> > >
> > > signatures and checksum: OK
> > > build 

[jira] [Updated] (OOZIE-3208) "It should never happen" error code when running an action when Sharelib doesn't exist

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3208:
-
Description: 
If you forget to install the sharelib and you try to run a workflow, the action 
will fail (as expected), but the error code will be "It should never happen".  
That's not a good error code :D

{noformat}
2018-03-29 14:35:02,361  WARN ActionStartXCommand:523 - 
SERVER[rkanter-MBP.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] 
JOB[000-180329142919509-oozie-rkan-W] 
ACTION[000-180329142919509-oozie-rkan-W@mr-node] Error starting action 
[mr-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message 
[File /user/rkanter/share/lib does not exist]
org.apache.oozie.action.ActionExecutorException: File /user/rkanter/share/lib 
does not exist
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:646)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:760)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:746)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:984)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1512)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:334)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:263)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
{noformat}
Error starting action [mr-node]. ErrorType [FAILED], ErrorCode [It should never 
happen]
{noformat}

  was:
If you forget to install the sharelib and you try to run a workflow, the action 
will fail (as expected), but the error code will be "It should never happen".  
That's not a good error code :D

{noformat}
2018-03-29 14:35:02,361  WARN ActionStartXCommand:523 - 
SERVER[rkanter-MBP.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] 
JOB[000-180329142919509-oozie-rkan-W] 
ACTION[000-180329142919509-oozie-rkan-W@mr-node] Error starting action 
[mr-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message 
[File /user/rkanter/share/lib does not exist]
org.apache.oozie.action.ActionExecutorException: File /user/rkanter/share/lib 
does not exist
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:646)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:760)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:746)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:984)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1512)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:334)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:263)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}


> "It should never happen" error code when running an action when Sharelib 
> doesn't exist
> --
>
> Key: OOZIE-3208
> URL: https://issues.apache.org/jira/browse/OOZIE-3208
> Project: Oozie
>  Issue Type: Bug
>    Aff

[jira] [Created] (OOZIE-3208) "It should never happen" error code when running an action when Sharelib doesn't exist

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3208:


 Summary: "It should never happen" error code when running an 
action when Sharelib doesn't exist
 Key: OOZIE-3208
 URL: https://issues.apache.org/jira/browse/OOZIE-3208
 Project: Oozie
  Issue Type: Bug
Affects Versions: 5.0.0
Reporter: Robert Kanter
 Fix For: 5.1.0


If you forget to install the sharelib and you try to run a workflow, the action 
will fail (as expected), but the error code will be "It should never happen".  
That's not a good error code :D

{noformat}
2018-03-29 14:35:02,361  WARN ActionStartXCommand:523 - 
SERVER[rkanter-MBP.local] USER[rkanter] GROUP[-] TOKEN[] APP[map-reduce-wf] 
JOB[000-180329142919509-oozie-rkan-W] 
ACTION[000-180329142919509-oozie-rkan-W@mr-node] Error starting action 
[mr-node]. ErrorType [FAILED], ErrorCode [It should never happen], Message 
[File /user/rkanter/share/lib does not exist]
org.apache.oozie.action.ActionExecutorException: File /user/rkanter/share/lib 
does not exist
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.addSystemShareLibForAction(JavaActionExecutor.java:646)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.addAllShareLibs(JavaActionExecutor.java:760)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.setLibFilesArchives(JavaActionExecutor.java:746)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:984)
at 
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1512)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at 
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:334)
at 
org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:263)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3207) Update ASF root pom version

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3207:


 Summary: Update ASF root pom version
 Key: OOZIE-3207
 URL: https://issues.apache.org/jira/browse/OOZIE-3207
 Project: Oozie
  Issue Type: Bug
  Components: build
Affects Versions: 5.0.0
Reporter: Robert Kanter
 Fix For: 5.1.0


The Oozie root pom uses the ASF root pom as it's parent (as required by ASF 
rules).  We're currently using version 17, which is from 2015.  The current 
version is 19, from earlier this year (2018).  We should update this.  

You can see more details about the ASF root pom, including diffs between each 
version here:
https://maven.apache.org/pom/asf/index.html

This may require some testing or at least looking into because the ASF root pom 
defines the versions of some things, especially maven plugins, and so things 
may break or be incompatible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3206) The README.txt file refers to the 4.3.0 Quickstart docs

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3206:


 Summary: The README.txt file refers to the 4.3.0 Quickstart docs
 Key: OOZIE-3206
 URL: https://issues.apache.org/jira/browse/OOZIE-3206
 Project: Oozie
  Issue Type: Bug
  Components: docs
Affects Versions: 5.0.0
Reporter: Robert Kanter
 Fix For: 5.1.0


The README.txt file refers to the 4.3.0 Quickstart docs.  It should refer to 
whatever the current version we're on is.
http://oozie.apache.org/docs/4.3.0/DG_QuickStart.html 

It might be a good idea to update our website with a /docs/current/*.html site 
so that we don't have to keep updating this, as long as we update the "current".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2339) Provide an API for writing jobs based on the XSD schemas

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16414845#comment-16414845
 ] 

Robert Kanter commented on OOZIE-2339:
--

I just left some more comments on ReviewBoard.  Though in the interest of time, 
I didn't look too carefully at some of the generating code like the 
{{DozerConverter}} classes and a lot of the unit tests.

One high-level thing I wanted to bring up is what are we calling this feature?  
It looks like it's currently "Jobs API", but I have a few concerns about that 
name:
# It's pretty generic-sounding and non-specific.  
# "Jobs" is a pretty overloaded term already.  It refers to Oozie Jobs (i.e. 
Workflows, Coordinators, and Bundles), Hadoop/Pig/Sqoop/etc Jobs, the Launcher 
Job, etc.  I'd like to avoid "Jobs" if possible, or at least qualify it with 
something else like "Programatic Jobs API" or something.

I don't have a great alternative for a name, other than "Programatic Jobs API", 
which isn't super great either.  Any ideas?
 

> Provide an API for writing jobs based on the XSD schemas
> 
>
> Key: OOZIE-2339
> URL: https://issues.apache.org/jira/browse/OOZIE-2339
> Project: Oozie
>  Issue Type: New Feature
>      Components: client
>Affects Versions: 4.3.0
>Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Major
> Fix For: 5.1.0
>
> Attachments: OOZIE-2339.001.patch, OOZIE-2339.002.patch, 
> OOZIE-2339.003.patch, OOZIE-2339.004.patch, OOZIE-2339.005.patch, 
> OOZIE-2339.006.patch, OOZIE-2339.008.patch, OOZIE-2339.010.patch, 
> OOZIE-2339.011.patch
>
>
> Users often complain about the XML they have to write for Oozie jobs.  It 
> would be nice if they could write them in something like Java, but we don't 
> want to have to maintain a separate Java API for this.  I was looking around 
> and saw that JAXB might be the right thing here.  From what I can tell, it 
> lets you create Java classes from XSD schemas.  So, we should be able to 
> auto-generate a Java API for writing Oozie jobs, without having to really 
> maintain it.
> We should investigate if this is feasible and, if so, implement it.
> Some useful looking links:
> * [JAXB 
> overview|https://en.wikipedia.org/wiki/Java_Architecture_for_XML_Binding]
> * [JAXB description|https://jaxb.java.net/2.2.11/docs/ch03.html]
> * [Maven JAXB plugin|https://java.net/projects/maven-jaxb2-plugin/pages/Home]
> * [Apache Falcon|https://falcon.apache.org]
> Key features:
> * must have:
> ** inside an {{oozie-jobs-api}} artifact
> ** able to create workflow / coordinator / bundle definitions programmatically
> ** synchronizing each and every XSD change on rebuild
> ** can write {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** cloneability of workflow etc. {{Object}} s
> ** perform cross checks, e.g. that the workflow graph is a DAG
> ** only latest XSD versions should be supported as must have
> * nice to have:
> ** XSD version(s) can be provided. When not provided, latest ones are 
> considered as valid
> ** implement a [*fluent API*|https://en.wikipedia.org/wiki/Fluent_interface]
> ** have a Python / Jython REPL to make it easy to experiment
> ** create documentation about usage
> ** can read {{workflow.xml}}, {{coordinator.xml}}, {{bundle.xml}}, and 
> {{jobs.properties}} artifacts of every XSD version
> ** can convert between XSD versions
> ** support XSD change on the fly (within REPL)
> ** support HDFS reads / writes
> ** support dry run on an Oozie server to perform checks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Robert Kanter via Review Board]

 for showing the new API.

It would be great if we could add some examples to the examples module so 
users could more easily play with it.  As you know, many users copy our example 
XML workflows as a starting place, so it would be good to provide some Java 
ones too.  This can be done as a followup JIRA: please file one.



docs/src/site/twiki/DG_JobsAPI.twiki
Lines 198 (patched)
<https://reviews.apache.org/r/64029/#comment280584>

"like this:"



docs/src/site/twiki/DG_JobsAPI.twiki
Lines 277 (patched)
<https://reviews.apache.org/r/64029/#comment280585>

Again, let's not say how bad XML is:

"tries to abstract away the task of assembly job"



docs/src/site/twiki/DG_JobsAPI.twiki
Lines 290-297 (patched)
<https://reviews.apache.org/r/64029/#comment280586>

These are empty



jobs/jobs-api/pom.xml
Lines 111-120 (patched)
<https://reviews.apache.org/r/64029/#comment280590>

It's unfortunate that we have to keep this manually :(

I hope we don't forget to update this when adding new schema versions.  
Perhaps there's some way to write a unit test that would fail if a developer 
added a newer schema version and didn't update this?  That would prevent this 
issue.



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/action/ActionAttributes.java
Lines 29 (patched)
<https://reviews.apache.org/r/64029/#comment280592>

Now that we're on Yarn, we should rename all jobTracker and similar 
to resourceManager and similar.  The earlier version of the schemas the 
Jobs API is supporting is 1.0 (and friends), which all allow for 
 so we should use that and just never even start with 
anything Job Tracker.  The generated XML should also use 
 only too.



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/action/ErrorHandler.java
Lines 23-34 (patched)
<https://reviews.apache.org/r/64029/#comment280593>

It would be good to add info about this to the twiki docs.  It's a common 
pattern to have an email action from an error transition before hitting a kill 
node.  Adding an example of that in the twiki docs would be great too :)



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/action/Launcher.java
Lines 38 (patched)
<https://reviews.apache.org/r/64029/#comment280594>

This constructor should be package private, right?



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/factory/SimpleWorkflowFactory.java
Lines 34 (patched)
<https://reviews.apache.org/r/64029/#comment280596>

I think this should either be in the test archive/package and/or in the 
example artifact.



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/oozie/dag/Decision.java
Lines 32 (patched)
<https://reviews.apache.org/r/64029/#comment280597>

Should we make this, and other nodes that the user shouldn't need to use, 
private (or at least package-private the constructor)?



jobs/jobs-client/pom.xml
Lines 39 (patched)
<https://reviews.apache.org/r/64029/#comment280600>

I don't think setting the  here should be necessary.



jobs/jobs-client/pom.xml
Lines 45 (patched)
<https://reviews.apache.org/r/64029/#comment280601>

I don't think setting the  here should be necessary.



jobs/jobs-client/pom.xml
Lines 72 (patched)
<https://reviews.apache.org/r/64029/#comment280599>

I don't think setting the  here should be necessary.



jobs/jobs-client/pom.xml
Lines 78 (patched)
<https://reviews.apache.org/r/64029/#comment280602>

I don't think setting the  here should be necessary.



jobs/jobs-client/pom.xml
Lines 90 (patched)
<https://reviews.apache.org/r/64029/#comment280603>

I don't think setting the  here should be necessary.



jobs/jobs-client/src/test/java/org/apache/oozie/jobs/client/examples/TestJavaMainExample.java
Lines 35 (patched)
<https://reviews.apache.org/r/64029/#comment280604>

What are the "Test___Example" classes for?  They look like they're unit 
tests but don't actually assert anything.  While it's nice to have examples of 
all of the actions, I don't think anyone's going to look at these.  Users won't 
because they won't know to look here for examples, and Oozie devs won't because 
these won't be failing (unless something is really wrong).  

Perhaps we should repurpose these for actual examples in the examples 
artifact instead?  I suggested in a different comment that it would be good to 
have these because users often start with our XML examples when writing 
workflows.  We can make this a followup JIRA if you want.

In their current state, I'm not sure these are helping anyone.


- Robert Kanter


On March 20, 2018, 9:58 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> --

[jira] [Commented] (OOZIE-2600) OYA: Update Documentation

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408454#comment-16408454
 ] 

Robert Kanter commented on OOZIE-2600:
--

Thanks for taking care of this!

> OYA: Update Documentation
> -
>
> Key: OOZIE-2600
> URL: https://issues.apache.org/jira/browse/OOZIE-2600
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: oya, 5.0.0
>    Reporter: Robert Kanter
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-2600.001.patch, OOZIE-2600.002.patch, 
> OOZIE-2600.003.patch, OOZIE-2600.004.patch
>
>
> Based on the work done in OOZIE-2896, document also the way how 
> {{oozie.launcher.override.\*}} and {{oozie.launcher.prepend.\*}} parameters 
> function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3197) Can you please guide me how to to exception handling at action level in oozie

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405182#comment-16405182
 ] 

Robert Kanter commented on OOZIE-3197:
--

[~kunu] JIRA is for bugs, feature requests, and other improvements.  For help 
_using_ Oozie, the user mailing list is more appropriate:
https://oozie.apache.org/mail-lists.html

You're using the Hortonworks distribution of Oozie, so you can also ask on 
Hortonwork's support and/or forums as well.

> Can you please guide me how to to exception handling at action level in oozie
> -
>
> Key: OOZIE-3197
> URL: https://issues.apache.org/jira/browse/OOZIE-3197
> Project: Oozie
>  Issue Type: Bug
>  Components: action
> Environment: i am using oozie 4.2 and distribution is hortonworks.
>Reporter: kunal kishore
>Priority: Blocker
>
> I am developing workflow and i would like to do exception handling at action 
> level, if it is possible to implement exception handling at action level.
>  
> Please guide me with example or code snippet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3197) Can you please guide me how to to exception handling at action level in oozie

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3197:
-
Priority: Minor  (was: Blocker)

> Can you please guide me how to to exception handling at action level in oozie
> -
>
> Key: OOZIE-3197
> URL: https://issues.apache.org/jira/browse/OOZIE-3197
> Project: Oozie
>  Issue Type: Bug
>  Components: action
> Environment: i am using oozie 4.2 and distribution is hortonworks.
>Reporter: kunal kishore
>Priority: Minor
>
> I am developing workflow and i would like to do exception handling at action 
> level, if it is possible to implement exception handling at action level.
>  
> Please guide me with example or code snippet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405175#comment-16405175
 ] 

Robert Kanter commented on OOZIE-3189:
--

I already updated the wiki :)

I think you have to file an INFRA JIRA to ask for edit permissions on the wiki. 
 

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch, OOZIE-3189.002.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16399576#comment-16399576
 ] 

Robert Kanter commented on OOZIE-3189:
--

Test failures are unrelated

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch, OOZIE-3189.002.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3189:
-
Attachment: OOZIE-3189.002.patch

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch, OOZIE-3189.002.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3189:
-
Priority: Blocker  (was: Major)

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Blocker
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390491#comment-16390491
 ] 

Robert Kanter commented on OOZIE-3189:
--

Test failures unrelated (no code changes).

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390339#comment-16390339
 ] 

Robert Kanter commented on OOZIE-3189:
--

The patch replaces the code generating the md5 with code generating the sha512 
as prescribed by the ASF.  I also ran ShellCheck and fixed a bunch of minor 
things it was complaining about.  I did run into a complication because the 
command to generate the sha512 has to use a {{>}} to redirect stdout, and that 
doesn't work so great with our {{run}} function.  I worked around it by 
duplicating a few things, but if there's a cleaner way, let me know.

I'll update the wiki shortly.

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3189:
-
Attachment: OOZIE-3189.001.patch

> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: OOZIE-3189.001.patch
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3189:
-
Description: 
Apache has updated it's policy on the release signatures, as per it's website 
[here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
recent email. Basically, all future releases should be providing a sha512 
checksum instead of an md5 one.

There are two tasks:
 # Update the release script to use sha512 instead of md5
 [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
 [https://www.apache.org/dev/release-signing#sha-checksum]
 # Update the wiki (requires committer/pmc permissions?)
 [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]

While we're updating the wiki, we should add details on:
 # Making sure the gpg key used for signing releases is 4096 bit RSA
 # Publishing your gpg public key to a key server 
([https://www.apache.org/dev/release-signing#keyserver])

  was:
Apache has updated it's policy on the release signatures, as per it's website 
[here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
recent email. Basically, all future releases should be providing a sha512 
checksum instead of an md5 one.

There are two tasks:
 # Update the release script to use sha512 instead of md5
 [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
 # Update the wiki (requires committer/pmc permissions?)
 [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]

While we're updating the wiki, we should add details on:
 # Making sure the gpg key used for signing releases is 4096 bit RSA
 # Publishing your gpg public key to a key server 
([https://www.apache.org/dev/release-signing#keyserver])


> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Major
> Fix For: 5.0.0
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  [https://www.apache.org/dev/release-signing#sha-checksum]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3189:
-
Description: 
Apache has updated it's policy on the release signatures, as per it's website 
[here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
recent email. Basically, all future releases should be providing a sha512 
checksum instead of an md5 one.

There are two tasks:
 # Update the release script to use sha512 instead of md5
 [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
 # Update the wiki (requires committer/pmc permissions?)
 [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]

While we're updating the wiki, we should add details on:
 # Making sure the gpg key used for signing releases is 4096 bit RSA
 # Publishing your gpg public key to a key server 
([https://www.apache.org/dev/release-signing#keyserver])

  was:
Apache has updated it's policy on the release signatures, as per it's website 
[here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
recent email.  Basically, all future releases should be providing a sha512 
checksum instead of an md5 one.

There are two tasks:
# Update the release script to use sha512 instead of md5
https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71
# Update the wiki (requires committer/pmc permissions?)
https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release

While we're updating the wiki, we should add details on:
# Making sure the gpg key used for signing releases is 4096 bit RSA
# Publishing your gpgp public key to a key server 
(https://www.apache.org/dev/release-signing#keyserver)


> Update the release script and wiki page to use sha512 instead of md5
> 
>
> Key: OOZIE-3189
> URL: https://issues.apache.org/jira/browse/OOZIE-3189
> Project: Oozie
>  Issue Type: Improvement
>  Components: scripts
>    Reporter: Robert Kanter
>    Assignee: Robert Kanter
>Priority: Major
> Fix For: 5.0.0
>
>
> Apache has updated it's policy on the release signatures, as per it's website 
> [here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
> recent email. Basically, all future releases should be providing a sha512 
> checksum instead of an md5 one.
> There are two tasks:
>  # Update the release script to use sha512 instead of md5
>  [https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71]
>  # Update the wiki (requires committer/pmc permissions?)
>  [https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release]
> While we're updating the wiki, we should add details on:
>  # Making sure the gpg key used for signing releases is 4096 bit RSA
>  # Publishing your gpg public key to a key server 
> ([https://www.apache.org/dev/release-signing#keyserver])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (OOZIE-3189) Update the release script and wiki page to use sha512 instead of md5

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3189:


 Summary: Update the release script and wiki page to use sha512 
instead of md5
 Key: OOZIE-3189
 URL: https://issues.apache.org/jira/browse/OOZIE-3189
 Project: Oozie
  Issue Type: Improvement
  Components: scripts
Reporter: Robert Kanter
Assignee: Robert Kanter
 Fix For: 5.0.0


Apache has updated it's policy on the release signatures, as per it's website 
[here|https://www.apache.org/dev/release-distribution#sigs-and-sums] and a 
recent email.  Basically, all future releases should be providing a sha512 
checksum instead of an md5 one.

There are two tasks:
# Update the release script to use sha512 instead of md5
https://github.com/apache/oozie/blob/master/bin/create-release-artifact#L71
# Update the wiki (requires committer/pmc permissions?)
https://cwiki.apache.org/confluence/display/OOZIE/How+To+Release

While we're updating the wiki, we should add details on:
# Making sure the gpg key used for signing releases is 4096 bit RSA
# Publishing your gpgp public key to a key server 
(https://www.apache.org/dev/release-signing#keyserver)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357250#comment-16357250
 ] 

Robert Kanter commented on OOZIE-3163:
--

Hmmm, it would be okay with me, but I think technically we would need a new RC 
because of the tagging, md5, etc.  Let's leave the RC as-is (so the release 
will have the old docs), but we can put the new skin on the 4.3.1 docs on the 
Oozie website.  I imagine most users look at the docs there anyway, rather than 
on the copy in the Oozie Server.

> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1(candidate 4)

[Robert Kanter]

Here's what I did:
- Verified md5
- Verified signature (gpg)
- Looked at rat report
- Looked at release log
- Verified Tomcat is version 6.0.53
- Built Oozie against Hadoop 2.7.2, using an empty local maven repo
- Ran a few of the examples, clicked around the UI, tried various CLI
commands, looked at the REST output, launcher log output, etc


+1

On Tue, Feb 6, 2018 at 7:24 PM, Artem Ervits  wrote:

> Attila, AFAIK, these properties are in my core-site.xml except for last
> one, mapreduce.framework.name is in mapred-site.xml. I'll have to double
> check on the proxyuser.root.hosts as I believe my service runs with user
> oozie not root and get back to you.
>
> On Tue, Feb 6, 2018 at 3:49 PM, Attila Sasvari 
> wrote:
>
> > +1 (binding)
> >
> > - Verified MD5 hash, signature, rat_report, release log are OK
> > - Built Oozie against Hadoop 2.7.5 (did not build against other Hadoop
> > versions this time)
> > - Installed Oozie sharelib
> > - Some example workflows I submitted to a 1-node Hadoop 2.7.5 (running in
> > pseudo distributed mode for tests) succeeded: ( aggregator bundle
> > coord-input-logic cron cron-schedule custom-main datelist-java-main demo
> > distcp hadoop-el hcatalog java-main map-reduce no-op pig shell sla sqoop
> > sqoop-freeform ssh streaming subwf )
> > - Verified basic functionality of the Web UI (version, Job DAG, Job logs)
> >
> >
> > Artem, have you specified conf/hadoop-conf/core-site.xml with all the
> > details Oozie needs to communicate with a Hadoop cluster?
> >
> > In my local environment, I use a pseudo Hadoop 2.7.5. My
> > conf/hadoop-conf/core-site.xml is as follows:
> > 
> > 
> > 
> > 
> > 
> >   
> > fs.defaultFS
> > hdfs://localhost:9000
> >   
> >   
> > hadoop.proxyuser.root.hosts
> > *
> >   
> >   
> > hadoop.proxyuser.root.groups
> > *
> >   
> > 
> > mapreduce.framework.name
> > yarn
> > 
> > 
> >
> >
> > On Tue, Feb 6, 2018 at 8:10 PM, Artem Ervits 
> > wrote:
> >
> > > jdk: 1.8.0_161
> > > hadoop: 2.7.5
> > > md5: OK
> > > curl http://localhost:11000/oozie/v1/admin/status
> > > {"systemMode":"NORMAL"}
> > >
> > > curl http://localhost:11000/oozie/v2/admin/status
> > > {"systemMode":"NORMAL"}
> > >
> > >
> > > I also got in oozie-ops.log
> > >
> > >   STARTUP MSG: Oozie BUILD_VERSION [4.3.1] compiled by [oozie] on
> > > [2018.02.06-16:30:43GMT]
> > >   STARTUP MSG:   revision [unavailable]@[unavailable]
> > >
> > > so perhaps we may want to add but not critical OOZIE-2004, OOZIE-3083,
> > > OOZIE-3152.
> > >
> > >  submitted a job and got but that is most likely my environment.
> > Otherwise
> > > +1
> > >
> > > 2018-02-06 18:09:37,635  WARN ActionStartXCommand:523 - SERVER[
> > > hadoop.example.com] USER[vagrant] GROUP[-] TOKEN[] APP[python-wf]
> > > JOB[000-180206180622504-oozie-oozi-W]
> > > ACTION[000-180206180622504-oozie-oozi-W@python-node] Error
> starting
> > > action [python-node]. ErrorType [ERROR], ErrorCode
> > > [UnsupportedOperationException], Message [
> UnsupportedOperationException:
> > > Accessing local file system is not allowed]
> > > org.apache.oozie.action.ActionExecutorException:
> > > UnsupportedOperationException: Accessing local file system is not
> allowed
> > > at
> > > org.apache.oozie.action.ActionExecutor.convertException(Acti
> > > onExecutor.java:446)
> > > at
> > > org.apache.oozie.action.hadoop.JavaActionExecutor.submitLaun
> > > cher(JavaActionExecutor.java:1204)
> > > at
> > > org.apache.oozie.action.hadoop.JavaActionExecutor.start(Java
> > > ActionExecutor.java:1386)
> > > at
> > > org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> > > onStartXCommand.java:234)
> > > at
> > > org.apache.oozie.command.wf.ActionStartXCommand.execute(Acti
> > > onStartXCommand.java:65)
> > > at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> > > at
> > > org.apache.oozie.service.CallableQueueService$CompositeCalla
> > > ble.call(CallableQueueService.java:332)
> > > at
> > > org.apache.oozie.service.CallableQueueService$CompositeCalla
> > > ble.call(CallableQueueService.java:261)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at
> > > org.apache.oozie.service.CallableQueueService$CallableWrappe
> > > r.run(CallableQueueService.java:179)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> > > Executor.java:1149)
> > > at
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> > > lExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > > Caused by: java.lang.UnsupportedOperationException: Accessing local
> file
> > > system is not allowed
> > > at
> > > org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalF
> > > ileSystem.java:48)
> > > at org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSys
> > > tem.java:47)
> > > at org.apache.hadoop.fs.FileSystem.createFileSystem(
> > FileSystem.java:2667)
> > > at 

[jira] [Commented] (OOZIE-3163) Improve documentation rendering: use fluido skin and better config

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356038#comment-16356038
 ] 

Robert Kanter commented on OOZIE-3163:
--

Wow!  That looks so much better.  Thanks for taking care of this.

> Improve documentation rendering: use fluido skin and better config
> --
>
> Key: OOZIE-3163
> URL: https://issues.apache.org/jira/browse/OOZIE-3163
> Project: Oozie
>  Issue Type: Task
>  Components: docs
>Affects Versions: 4.3.0
>Reporter: Hervé Boutemy
>Assignee: Hervé Boutemy
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: 34.patch, oozie-docs-coordinator-fs-fluido.png, 
> oozie-docs-coordinator-fs-old.png, oozie-docs-index-fluido.png, 
> oozie-docs-index-old.png
>
>
> Current output is really ugly: as Maven maintainer, I can't let Oozie with 
> such awful result :)
> Even with old maven-site-plugin 2.0-beta-6 (we'll see later how to upgrade), 
> using Fluido skin would improve output a lot
> Pull Request coming



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349303#comment-16349303
 ] 

Robert Kanter commented on OOZIE-2814:
--

Good thing we're trying to use the new schemas ;)

I'd also say we should change {{jobTracker}} to {{resourceManager}} as well:
{code:xml}
${resourceManager}
{code}


> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>    Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Welcoming new Oozie PMC members: Attila Sasvari, Peter Cseh, and Peter Bacsko

[Robert Kanter]

Hello Oozie community,

It's my pleasure to announce that Attila Sasvari, Peter Cseh, and Peter
Bascko
have become the newest additions to the Oozie PMC.

Please join me congratulating them.
Congrats!


- Robert, on behalf of the Oozie PMC

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347605#comment-16347605
 ] 

Robert Kanter commented on OOZIE-2814:
--

I think we should also update {{}} to {{}} to 
encourage users to start using it.

> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>    Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Blocker
> Attachments: OOZIE-2814-00.patch
>
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (OOZIE-2714) Detect conflicting resources during class loading

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345566#comment-16345566
 ] 

Robert Kanter commented on OOZIE-2714:
--

Plus, I discovered not long ago that Hadoop's Configuration does some funny 
stuff with the classloader too.  

Anyway, I don't know much about it, but JHades sounds promising.

> Detect conflicting resources during class loading
> -
>
> Key: OOZIE-2714
> URL: https://issues.apache.org/jira/browse/OOZIE-2714
> Project: Oozie
>  Issue Type: New Feature
>  Components: core
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: ClassLoaderTest.java, OOZIE-2714-POC01.patch, 
> OOZIE-2714-POC02.patch
>
>
> There are a bunch of issues in Oozie which are related to class loading. 
> The main problem is that the classpath is constructed in a way which is very 
> specific to Oozie:
> - Hadoop lib jars
> - Sharelib jars
> - User-defined jars
> Sometimes there is a conflict between sharelib and hadoop lib version. Also, 
> users can add their own jars which sometimes contain a different version of 
> popular libraries such as Guava, Apache commons, etc.
> We should be able to detect these conflicts and print exact error message so 
> that Oozie users can take appropriate actions to resolve the problem.
> A possible approach is the following:
> * start the execution of an action on a different thread
> * replace the thread's context classloader with a classloader which can 
> detect conflicts
> * when the JVM invokes the {{loadClass()}} method of the classloader, it  
> scans through the jars (which are available as {{URLClassPath}} objects). If 
> it finds the given resource in at least two jars, it can do different things 
> depending on the setup:
> ** throws an error immediately, mentioning the conflicting jars (this is 
> probably too strict - but still an option)
> ** loads the two resource into a byte array and compares them - it only 
> throws an error if there is difference
> ** compares the jars but only emits an error message if there is a conflict
> ** something else (user defined action?)
> Implementing such a classloader is not difficult and would greatly enhance 
> the supportability of Oozie. It could work in multiple modes depending on the 
> setup - perhaps being able to control it from a workflow config is desirable. 
> If there's any problem, we should be able to turn it off completely, too.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 65385: OOZIE-1717 add indices to speed up db queries

[Robert Kanter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65385/#review196444
---




tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
Line 64 (original), 64 (patched)
<https://reviews.apache.org/r/65385/#comment276097>

It looks like we started using DB version 3 in Oozie 4.1.  Instead of 
DB_VERSION_PRE_5_0, let's rename this to DB_VERSION_FOR_4_1.  

Also, why are these two not private?



tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java
Lines 750 (patched)
<https://reviews.apache.org/r/65385/#comment276100>

Shouldn't we write the statements to a file in this case?


- Robert Kanter


On Jan. 29, 2018, 3:04 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65385/
> ---
> 
> (Updated Jan. 29, 2018, 3:04 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Add indices to speed up SELECT operations.
> 
> 
> Index user_name column
> 
> for purge service OOZIE-1803:
> CREATE INDEX "OOZIEDB"."BUNDLE_JOBS_END_TIME_IDX" ON "BUNDLE_JOBS" 
> ("END_TIME");
> CREATE INDEX "OOZIEDB"."COORD_JOBS_END_TIME_IDX" ON "COORD_JOBS" ("END_TIME");
> CREATE INDEX "OOZIEDB"."WF_JOBS_END_TIME_IDX" ON "WF_JOBS" ("END_TIME");
> CREATE INDEX "OOZIEDB"."COORD_ACTIONS_JOB_ID_IDX" ON "COORD_ACTIONS" 
> ("JOB_ID");
> CREATE INDEX "OOZIEDB"."COORD_JOBS_BUNDLE_ID_IDX" ON "COORD_JOBS" 
> ("BUNDLE_ID");
> CREATE INDEX "OOZIEDB"."WF_ACTIONS_WF_ID_IDX" ON "WF_ACTIONS" ("WF_ID");
> CREATE INDEX "OOZIEDB"."WF_JOBS_PARENT_ID_IDX" ON "WF_JOBS" ("PARENT_ID");
> CREATE INDEX "OOZIEDB"."BUNDLE_ACTIONS_BUNDLE_ID_IDX" ON "BUNDLE_ACTIONS" 
> ("BUNDLE_ID");
> 
> Composite indices to speed up operations on the web UI
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/BundleJobBean.java 
> b42f12841c2f9cb6d508cd2ac2f393e6407d1cf0 
>   core/src/main/java/org/apache/oozie/CoordinatorJobBean.java 
> 99c7d3c474d9e1e22e2f5a927c03d71b9292b663 
>   core/src/main/java/org/apache/oozie/WorkflowJobBean.java 
> cd9a6df42e552a0607606a244b81978747d89b41 
>   tools/src/main/java/org/apache/oozie/tools/OozieDBCLI.java 
> a8b2234966c967f9ad9d722628bd486012b14bd3 
> 
> 
> Diff: https://reviews.apache.org/r/65385/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: [ANNOUNCE] Apache Oozie 5.0.0-beta1 released

[Robert Kanter]

Thanks Attila and everyone!  It's good to see Oozie 5 moving forward.


- Robert

On Fri, Jan 26, 2018 at 8:14 AM, Peter Cseh  wrote:

> Thanks everybody who has been working on this release! Special thanks for
> Attila for putting the release together!
>
> Gp
>
>
>
> On Jan 25, 2018 21:33, "Attila Sasvári"  wrote:
>
> > Hello Oozie Community,
> >
> > The Apache Oozie team is pleased to announce the release of Oozie
> > 5.0.0-beta1.
> > Oozie 5.0.0-beta1 includes new features, bug fixes and minor
> improvements.
> >
> > The details of the release and download location can be found at
> > http://oozie.apache.org/
> >
> > The full release notes can be found at the download location, but some of
> > the highlights of the release include:
> >
> > - Migrated Oozie Launcher from MapReduce to YARN AM for significant
> > performance improvements and simplicity
> > - Dropped support for Hadoop 1.x
> > - Added new 1.0 workflow schemas
> > - Switched from Tomcat to embedded Jetty for easier management and better
> > security
> > - Completely rewritten Job DAG renderer to support SVG and DOT formats as
> > well as better performance
> > - Better resiliency to transient database outages
> > - Requires Java 8
> >
> > Many thanks to all the developers, QA's, users and committers for making
> > this release happen.
> >
> > Regards,
> > - Attila
> >
>

[jira] [Resolved] (OOZIE-1266) Oozie MapReduce failing to run

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter resolved OOZIE-1266.
--
   Resolution: Cannot Reproduce
Fix Version/s: (was: 3.3.1)

> Oozie MapReduce failing to run 
> ---
>
> Key: OOZIE-1266
> URL: https://issues.apache.org/jira/browse/OOZIE-1266
> Project: Oozie
>  Issue Type: Bug
>  Components: examples
>Affects Versions: 3.3.1
>Reporter: srikanth ayalasomayajulu
>Priority: Major
>  Labels: hadoop023, mapreduce, oozie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Installed oozie 3.3.1 on apache hadoop 0.23.3, when trying to run an 
> mapreduce example on the oozie, the following error is occurring.
> 2013-03-11 15:29:27,760  INFO ActionStartXCommand:539 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@:start:] Start action 
> [000-130311152909237-oozie-srik-W@:start:] with user-retry state : 
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2013-03-11 15:29:27,761  WARN ActionStartXCommand:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@:start:] 
> [***000-130311152909237-oozie-srik-W@:start:***]Action status=DONE
> 2013-03-11 15:29:27,762  WARN ActionStartXCommand:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@:start:] 
> [***000-130311152909237-oozie-srik-W@:start:***]Action updated in DB!
> 2013-03-11 15:29:28,106  INFO ActionStartXCommand:539 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] Start action 
> [000-130311152909237-oozie-srik-W@mr-node] with user-retry state : 
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2013-03-11 15:29:28,447  WARN MapReduceActionExecutor:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] credentials is null for 
> the action
> 2013-03-11 15:29:31,521  INFO MapReduceActionExecutor:539 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] checking action, 
> external ID [job_1362986780726_0001] status [RUNNING]
> 2013-03-11 15:29:31,524  WARN ActionStartXCommand:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] 
> [***000-130311152909237-oozie-srik-W@mr-node***]Action status=RUNNING
> 2013-03-11 15:29:31,524  WARN ActionStartXCommand:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] 
> [***000-130311152909237-oozie-srik-W@mr-node***]Action updated in DB!
> 2013-03-11 15:40:26,462  INFO MapReduceActionExecutor:539 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] action completed, 
> external ID [job_1362986780726_0001]
> 2013-03-11 15:40:26,483  WARN MapReduceActionExecutor:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] LauncherMapper died, 
> check Hadoop log for job [localhost:8040:job_1362986780726_0001]
> 2013-03-11 15:40:26,615  INFO ActionEndXCommand:539 - USER[srikanth] GROUP[-] 
> TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@mr-node] ERROR is considered as 
> FAILED for SLA
> 2013-03-11 15:40:26,669  INFO ActionStartXCommand:539 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@fail] Start action 
> [000-130311152909237-oozie-srik-W@fail] with user-retry state : 
> userRetryCount [0], userRetryMax [0], userRetryInterval [10]
> 2013-03-11 15:40:26,670  WARN ActionStartXCommand:542 - USER[srikanth] 
> GROUP[-] TOKEN[] APP[map-reduce-wf] JOB[000-130311152909237-oozie-srik-W] 
> ACTION[000-130311152909237-oozie-srik-W@fail] 
> [***000-130311152909237-oozie-srik-W@fail***]Action status=DONE
> 2013-03-11 15:40:26,67

[jira] [Updated] (OOZIE-3169) Create a download page for releases

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3169:
-
Description: 
Today to download a release, you need to click on the Releases tab and select a 
release. It will take a user to an external page where you can download the 
release from an Apache mirrors (e.g.  
[http://www.apache.org/dyn/closer.lua/oozie/5.0.0-beta1).]  

It would be better to create a page on the Oozie site to download releases and 
related hash and PGP signature files (these are often not replicated to 
mirrors, and they should be hosted on Apache anyway).  This way a visitor could 
stay on the Oozie site.


Examples:
[http://hadoop.apache.org/releases.html]

[https://flume.apache.org/releases/]

 

  was:
Today to download a release, you need to click on the Releases tab and select a 
release. It will take a user to an external page where you can download the 
release from an Apache mirrors (e.g.  
[http://www.apache.org/dyn/closer.lua/oozie/5.0.0-beta1).]  

It would be better to create a page on the Oozie site to download releases and 
related hash and PGP signature files.  This way a visitor could stay on the 
Oozie site.


Examples:
[http://hadoop.apache.org/releases.html]

[https://flume.apache.org/releases/]

 


> Create a download page for releases
> ---
>
> Key: OOZIE-3169
> URL: https://issues.apache.org/jira/browse/OOZIE-3169
> Project: Oozie
>  Issue Type: New Feature
>  Components: site
>Reporter: Attila Sasvari
>Priority: Minor
>
> Today to download a release, you need to click on the Releases tab and select 
> a release. It will take a user to an external page where you can download the 
> release from an Apache mirrors (e.g.  
> [http://www.apache.org/dyn/closer.lua/oozie/5.0.0-beta1).]  
> It would be better to create a page on the Oozie site to download releases 
> and related hash and PGP signature files (these are often not replicated to 
> mirrors, and they should be hosted on Apache anyway).  This way a visitor 
> could stay on the Oozie site.
> Examples:
> [http://hadoop.apache.org/releases.html]
> [https://flume.apache.org/releases/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release Oozie 4.3.1 (candidate 2)

[Robert Kanter]

Here's what I did:
- Verified md5
- Verified signature (gpg)
- Looked at rat report
- Looked at release log
- Verified Tomcat is version 6.0.53
- Built Oozie against Hadoop 2.7.2, using an empty local maven repo


I ran into another problem: OOZIE-2533.  Basically, newer versions of Java
+ newer versions of Tomcat 6 break JSP compilation so the Oozie Web UI is
completely broken.  The JIRA cites Java 8u91 and later, but I still ran
into this even when I tried with Java 7u79.  I guess whatever breaks this
in 8u91 must have been backported to 7u79.  I heard that it was for some
Java security fix so there's not a lot of detail on it.  Applying the patch
from OOZIE-2533 appears to fix it.  We've actually been using that patch in
CDH for almost 2 years now.  I apologize for forgetting about this when I
reported that we should upgrade Tomcat in RC1.  The patch is pretty minor
and basically just replaces the jsp page with a static html page.  The only
downside is that the SLA, Instrumentation, and Metrics pages are always
showing, even if those features are disabled (because that's what JSP was
used for).


- Robert

On Wed, Jan 24, 2018 at 4:18 PM, Artem Ervits  wrote:

> Satish, I have ansible scripts to pull a git branch and run through
> mkdistro and install steps and I wrote another recipe to stand up a cluster
> based on binary release. The problem I just realized is that my compile
> from source script relies on a branch which doesn't exist, is it your
> private repo? If so, can you publish that and I'll test it with my script?
> Otherwise, can you publish the binary release and I'll try to test it as
> well. I have more confidence with compile from source script but that won't
> work on 4.3.1 until you create the branch on github. That said, results of
> my test are the following
>
> verified md5 +1
> tried to build with
>
> *bin/mkdistro.sh -DskipTests -Dhadoop.version=2.7.5*
>
> found the following error
>
> Downloading:
> http://repo1.maven.org/maven2/org/apache/apache-jar-
> resource-bundle/1.4/apache-jar-resource-bundle-1.4.jar
> Downloaded:
> http://repo1.maven.org/maven2/org/apache/apache-jar-
> resource-bundle/1.4/apache-jar-resource-bundle-1.4.jar
> (13 KB at 325.1 KB/sec)
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Oozie Main . FAILURE
> [24.265s]
> [INFO] Apache Oozie Hadoop Utils hadoop-2-4.3.1 .. SKIPPED
> [INFO] Apache Oozie Hadoop Distcp hadoop-2-4.3.1 . SKIPPED
> [INFO] Apache Oozie Hadoop Auth hadoop-2-4.3.1 Test .. SKIPPED
> [INFO] Apache Oozie Hadoop Libs .. SKIPPED
> [INFO] Apache Oozie Client ... SKIPPED
> [INFO] Apache Oozie Share Lib Oozie .. SKIPPED
> [INFO] Apache Oozie Share Lib HCatalog ... SKIPPED
> [INFO] Apache Oozie Share Lib Distcp . SKIPPED
> [INFO] Apache Oozie Core . SKIPPED
> [INFO] Apache Oozie Share Lib Streaming .. SKIPPED
> [INFO] Apache Oozie Share Lib Pig  SKIPPED
> [INFO] Apache Oozie Share Lib Hive ... SKIPPED
> [INFO] Apache Oozie Share Lib Hive 2 . SKIPPED
> [INFO] Apache Oozie Share Lib Sqoop .. SKIPPED
> [INFO] Apache Oozie Examples . SKIPPED
> [INFO] Apache Oozie Share Lib Spark .. SKIPPED
> [INFO] Apache Oozie Share Lib  SKIPPED
> [INFO] Apache Oozie Docs . SKIPPED
> [INFO] Apache Oozie WebApp ... SKIPPED
> [INFO] Apache Oozie Tools  SKIPPED
> [INFO] Apache Oozie MiniOozie  SKIPPED
> [INFO] Apache Oozie Distro ... SKIPPED
> [INFO] Apache Oozie ZooKeeper Security Tests . SKIPPED
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 55.955s
> [INFO] Finished at: Thu Jan 25 00:14:08 UTC 2018
> [INFO] Final Memory: 20M/48M
> [INFO]
> 
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
> (default) on project oozie-main: Error finding remote resources manifests:
> /opt/oozie/oozie-4.3.1/target/maven-shared-archive-
> resources/META-INF/NOTICE
> (No such file or directory) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the 

Re: [VOTE] Release Oozie 5.0.0-beta1 (candidate 2)

[Robert Kanter]

Here's what I did:
- Verified md5
- Verified signature (gpg)
- Looked at rat report
- Looked at release log
- Built Oozie against Hadoop 2.7.2
- Ran a few of the examples, clicked around the UI, tried various CLI
commands, looked at the REST output, launcher log output, etc

+1 (binding)

On Thu, Jan 18, 2018 at 3:58 AM, Harsh J  wrote:

> +1 (binding)
>
> - Verified the md5 and sig (asc) against the tar
> - Executed a build with tests
>
> On Wed, 17 Jan 2018, 21:15 Attila Sasvári,  wrote:
>
> > Hi,
> >
> > I have created a build for Oozie 5.0.0-beta1, release candidate 2.
> >
> > This release introduces major changes:
> > - moving launcher from MapReduce AM to Oozie AM,
> > - dropped support for Hadoop 1.x,
> > - added new 1.0 workflow schemas,
> > - switching from Tomcat to embedded Jetty,
> > - completely rewritten graph generator,
> > - local sharelib,
> > - JDK 8 support.
> >
> > It also includes OOZIE-3083, OOZIE-3159 (beta1 blockers).
> >
> > Keys to verify the signature of the release artifact are available at
> >
> >  http://www.apache.org/dist/oozie/KEYS
> >
> > Please download, test, and try it out:
> >
> >   http://people.apache.org/~asasvari/oozie-5.0.0-beta1-rc2/
> >
> > The release, md5 signature, gpg signature, and rat report can all be
> found
> > at the above address.
> >
> > The vote is open until Tuesday, January 23, 2018 at 9:00 CEST.
> >
> > Thanks,
> > Attila Sasvari
> >
>

Re: [VOTE] Release Oozie 4.3.1 (candidate 1)

[Robert Kanter]

Here's what I did:
- Verified md5
- Verified signature (gpg)
- Looked at rat report
- Looked at release log
- Built Oozie against Hadoop 2.7.2
- Ran a few of the examples, clicked around the UI, tried various CLI
commands, looked at the REST output

I think we should roll another RC because we're currently using Tomcat
6.0.47 and the latest Tomcat 6 is 6.0.53 (the theoretically final Tomcat 6
release).
There are some bug and security fixes:
https://tomcat.apache.org/tomcat-6.0-doc/changelog.html
https://tomcat.apache.org/security-6.html


thanks
- Robert


On Fri, Jan 19, 2018 at 3:44 AM, Peter Cseh  wrote:

> +1 (non-binding)
>
> - Verified md5, signature and rat report
> - Distro builds.
> - Ran some of the examples and they are working (MR, Spark, Pig)
>
> gp
>
> On Fri, Jan 19, 2018 at 5:43 AM, Rohini Palaniswamy <
> rohini.adi...@gmail.com
> > wrote:
>
> > +1 (binding)
> >
> > - Verified md5, signature, release log and rat report
> > - Ran the unit tests and they are all good. There was one known flaky
> test
> > which was fine on rerun.
> > - Executed a simple workflow
> >
> > Regards,
> > Rohini
> >
> > On Wed, Jan 17, 2018 at 5:57 AM, Attila Sasvari 
> > wrote:
> >
> > > Many thanks Satish.
> > >
> > > +1 (non-binding)
> > >
> > > Testing done:
> > > - Verified md5 and signature, release-log.txt is up to date
> > > - Created distro with the mkdistro script using -Puber -Phadoop-2
> > > -Dhadoop.version-2.6.0
> > > - Executed example workflows (all except Hive related ones) /
> > coordinators
> > > / bundles on a pseudo Hadoop 2.6.0 - each succeeded
> > > - Submitted Spark example workflow no only in local mode, but also on
> > YARN
> > > (master=yarn, mode=client/cluster)
> > > - Tested FS action
> > > - Clicked through the Web UI. Job logs can be retrieved, DAG is
> > displayed,
> > > version is correct
> > >
> > >
> > >
> > > On Wed, Jan 17, 2018 at 6:18 AM, satish saley 
> > > wrote:
> > >
> > > > Hi,
> > > > I have created a build for Oozie 4.3.1, candidate 1.
> > > > It includes all the changes decided earlier except OOZIE-3018 and
> > > > OOZIE-3072. OOZIE-3018 makes changes to SchemaCheckerService which we
> > are
> > > > not picking. OOZIE-3072 makes changes to files created in OOZIE-1770.
> > > > Keys to verify the signature of the release artifact are available at
> > > >   http://www.apache.org/dist/oozie/KEYS
> > > > Please download, test, and try it out:
> > > >   http://people.apache.org/~satishsaley/oozie-4.3.1-rc1
> > > > The release, md5 signature, gpg signature, and rat report can allbe
> > found
> > > > at the above address.
> > > > Vote closes on 01/19/2018 11:59 PM PT.
> > > > -Satish Saley
> > >
> > >
> > >
> > >
> > > --
> > > --
> > > Attila Sasvari
> > > Software Engineer
> > > 
> > >
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> 
>

[jira] [Created] (OOZIE-3158) Find a replacement for Xerces

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3158:


 Summary: Find a replacement for Xerces
 Key: OOZIE-3158
 URL: https://issues.apache.org/jira/browse/OOZIE-3158
 Project: Oozie
  Issue Type: Improvement
Affects Versions: trunk
Reporter: Robert Kanter


Oozie currently relies on Apache Xerces for XML validation and related tasks.  
The current version of Xerces-J is 2.11.0, which Oozie is using.  However, it 
was released in *2010*!  It might be a good idea to look into an alternative 
library that is being actively maintained to make sure we continue to get bug, 
security, performance, etc fixes.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 64029: OOZIE-2339 Provide an API for writing jobs based on the XSD schemas

[Robert Kanter via Review Board]

be 
oozie.jobs-api.application.generated.path or something like that?



core/src/test/java/org/apache/oozie/client/TestOozieCLI.java
Line 21 (original), 21 (patched)
<https://reviews.apache.org/r/64029/#comment274376>

No *



docs/src/site/twiki/DG_CommandLineTool.twiki
Lines 1043 (patched)
<https://reviews.apache.org/r/64029/#comment274377>

Is this going to make it for the 5.0 GA?  We should probably push this to 
5.1 to allow more time, especially for testing.



docs/src/site/twiki/DG_CommandLineTool.twiki
Lines 1058 (patched)
<https://reviews.apache.org/r/64029/#comment274381>

The DG_JobsAPI.twiki file seems to be missing.



jobs/jobs-api/pom.xml
Lines 49 (patched)
<https://reviews.apache.org/r/64029/#comment274378>

All of the version numbers in the poms should be removed (they get 
inherited from the root pom).  If any new dependencies are added, they should 
be added to the root pom.



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/GraphVisualization.java
Lines 38 (patched)
<https://reviews.apache.org/r/64029/#comment274379>

Why is this here?



jobs/jobs-api/src/main/java/org/apache/oozie/jobs/api/action/ActionAttributes.java
Lines 28 (patched)
<https://reviews.apache.org/r/64029/#comment274380>

Is this something we're going to have to manually maintain going forward if 
the XML schema changes?  If so, and/or if there are other cases of that, we 
should have some comments or developer docs or something explaining what to do 
and when.  Otherwise, I can easily see this not getting updated when someone 
updates the schema.

Even better would be to somehow have a unit test that fails if these get 
out of sync.



pom.xml
Lines 48-50 (original), 48-50 (patched)
<https://reviews.apache.org/r/64029/#comment274355>

We should leave this at 1.8



pom.xml
Lines 131 (patched)
<https://reviews.apache.org/r/64029/#comment274356>

I wonder if we should call this something else.  "jobs" isn't super clear, 
and we already have a number of things called jobs (e.g. Oozie Jobs (workflows, 
coords, etc), Hadoop Jobs (apps), etc).  Maybe "jobs-api"?  Though that's not 
much different.  I'm not sure of a good name.


- Robert Kanter


On Jan. 8, 2018, 10:31 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64029/
> ---
> 
> (Updated Jan. 8, 2018, 10:31 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2339 Provide an API for writing jobs based on the XSD schemas
> 
> 
> Diffs
> -
> 
>   .gitignore a4285a0d77ac2e4e8fc4afbfa21bf9fd287b3e4d 
>   client/pom.xml 3b67d1286499337d61c269f9dbd1ff65ba23b903 
>   client/src/main/java/org/apache/oozie/cli/ApiJarFactory.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/cli/ApiJarLoader.java PRE-CREATION 
>   client/src/main/java/org/apache/oozie/cli/OozieCLI.java 
> 4abc7502c0c9d8b59ded2baaed30c407ad073008 
>   client/src/main/java/org/apache/oozie/cli/WorkflowFactoryCompiler.java 
> PRE-CREATION 
>   client/src/main/java/org/apache/oozie/client/OozieClient.java 
> 7e41ef89c784882c26f4af0da0110101fa1a61a0 
>   core/src/main/java/org/apache/oozie/BaseLocalOozieClient.java 
> 38fb006980c0acb582b4f91175be1ced8a1afa79 
>   core/src/main/java/org/apache/oozie/local/LocalOozie.java 
> 9ab646cd53223d903e387eb92aa103a7b6577d0f 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobServlet.java 
> e1bd3cf61aabfa4311881b2944fc9f10ffe9145d 
>   core/src/main/java/org/apache/oozie/servlet/BaseJobsServlet.java 
> d4b08711223356c8d4f58a3391b6b87efa0a2b73 
>   core/src/main/java/org/apache/oozie/servlet/ServletUtilities.java 
> 9cdd915c02a3e34ad211cf6cced2e64f3a1196a0 
>   core/src/main/java/org/apache/oozie/servlet/V0JobsServlet.java 
> ff50f3d121d8af7d607124424915e1bf30ffb109 
>   core/src/main/java/org/apache/oozie/servlet/V1JobsServlet.java 
> c1ca65fd2f8bf0f3e5c618dddcba3bf52eaf5420 
>   core/src/main/java/org/apache/oozie/servlet/V2ValidateServlet.java 
> 7bf8fb3f0ce9253d553c7b1883666811686f6a9f 
>   core/src/main/resources/oozie-default.xml 
> 830893fc50f0eb044df91c15ba87ed17761836d4 
>   core/src/test/java/org/apache/oozie/client/TestOozieCLI.java 
> cf7663102b2054c6f20a7bb09800f06bd11cfffb 
>   core/src/test/java/org/apache/oozie/test/MiniOozieTestCase.java 
> 12ede028655f2a30613a9f802e48cd3a27af6ed9 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> 24fb1c99101b20c2ce2a6f103288fc090ceab3e7 
>   jobs/jobs-api/pom.xml PRE-CREATION 
>   jobs/jobs-api/src/main/java/org/apach

Re: [VOTE] Release Oozie 5.0.0-beta1 (candidate 1)

[Robert Kanter]

Here's what I did:
- Verified md5
- Verified signature (gpg)
- Built Oozie against Hadoop 2.7.2
- Ran a few of the examples, clicked around the UI, tried various CLI
commands, looked at the REST output
- Verified that OOZIE-3083 is fixed
- I also saw the issue Kinga found in OOZIE-3155, and I agree that's not a
blocker for the beta, but I've made it "critical" for GA as that would be
good to fix

+1 (binding)

thanks
- Robert


On Wed, Jan 10, 2018 at 9:06 AM, Julia Kinga Marton 
wrote:

> Hi,
>
> I made some testing on the RC, and it seems OK.
>
> What I checked:
> - md5
> - gpg signature
> - rat report
> - built Oozie against Hadoop-2.6.0
> - ran a few examples (wf and coordinator), clicked the Web UI, ran some CLI
> commands
>
> I found a small issue related the Job DAG (OOZIE-3155
> ), but I don`t consider
> it a blocker.
>
> +1 from my side (non-binding)
>
> Best regards,
> Kinga
>
> On Tue, Jan 9, 2018 at 5:53 PM, Attila Sasvári 
> wrote:
>
> > It is release candidate 1.
> >
> > On Tue, Jan 9, 2018 at 5:50 PM, Attila Sasvári 
> > wrote:
> >
> > > Hi,
> > >
> > > I have created a build for Oozie 5.0.0-beta1, release candidate 0.
> > >
> > > This release introduces major changes:
> > > - moving launcher from MapReduce AM to Oozie AM,
> > > - dropped support for Hadoop 1.x,
> > > - added new 1.0 workflow schemas,
> > > - switching from Tomcat to embedded Jetty,
> > > - completely rewritten graph generator,
> > > - local sharelib,
> > > - JDK 8 support.
> > >
> > > It also includes OOZIE-3083.
> > >
> > > Keys to verify the signature of the release artifact are available at
> > >
> > >  http://www.apache.org/dist/oozie/KEYS
> > >
> > > Please download, test, and try it out:
> > >
> > >   http://people.apache.org/~asasvari/oozie-5.0.0-beta1-rc1/
> > >
> > > The release, md5 signature, gpg signature, and rat report can all be
> > found
> > > at the above address.
> > >
> > > The vote is open until Friday, January 12, 2018 at 18:00 CEST.
> > >
> > > Thanks,
> > > Attila Sasvari
> > >
> >
>

[jira] [Updated] (OOZIE-3155) Job DAG is not refreshed when a job is finished

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3155:
-
Priority: Critical  (was: Minor)

> Job DAG is not refreshed when a job is finished
> ---
>
> Key: OOZIE-3155
> URL: https://issues.apache.org/jira/browse/OOZIE-3155
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: trunk, 5.0.0b1
>Reporter: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.0.0
>
> Attachments: Screen Shot 2018-01-10 at 16.23.28.png
>
>
> When opening a Job DAG when the job is still in progress, the DAG is OK, but 
> when the job will finish, this diagram is not refreshed.
> Steps to reproduce: 
> - submit a workflow (e.g. the subwf from the examples) - the wf should not 
> end very quickly, to have time to catch it when it is still running
> - check the Job DAG when the job is still running
> - wait for the job to finish
> - check again the Job DAG -> is the same with the running one
> Note: if check it in a new tab, is OK, and also a browser refresh will solve 
> it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-3155) Job DAG is not refreshed when a job is finished

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-3155:
-
Affects Version/s: 5.0.0b1
Fix Version/s: 5.0.0

> Job DAG is not refreshed when a job is finished
> ---
>
> Key: OOZIE-3155
> URL: https://issues.apache.org/jira/browse/OOZIE-3155
> Project: Oozie
>  Issue Type: Bug
>  Components: client
>Affects Versions: trunk, 5.0.0b1
>Reporter: Julia Kinga Marton
>Priority: Critical
> Fix For: 5.0.0
>
> Attachments: Screen Shot 2018-01-10 at 16.23.28.png
>
>
> When opening a Job DAG when the job is still in progress, the DAG is OK, but 
> when the job will finish, this diagram is not refreshed.
> Steps to reproduce: 
> - submit a workflow (e.g. the subwf from the examples) - the wf should not 
> end very quickly, to have time to catch it when it is still running
> - check the Job DAG when the job is still running
> - wait for the job to finish
> - check again the Job DAG -> is the same with the running one
> Note: if check it in a new tab, is OK, and also a browser refresh will solve 
> it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3149) Delete work.log

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308469#comment-16308469
 ] 

Robert Kanter commented on OOZIE-3149:
--

Thanks [~asasvari].  Committed to master!  

> Delete work.log
> ---
>
> Key: OOZIE-3149
> URL: https://issues.apache.org/jira/browse/OOZIE-3149
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>    Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Trivial
>  Labels: newbie
> Attachments: OOZIE-3149-01.patch
>
>
> The root dir has a file named {{work.log}} which appears to be an old version 
> of {{release-log.txt}}, up to Oozie 3.0.1.  It should be safe to simply 
> delete it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3149) Delete work.log

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308464#comment-16308464
 ] 

Robert Kanter commented on OOZIE-3149:
--

+1

> Delete work.log
> ---
>
> Key: OOZIE-3149
> URL: https://issues.apache.org/jira/browse/OOZIE-3149
> Project: Oozie
>  Issue Type: Bug
>Affects Versions: 5.0.0
>    Reporter: Robert Kanter
>Assignee: Attila Sasvari
>Priority: Trivial
>  Labels: newbie
> Attachments: OOZIE-3149-01.patch
>
>
> The root dir has a file named {{work.log}} which appears to be an old version 
> of {{release-log.txt}}, up to Oozie 3.0.1.  It should be safe to simply 
> delete it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Release Oozie 5.0.0-beta1 (candidate 0)

[Robert Kanter]

Thanks for putting this together.

Another important thing to note in the highlights that as part of moving
the launcher, we also dropped support for Hadoop 1.x and we added new 1.0
schemas.

+1

Here's what I did:
- Verified md5
- Verified signature (gpg)
- Built Oozie against Hadoop 2.7.2
- Ran a few of the examples, clicked around the UI and tried various CLI
commands

Here's some issues I ran into; none are blockers for 5b1:
- Filed OOZIE-3149 to delete work.log, which is really old (not a blocker)
- Filed OOZIE-3150 to update NOTICE.txt (I made this a blocker for 5.0.0)
- The examples are still the older schemas, which is fine for the beta, but
I made OOZIE-2814 a blocker for 5.0.0 because it will encourage users to
use the new schemas

- Robert


On Thu, Dec 28, 2017 at 1:23 PM, Peter Cseh  wrote:

> Hi Attila!
>
> It looks like I've messed up the extraction of the tar.gz and I was looking
> at empty folders :)
> Will validate the release tomorrow.
>
> Sorry for causing confusion!
> gp
>
> On Thu, Dec 28, 2017 at 2:08 PM, Attila Sasvari 
> wrote:
>
> > @Andras: many thanks. Please let me know if you notice anything strange
> > with hashes or run into other problems.
> >
> > @gp: thanks a lot. This is actually what the bin/create-release-artifact
> > script has generated. Comparing it with previous release candidate (4.3.0
> > rc1), file size of oozie-5.0.0-beta1.tar.gz looks correct to me
> >
> >   $ wget
> > https://dist.apache.org/repos/dist/dev/oozie/oozie-4.3.0-
> > rc1/oozie-4.3.0.tar.gz
> >   $ wget
> > http://people.apache.org/\~asasvari/oozie-5.0.0-beta1-
> > rc0/oozie-5.0.0-beta1.tar.gz
> >
> >   $ ls -lrt oozie-*tar.gz
> >   -rw-r--r--  1 asasvari  staff  2392205 Oct 25  2016 oozie-4.3.0.tar.gz
> >   -rw-r--r--  1 asasvari  staff  2539718 Dec 28 00:12
> > oozie-5.0.0-beta1.tar.gz
> >
> > oozie-5.0.0-beta1.tar.gz contains much more files (it contains all the
> > sources):
> >
> >   $  tar tvf oozie-5.0.0-beta1.tar.gz | grep "\.java" | head -5
> >   -rw-r--r--  0 asasvari wheel   10328 Dec 28 00:02
> > oozie-5.0.0-beta1/zookeeper-security-tests/src/test/java/
> > org/apache/oozie/util/TestZKUtilsWithSecurity.java
> >   -rw-r--r--  0 asasvari wheel7357 Dec 28 00:02
> > oozie-5.0.0-beta1/zookeeper-security-tests/src/test/java/
> > org/apache/oozie/test/ZKXTestCaseWithSecurity.java
> >   -rw-r--r--  0 asasvari wheel   32589 Dec 28 00:02
> > oozie-5.0.0-beta1/tools/src/test/java/org/apache/oozie/
> > tools/FakeConnection.java
> >   -rw-r--r--  0 asasvari wheel1921 Dec 28 00:02
> > oozie-5.0.0-beta1/tools/src/test/java/org/apache/oozie/
> > tools/FakeDriver.java
> >   -rw-r--r--  0 asasvari wheel2230 Dec 28 00:02
> > oozie-5.0.0-beta1/tools/src/test/java/org/apache/oozie/
> > tools/LauncherSecurityManager.java
> >
> > If you download the tarball and run bin/mkdistro.sh, it will probably
> fail
> > due to known flaky tests. Running tests with bin/mkdistro.sh
> > -Dsurefire.rerunFailingTestsCount=2, they shall pass (printing things
> > like [WARNING] Tests run: 1841, Failures: 0, Errors: 0, Skipped: 2,
> Flakes:
> > 2 ).
> >
> > - Attila
> >
> > On Thu, Dec 28, 2017 at 10:57 AM, Peter Cseh 
> > wrote:
> >
> > > Thanks Attila!
> > > The tar.gz does not contain the release, it only contains the pom
> files.
> > > The whole thing is 2.4 Mb, an Oozie was bigger the last time I've
> checked
> > > :)
> > >
> > > gp
> > >
> > > On Thu, Dec 28, 2017 at 12:53 AM, Attila Sasvári 
> > > wrote:
> > >
> > > > Hi there,
> > > >
> > > > I have created a build for Oozie 5.0.0-beta1, release candidate 0.
> > > >
> > > > This release introduces major changes:
> > > > - moving launcher from MapReduce AM to Oozie AM,
> > > > - switching from Tomcat to embedded Jetty,
> > > > - completely rewritten graph generator,
> > > > - local sharelib,
> > > > - JDK 8 support.
> > > >
> > > > Keys to verify the signature of the release artifact are available at
> > > >
> > > >  http://www.apache.org/dist/oozie/KEYS
> > > >
> > > > Please download, test, and try it out:
> > > >
> > > >   http://people.apache.org/~asasvari/oozie-5.0.0-beta1-rc0/
> > > >
> > > > The release, md5 signature, gpg signature, and rat report can all be
> > > found
> > > > at the above address.
> > > >
> > > > The vote is open until Thursday, January 04, 2018 at 12:00 CEST.
> > > >
> > > > Thanks,
> > > > Attila Sasvari
> > > >
> > >
> > >
> > >
> > > --
> > > Peter Cseh
> > > Software Engineer
> > > 
> > >
> >
> >
> >
> > --
> > --
> > Attila Sasvari
> > Software Engineer
> > 
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> 
>

[jira] [Commented] (OOZIE-2814) OYA: Update example workflows to newest schemas

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306578#comment-16306578
 ] 

Robert Kanter commented on OOZIE-2814:
--

Making this a blocker.  It'll help get users to start using the newer schema 
with all of the OYA changes.

> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>    Reporter: Robert Kanter
>Priority: Blocker
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2814) OYA: Update example workflows to newest schemas

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2814:
-
Priority: Blocker  (was: Critical)

> OYA: Update example workflows to newest schemas
> ---
>
> Key: OOZIE-2814
> URL: https://issues.apache.org/jira/browse/OOZIE-2814
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>    Reporter: Robert Kanter
>Priority: Blocker
>
> OOZIE-2687 will add in a whole set of newer schemas that are more 
> Yarn-centric.  Most, if not all, examples currently use pretty old versions 
> of the schemas.  We should take this opportunity to update the examples to 
> the newest schemas added by OOZIE-2687, which should help demonstrate them.
> It may make sense to keep the older schemas too, to show that older workflows 
> still work.  We have some examples where we have two workflow XML files that 
> do the same thing in different ways.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3150) Update NOTICE.txt

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3150:


 Summary: Update NOTICE.txt
 Key: OOZIE-3150
 URL: https://issues.apache.org/jira/browse/OOZIE-3150
 Project: Oozie
  Issue Type: Bug
Affects Versions: 5.0.0b1
Reporter: Robert Kanter
Priority: Blocker
 Fix For: 5.0.0


NOTICE.txt currently contains:
{noformat}
===
NOTICE file for use with, and corresponding to Section 4 of,
the Apache License, Version 2.0,
in this case for the Oozie, Workflow Engine for Hadoop project
===

Copyright 2011 The Apache Software Foundation

This product includes software developed by The Apache Software
Foundation (http://www.apache.org/).

This product includes org.json (http://www.json.org/java/index.html),
Copyright (c) 2002 JSON.org

Detailed License information for all components can be found in the
documentation in the ooziedocs.war at index.html##LicenseInfo

Oozie source contains 3 binaries (wordcount-simple_Linux-amd64-64_h20,
wordcount-simple_Linux-i386-32_h20, wordcount-simple_Mac_OS_X-x86_64-64_h20) for
testing purposes, these binaries are compiled artifacts from Apache Hadoop.
{noformat}

A few problems:
- There is no ooziedocs.war
- Oozie doesn't contain any binaries AFAIK and hasn't for a long time
- We should see if there is anything else we need to add



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (OOZIE-3149) Delete work.log

[Robert Kanter (JIRA)]

Robert Kanter created OOZIE-3149:


 Summary: Delete work.log
 Key: OOZIE-3149
 URL: https://issues.apache.org/jira/browse/OOZIE-3149
 Project: Oozie
  Issue Type: Bug
Affects Versions: 5.0.0
Reporter: Robert Kanter
Priority: Trivial


The root dir has a file named {{work.log}} which appears to be an old version 
of {{release-log.txt}}, up to Oozie 3.0.1.  It should be safe to simply delete 
it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Oozie 5.0.0-beta1

[Robert Kanter]

Thanks for looking into that and filing some new JIRAs.
If we have some known flakey tests, then I think it's fine to leave those
for the beta.  But we should try to fix them for the GA (plus, that'll make
development easier too).

On Wed, Dec 27, 2017 at 10:01 AM, Attila Sasvari <asasv...@cloudera.com>
wrote:

> Thanks Robert for the ideas. Some explanation: we rerun all the tests
> within a test class in case there is a failing test case because surefire
> does not allow running individual test cases from multiple test classes
> (see bin/test-patch-20-tests). In other words: passing
> -Dtest=Test1#testMethod1,Test2#testMethod2 to mvn test does not work,
> while
> -DTest1,Test2 works at the cost of running all other tests in Test1 an
> Test2.
>
> Clean up before and after tests is a good idea. In fact, I have seen
> multiple undeleted test resources (after mvn clean too):
> core/activemq-data/
> core/dist2.txt
> core/dist3.txt
> core/distcp-log4j.properties
> core/distcp-oozie-1514391729048.log
> core/dst1.txt
> core/test-invalid-workflow-app.xml
> core/test-workflow-app.xml
>
> I filed some new JIRA-s:
> - OOZIE-3145 TestDistcpMain shall remove created files after test execution
> - OOZIE-3148 Rerun Failing Tests through Maven surefire
> ( Right now I am running again mkdistro.sh with
> -Dsurefire.rerunFailingTestsCount=2. I will look into mini cluster logs if
> there are failures.)
>
> Unfortunately, there are some known flaky tests in Oozie (tracked by
> OOZIE-3111 umbrella JIRA). I am not sure if we need to fix all of them
> before the first 5.0.0-beta1 release candidate; just out of curiosity, I
> tried to run bin/mkdistro.sh on release-4.3.0 and some tests failed too.
>
> - Attila
>
> On Fri, Dec 22, 2017 at 1:20 AM, Robert Kanter <rkan...@cloudera.com>
> wrote:
>
> > I took a look at the latest PreCommit job
> > <https://builds.apache.org/job/PreCommit-OOZIE-Build/288/consoleFull>
> and
> > it reported 55 rerun tests.
> > Tests rerun: 55
> > Tests failed at first run:
> > org.apache.oozie.action.hadoop.TestJavaActionExecutor,
> >
> > However, looking through the actual output, I only see 1 test that failed
> > (and was rerun): TestJavaActionExecutor. testCredentialsSkip.
> >
> > [INFO] Running org.apache.oozie.action.hadoop.TestJavaActionExecutor
> > [ERROR] Tests run: 55, Failures: 0, Errors: 1, Skipped: 0, Time
> > elapsed: 126.213 s <<< FAILURE! - in
> > org.apache.oozie.action.hadoop.TestJavaActionExecutor
> > [ERROR] testCredentialsSkip(org.apache.oozie.action.hadoop.
> > TestJavaActionExecutor)
> >  Time elapsed: 0.532 s  <<< ERROR!
> > org.apache.oozie.action.ActionExecutorException: JA020: Could not load
> > credentials of type [abc] with name [abcname]]; perhaps it was not
> > defined in oozie-site.xml?
> > at org.apache.oozie.action.hadoop.TestJavaActionExecutor.
> > _testCredentialsSkip(TestJavaActionExecutor.java:1106)
> > at org.apache.oozie.action.hadoop.TestJavaActionExecutor.
> > testCredentialsSkip(TestJavaActionExecutor.java:1006)
> >
> >
> > In fact, the report only lists the one test class, not 55 of them.  So I
> > think there's something wrong with our reporting here.
> >
> > Anyway, typically, when we see test that succeed on their own but fail
> when
> > run all together, that means that (likely some other) test is not
> cleaning
> > up properly.  This is unfortunately tricky to debug because it's hard to
> > figure out what the other test is.  A long time ago, a big example of
> this
> > problem was not properly shutting down the Services singleton, so we'd
> have
> > duplicates and other issues.
> >
> > For these specific issues, some hints:
> > # RUNNINGWITHERROR: This problem means that a yarn job in the mini
> cluster
> > has failed.  To find out why, you should be able to dig out the app id
> from
> > the test output, and then find it's yarn logs somewhere (there's a
> > minicluster logs dir, but I forget where).  That'll hopefully make it
> > obvious what's going on.
> > # Credentials: There's probably an oozie-site or Configuration class
> > leaking from somewhere or not properly cleaned up by a previous test or
> > setup by this test.  The Credentials class is missing.
> > # TestJMSAccessorService: Sounds like something didn't get cleaned up.
> >
> > One way that might be easier to fix this is to have the setUp methods
> > ensure that things are clean just in case.  We actually have

Re: Oozie 5.0.0-beta1

[Robert Kanter]

t;> > Hey Attila,
> >>> >
> >>> > I won't be able to work on the release for a couple weeks now.
> >>> > Thanks for getting the release rolling!
> >>> >
> >>> > Cheers,
> >>> > gp
> >>> >
> >>> >
> >>> > On Mon, Dec 18, 2017 at 4:00 PM, Attila Sasvari <
> asasv...@cloudera.com
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Hi everyone,
> >>> > >
> >>> > > I would like to create the release branch, branch-5.0.0-beta1
> >>> (following
> >>> > > Hadoop release versioning), earlier.
> >>> > >
> >>> > > Looking at https://issues.apache.org/jira/projects/OOZIE/versions/
> >>> > 12342048
> >>> > > there are 3 issues in progress (OOZIE-2231, OOZIE-2942, OOZIE-2974)
> >>> and 3
> >>> > > issues to do (OOZIE-2600, OOZIE-3093, OOZIE-1987). I will push
> those
> >>> out
> >>> > to
> >>> > > 5.0.0 if there are no objections.
> >>> > >
> >>> > > At the same time, I am volunteering to be the release manager if
> >>> Peter
> >>> > Cseh
> >>> > > does not mind.
> >>> > >
> >>> > > Regards,
> >>> > > Attila
> >>> > >
> >>> > > On Wed, Dec 6, 2017 at 8:28 PM, Robert Kanter <
> rkan...@cloudera.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Sounds good to me!
> >>> > > >
> >>> > > > On Wed, Dec 6, 2017 at 5:17 AM, Andras Piros <
> >>> > andras.pi...@cloudera.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Good idea Gezapeti!
> >>> > > > >
> >>> > > > > Time to wrap things up towards a stable 5.0.0 - a release
> >>> candidate
> >>> > on
> >>> > > > > 5.0.0b1 is a good first step.
> >>> > > > >
> >>> > > > > Since other components that Oozie uses like Pig and Hive do not
> >>> > (fully)
> >>> > > > > support Hadoop 3, we have to wait with OOZIE-2973
> >>> > > > > <https://issues.apache.org/jira/browse/OOZIE-2973>.
> >>> > > > >
> >>> > > > > Thanks,
> >>> > > > >
> >>> > > > > Andras
> >>> > > > >
> >>> > > > > On Wed, Dec 6, 2017 at 1:50 PM, Peter Cseh <
> >>> gezap...@cloudera.com>
> >>> > > > wrote:
> >>> > > > >
> >>> > > > > > Hi everyone!
> >>> > > > > >
> >>> > > > > > Now that OOZIE-2969 <https://issues.apache.org/
> >>> > > jira/browse/OOZIE-2969>
> >>> > > > > is
> >>> > > > > > committed I'd like to start the process of creating the
> branch
> >>> for
> >>> > > > > 5.0.0b1
> >>> > > > > > and building a release from there.
> >>> > > > > > It's unfortunate that we won't be able to support Hadoop 3 in
> >>> the
> >>> > > beta
> >>> > > > > for
> >>> > > > > > reasons described in OOZIE-2973
> >>> > > > > > <https://issues.apache.org/jira/browse/OOZIE-2973>
> >>> > > > > > I don't see any more blockers for the beta1 and I hope we
> won't
> >>> > find
> >>> > > > > > hard-to-fix major issues so we can release Oozie 5.0.0 in
> early
> >>> > 2018.
> >>> > > > > > Please let me know if you have any suggestions.
> >>> > > > > > Thanks
> >>> > > > > > gp
> >>> > > > > >
> >>> > > > > > --
> >>> > > > > > Peter Cseh
> >>> > > > > > Software Engineer
> >>> > > > > > <http://www.cloudera.com>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > --
> >>> > > Attila Sasvari
> >>> > > Software Engineer
> >>> > > <http://www.cloudera.com/>
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Peter Cseh
> >>> > Software Engineer
> >>> > <http://www.cloudera.com>
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> --
> >> Attila Sasvari
> >> Software Engineer
> >> <http://www.cloudera.com/>
> >>
> >
> >
> >
> > --
> > --
> > Attila Sasvari
> > Software Engineer
> > <http://www.cloudera.com/>
> >
>
>
>
> --
> --
> Attila Sasvari
> Software Engineer
> <http://www.cloudera.com/>
>

Welcoming new Oozie PMC member Satish Saley

[Robert Kanter]

Hello Oozie community,

It's my pleasure to announce that Satish Saley has become the newest
addition to the Oozie PMC.

Please join me congratulating him.
Congrats!


- Robert, on behalf of the Oozie PMC

Re: Oozie 5.0.0-beta1

[Robert Kanter]

Sounds good to me!

On Wed, Dec 6, 2017 at 5:17 AM, Andras Piros 
wrote:

> Good idea Gezapeti!
>
> Time to wrap things up towards a stable 5.0.0 - a release candidate on
> 5.0.0b1 is a good first step.
>
> Since other components that Oozie uses like Pig and Hive do not (fully)
> support Hadoop 3, we have to wait with OOZIE-2973
> .
>
> Thanks,
>
> Andras
>
> On Wed, Dec 6, 2017 at 1:50 PM, Peter Cseh  wrote:
>
> > Hi everyone!
> >
> > Now that OOZIE-2969 
> is
> > committed I'd like to start the process of creating the branch for
> 5.0.0b1
> > and building a release from there.
> > It's unfortunate that we won't be able to support Hadoop 3 in the beta
> for
> > reasons described in OOZIE-2973
> > 
> > I don't see any more blockers for the beta1 and I hope we won't find
> > hard-to-fix major issues so we can release Oozie 5.0.0 in early 2018.
> > Please let me know if you have any suggestions.
> > Thanks
> > gp
> >
> > --
> > Peter Cseh
> > Software Engineer
> > 
> >
>

Re: Review Request 63875: OOZIE-2900 Retrieve tokens for oozie.launcher.mapreduce.job.hdfs-servers before submission

[Robert Kanter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63875/#review191285
---




core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java
Lines 63-66 (patched)
<https://reviews.apache.org/r/63875/#comment269046>

Given that this stuff is only needed if jobNameNodes is not null, we should 
move it inside the if block so we don't do it unnecessarily.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1404 (patched)
<https://reviews.apache.org/r/63875/#comment269050>

This probably shouldn't be hardcoded here :)


- Robert Kanter


On Nov. 16, 2017, 1:50 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63875/
> ---
> 
> (Updated Nov. 16, 2017, 1:50 p.m.)
> 
> 
> Review request for oozie, Peter Bacsko and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> Before Oozie on YARN, ``JobSubmitter`` from MapReduce (more precisely 
> ``TokenCache.obtainTokensForNamenodes``) took care of obtaining delegation 
> tokens for HDFS nodes specified by 
> ``oozie.launcher.mapreduce.job.hdfs-servers`` before submitting the Oozie 
> launcher job.
> 
> Oozie launcher is now a Yarn Application Master. It needs HDFS delegation 
> tokens to be able to copy files between secure clusters via the Oozie DistCp 
> action. 
> 
> Changes:
> - ``JavaActionExecutor`` was modified to handle Distcp related parameters 
> like (``oozie.launcher.mapreduce.job.hdfs-servers`` and 
> ``oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude``)
> - ``HDFSCredentials`` was changed to reuse 
> ``TokenCache.obtainTokensForNamenodes`` to obtain HDFS delegation tokens.
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/HDFSCredentials.java 
> 92a7ebe 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 8cb76cf 
> 
> 
> Diff: https://reviews.apache.org/r/63875/diff/1/
> 
> 
> Testing
> ---
> 
> Tested on a secure cluster that Oozie dist cp action can copy file from 
> another secure cluster where different Kerberos realm was used.
> 
> - workflow:
> ```
> 
> 
> 
> Action failed, error 
> message[${wf:errorMessage(wf:lastErrorNode())}]
> 
> 
> 
> ${jobTracker}
> ${nameNode}
> 
> 
>   
> 
> oozie.launcher.mapreduce.job.dfs.namenode.kerberos.principal.pattern
> *
>   
> 
>   oozie.launcher.mapreduce.job.hdfs-servers
>   hdfs://oozie.test1.com:8020,hdfs://remote.test2.com:8020
> 
>  
> 
> 
> oozie.launcher.mapreduce.job.hdfs-servers.token-renewal.exclude
> remote.test2.com
> 
> 
>   hdfs://remote.test2.com:8020/tmp/1
>   hdfs://oozie.test1.com:8020/tmp/2
> 
> 
> 
> 
> 
> 
> ```
> 
> Prior to executing the workflow I had to setup cross realm trust between the 
> test secure clusters. It involved:
> - changing Kerberos configuration ``/etc/krb5.conf`` (adding realms and 
> setting additional properties like ``udp_preference_limit = 1``)
> - regenerating service credentials
> - changing HDFS settings (such as 
> ``dfs.namenode.kerberos.principal.pattern``) and setting hadoop auth to local 
> rule like ``RULE:[2:$1](.*)s/(.*)/$1/g``
> - additional configuration to enable trust between the test hadoop clusters
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>

Re: Oozie 4.3.1. release

[Robert Kanter]

Sounds good to me.

On Tue, Nov 14, 2017 at 10:09 AM, Peter Bacsko  wrote:

> Good idea.
>
> If I'm not mistaken, there's also a pending review/patch submitted by
> someone (apologies, but I forgot how it was) that cannot be applied to the
> master anymore, at least not easily. It's worth checking that too (perhaps
> there are more than one patch).
>
> Peter
>
> On Tue, Nov 14, 2017 at 5:07 PM, Peter Cseh  wrote:
>
> > Hey!
> >
> > I agree, users wanting stability would appreciate a more stable 4.3.1 or
> > even 4.4.0 to be released with improvements.
> > I think we could include the following issues as well:
> > OOZIE-3100 Upgrade javax.mail:mail to 1.4.7 (andras.piros)
> > OOZIE-3077 SLA documentation fixes (satishsaley)
> > OOZIE-2886 Ensure consistent versioning of hadoop jars in sharelibs
> > (dbist13 via rkanter)
> > OOZIE-2747 README.txt is out of date (Jan Hentschel via rkanter)
> > OOZIE-2825 Custom Authentication doc page is not well formatted (Jan
> > Hentschel via rkanter)
> > OOZIE-3004 Forked action retry info is not working
> > Most of these are minor improvements or user-facing fixes which would be
> > nice to get through the door.
> >
> > BRs,
> > gp
> >
> > On Tue, Nov 14, 2017 at 12:56 AM, Rohini Palaniswamy <
> > rohini.adi...@gmail.com> wrote:
> >
> > > Hi folks,
> > >I think this would be a good time to do a patch release on 4.3
> branch.
> > > The 5.0 release is going to be a major change with the move of launcher
> > > from Mapreduce AM to Oozie AM and might take time to stabilize. So it
> > would
> > > be good to do one final patch release on 4.3 branch pulling in some
> > > important fixes and Spark action related fixes (Spark 2.x support) that
> > > have gone in master so far.  Satish has volunteered to be the Release
> > > Manager.
> > >
> > > Below is a initial list of jiras that I identified.
> > >
> > >
> > > OOZIE-3095 Job submit command is not logged in audit log (puru)
> > > OOZIE-3031 Coord job with only unresolved dependencies doesn't timeout
> > > (puru via satishsaley)
> > > OOZIE-3079 Filtering coordinators returns bundle id as null
> (satishsaley)
> > > OOZIE-3078 PasswordMasker throws NPE with null arguments (asasvari)
> > > OOZIE-3072 oozie.service.HadoopAccessorService.action.configurations
> > >  should overwrite default values set in Hadoop's configuration files
> > > (gezapeti)
> > > OOZIE-2885 Running Spark actions should not need Hive on the classpath
> > > (satishsaley)
> > > OOZIE-3058 nocleanup option is missing in oozie-coordinator-0.5.xsd
> > > (satishsaley)
> > > OOZIE-3036 Spark 2.2.0 support: tell Spark not to get any delegation
> > tokens
> > > (andras.piros via gezapeti)
> > > OOZIE-3028 Oozie Pig Action fails with no python dependencies (dbist13
> > via
> > > rohini)
> > > OOZIE-3018 Use Hadoop's CredentialProvider for passwords in
> > > SchemaCheckerService (andras.piros via gezapeti)
> > > OOZIE-2984 Parse spark-defaults.conf values with spaces without needing
> > the
> > > quotes (andras.piros via gezapeti)
> > > OOZIE-2923 Improve Spark options parsing (andras.piros via gezapeti)
> > > OOZIE-2863 SLACalculatorMemory.loadOnRestart causing delay in server
> > start
> > > (puru via satishsaley)
> > > OOZIE-2862 Coord change command doesn't change job to running if job
> was
> > > killed without creating any actions (puru)
> > > OOZIE-2807 Oozie gets RM delegation token even for checking job status
> > > (satishsaley)
> > > OOZIE-2811 Add support for filtering out properties from
> > > SparkConfigurationService (gezapeti via rkanter)
> > > OOZIE-2802 Spark action failure on Spark 2.1.0 due to duplicate
> sharelibs
> > > (gezapeti via rkanter)
> > > OOZIE-2803 Mask passwords when printing out configs/args in
> MapReduceMain
> > > and SparkMain (pbacsko via rkanter)
> > > OOZIE-2799 Setting log location for spark sql on hive (satishsaley)
> > > OOZIE-2792 Hive2 action is not parsing Spark application ID from log
> file
> > > properly when Hive is on Spark (zhengxb2005 via rkanter)
> > > OOZIE-2786 Pass Oozie workflow ID and settings to Spark application
> > > configuration (zhengxb2005 via rkanter)
> > > OOZIE-2790 log4j configuration is not passed to spark executors
> > > (satishsaley)
> > > OOZIE-2787 Oozie distributes application jar twice making the spark job
> > > fail (satishsaley)
> > > OOZIE-2777 Config-default.xml longer than 64k results in
> > > java.io.UTFDataFormatException (gezapeti via harsh)
> > > OOZIE-2771 Allow retrieving keystore and truststore passwords from
> Hadoop
> > > Credential Provider (asasvari via abhishekbafna)
> > > OOZIE-2748 NPE in LauncherMapper.printArgs() (pbacsko via rkanter)
> > > OOZIE-2654 Zookeeper dependent services should not depend on
> > > Connectionstate to be valid before cleaning up (venkatnrangan via
> > > abhishekbafna)
> > > OOZIE-2690 OOZIE NPE while executing kill() (abhishekbafna via
> > > jaydeepvishwakarma)
> > >
> > >
> > > Please reply to this 

[jira] [Resolved] (OOZIE-3103) Improve log message regarding max concurrency for coordinators

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter resolved OOZIE-3103.
--
   Resolution: Fixed
 Assignee: Attila Sasvari
Fix Version/s: 5.0.0b1

Thanks [~asasvari].  Committed to master!

> Improve log message regarding max concurrency for coordinators
> --
>
> Key: OOZIE-3103
> URL: https://issues.apache.org/jira/browse/OOZIE-3103
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Assignee: Attila Sasvari
>Priority: Trivial
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-3103-001.patch, OOZIE-3103-002.patch
>
>
> Currently, when maximum concurrency is reached for coordinators, the log 
> message is as follows:
> {code}
> 2017-09-25 06:00:00,069 WARN 
> org.apache.oozie.command.coord.CoordActionReadyXCommand: 
> SERVER[dsfsdfsdfds.com] USER[testclu0] GROUP[-] TOKEN[] 
> APP[test->dev->xyzabc] JOB[065-191474932323346-oozie-oozi-C] ACTION[] No 
> actions to start for jobId=065-191474932323346-oozie-oozi-C as max 
> concurrency reached!
> {code}
> It's confusing as-is and sounds like something is wrong, even though it's 
> perfectly fine. 
> We should make it an INFO message and say something more specific and less 
> scary sounding (no exclamation ). Maybe something like "Not starting any 
> additional actions because max concurrency \[] for \[] has 
> been reached"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3103) Improve log message regarding max concurrency for coordinators

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240633#comment-16240633
 ] 

Robert Kanter commented on OOZIE-3103:
--

+1

> Improve log message regarding max concurrency for coordinators
> --
>
> Key: OOZIE-3103
> URL: https://issues.apache.org/jira/browse/OOZIE-3103
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Priority: Trivial
> Attachments: OOZIE-3103-001.patch, OOZIE-3103-002.patch
>
>
> Currently, when maximum concurrency is reached for coordinators, the log 
> message is as follows:
> {code}
> 2017-09-25 06:00:00,069 WARN 
> org.apache.oozie.command.coord.CoordActionReadyXCommand: 
> SERVER[dsfsdfsdfds.com] USER[testclu0] GROUP[-] TOKEN[] 
> APP[test->dev->xyzabc] JOB[065-191474932323346-oozie-oozi-C] ACTION[] No 
> actions to start for jobId=065-191474932323346-oozie-oozi-C as max 
> concurrency reached!
> {code}
> It's confusing as-is and sounds like something is wrong, even though it's 
> perfectly fine. 
> We should make it an INFO message and say something more specific and less 
> scary sounding (no exclamation ). Maybe something like "Not starting any 
> additional actions because max concurrency \[] for \[] has 
> been reached"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3091) Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper"

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227287#comment-16227287
 ] 

Robert Kanter commented on OOZIE-3091:
--

Done.

> Oozie Sqoop Avro Import fails with "java.lang.NoClassDefFoundError: 
> org/apache/avro/mapred/AvroWrapper"
> ---
>
> Key: OOZIE-3091
> URL: https://issues.apache.org/jira/browse/OOZIE-3091
> Project: Oozie
>  Issue Type: Bug
>  Components: action
>Affects Versions: 4.2.0
>Reporter: Prabhu Joseph
>
> Oozie Sqoop Action which does Import as avro fails with below. 
> avro-mapred-1.8.0-hadoop2.jar need to be included in Oozie Sqoop Sharelib
> {code}
> 2017-10-19 09:45:25,349 WARN [main] org.apache.hadoop.mapred.YarnChild: 
> Exception running child : java.lang.RuntimeException: 
> java.lang.reflect.InvocationTargetException
> at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:134)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
> ... 7 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroWrapper
> at 
> org.apache.sqoop.mapreduce.AvroImportMapper.(AvroImportMapper.java:43)
> ... 12 more
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.avro.mapred.AvroWrapper
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 13 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (OOZIE-3103) Improve log message regarding max concurrency for coordinators

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227189#comment-16227189
 ] 

Robert Kanter edited comment on OOZIE-3103 at 10/31/17 5:47 PM:


We should use the \{0} syntax instead of concatenating strings.  The message 
sounds good though.  And we usually put values in "[" and "]" when logging.


was (Author: rkanter):
We should use the {{\{0\}}} syntax instead of concatenating strings.  The 
message sounds good though.

> Improve log message regarding max concurrency for coordinators
> --
>
> Key: OOZIE-3103
> URL: https://issues.apache.org/jira/browse/OOZIE-3103
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Priority: Trivial
> Attachments: OOZIE-3103-001.patch
>
>
> Currently, when maximum concurrency is reached for coordinators, the log 
> message is as follows:
> {code}
> 2017-09-25 06:00:00,069 WARN 
> org.apache.oozie.command.coord.CoordActionReadyXCommand: 
> SERVER[dsfsdfsdfds.com] USER[testclu0] GROUP[-] TOKEN[] 
> APP[test->dev->xyzabc] JOB[065-191474932323346-oozie-oozi-C] ACTION[] No 
> actions to start for jobId=065-191474932323346-oozie-oozi-C as max 
> concurrency reached!
> {code}
> It's confusing as-is and sounds like something is wrong, even though it's 
> perfectly fine. 
> We should make it an INFO message and say something more specific and less 
> scary sounding (no exclamation ). Maybe something like "Not starting any 
> additional actions because max concurrency \[] for \[] has 
> been reached"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-3103) Improve log message regarding max concurrency for coordinators

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-3103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16227189#comment-16227189
 ] 

Robert Kanter commented on OOZIE-3103:
--

We should use the {{\{0\}}} syntax instead of concatenating strings.  The 
message sounds good though.

> Improve log message regarding max concurrency for coordinators
> --
>
> Key: OOZIE-3103
> URL: https://issues.apache.org/jira/browse/OOZIE-3103
> Project: Oozie
>  Issue Type: Improvement
>Reporter: Attila Sasvari
>Priority: Trivial
> Attachments: OOZIE-3103-001.patch
>
>
> Currently, when maximum concurrency is reached for coordinators, the log 
> message is as follows:
> {code}
> 2017-09-25 06:00:00,069 WARN 
> org.apache.oozie.command.coord.CoordActionReadyXCommand: 
> SERVER[dsfsdfsdfds.com] USER[testclu0] GROUP[-] TOKEN[] 
> APP[test->dev->xyzabc] JOB[065-191474932323346-oozie-oozi-C] ACTION[] No 
> actions to start for jobId=065-191474932323346-oozie-oozi-C as max 
> concurrency reached!
> {code}
> It's confusing as-is and sounds like something is wrong, even though it's 
> perfectly fine. 
> We should make it an INFO message and say something more specific and less 
> scary sounding (no exclamation ). Maybe something like "Not starting any 
> additional actions because max concurrency \[] for \[] has 
> been reached"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2683) Rewrite the Oozie Web UI

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226001#comment-16226001
 ] 

Robert Kanter commented on OOZIE-2683:
--

I think people will be happy with pretty much anything :)

Just pick something that's (a) easy to develop with, (b) easy to deploy 
(ideally, automatically), and (c) license compatible.

> Rewrite the Oozie Web UI
> 
>
> Key: OOZIE-2683
> URL: https://issues.apache.org/jira/browse/OOZIE-2683
> Project: Oozie
>  Issue Type: New Feature
>    Reporter: Robert Kanter
>Assignee: Andras Piros
>
> We're currently relying on a version of ExtJS that's so old, it's not linked 
> on their official website, and is often temporarily deleted.  Looks like it's 
> now been deleted for good (OOZIE-2622).  It also has a GPL license, so we 
> can't include it with Oozie, and users are forced to go and download it 
> themselves.  And finally, it's a really outdated UI that's not very good.
> We should invest in a new Web UI that's more modern and compatibly licensed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-2929) Coordinator SLA should support configuration variable

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter reassigned OOZIE-2929:


Assignee: Chi Wen

> Coordinator SLA should support configuration variable
> -
>
> Key: OOZIE-2929
> URL: https://issues.apache.org/jira/browse/OOZIE-2929
> Project: Oozie
>  Issue Type: Bug
>  Components: coordinator
>Affects Versions: 4.3.0
>Reporter: Chi Wen
>Assignee: Chi Wen
> Attachments: OOZIE-2929-001.patch, OOZIE-2929-002.patch
>
>
> if coordinator.xml define SLA like this:
>   
> ${nominal_time}
> ${ 1 * HOURS}
> ${1 * MINUTES}
>   
> In job.properties define: nominal_time=${coord:nominalTime()}
> Then coordinator will run failed for "Coord Job Materialization Error: Could 
> not parse [${coord:nominalTime()}] using [-MM-dd'T'HH:mm+0800] mask";
> Oozie should support use define a variable for SLA;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (OOZIE-3098) Allow to use the rest api to get the creation time of the Coordinator

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter reassigned OOZIE-3098:


Assignee: Chi Wen

Done.

> Allow to use the rest api to get the creation time of the Coordinator
> -
>
> Key: OOZIE-3098
> URL: https://issues.apache.org/jira/browse/OOZIE-3098
> Project: Oozie
>  Issue Type: Improvement
>  Components: coordinator
>Affects Versions: 4.3.0
>Reporter: Chi Wen
>Assignee: Chi Wen
> Fix For: 5.0.0
>
> Attachments: OOZIE-3098-001.patch
>
>
> It seems that the rest api cannot get the creation time of the Coordinator;
> RB link: https://reviews.apache.org/r/63296/



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (OOZIE-2231) Upgrade curator to latest version 2.12.0

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219752#comment-16219752
 ] 

Robert Kanter commented on OOZIE-2231:
--

We unfortunately have to be careful about upgrading guava due to Hadoop.  If 
{{Closeables.closeQuietly(yarnClient);}} is the only issue, then I'd suggest 
copying that into Oozie.  In fact, we already have our own version in 
{{IOUtils}} 
[here|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/util/IOUtils.java#L270]
 :)

> Upgrade curator to latest version 2.12.0
> 
>
> Key: OOZIE-2231
> URL: https://issues.apache.org/jira/browse/OOZIE-2231
> Project: Oozie
>  Issue Type: Bug
>  Components: HA
>Reporter: Purshotam Shah
>Assignee: Artem Ervits
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2231-00.patch, OOZIE-2231-01.patch, 
> OOZIE-2231-02.patch, OOZIE-2231-02.patch, OOZIE-2231-03.patch, 
> OOZIE-2231-04.patch
>
>
> It have some fix related to InterProcessReadWriteLock, ChildReaper, 
> LeaderSelector which we use.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2896:
-
Fix Version/s: (was: 5.0.0)
   5.0.0b1

> Ensure compatibility for existing LauncherMapper settings
> -
>
> Key: OOZIE-2896
> URL: https://issues.apache.org/jira/browse/OOZIE-2896
> Project: Oozie
>  Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Bacsko
>Assignee: Andras Piros
>Priority: Blocker
> Fix For: 5.0.0b1
>
> Attachments: OOZIE-2896.001.patch, OOZIE-2896.002.patch, 
> OOZIE-2896.002.patch, OOZIE-2896.003.patch, OOZIE-2896.004.patch, 
> OOZIE-2896.005.patch, OOZIE-2896.006.patch
>
>
> There are a couple of LauncherMapper settings that we should support for 
> backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, 
> {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since 
> there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2973:
-
Fix Version/s: 5.0.0b1

> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
> Fix For: 5.0.0b1
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (OOZIE-2973) Make sure Oozie works with Hadoop 3

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/OOZIE-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated OOZIE-2973:
-
Priority: Blocker  (was: Major)

> Make sure Oozie works with Hadoop 3 
> 
>
> Key: OOZIE-2973
> URL: https://issues.apache.org/jira/browse/OOZIE-2973
> Project: Oozie
>  Issue Type: Bug
>Reporter: Attila Sasvari
>Priority: Blocker
> Fix For: 5.0.0b1
>
>
> This JIRA is to track that Oozie works with Hadoop 3. 
> - Build Oozie with latest hadoop 3 (e.g. assuming Apache Hadoop 3.0.0-alpha3 
> pass {{-D-Dhadoop.version=3.0.0-alpha3}} to maven) and execute tests
> - It would be nice to create an Apache jenkins job that runs Oozie tests with 
> hadoop 3 (also with other component) 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Welcome new Oozie Committer - Andras Piros

[Robert Kanter]

Hi everyone,

It is my pleasure to announce that the Oozie PMC has invited
Andras Piros to become an Oozie committer and he has
accepted our invitation.

Please join me in congratulating him.
Congrats!


- Robert, on behalf of the Oozie PMC

[jira] [Commented] (OOZIE-2296) Add an Oozie diagnostic bundle tool

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206286#comment-16206286
 ] 

Robert Kanter commented on OOZIE-2296:
--

You mentioned in one of the comments on RB the idea of getting the DAG image, 
especially now that OOZIE-2406 is done and we can get a dot file now too.  This 
is a great idea!

However, we should probably wait for OOZIE-3059 first.  It's to add a CLI 
command to get the image.  While you could have the tool use the REST API 
directly to do that, that's going to be 90% overlap with the code needed for 
OOZIE-3059, so you may as well do OOZIE-3059 instead.

I'm fine with getting the image being a followup JIRA if you don't want to 
block the rest of this on OOZIE-3059.

> Add an Oozie diagnostic bundle tool
> ---
>
> Key: OOZIE-2296
> URL: https://issues.apache.org/jira/browse/OOZIE-2296
> Project: Oozie
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: trunk
>    Reporter: Robert Kanter
>Assignee: Attila Sasvari
> Attachments: OOZIE-2296-002.patch, OOZIE-2296-003.patch, 
> OOZIE-2296-004.patch, OOZIE-2296-005.patch, OOZIE-2296-006.patch, 
> OOZIE-2296-009.patch, OOZIE-2296-010.patch, OOZIE-2296-011.patch, 
> OOZIE-2296-012.patch, OOZIE-2296-013.patch, OOZIE-2296-014.patch, 
> OOZIE-2296-015.patch
>
>
> To help with our support cases, I've built a tool that collects a bunch of 
> job and other information from Oozie that throws it all in a tarball.  The 
> idea is that the user can just click a button in Cloudera Manager, and it 
> will run this tool.  
> This tool could be useful for others as an easy way to get information out of 
> Oozie, so I thought I'd contribute it here.  It's built as a "tool" (so it 
> sits next to the sharelib and database tools), and simply uses the Oozie 
> client for getting pretty much everything, so it doesn't require anything 
> special.
> Here's the information that it can get:
> # Sharelib: {{ooze admin -shareliblist}} and {{oozie admin -shareliblist 
> }}
> # Oozie Server's resolved loaded configuration (from admin endpoint)
> # Other admin commands output (queue dump, env vars, etc)
> # Thread dump (HOST:11000/oozie/admin/jvminfo.jsp)
> # Details from last n jobs and/or specific list of jobs
> #- job.properties contents
> #- XML definition
> #- verbose status for each job and each action etc
> #- Oozie logs
> #- Unfortunately, we can't get the launcher jobs' logs from Hadoop
> # Metrics/Instrumentation



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

[Robert Kanter via Review Board]

> On Oct. 12, 2017, 11:36 p.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
> > Lines 1197 (patched)
> > <https://reviews.apache.org/r/62936/diff/1/?file=1853365#file1853365line1233>
> >
> > This is fine to leave for a different JIRA, but we should consider 
> > dropping the "oozie:launcher:" part of the name.  The job type is now 
> > OOZIE, so it's probably redundent and we can shorten the names.
> 
> Peter Cseh wrote:
> OOZIE-2893

Oh I forgot about that Haha.  At least I'm being consistent with the name :)


- Robert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review187893
---


On Oct. 12, 2017, 3:33 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated Oct. 12, 2017, 3:33 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
>  PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 2389b993b7b1769521d6dd0c7386e99e7516210a 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationFilter.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/1/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

[jira] [Commented] (OOZIE-2406) Completely rewrite GraphGenerator code

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/OOZIE-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203850#comment-16203850
 ] 

Robert Kanter commented on OOZIE-2406:
--

Yay!

> Completely rewrite GraphGenerator code
> --
>
> Key: OOZIE-2406
> URL: https://issues.apache.org/jira/browse/OOZIE-2406
> Project: Oozie
>  Issue Type: New Feature
>Affects Versions: 4.2.0
>    Reporter: Robert Kanter
>Assignee: Andras Piros
> Fix For: 5.0.0
>
> Attachments: OOZIE-2406.004.patch, OOZIE-2406.004.patch, 
> OOZIE-2406.005.patch, OOZIE-2406.006.patch, OOZIE-2406.007.patch, 
> OOZIE-2406.008.patch, OOZIE-2406.009.patch, OOZIE-2406.010.patch, 
> OOZIE-2406.011.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch, 
> OOZIE-2406.012.patch, OOZIE-2406.012.patch, OOZIE-2406.012.patch, 
> OOZIE-2406.013.patch, Screen Shot 2017-10-02 at 1.55.58 PM.png, Screen Shot 
> 2017-10-02 at 1.56.33 PM.png, graphviz-10-action-nodes.png, 
> graphviz-25-action-nodes.png, graphviz-50-action-nodes.png, 
> graphviz-many-nodes.png, jung-many-nodes.png
>
>
> The Web UI currently generates a graph of the workflow DAG as a png image 
> that it can show to the user ({{GraphGenerator}} class).  Unfortunately, 
> there are a number of downsides to the current implementation:
> # The image is generated server-side, which doesn't scale well and eats up 
> lots of memory
> #- To help combat this issue, we only generate graphs for workflows with less 
> than 25 nodes and we disabled the refresh button in the UI
> # It's slow
> # It uses a library ({{net.sf.jung}} from http://jung.sourceforge.net), which 
> hasn't been updated since 2010.  
> #- This library also has a dependency on a fork of Commons-Collections 
> ({{net.sourceforge.collections}} from 
> http://sourceforge.net/projects/collections/), which similarly hasn't been 
> updated since 2010.  This is a problem because we can't update 
> Commons-Collections when there are security concerns such as COLLECTIONS-580 
> (though Oozie is *not* susceptible to this attack).
> It would be good to do a complete rewrite, using a different library and 
> remove Jung and the Commons-Collections fork.  Whatever we choose, it should 
> an svg which will draw the image on the browser.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 62936: OOZIE-2896 Ensure compatibility for existing LauncherMapper settings

[Robert Kanter via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62936/#review187893
---




core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Line 17 (original), 17 (patched)
<https://reviews.apache.org/r/62936/#comment264948>

Don't forget to add documentation.  I'm not sure where is the best place 
though.  I guess somewhere on the workflow page?



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1154 (patched)
<https://reviews.apache.org/r/62936/#comment264936>

Shouldn't we check the workflow.xml first?  Same with the others.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1197 (patched)
<https://reviews.apache.org/r/62936/#comment264938>

This is fine to leave for a different JIRA, but we should consider dropping 
the "oozie:launcher:" part of the name.  The job type is now OOZIE, so it's 
probably redundent and we can shorten the names.



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1211 (patched)
<https://reviews.apache.org/r/62936/#comment264942>

Should we log something here?  A stack trace is unnecessary, but maybe 
something like "Invalid integer [X] specified for max attempts, using default 
[Y]".



core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
Lines 1217-1221 (patched)
<https://reviews.apache.org/r/62936/#comment264944>

The log message might be misleading if you had a negative value.  It would 
say that we're setting it, but then the if statement would make it not.

Also, should we log something in the case of a negative value?



core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
Lines 29 (patched)
<https://reviews.apache.org/r/62936/#comment264946>

Should we make this all static?  It doesn't really have any state and we're 
always calling it like:

new LauncherConfigurationFilter(actionDefaultConf).filter(launcherConf);


We could avoid creating the object by doing:

LauncherConfigurationFilter.filter(actionDefaultConf, launcherConf);



- Robert Kanter


On Oct. 12, 2017, 3:33 p.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62936/
> ---
> 
> (Updated Oct. 12, 2017, 3:33 p.m.)
> 
> 
> Review request for oozie, Attila Sasvari, Peter Cseh, and Peter Bacsko.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2896 Ensure compatibility for existing LauncherMapper settings
> 
> 
> Diffs
> -
> 
>   core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 
> 76d0daa4f7528de6fb3c8993bae4dd28b6f7865b 
>   
> core/src/main/java/org/apache/oozie/action/hadoop/LauncherConfigurationFilter.java
>  PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 2389b993b7b1769521d6dd0c7386e99e7516210a 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 
> a7bd357f8938d148a150db43135a7a8fe7f0da4c 
>   
> core/src/test/java/org/apache/oozie/action/hadoop/TestLauncherConfigurationFilter.java
>  PRE-CREATION 
>   sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherAM.java 
> ee1a32ac9b071009e1dbe78396520d778f58bddc 
> 
> 
> Diff: https://reviews.apache.org/r/62936/diff/1/
> 
> 
> Testing
> ---
> 
> JUnit tests:
> 
> * `TestJavaActionExecutor`
> * `TestLauncherConfigurationFilter`
> 
> Functional tests: submit and run `examples/apps/java-main/workflow.xml` in 
> three modes:
> 
> * only `oozie.launcher.` parameters are present: `oozie.launcher.` parameters 
> are passed through
> * `oozie.launcher.` and override (`yarn.`, `mapreduce.`, and `mapred.`) 
> parameters are also present: `oozie.launcher.` parameters are passed through
> * only override (`yarn.`, `mapreduce.`, and `mapred.`) parameters are 
> present: override parameters are passed through
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62352: OOZIE-2406 Completely rewrite GraphGenerator code

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62352/#review187224
---


Fix it, then Ship it!




Looks great!  Just some trivial things.


core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 24 (patched)
<https://reviews.apache.org/r/62352/#comment264158>

Let's not use * imports



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 51 (patched)
<https://reviews.apache.org/r/62352/#comment264156>

Extra space before @param



core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java
Lines 67 (patched)
<https://reviews.apache.org/r/62352/#comment264157>

This isn't always a PNG stream, right?



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 39 (patched)
<https://reviews.apache.org/r/62352/#comment264159>

Let's not use * imports



core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java
Lines 53 (patched)
<https://reviews.apache.org/r/62352/#comment264160>

oozie.graphviz.timeout.seconds should be a constant.  No need to 
put a default value here because it's in oozie-default.xml and we want to keep 
only one source of truth on those.
ConfigurationService.getLong(OOZIE_GRAPHVIZ_TIMEOUT_SECONDS);



core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java
Lines 394 (patched)
<https://reviews.apache.org/r/62352/#comment264162>

// format=svg -> OutputFormat.SVG


- Robert Kanter


On Oct. 5, 2017, 11:02 a.m., András Piros wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62352/
> ---
> 
> (Updated Oct. 5, 2017, 11:02 a.m.)
> 
> 
> Review request for oozie and Robert Kanter.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> OOZIE-2406 Completely rewrite GraphGenerator code
> 
> 
> Diffs
> -
> 
>   client/src/main/java/org/apache/oozie/client/rest/RestConstants.java 
> 74843dc9a3cbb8cef38e97f81e34731f191d9aab 
>   core/pom.xml 6f9adb66af9344ac7d2212cdc31aa203ec06c286 
>   core/src/main/java/org/apache/oozie/servlet/JsonRestServlet.java 
> 059d3cf6dc251b49940af29d82cbdd817043a176 
>   core/src/main/java/org/apache/oozie/servlet/V1JobServlet.java 
> 10812c675ebc0cc4aefca9f4a85ef2fc26d143e1 
>   core/src/main/java/org/apache/oozie/util/GraphGenerator.java 
> 6ded2c6dc15c9e8453ff800407ff0324be185f41 
>   core/src/main/java/org/apache/oozie/util/graph/GraphGenerator.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/GraphvizRenderer.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/OutputFormat.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowActionNode.java 
> PRE-CREATION 
>   core/src/main/java/org/apache/oozie/util/graph/WorkflowGraphHandler.java 
> PRE-CREATION 
>   core/src/main/resources/oozie-default.xml 
> 2389b993b7b1769521d6dd0c7386e99e7516210a 
>   core/src/test/java/org/apache/oozie/servlet/TestV1JobServlet.java 
> ee9ab556c9b6930c406ca5dcd54065cbbebb86c9 
>   core/src/test/java/org/apache/oozie/util/TestGraphGenerator.java 
> 002e925b57cd830ea6d83a87cea4383165116b80 
>   core/src/test/java/org/apache/oozie/util/graph/TestGraphGenerator.java 
> PRE-CREATION 
>   core/src/test/resources/graph-workflow-decision-fork-join.xml PRE-CREATION 
>   core/src/test/resources/graph-workflow-simple.xml PRE-CREATION 
>   core/src/test/resources/graphWF.xml 
> 6a7b0427a9951835a7533a04b66258ded369d5bf 
>   core/src/test/resources/graphWF_26_actions.xml 
> a091be0f3559ede195ccc3339adee4478a8da8c0 
>   core/src/test/resources/invalidGraphWF.xml  
>   docs/src/site/twiki/WebServicesAPI.twiki 
> ef3e60242512decd48beb3d8c9ac747b7d128eda 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   webapp/src/main/webapp/oozie-console.js 
> 72c8a198a4ffe60f74a9f700831f65efcb3066c4 
> 
> 
> Diff: https://reviews.apache.org/r/62352/diff/7/
> 
> 
> Testing
> ---
> 
> `TestGraphGenerator`, `TestV1JobServlet`
> 
> 
> Thanks,
> 
> András Piros
> 
>

Re: Review Request 62459: OOZIE-2296: Add an Oozie diagnostic bundle tool

[Robert Kanter]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62459/#review187223
---




docs/src/site/twiki/DG_CommandLineTool.twiki
Lines 1989 (patched)
<https://reviews.apache.org/r/62459/#comment264151>

Shouldn't this "Done" show up on the previous line like the rest of them?



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 173-174 (patched)
<https://reviews.apache.org/r/62459/#comment264153>

LogAggregationUtils is marked @Private so we shouldn't be 
using it.  Hadoop can change things incompatibly here.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 180 (patched)
<https://reviews.apache.org/r/62459/#comment264152>

var20 is a funny name for a variable :)



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 241 (patched)
<https://reviews.apache.org/r/62459/#comment264155>

storeOozieLauncherLog should be called per Action, not per Job.  
Launcher logs are an Action-level concept.

Here, you're just picking the first action's external id to use for the 
job, which may not even exist (e.g. the first action is a decision node or an 
hdfs node.



tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java
Lines 242 (patched)
<https://reviews.apache.org/r/62459/#comment264154>

We should get this from the WorkflowJob.  The log reading code 
computes a path based on the user who ran the job, which is likely to be 
different from the user running this tool.


- Robert Kanter


On Oct. 4, 2017, 2:18 p.m., Attila Sasvari wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62459/
> ---
> 
> (Updated Oct. 4, 2017, 2:18 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Repository: oozie-git
> 
> 
> Description
> ---
> 
> A diagnostic tool that collects a bunch of job and other information from 
> Oozie in a zip file.
> 
> 
> Diffs
> -
> 
>   docs/src/site/twiki/DG_CommandLineTool.twiki 
> d4047671876dcc3279a2ec379bc1d003f5e6f1aa 
>   pom.xml 0b94484da1c97618e9168cea0ebbfff7f70f723c 
>   tools/pom.xml 7306a14e7b237977be00f8fe28e34573540fd508 
>   tools/src/main/bin/oozie-diag-bundle-collector.sh PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/AppInfoCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ArgParser.java PRE-CREATION 
>   
> tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCollectorDriver.java
>  PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleCompressor.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagBundleEntryWriter.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/DiagOozieClient.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/MetricsCollector.java 
> PRE-CREATION 
>   tools/src/main/java/org/apache/oozie/tools/diag/ServerInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestAppInfoCollector.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestArgParser.java 
> PRE-CREATION 
>   tools/src/test/java/org/apache/oozie/tools/diag/TestMetricsCollector.java 
> PRE-CREATION 
>   
> tools/src/test/java/org/apache/oozie/tools/diag/TestServerInfoCollector.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/62459/diff/10/
> 
> 
> Testing
> ---
> 
> - new unit tests: TestOozieDiagBundleCollector
> - started Oozie with a pseudo hadoop cluster, submitted a couple workflows, 
> and executed the following commands: 
> -- ``bin/oozie-diag-bundle-collector.sh`` (usage info printed),
> -- ``bin/oozie-diag-bundle-collector.sh  -numworkflows 2000 -oozie 
> http://localhost:11000/oozie -output /tmp``, 
> -- ``bin/oozie-diag-bundle-collector.sh  -jobs 
> 001-170918144116149-oozie-asas-W -oozie http://localhost:11000/oozie 
> -output .`` (verified zip the tool generated).
> 
> 
> Thanks,
> 
> Attila Sasvari
> 
>