Re: Oozie on Kerberos

[Andras Piros]

Hi Namit,

for Oozie latest 5.1.0, you can try the following:

   1. setting up proper Hadoop configuration
   ,
also while
   installing
   

   2. server setup if you need SSL
   

   3. action-level Hadoop authentication setup
   .
   Here's an example
   

   4. CLI authentication
   

Hope that helps,

Andras

On Wed, Feb 27, 2019 at 10:03 AM Namit Kabra  wrote:

> Hi,
>
> I am searching for some documentation on how to run a mapreduce job using
> oozie on a kerberized cluster?
> Any help would be appreciated.
>
> Regards,
> Namit Kabra
>
> Note: Here are the references that I tried:
>
>1. It may be possible to run oozie using curl command on the Kereberized
>cluster by using: oozie jobs -auth KERBEROS  -oozie
>https://[redacted-server-same]:11443/oozie (Reference:
>
> https://community.cloudera.com/t5/Batch-Processing-and-Workflow/oozie-Authentication-exception-GSSException-No-valid/td-p/66172
>)
>2. https://oozie.apache.org/docs/4.3.1/ENG_Custom_Authentication.html
>3.
>
> https://ambari.apache.org/1.2.5/installing-hadoop-using-ambari/content/ambari-kerb-2-3-4.html
>( Configuring Secure Oozie)
>4.
>
> https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_sg_oozie_server_kerberos.html
>(Configuring Kerberos Authentication for the Oozie Server for Cloudera)
>5. https://oozie.apache.org/docs/3.3.2/AG_Install.html (Setting Up
> Oozie
>with HTTPS (SSL))
>6.
> https://oozie.apache.org/docs/3.2.0-incubating/DG_CommandLineTool.html
>(Oozie CLI Utilities)
>

Re: Oozie actions staying in PREP status

[Andras Piros]

Hello,

in case there are some Oozie launcher / action applications inside YARN RM
WebUI but these haven't been run, chances are you need to set
oozie.launcher.mapreduce.map.memory.mb to some high value in your
workflow.xml file like this

.

Regards,

Andras

On Fri, Oct 19, 2018 at 5:34 PM lune voo  wrote:

> Hello everyone.
>
> Sending this mail because I have a problem with my Oozie server.
> Currently using Oozie 4.2 in HDP 2.6.2 in a cluster of 10 worker nodes, 2
> master with HA components and multiples edges with one containing the oozie
> server.
>
> I send this mail because I have a problem in the cluster.
>
> All the actions of all the workflows are stuck in PREP state.
> I cannot see any jobs running in Yarn, except some of them not using Oozie.
>
> The problem begun this morning, before it was working well.
>
> I checked oozie server logs and don't see anything special (no ERROR or
> WARN) even in DEBUG log mode.
>
> I checked the oozie admin web page to see if it was a JVM problem, but I
> cannot see anything special :
> ###
> Memory Information:
>
>- Heap Memory:
>INIT=197132288  USED=309877976  COMMITTED=832569344  MAX=3817865216
>- NonHeap Memory:
> INIT=2555904  USED=139238472  COMMITTED=142237696  MAX=-1
>- ClassLoading:
> Total Loaded Classes=10656  Loaded Classes=10564  Unloaded
>Classes=92
>- Threads:
> Thread Count=62  Peak Thread Count=135  Total Started
>Threads=2708300  Deamon Threads=40
>
>RUNNABLE=6BLOCKED=0WAITING=39TIMED_WAITING=17
> NEW=0TERMINATED=0
>Time of Thread Dump=Fri Oct 19 17:32:43 CEST 2018
>
> ###
>
> As an additionnal note, my environment is kerberized and everything except
> Oozie works fine.
>
> I was wondering if one of you had already seen this behaviour ?
>
> Best regards.
>
> Lune
>

Re: Oozie 5.1.0 release and plans for the year

[Andras Piros]

Hi everyone,

Gp, I can take over 5.1.0 release management. AFAIK we're almost a go for
RC0.

Regards,

Andras

On Tue, Jun 19, 2018 at 10:32 PM Artem Ervits  wrote:

> +1 on the plans, I won't get to the jiras I'm assigned to until next week.
> Don't want to be a blocker.
>
> On Tue, Jun 19, 2018, 4:48 AM Andras Piros  wrote:
>
> > Good idea Gp!
> >
> > Thanks for volunteering as the release manager for 5.1.0. I can see *a
> few
> > blockers
> > <
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20OOZIE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%205.1.0%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20%20%20priority%20DESC%2C%20updated%20DESC
> > >*
> > for 5.1.0 now, of which *OOZIE-3178
> > <https://issues.apache.org/jira/browse/OOZIE-3178>* seems to be a real
> > one.
> >
> > Regards,
> >
> > Andras
> >
> > On Tue, Jun 19, 2018 at 12:50 PM Gézapeti Cseh 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Now, that Oozie-2339 is in (with other fixes as well), I think it would
> > be
> > > nice to release it as part of Oozie 5.1.0 so more people will try it
> out
> > > before we jump in to do the coordinator/bunlde part as well.
> > >
> > > Also, we're planning to pick up on new action types with Andras and
> Peter
> > > and probably will do a bunch of new releases as they are ready later
> this
> > > year. The git action is the closest one, but there are others in the
> > > pipeline like Maven and callback.
> > >
> > > Is anyone aware of issues in which we should wait for to start working
> on
> > > 5.1.0?
> > > If nobody else does, I can volunteer to be the release manager for it
> and
> > > start the branching process in a week or so.
> > >
> > > thanks
> > > gp
> > >
> >
>

Re: oozie install failed due to not found oozie-sharelib.tar.gz

[Andras Piros]

Hi Lian Jiang,

you could set TRACE level logging
<https://oozie.apache.org/docs/4.3.1/AG_OozieLogging.html> for package
org.apache.oozie.service
<https://github.com/apache/oozie/blob/release-4.3.1/core/src/main/java/org/apache/oozie/service/Services.java>
for better log information.

Can you please also post stderr content? Should be somewhere around
catalina.log.

Regards,

Andras

On Tue, Aug 21, 2018 at 7:02 PM Lian Jiang  wrote:

> Thanks Andras.
>
> /usr/hdp/current/oozie-server/oozie-sharelib.tar.gz missing does not 100%
> repro. I will cut the log when I repro it again.
>
> In my new repro, /usr/hdp/current/oozie-server/oozie-sharelib.tar.gz
> presents. However, oozie server cannot be started and the logs are not
> obvious:
>
>
>
> ***
>
>
>   STARTUP MSG: Oozie BUILD_VERSION [4.3.1.3.0.0.0-1634] compiled by
> [jenkins] on [${build.time}]
>
>   STARTUP MSG:   revision [${vc.revision}]@[${vc.url}]
>
>
> ***
>
> 2018-08-21 05:59:46,794  INFO XLogService:520 - SERVER[] Log4j
> configuration file [oozie-log4j.properties]
>
> 2018-08-21 05:59:46,795  INFO XLogService:520 - SERVER[] Log4j
> configuration file loaded from [/usr/hdp/current/oozie-server/conf]
>
> 2018-08-21 05:59:46,795  INFO XLogService:520 - SERVER[] Log4j reload
> interval [10 sec]
>
> 2018-08-21 05:59:46,796  INFO ConfigurationService:520 - SERVER[] Oozie
> home dir  [/usr/hdp/3.0.0.0-1634/oozie]
>
> 2018-08-21 05:59:46,796  INFO ConfigurationService:520 - SERVER[] Oozie
> conf dir  [/usr/hdp/current/oozie-server/conf]
>
> 2018-08-21 05:59:46,797  INFO ConfigurationService:520 - SERVER[] Oozie
> conf file [oozie-site.xml]
>
> 2018-08-21 05:59:46,810  WARN ConfigurationService:523 - SERVER[] Invalid
> configuration defined, [use.system.libpath.for.mapreduce.and.pig.jobs]
>
> 2018-08-21 05:59:46,811  WARN ConfigurationService:523 - SERVER[] Invalid
> configuration defined, [oozie.email.smtp.password]
>
> 2018-08-21 05:59:46,811  WARN ConfigurationService:523 - SERVER[] Invalid
> configuration defined, [oozie.email.smtp.username]
>
> 2018-08-21 05:59:46,812  WARN ConfigurationService:523 - SERVER[] Invalid
> configuration defined,
> [oozie.service.AuthorizationService.security.enabled]
>
>
> 2018-08-21 05:59:46,813  WARN ConfigurationService:523 - SERVER[] Invalid
> configuration defined, [oozie.authentication.jwt.cookie]
>
> 2018-08-21 05:59:46,813  INFO Services:520 - SERVER[] Exiting null Entering
> NORMAL
>
> 2018-08-21 05:59:46,814  INFO Services:520 - SERVER[] Initialized runtime
> directory [/tmp/oozie-root151166298688308409.dir]
>
> 2018-08-21 05:59:46,815  INFO ConfigurationService:520 - SERVER[]
> Programmatic configuration change,
> property[oozie.services]=[org.apache.oozie.service.HadoopAccessorService]
>
> 2018-08-21 05:59:46,815  INFO ConfigurationService:520 - SERVER[]
> Programmatic configuration change, property[oozie.services.ext]=[]
>
> 2018-08-21 05:59:46,816  INFO HadoopAccessorService:520 - SERVER[]
> JOB_TRACKER_WHITELIST :[], Total entries :0
>
> 2018-08-21 05:59:46,816  INFO HadoopAccessorService:520 - SERVER[]
> NAME_NODE_WHITELIST :[], Total entries :0
>
> 2018-08-21 05:59:46,816  INFO HadoopAccessorService:520 - SERVER[] Oozie
> Kerberos Authentication [disabled]
>
> 2018-08-21 05:59:46,841  INFO Services:520 - SERVER[] Initialized
>
> 2018-08-21 05:59:46,841  INFO Services:520 - SERVER[] Running with JARs for
> Hadoop version [3.1.0.3.0.0.0-1634]
>
> 2018-08-21 05:59:46,841  INFO Services:520 - SERVER[] Oozie System ID
> [oozie-root] started!
>
> 2018-08-21 05:59:46,842  INFO Services:520 - SERVER[] Shutdown
>
>
>
> As you can see, oozie server started and shutdown, causing port 11000
> cannot be connected. The warnings should be benign because it worked in
> HDP2.6.
>
>
> Again, I *sometimes *can start oozie server successfully and port 11000
> works fine. However, oozie installation is extremely unreliable due to
> multiple issues (e.g. missing oozie-sharelib.tar.gz, oozie server
> shutdown).
>
>
> Any idea why oozie server shutdown? How to debug? Thanks a lot.
>
>
>
> On Tue, Aug 21, 2018 at 1:10 AM Andras Piros  wrote:
>
> > Hi Lian Jiang,
> >
> > as I'm not familiar w/ HDP, can you please post exception logs? I believe
> > HDP 3.0 is based on Oozie 4.3.1.
> >
> > For a general introduction how Oozie sharelib works, please see relevant
> > parts of the Apache Oozie documentation
> > <https://oozie.apache.org/docs/4.3.1/AG_Install.html#Oozie_Share_Lib>
> and
> &

Re: oozie install failed due to not found oozie-sharelib.tar.gz

[Andras Piros]

Hi Lian Jiang,

as I'm not familiar w/ HDP, can you please post exception logs? I believe
HDP 3.0 is based on Oozie 4.3.1.

For a general introduction how Oozie sharelib works, please see relevant
parts of the Apache Oozie documentation
 and this
blog entry

.

Regards,

Andras

On Tue, Aug 21, 2018 at 12:01 AM Lian Jiang  wrote:

> Hi,
>
> I am using HDP3.0 and ambari 2.7 blueprint to install a cluster. Everything
> but oozie worked. The oozie failed to start due to the fact that
> /usr/hdp/current/oozie-server/oozie-sharelib.tar.gz is not generated.
>
> oozie-sharelib.tar.gz is 1.6G and Our production environment has very small
> /tmp partition (1G). I guess this is the reason why oozie-sharelib.tar.gz
> is not generated using ambari cluster installation. I hacked oozie-setup.sh
> to use /u01/tmp (a bigger partition) instead of /tmp and assumed
> "oozie-setup.sh prepare-war" will generate oozie-sharelib.tar.gz. However,
> I still did not get oozie-sharelib.tar.gz.
>
> Any idea how to install oozie in this case? Thanks.
>

Re: Oozie 5.1.0 release and plans for the year

[Andras Piros]

Good idea Gp!

Thanks for volunteering as the release manager for 5.1.0. I can see *a few
blockers
*
for 5.1.0 now, of which *OOZIE-3178
* seems to be a real one.

Regards,

Andras

On Tue, Jun 19, 2018 at 12:50 PM Gézapeti Cseh  wrote:

> Hi everyone,
>
> Now, that Oozie-2339 is in (with other fixes as well), I think it would be
> nice to release it as part of Oozie 5.1.0 so more people will try it out
> before we jump in to do the coordinator/bunlde part as well.
>
> Also, we're planning to pick up on new action types with Andras and Peter
> and probably will do a bunch of new releases as they are ready later this
> year. The git action is the closest one, but there are others in the
> pipeline like Maven and callback.
>
> Is anyone aware of issues in which we should wait for to start working on
> 5.1.0?
> If nobody else does, I can volunteer to be the release manager for it and
> start the branching process in a week or so.
>
> thanks
> gp
>

Re: Oozie for spark jobs without Hadoop

[Andras Piros]

Hi,

Oozie needs HDFS to store workflow, coordinator, or bundle definitions, as
well as sharelib files in a safe, distributed and scalable way. Oozie needs
YARN to run almost all of its actions, Spark action being no exception.

At the moment it's not feasible to install Oozie without those Hadoop
components. How to install Oozie please *find here
*.

Regards,

Andras

On Tue, May 15, 2018 at 4:11 PM, purna pradeep 
wrote:

> Hi,
>
> Would like to know if I can use sparkaction in oozie without having Hadoop
> cluster?
>
> I want to use oozie to schedule spark jobs on Kubernetes cluster
>
> I’m a beginner in oozie
>
> Thanks
>

Re: Oozie Presentation - New Features Overview and Ambari GUI

[Andras Piros]

Thanks Clay, nice job!

Especially good to see how Oozie can be used for cluster deployment - never
would have thought of it.

On Mon, Apr 30, 2018 at 7:42 PM, Mohammad Islam 
wrote:

>  Good work Clay!
> On Monday, April 30, 2018, 9:15:28 AM PDT, Peter Cseh <
> gezap...@cloudera.com> wrote:
>
>  Thanks Clay for sharing these!
>
> On Sun, Apr 29, 2018 at 8:44 PM, Clay B.  wrote:
>
> > Hi Oozie Users,
> >
> > Recently the ever productive Artem Ervits[1] and I presented at DataWorks
> > Summit Berlin on some of the recent Oozie community work. We focused on
> new
> > features to Oozie and the Ambari Workflow Manager for building and
> managing
> > Oozie workflows all within a GUI.
> >
> > To see our slides, please see: http://bit.ly/DataWorks_Breath
> > ing_New_Life_into_Oozie
> >
> > Also, if of interest, please see more from my past year's presentations:
> > * HBase and Oozie[2] (specifically ideas around using HBase delegation
> >  tokens in Java actions and using Oozie as a controlled privilege
> >  escalation for HBase export snapshot)
> > * Continuous delivery with Oozie (particularly ideas around
> >  OOZIE-2877)[3].
> >
> > Cheers,
> > Clay
> >
> > [1]: Artem Ervits:
> > * JIRAs: http://bit.ly/artems_oozie_jiras
> > * LinkedIn: https://twitter.com/dbist/status/987160309264801792
> > * Tweet of the event: https://twitter.com/dbist/
> status/987160309264801792
> > (This is only half the audience and does not include the folks standing
> > too!)
> >
> > [2]: DataWorks Summit San Jose 2017: "Multitenancy At Bloomberg - HBase
> > and Oozie":
> > Slides: http://bit.ly/DataWorks_Multitenancy_at_Bloomberg
> > Video: https://www.youtube.com/watch?v=iPCA1ZTitQk
> >
> > [3]: Apache: Big Data North America 2017: "Cluster Continuous Delivery
> > with Oozie":
> > Slides: http://bit.ly/ApacheCon_Cluster_Continuous_Delivery_with_Oozie
> >
>
>
>
> --
> *Peter Cseh *| Software Engineer
> cloudera.com 
>
> [image: Cloudera] 
>
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image: Cloudera
> on LinkedIn] 
> --
>
>

[ANNOUNCE] Apache Oozie 5.0.0 released

[Andras Piros]

Hello Oozie community,


Apache Oozie is a workflow scheduler system to manage Apache Hadoop jobs.


The Apache Oozie team is pleased to announce the release of Oozie 5.0.0.

Oozie 5.0.0 includes new features, bug fixes and minor improvements.


The details of the release and download location can be found at
http://oozie.apache.org.



The full release notes can be found at the download location, but some
highlights of this release include:


- moved launcher from MapReduce AM to YARN AM,
- switched from Tomcat to embedded Jetty,
- updated third party libraries,
- completely rewritten workflow graph generator,
- JDK 8 support,
- deprecated Instrumentation in favor of Metrics,
- added indexes to speed up DB queries,
- fixed CVE-2017-15712

Thanks to all developers, QA's, users, and committers for making this
release happen.

Regards,

Andras

Re: Adding --jars to oozie workflow

[Andras Piros]

Hi Fawze,

actually, the  element of the  action can contain a
comma-separated list of JAR or ZIP files. So you actually already can add
more than one JAR files to the Spark application.

Please see documentation on *Spark action details
*.

Regards,

Andras

On Sun, Mar 25, 2018 at 8:20 PM, Fawze Abujaber  wrote:

> Hi All,
>
> I'm using oozie action workflow for spark job and i want to add to my oozie
> workflow --jars options, i'm aware it's not a straightforward solution.
>
> Here is my workflow and i want to add --jars to the spark-opts
>
>  
> 
> ${jobTracker}
> ${nameNode}
> 
> 
> mapred.job.queue.name
> ${queue}
> 
> 
> ${master}
> ${mode}
> aggregation_group_3
> xx
>
> hdfs:///tmp/${jarVersion}/aggregator-code-${jarVersion}
> -jar-with-dependencies.jar
> --driver-memory ${driver_memory} --num-executors
> ${num_executors_3} --executor-cores ${executor_cores_3} --executor-memory
> ${executor_memory_3} --queue ${queue} ${files} --driver-java-options
> -Dcp.days.lookback=${days_lookback}
> ${applicationConf}
>
> hdfs://tmp/conf/rpt_agg_15m_skill_camp_distinct.yaml,
> hdfs://tmp/conf/rpt_agg_15m_skill_distinct.yaml,hdfs://
> tmp/conf/rpt_agg_15m_lob_distinct.yaml
> 
> 
> 
> 
> 
>

Re: Oozie server start failed with "Invalid TimeZone: UTC" error

[Andras Piros]

Hi Michael,

what's your JRE / JDK version? My first thought is that you should check
your JRE's timezone settings / timezone database.

For *Oracle JREs
*,
there is an *extensive FAQ
*
and a *TZupdater
tool
*
.

My second two cents are :) that it has to do something w/ the daylight
saving changes: US timezones have just changed to DST meanwhile your local
timezone Europe/Berlin still hasn't. Does the problem occur if you set your
server's date to be before Mar 11 2018 or after Mar 25 2018?

Does the issue happen w/ Oozie 5.0.0-beta1 / master?

Regards,

Andras

On Wed, Mar 14, 2018 at 7:59 AM, Michael Raj 
wrote:

> I am using Oozie 4.2.0 package(http://archive.apache.org/dist/oozie/4.2.0/).
> I built this Oozie package source and used it in windows and Linux machine
> over past 2 years. But, today I am getting the following error when
> starting oozie server (/bin/oozied.sh run) in my Linux machine.
>
> *Validate DB Connection*
> *Exception in thread "main" java.lang.ExceptionInInitializerError*
> *at org.apache.oozie.service.Services.(Services.java:114)*
> *at
> org.apache.oozie.tools.OozieDBCLI.getJdbcConf(OozieDBCLI.java:164)*
> *at
> org.apache.oozie.tools.OozieDBCLI.createConnection(OozieDBCLI.java:853)*
> *at
> org.apache.oozie.tools.OozieDBCLI.validateConnection(OozieDBCLI.java:861)*
> *at
> org.apache.oozie.tools.OozieDBCLI.createDB(OozieDBCLI.java:182)*
> *at org.apache.oozie.tools.OozieDBCLI.run(OozieDBCLI.java:126)*
> *at org.apache.oozie.tools.OozieDBCLI.main(OozieDBCLI.java:77)*
> *Caused by: java.lang.IllegalArgumentException: Invalid TimeZone: UTC*
> *at
> org.apache.oozie.util.DateUtils.getTimeZone(DateUtils.java:146)*
> *at org.apache.oozie.util.DateUtils.(DateUtils.java:44)*
> *... 7 more*
>
> When I am tracing this error in Oozie source file. The DateUtils.java file
> like below.
>
>
>
> Here run time values for the following,
> tzId = "UTC"
> tz = sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,us
> eDaylight=false,transitions=0,lastRule=null]
> tz.getID() = "GMT"
>
> So, when checking the following condition in DateUtils.java. It always
> succeeds and throw IllegalArgumentException in my case.
>
> *if (!tz.getID().equals(tzId)) {*
> *throw new IllegalArgumentException("Invalid TimeZone: " +
> tzId);*
> *}*
>
>
> My question is why my machine time zone is detecting timezone id "GMT" in
> java code. And why Apache oozie is checking this condition.
> Please, help me to to start Oozie server. Thanks in Advance.
>
>
> *Machine Environment Details:*
>
>- Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-116-generic x86_64)
>- locale - LANG=en_US.UTF-8
>- Timezone - Europe/Berlin
>
>
>

Re: oozie 4.3.0 can run with hadoop3.0?

[Andras Piros]

Hi there,

for Oozie to run w/ Hadoop 3.0.0 we need to fix *OOZIE-2973
*. This is expected to
happen in Oozie 5.0.0 which drops support for Hadoop 1.x.

So no, Oozie 4.3.0 doesn't support Hadoop 3.0.0.

Regards.

Andras

On Fri, Jan 5, 2018 at 3:25 AM, WangYQ  wrote:

> in oozie doc:
> hadoop.version (*) : indicates the Hadoop version(Hadoop-1 or Hadoop-2)
> you wish to build Oozie against specifically. It will substitute this value
> in the Oozie POM properties and pull the corresponding Hadoop artifacts
> from Maven. Default version is 1.2.1 for Hadoop-1 (the most common case).
> For Hadoop-2, the version you can pass is 2.4.0 .
>
>
>
>
>
>
> oozie  4.3.0 can run with hadoop 3.0?

Re: Oozie coordinator and kafka

[Andras Piros]

@Edward can you create upstream Oozie JIRA for that HivePartition using
stuff? Maybe providing your bits as patch ;)

@Serega @Artem do you have ideas where Oozie HDFS path handling is way
inflexible?

Thanks,

Andras

On Tue, Dec 19, 2017 at 8:42 PM, Edward Capriolo <edlinuxg...@gmail.com>
wrote:

> The point is valid. Oozie has a good amount of specific HDFS sauce baked
> in, and not in a good way. I was contemplating DataFileDependencies that
> use HivePartition information. It was not easy to do.
>
> On Tue, Dec 19, 2017 at 10:38 AM, Serega Sheypak <serega.shey...@gmail.com
> >
> wrote:
>
> > Don't push NiFi :) Please don't :)
> >
> > 2017-12-19 16:16 GMT+01:00 Artem Ervits <artemerv...@gmail.com>:
> >
> > > you're probably better off using Apache Nifi at that point, Kafka Nifi
> is
> > > better integrated today than Oozie and Kafka. Probably wrong mailing
> list
> > > to push Nifi but it's a common use case in that realm.
> > >
> > > Thanks
> > >
> > > On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak <
> > serega.shey...@gmail.com>
> > > wrote:
> > >
> > > > Hi, I know default coordinator functionality, but it's limited
> (almost)
> > > to
> > > > HDFS.
> > > > Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes
> > > > integration contract much more flexible.
> > > > I could have traceability, debuggability, transparency, throttling,
> > > > concurrency of oozie and push coordinator job on demand. And I'm not
> > > > limited to strict HDFS path pattern.
> > > >
> > > >
> > > > 2017-12-18 18:14 GMT+01:00 Andras Piros <andras.pi...@cloudera.com>:
> > > >
> > > > > Hi Serega,
> > > > >
> > > > > not to my knowledge. Would be interested on your use case, though.
> > > > >
> > > > > Would start w/ *Coordinator Input Events / Datasets
> > > > > <https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.
> > > > > html#a5._Dataset>*
> > > > > .
> > > > >
> > > > > Andras
> > > > >
> > > > > On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak <
> > > > serega.shey...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi, did anyone try to integrate oozie coordinator with kafka?
> > > > > > use case:
> > > > > >
> > > > > > System publishes message to kafka topic (sample message)
> > > > > > - cluster: hdfs://prod-cluster
> > > > > > - path: /my/input/data
> > > > > > - format: avro
> > > > > >
> > > > > > Oozie coordinator listens to kafka topic, consumes message and
> > starts
> > > > > > workflow.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Oozie coordinator and kafka

[Andras Piros]

Hi Serega,

not to my knowledge. Would be interested on your use case, though.

Would start w/ *Coordinator Input Events / Datasets
*
.

Andras

On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak 
wrote:

> Hi, did anyone try to integrate oozie coordinator with kafka?
> use case:
>
> System publishes message to kafka topic (sample message)
> - cluster: hdfs://prod-cluster
> - path: /my/input/data
> - format: avro
>
> Oozie coordinator listens to kafka topic, consumes message and starts
> workflow.
>

Re: Welcoming new Oozie PMC member Satish Saley

[Andras Piros]

Congrats Satish! To infinity and beyond!

Artem Ervits  (időpont: 2017. dec. 11., H, 21:06)
ezt írta:

> Well deserved, congratulations Satish!
>
> On Dec 11, 2017 2:49 PM, "Robert Kanter"  wrote:
>
> > Hello Oozie community,
> >
> > It's my pleasure to announce that Satish Saley has become the newest
> > addition to the Oozie PMC.
> >
> > Please join me congratulating him.
> > Congrats!
> >
> >
> > - Robert, on behalf of the Oozie PMC
> >
>

Re: how to set the doneflag of input event with batch number and other variable?

[Andras Piros]

Hi there,

yes,  should support user defined variable substitution, as you
can see in TestCoordSubmitXCommand#testSubmitWithDoneFlag()

.

Further usage of  and  please see the Oozie Coordinator
Functional Specification

.

Regards,

Andras

On Sun, Oct 1, 2017 at 6:41 AM, 罗 辉  wrote:

> Hi there:
>  Does the  of input event support batch number and other
> variable? How to set the doneflag of input event with batch number and
> other variable?
> For example, I have a multi-batch job, at about 5minutes interval, every
> batch has a trigger file which follows below naming rule:
> ${uppersystem}_${batchno}_ trigger
>
> Here is a part of my coordinator.xml
> 
>  initial-instance="2017-10-01T00:00Z" timezone="UTC">
> ${inputPath}/${tableName}/${YEAR}${MONTH}${
> DAY}
>  ${uppersystem}_${batchno}_
> trigger
> 
>  initial-instance="2017-10-01T01:00Z" timezone="UTC">
> ${outputPath}/${tableName}/${YEAR}${MONTH}${
> DAY}
> 
> 
> 
> 
> ${coord:current(0)}
> 
> 
> 
> 
> ${coord:current(0)}
> 
> 
>
>
>
> Any ideas is welcome
>
>
> Hui Luo
>
>
>

Re: Welcome new Oozie Committer - Attila Sasvari

[Andras Piros]

Yy! Great job, well done, Sasi!

Congrats!

Andras

On Thu, Jul 20, 2017 at 6:01 PM, Robert Kanter  wrote:

> Hi everyone,
>
> It is my pleasure to announce that the Oozie PMC has invited
> Attila Sasvari to become an Oozie committer and he has
> accepted our invitation.
>
> Please join me in congratulating him.
> Congrats!
>
>
> - Robert, on behalf of the Oozie PMC
>

Re: Minitest failing with java.lang.NoClassDefFoundError: Could not initialize class org.apache.oozie.util.DateUtils

[Andras Piros]

Hi Stanislavs,

as per current standing, JDK1.7.0 is discouraged. Some part of
maven-findbugs-plugin won't work anything other than JDK1.8.0.

Regards,

Andras

On Wed, Jul 19, 2017 at 1:51 PM, Staņislavs Rogozins <
stanislavs.rogoz...@gmail.com> wrote:

> Apache Maven 3.3.9
> Maven home: /usr/share/maven
> Java version: 1.7.0_95, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.4.0-62-generic", arch: "amd64", family:
> "unix"
>
> Distro is Ubuntu 16.04.2 LTS
>
> On Wed, Jul 19, 2017 at 2:32 PM, Andras Piros <andras.pi...@cloudera.com>
> wrote:
>
> > Hi Stanislavs,
> >
> > what is the OS / JDK version you're trying with?
> >
> > For me, mvn clean test succeeds from minitest folder:
> >
> > >
> > > ---
> > >  T E S T S
> > > ---
> > > Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> > > MaxPermSize=512m; support was removed in 8.0
> > > Concurrency config is parallel='classes', perCoreThreadCount=false,
> > > threadCount=1, useUnlimitedThreads=false
> > > objc[66917]: Class JavaLaunchHelper is implemented in both
> > > /Library/Java/JavaVirtualMachines/jdk1.8.0_
> > 102.jdk/Contents/Home/jre/bin/java
> > > (0x10855d4c0) and
> > > /Library/Java/JavaVirtualMachines/jdk1.8.0_
> > 102.jdk/Contents/Home/jre/lib/libinstrument.dylib
> > > (0x1208664e0). One of the two will be used. Which one is undefined.
> > > Running org.apache.oozie.test.TestParallelJPAOperationRetries
> > > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.583
> > sec
> > > Running org.apache.oozie.test.TestWorkflow
> > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.711
> > sec
> > > Running org.apache.oozie.test.TestWorkflowRetries
> > > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.895
> > sec
> > > Results :
> > > Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
> >
> > I have Oracle JDK 1.8.0_102 and Mac OS X current version.
> >
> > Regards,
> >
> > Andras
> >
> > On Wed, Jul 19, 2017 at 12:55 PM, Staņislavs Rogozins <
> > stanislavs.rogoz...@gmail.com> wrote:
> >
> > > I'm trying to follow the instructions here to setup testing environment
> > for
> > > Oozie: https://oozie.apache.org/docs/4.3.0/ENG_MiniOozie.html
> > >
> > > When I run `mvn clean test`, the test phase fails. I found the
> following
> > > error in target/surefire-reports/org.apache.oozie.test.
> WorkflowTest.txt:
> > >
> > > >
> > > > Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 24.68
> > sec
> > > > <<< FAILURE!
> > > > testWorkflowRun(org.apache.oozie.test.WorkflowTest)  Time elapsed:
> > 0.156
> > > > sec  <<< ERROR!
> > > > java.lang.ExceptionInInitializerError
> > > > at org.apache.oozie.service.Services.(Services.java:
> 114)
> > > > at
> > > > org.apache.oozie.test.XTestCase.cleanUpDBTables(XTestCase.java:804)
> > > > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:431)
> > > > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:280)
> > > > at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.
> > java:61)
> > > > at
> > > > org.apache.oozie.test.MiniOozieTestCase.setUp(
> > MiniOozieTestCase.java:61)
> > > > at org.apache.oozie.test.WorkflowTest.setUp(
> > > WorkflowTest.java:46)
> > > > at junit.framework.TestCase.runBare(TestCase.java:132)
> > > > at junit.framework.TestResult$1.protect(TestResult.java:110)
> > > > at junit.framework.TestResult.runProtected(TestResult.java:
> > 128)
> > > > at junit.framework.TestResult.run(TestResult.java:113)
> > > > at junit.framework.TestCase.run(TestCase.java:124)
> > > > at junit.framework.TestSuite.runTest(TestSuite.java:243)
> > > > at junit.framework.TestSuite.run(TestSuite.java:238)
> > > > at
> > > > org.junit.internal.runners.JUnit38ClassRunner.run(
> > > JUnit38ClassRunner.java:83)
> > > > at org.junit.runners.Suite.runChild(Suite.java:

Re: Minitest failing with java.lang.NoClassDefFoundError: Could not initialize class org.apache.oozie.util.DateUtils

[Andras Piros]

Hi Stanislavs,

what is the OS / JDK version you're trying with?

For me, mvn clean test succeeds from minitest folder:

>
> ---
>  T E S T S
> ---
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=512m; support was removed in 8.0
> Concurrency config is parallel='classes', perCoreThreadCount=false,
> threadCount=1, useUnlimitedThreads=false
> objc[66917]: Class JavaLaunchHelper is implemented in both
> /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/bin/java
> (0x10855d4c0) and
> /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/lib/libinstrument.dylib
> (0x1208664e0). One of the two will be used. Which one is undefined.
> Running org.apache.oozie.test.TestParallelJPAOperationRetries
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.583 sec
> Running org.apache.oozie.test.TestWorkflow
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.711 sec
> Running org.apache.oozie.test.TestWorkflowRetries
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.895 sec
> Results :
> Tests run: 7, Failures: 0, Errors: 0, Skipped: 0

I have Oracle JDK 1.8.0_102 and Mac OS X current version.

Regards,

Andras

On Wed, Jul 19, 2017 at 12:55 PM, Staņislavs Rogozins <
stanislavs.rogoz...@gmail.com> wrote:

> I'm trying to follow the instructions here to setup testing environment for
> Oozie: https://oozie.apache.org/docs/4.3.0/ENG_MiniOozie.html
>
> When I run `mvn clean test`, the test phase fails. I found the following
> error in target/surefire-reports/org.apache.oozie.test.WorkflowTest.txt:
>
> >
> > Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 24.68 sec
> > <<< FAILURE!
> > testWorkflowRun(org.apache.oozie.test.WorkflowTest)  Time elapsed: 0.156
> > sec  <<< ERROR!
> > java.lang.ExceptionInInitializerError
> > at org.apache.oozie.service.Services.(Services.java:114)
> > at
> > org.apache.oozie.test.XTestCase.cleanUpDBTables(XTestCase.java:804)
> > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:431)
> > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:280)
> > at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:61)
> > at
> > org.apache.oozie.test.MiniOozieTestCase.setUp(MiniOozieTestCase.java:61)
> > at org.apache.oozie.test.WorkflowTest.setUp(
> WorkflowTest.java:46)
> > at junit.framework.TestCase.runBare(TestCase.java:132)
> > at junit.framework.TestResult$1.protect(TestResult.java:110)
> > at junit.framework.TestResult.runProtected(TestResult.java:128)
> > at junit.framework.TestResult.run(TestResult.java:113)
> > at junit.framework.TestCase.run(TestCase.java:124)
> > at junit.framework.TestSuite.runTest(TestSuite.java:243)
> > at junit.framework.TestSuite.run(TestSuite.java:238)
> > at
> > org.junit.internal.runners.JUnit38ClassRunner.run(
> JUnit38ClassRunner.java:83)
> > at org.junit.runners.Suite.runChild(Suite.java:128)
> > at org.junit.runners.Suite.runChild(Suite.java:24)
> > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.IllegalArgumentException: Invalid TimeZone: UTC
> > at org.apache.oozie.util.DateUtils.getTimeZone(
> DateUtils.java:146)
> > at org.apache.oozie.util.DateUtils.(DateUtils.java:44)
> > ... 23 more
> > testWorkflowRunFromFile(org.apache.oozie.test.WorkflowTest)  Time
> elapsed:
> > 0.001 sec  <<< ERROR!
> > java.lang.NoClassDefFoundError: Could not initialize class
> > org.apache.oozie.util.DateUtils
> > at org.apache.oozie.service.Services.(Services.java:114)
> > at
> > org.apache.oozie.test.XTestCase.cleanUpDBTables(XTestCase.java:804)
> > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:431)
> > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:280)
> > at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:61)
> > at
> > org.apache.oozie.test.MiniOozieTestCase.setUp(MiniOozieTestCase.java:61)
> > at org.apache.oozie.test.WorkflowTest.setUp(
> WorkflowTest.java:46)
> > at junit.framework.TestCase.runBare(TestCase.java:132)
> > at junit.framework.TestResult$1.protect(TestResult.java:110)
> > at junit.framework.TestResult.runProtected(TestResult.java:128)
> > at 

Re: OYA! - no more LauncherMappers

[Andras Piros]

Hi Dongying,

while implementing Oozie On YARN, we tried to avoid the most backward
incompatibility issues possible. However, there are some of these still
remaining as part as *OOZIE-2889
*. It is mostly targeted
for Oozie 5.0.0.

Can you please file a JIRA under this umbrella issue w/ the exact steps to
reproduce?

Thanks,

Andras

On Mon, Jun 12, 2017 at 8:44 AM, Dongying Jiao 
wrote:

> Hi:
> After the merge of OOZIE-1770, it seems the previous workflow can't run
> successfully. I tried map-reduce apps in oozie examples, it failed. So does
> it mean we need to modify existing workflows in next oozie release?
>
> Thanks
>
>
>
> 2017-05-31 3:00 GMT+08:00 Robert Kanter :
>
> > Thanks for getting this in.  I think this is going to greatly improve
> > things for Oozie going forward, and allow us to address common issues we
> > couldn't before.
> >
> > On Fri, May 26, 2017 at 4:23 AM, Peter Bacsko 
> > wrote:
> >
> > > Finally, it's a milestone :)
> > >
> > > By the way, we should summarize what changed and how it affects current
> > > Oozie workflows (perhaps adding something to the Oozie website makes
> > sense
> > > too). Right now OYA lacks good backward compatibility, which means that
> > MR
> > > properties don't have any effect. This will change though.
> > >
> > > Another thing that we have to agree on is patching: lots of stuff
> changed
> > > inside JavaActionExecutor, LauncherMapper no longer exists, etc.
> > Therefore
> > > if someone discovers a problem in these classes (or just wants to add
> an
> > > improvement), it will not be possible to patch them. I suggest
> branching
> > > off from the commit before OYA and then we can start maintaining two
> > > branches. But this is just an idea.
> > >
> > > Peter
> > >
> > > On Fri, May 26, 2017 at 12:40 PM, Peter Cseh 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > OOZIE-1770 - Create Oozie Application Master for YARN is committed to
> > > > master!
> > > > I would like to thank the effort of everybody who was involved in the
> > > > design, the development or provided feedback in Jira or on
> ReviewBoard.
> > > >
> > > > This is a big change in how Oozie works and there is still a lot to
> do:
> > > you
> > > > can check out OOZIE-2889 for details.
> > > >
> > > > Thanks again everyone!
> > > > gp
> > > >
> > > >
> > > > --
> > > > Peter Cseh
> > > > Software Engineer
> > > > 
> > > >
> > >
> >
>

Re: Oozie 4.3.0 Launcher Uber Mode + Preemption

[Andras Piros]

Hi Rohit,

OOZIE-2123  is that when
those parameters mapreduce.job.classloader or mapreduce.user.classpath.first
are set by the user, they take precedence over oozie.action.launcher.
mapreduce.job.ubertask.enable one - it will be silently set to false. The
reasoning behind is that those parameters should have no effects when the
Oozie launcher map-only MapTask runs in the same JVM of the YARN
ApplicationMaster.

Actually Oozie On YARN  is
already merged to master, meaning this uber job enable switch won't have an
effect for the upcoming Oozie 5.0.0.

As for how to influence YARN AM preemption, please see the
CapacityScheduler documentation
.
on how to use yarn.scheduler.capacity.maximum-am-resource-percent.

Regards,

Andras

On Mon, Jun 5, 2017 at 9:28 PM, Menon, Rohit  wrote:

> Hello,
>
> I had a query regarding https://issues.apache.org/jira/browse/OOZIE-2123.
>
> We want to enable uber mode for oozie launchers as a prerequisite for
> preemption. However we do use mapreduce.user.classpath.first configuration
> in our environment. What is the reasoning/concern behind disabling uber
> mode when the above configuration is set?
>
> Any other suggestions for using preemption with oozie?
>
> We are using oozie 4.3.0 + Hadoop 2.7.3
>
> Thanks,
> Rohit
>

Re: is there any doc or jira to show why we need 'timezone' in coordinator.xml

[Andras Piros]

Hi Wang,

here u go w/ reading the *the appropriate chapter
*
of Coordinator Functional Specification.

Regards,

Andras

On Sat, Apr 8, 2017 at 10:58 AM, WangYQ  wrote:

> is there any doc or jira to show why we need  'timezone'  in
> coordinator.xml
>
>

Re: Amazon EMR

[Andras Piros]

Hi Hitesh,

yes, you can provide your shell script to Oozie's Shell action within
the  element.

As for installation and configuration, I strongly suggest you read *the
appropriate documentation section
<https://oozie.apache.org/docs/4.3.0/AG_Install.html>*. Maybe
beginning w/ *Quick
start <https://oozie.apache.org/docs/4.3.0/DG_QuickStart.html>*.

Regards,

Andras

On Thu, Apr 6, 2017 at 2:44 PM, Hitesh Goyal <hitesh.go...@nlpcaptcha.com>
wrote:

> Thanks for your response Andras
>
> So can I put my shell script file (which contains command to fire up the
> cluster) in the  tag at the very beginning.
> Where should I install oozie ? Can I install it on my web server ?
>
> -----Original Message-
> From: Andras Piros [mailto:andras.pi...@cloudera.com]
> Sent: Thursday, April 6, 2017 6:01 PM
> To: user@oozie.apache.org
> Subject: Re: Amazon EMR
>
> Hi Hitesh,
>
> thanks for your interest towards Oozie!
>
> Please take a look at the *Shell Action documentation <
> https://oozie.apache.org/docs/4.3.0/DG_ShellActionExtension.html>*.
>
> Regards,
>
> Andras
>
> On Thu, Apr 6, 2017 at 2:19 PM, Hitesh Goyal <hitesh.go...@nlpcaptcha.com>
> wrote:
>
> > Hi team,
> >
> > I am new to oozie Workflow engine.
> > I want to use it is as a service which starts EMR clusters and then
> > run jobs on it and after completing the jobs it would terminate the
> clusters.
> > Can I use oozie for this procedure.
> > Please let me know a solution for this.
> >
> > Thanks,
> > Hitesh
> >
>

Re: Amazon EMR

[Andras Piros]

Hi Hitesh,

thanks for your interest towards Oozie!

Please take a look at the *Shell Action documentation
*.

Regards,

Andras

On Thu, Apr 6, 2017 at 2:19 PM, Hitesh Goyal 
wrote:

> Hi team,
>
> I am new to oozie Workflow engine.
> I want to use it is as a service which starts EMR clusters and then run
> jobs on it and after completing the jobs it would terminate the clusters.
> Can I use oozie for this procedure.
> Please let me know a solution for this.
>
> Thanks,
> Hitesh
>

Re: Immutable vs mutable parameters in Oozie 4.2.0

[Andras Piros]

Hi Toby,

thanks for your interest on using Oozie!

In order I can help you further, can you please:

   - add *Apache GitHub * git hashes of
   Oozie versions for each HDP version
   - add a small example of two workflows, one sets a parameter, the other
   reads the changed parameter - that was working in HDP versions 2.3 and 2.4,
   but not anymore in 2.5?

Regards,

Andras

On Tue, Apr 4, 2017 at 4:38 PM, Toby Evans  wrote:

> Hi there,
>
> We've got a big Oozie pipeline, which I'm having a little difficulty
> upgrading from Hortonworks 2.3 > 2.4 > 2.5
>
> HDP Oozie
> 2.3 4.2.0.2.3.2.0-2950
> 2.4 4.2.0.2.4.2.0-258
> 2.5 4.2.0.2.5.0.0-1245
>
>
> So, we ingest and process data from a range of sources, eg databases, FTP,
> SQL dbs etc.   These are all different, but share so many common features
> that we have grown various heavily templated workflows. Each of our
> workflows has a set of parameters (eg date range, source type,
> hasCustomStep) all that sort of thing, all stored in "profile" workflows.
> We first call our "profile" workflow, which sets up the various parameters
> and then calls the template workflow, which then fills in the values
> supplied from the profile.
>
> This has run fine for a couple of years in HDP 2.3, seems OK in 2.4, but
> definitely doesn't in 2.5. The problem is that once a parameter has been
> set in a workflow chain, then it stays set. I understand that you don't
> want to be changing a parameter inside the same workflow, and we've
> refactored so that we don't have a parameter value passed to a workflow,
> which then changes the parameter value then calls another sub-workflow.
> However, we're getting caught out by parameters being set by a workflow and
> then not being reset much further down the pipeline when used by a
> different "profile", which should be isolated but clearly isn't
>
> Has there been a code change between the versions, and is there a config
> setting that allows us to switch off cross-workflow parameter immutability?
>
> thanks
>
> Toby
>

Re: Welcome new Oozie Committers - Peter Bacsko and Peter Cseh

[Andras Piros]

Congratulations Peter (Geza) and Peter Bacsko!

On Mon, Mar 20, 2017 at 11:37 PM, Robert Kanter  wrote:

> Hi everyone,
>
> It is my pleasure to announce that the Oozie PMC has invited
> Peter Bacsko and Peter (Geza) Cseh to become Oozie committers
> and they have both accepted our invitation.
>
> Please join me congratulating them.
> Congrats!
>
>
> - Robert, on behalf of the Oozie PMC
>

Re: Web UI Error

[Andras Piros]

OK, Suresh, thanks for your input.

When deploying not to Amazon EMR but to a local Tomcat 6.0.45, do you
encounter the same problem? Actually it seems to me that some Jetty
classpaths might be *lurking in
<http://stackoverflow.com/questions/17602988/java-lang-nosuchmethoderror-org-eclipse-jdt-internal-compiler-compilationresult>*
.

Andras

On Mon, Mar 13, 2017 at 10:26 PM, Suresh V <verdi...@gmail.com> wrote:

> Hello Andras,
>
> I can reproduce it by just typing the http://:11000 on the
> browser. This is an AWS EMR 5.0.3 with Oozie version 4.2.0.
> We created 3 EMR clusters recently of this version, and all three have this
> problem. I'm about to take this to AWS but wanted to check here if anyone
> has encountered it.
>
> The Tomcat page on port 8080 shows up very fine.
>
> Thank you for assisting with this,
> Suresh.
>
>
> On Mon, Mar 13, 2017 at 1:46 AM, Andras Piros <andras.pi...@cloudera.com>
> wrote:
>
> > Hi Suresh,
> >
> > can you please provide following details:
> >
> >- what are the exact and minimal steps to reproduce, aka what does
> that
> >exactly mean "when accessing the Oozie web client"?
> >- what is the Oozie version you're trying? Or did you build that from
> >master?
> >- when trying w/ Tomcat 6.0.45 locally, do you encounter any problems?
> >
> > Thanks,
> >
> > Andras
> >
> > On Sat, Mar 11, 2017 at 1:58 PM, Suresh V <verdi...@gmail.com> wrote:
> >
> > > Hello all,
> > >
> > > We are seeing below error in the browser when accessing the Oozie web
> > > client. This is out of the box from an AWS EMR cluster.
> > > Any pointers?
> > >
> > > HTTP Status 500 - java.lang.NoSuchMethodError:
> > > org.eclipse.jdt.internal.compiler.CompilationResult.
> > > getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;
> > > --
> > >
> > > *type* Exception report
> > >
> > > *message* *java.lang.NoSuchMethodError:
> > > org.eclipse.jdt.internal.compiler.CompilationResult.
> > > getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;*
> > >
> > > *description* *The server encountered an internal error that prevented
> it
> > > from fulfilling this request.*
> > >
> > > *exception*
> > >
> > > javax.servlet.ServletException: java.lang.NoSuchMethodError:
> > > org.eclipse.jdt.internal.compiler.CompilationResult.
> > > getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;
> > > org.apache.jasper.servlet.JspServlet.service(JspServlet.
> > java:273)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > > org.apache.oozie.servlet.AuthFilter$2.doFilter(
> > > AuthFilter.java:171)
> > > org.apache.hadoop.security.authentication.server.
> > > AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> > > org.apache.hadoop.security.authentication.server.
> > > AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> > > org.apache.oozie.servlet.AuthFilter.doFilter(
> > AuthFilter.java:176)
> > > org.apache.oozie.servlet.HostnameFilter.doFilter(
> > > HostnameFilter.java:86)
> > >
> > > *root cause*
> > >
> > > java.lang.NoSuchMethodError:
> > > org.eclipse.jdt.internal.compiler.CompilationResult.
> > > getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;
> > > org.apache.jasper.compiler.JDTCompiler$2.acceptResult(
> > > JDTCompiler.java:354)
> > > org.eclipse.jdt.internal.compiler.Compiler.compile(
> > > Compiler.java:480)
> > > org.apache.jasper.compiler.JDTCompiler.generateClass(
> > > JDTCompiler.java:425)
> > > org.apache.jasper.compiler.Compiler.compile(Compiler.java:298)
> > > org.apache.jasper.compiler.Compiler.compile(Compiler.java:277)
> > > org.apache.jasper.compiler.Compiler.compile(Compiler.java:265)
> > > org.apache.jasper.JspCompilationContext.compile(
> > > JspCompilationContext.java:564)
> > > org.apache.jasper.servlet.JspServletWrapper.service(
> > > JspServletWrapper.java:299)
> > > org.apache.jasper.servlet.JspServlet.serviceJspFile(
> > > JspServlet.java:315)
> > > org.apache.jasper.servlet.JspServlet.service(JspServlet.
> > java:265)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > > org.apache.oozie.servlet.AuthFilter$2.doFilter(
> > > AuthFilter.java:171)
> > > org.apache.hadoop.security.authentication.server.
> > > AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> > > org.apache.hadoop.security.authentication.server.
> > > AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> > > org.apache.oozie.servlet.AuthFilter.doFilter(
> > AuthFilter.java:176)
> > > org.apache.oozie.servlet.HostnameFilter.doFilter(
> > > HostnameFilter.java:86)
> > >
> > > *note* *The full stack trace of the root cause is available in the
> Apache
> > > Tomcat/6.0.45 logs.*
> > > --
> > > Apache Tomcat/6.0.45
> > >
> > >
> > > Thank you
> > >
> > > Suresh.
> > >
> >
>

Re: Web UI Error

[Andras Piros]

Hi Suresh,

can you please provide following details:

   - what are the exact and minimal steps to reproduce, aka what does that
   exactly mean "when accessing the Oozie web client"?
   - what is the Oozie version you're trying? Or did you build that from
   master?
   - when trying w/ Tomcat 6.0.45 locally, do you encounter any problems?

Thanks,

Andras

On Sat, Mar 11, 2017 at 1:58 PM, Suresh V  wrote:

> Hello all,
>
> We are seeing below error in the browser when accessing the Oozie web
> client. This is out of the box from an AWS EMR cluster.
> Any pointers?
>
> HTTP Status 500 - java.lang.NoSuchMethodError:
> org.eclipse.jdt.internal.compiler.CompilationResult.
> getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;
> --
>
> *type* Exception report
>
> *message* *java.lang.NoSuchMethodError:
> org.eclipse.jdt.internal.compiler.CompilationResult.
> getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;*
>
> *description* *The server encountered an internal error that prevented it
> from fulfilling this request.*
>
> *exception*
>
> javax.servlet.ServletException: java.lang.NoSuchMethodError:
> org.eclipse.jdt.internal.compiler.CompilationResult.
> getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:273)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> org.apache.oozie.servlet.AuthFilter$2.doFilter(
> AuthFilter.java:171)
> org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> org.apache.oozie.servlet.HostnameFilter.doFilter(
> HostnameFilter.java:86)
>
> *root cause*
>
> java.lang.NoSuchMethodError:
> org.eclipse.jdt.internal.compiler.CompilationResult.
> getProblems()[Lorg/eclipse/jdt/core/compiler/IProblem;
> org.apache.jasper.compiler.JDTCompiler$2.acceptResult(
> JDTCompiler.java:354)
> org.eclipse.jdt.internal.compiler.Compiler.compile(
> Compiler.java:480)
> org.apache.jasper.compiler.JDTCompiler.generateClass(
> JDTCompiler.java:425)
> org.apache.jasper.compiler.Compiler.compile(Compiler.java:298)
> org.apache.jasper.compiler.Compiler.compile(Compiler.java:277)
> org.apache.jasper.compiler.Compiler.compile(Compiler.java:265)
> org.apache.jasper.JspCompilationContext.compile(
> JspCompilationContext.java:564)
> org.apache.jasper.servlet.JspServletWrapper.service(
> JspServletWrapper.java:299)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(
> JspServlet.java:315)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> org.apache.oozie.servlet.AuthFilter$2.doFilter(
> AuthFilter.java:171)
> org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:594)
> org.apache.hadoop.security.authentication.server.
> AuthenticationFilter.doFilter(AuthenticationFilter.java:553)
> org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:176)
> org.apache.oozie.servlet.HostnameFilter.doFilter(
> HostnameFilter.java:86)
>
> *note* *The full stack trace of the root cause is available in the Apache
> Tomcat/6.0.45 logs.*
> --
> Apache Tomcat/6.0.45
>
>
> Thank you
>
> Suresh.
>

Re: is there any simple demo to show how to submit a job to a oozie server with kerberos on

[Andras Piros]

Hi Wang,

for details on how to configure Oozie user and Hadoop authentication,
please see *this
*
.

For Oozie SSL configuration, please take a look at *this
*
.

For details on how to modify your Oozie actions to run in a secure cluster,
please see *that
*.

Andras

On Thu, Mar 9, 2017 at 5:26 AM, WangYQ  wrote:

> i conf the oozie-site.xml to open kerberos on oozie server
> submit job to oozie server, can not run successfully
> maybe my client submit job code is not correct
>
>
>
>
> is there any simple demo to show how to submit a job to a oozie server
> with kerberos on
>
>

Re: oozie ha

[Andras Piros]

Hi WangYQ,

yes, you're basically right - best is you set up a load balancer for each
of the Oozie instances and add its address to all the configuration entries
where a single Oozie server address used to be.

E.g. http://load-balancer-host:11000/oozie or
https://load-balancer-host:11443/oozie.

Regards,

Andras

--
Andras PIROS
Software Engineer
<http://www.cloudera.com/>

On Tue, Dec 20, 2016 at 1:10 PM, WangYQ <wangyongqiang0...@163.com> wrote:

> in
> http://blog.cloudera.com/blog/2014/03/inside-apache-oozie-ha/
>
>
> Architecture: Access
>
> Usually, when you use the Oozie client, REST API, or Web UI, there’s a
> single address to use (http://myhost:11000/oozie, for example). But now
> that you have multiple Oozie servers, you have multiple addresses to which
> users can connect — so what happens if the one they pick goes down?  There
> are also many clients or tools that only support a single entry point for
> Oozie, such as the JobTracker. To fix this issue, you need to provide a
> single address that will round-robin between the Oozie servers.
>
> You can use a load balancer, a virtual IP address, or DNS round-robin for
> this purpose. As with the database, this setup technically needs to be HA
> as well.
>
>
>
>
>
>
>
> we need make a single address to oozie by ourselves?
>
> such as  load balancer, a virtual IP address, or DNS round-robin
>
> does oozie provide a simple method?
>
>
>
>
>
>

Re: Exception when setup oozie HA using virtual IP

[Andras Piros]

Hi Dongying,

this seems like a bug in ZKJobsConcurrencyService - in case numOozies is
zero isJobIdForThisServer() should emit a WARN log stating that the other
Oozie instance might be missing and return true rather than throwing a
RuntimeException.

Can you please file a bug under Apache JIRA.

Thanks, and regards,

Andras

--
Andras PIROS
Software Engineer
<http://www.cloudera.com/>

On Tue, Dec 6, 2016 at 4:33 AM, Dongying Jiao <pineapple...@gmail.com>
wrote:

> Hi:
> Do you have the detail steps on setting up oozie HA using virtual IP?
> I setup oozie HA using virtual IP, server-1 and server-2(active-active),
> when we take down server-1 any oozie job submitted fails with below
> stacktrace. If both are up , there is no issue.
> ERROR RecoveryService$RecoveryRunnable:517 - SERVER[] USER[-] GROUP[-]
> TOKEN[-] APP[-] JOB[-] ACTION[-] Exception, / by zero
> java.lang.ArithmeticException: / by zero
> at
> org.apache.oozie.service.ZKJobsConcurrencyService.checkJobIdForServer(
> ZKJobsConcurrencyService.java:167)
> at
> org.apache.oozie.service.ZKJobsConcurrencyService.isJobIdForThisServer(
> ZKJobsConcurrencyService.java:129)
> at
> org.apache.oozie.service.RecoveryService$RecoveryRunnable.
> runWFRecovery(RecoveryService.java:362)
> at
> org.apache.oozie.service.RecoveryService$RecoveryRunnable.run(
> RecoveryService.java:146)
> at
> org.apache.oozie.service.SchedulerService$2.run(SchedulerService.java:175)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> It seems server-2 can't get oozie server list from zookeeper. Zookeeper
> connection string is already added to oozie site.
>
> Thanks
>
> Best Regards,
> Dongying Jiao
>

Re: Change NN and JobTracker dynamically during runtime

[Andras Piros]

Hi Dipesh,

during workflow / job submission you can define variables inside
job.properties coming e.g. from env vars that are used in workflow.xml. So
much for the flexibility.

Can you tell me a use case where runtime routing to different JT / NN
instances via Oozie (and not e.g. coming from a load balancer setting
configured runtime) is better?

Thanks,

Andras

--
Andras PIROS
Software Engineer
<http://www.cloudera.com/>

On Mon, Dec 5, 2016 at 7:45 PM, mdk-swandha <dipeshsoftw...@gmail.com>
wrote:

> Hi Alex,
>
> The idea is to call this external service which will find the best cluster
> and inform the caller. So today this caller is Oozie, tomorrow it will be
> Zeppelin or any other application.
>
> How can I provide multiple JT and NN addresses in job.properties? You mean
> during job/workflow creation? I will still need to overwrite job.properties
> or provide these values somewhere dynamically?
>
> Thanks.
> -Dipesh
>
> On Mon, Dec 5, 2016 at 5:24 AM, Andras Piros <andras.pi...@cloudera.com>
> wrote:
>
> > Hi Dipesh,
> >
> > seems like a bad idea to programmatically change job-tracker or
> > name-node properties
> > - it's just not the task of Oozie to determine what are the exact JT or
> NN
> > instances Oozie should use.
> >
> > Instead, I'd rather setup a load balancer for JT and another one for NN,
> > and provide those addresses to Oozie's job.properties. That way, we
> > separate concerns - the load balancer can choose the JT or NN node
> runtime,
> > e.g. on a round robin basis.
> >
> > Regards,
> >
> > Andras
> >
> > --
> > Andras PIROS
> > Software Engineer
> > <http://www.cloudera.com/>
> >
> > On Thu, Dec 1, 2016 at 9:29 PM, mdk-swandha <dipeshsoftw...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I have a use case like this - in a multi cluster (hadoop cluster)
> > > environment if I would like to send a job/oozie workflow to a desired
> > > cluster during runtime, how can this be done.
> > >
> > > I see that there is JavaActionExecutor class which read NN and
> JobTracker
> > > in createBaseHadoopConf method
> > >
> > > All HadoopActionExectors are derived from JavaActionExecutor so this
> > seems
> > > to be a place wherein I can insert my code. How can I do this without
> > > disrupting the original flow by adding my hook.
> > >
> > > One option is to to derive my new JavaActionExecutor and over ride
> > > createBaseHadoopConf method and then derive all ActionExecutors from my
> > new
> > > JavaActionExecutor. It doesn't seem to be elegant to me, so thought to
> > ask
> > > out here.
> > >
> > > Any input will be useful.
> > >
> > > Thanks.
> > > -Dipesh
> > >
> >
>

Re: MiniOozie for local dryrun or other options for doing dryrun of oozie workflows?

[Andras Piros]

Hi Serega,

as per *Oozie documentation
<http://oozie.apache.org/docs/4.1.0/DG_CommandLineTool.html#Dryrun_of_Workflow_Job>*
we
can see that with -dryrun option does not create nor run a job.

So for the killer feature request, I think it's not possible ATM.

Regards,

Andras

--
Andras PIROS
Software Engineer
<http://www.cloudera.com/>

On Thu, Dec 1, 2016 at 8:33 PM, Serega Sheypak <serega.shey...@gmail.com>
wrote:

> Hi, did anyone make it work property in his project?
> I need to do dry run for my workflows.
> The usecase is:
> User writes workflow and wants to:
> 1. Check if it valid
> 2. do dryrun, see how it flows without executing steps.
>
> Let say I have wflow with three steps:
>
> 1. disctp data from $A to $B
> 2. run spark action with $B as input
> 3. disctp $B to $C
>
> I want to do dryrun and check how my variables were interpolated it wflow.
> The killer feature is: I want to imitate spark action failure and check how
> my kill node looks like.
>

Re: Spark oozie job failed on launching

[Andras Piros]

Hi Saurabh,

apparently there is some classpath issue along w/ the runtimes used / the
common jar.

In order to track down class loading issues like this one the best you can
do is to give extra parameters like -verbose:class (*reference
<http://www.oracle.com/technetwork/java/javase/clopts-139448.html#gbmtm>*),
or just list the contents of each and every jar file loaded by the jvm
trying to run MRAppMaster.main().

Regards,

Andras

--
Andras PIROS
Software Engineer
<http://www.cloudera.com/>

On Fri, Nov 4, 2016 at 7:10 AM, Saurabh Malviya (samalviy) <
samal...@cisco.com> wrote:

> Hi,
>
> Spark oozie jobs keep failing on deployment, although it works fine for
> the first team. We are using common jar for all jobs
> Oozie.libpath= jar
>
> Log Type: syslog
> Log Upload Time: Fri Nov 04 05:24:17 + 2016
> Log Length: 432
> 2016-11-04 05:24:15,845 ERROR [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
> Error starting MRAppMaster
> java.lang.NoSuchMethodError: org.apache.hadoop.yarn.api.
> records.ContainerId.fromString(Ljava/lang/String;)
> Lorg/apache/hadoop/yarn/api/records/ContainerId;
> at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(
> ConverterUtils.java:179)
> at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(
> MRAppMaster.java:1343)
>  ===
>