Re: Oozie 5.1.0 release and plans for the year

[Artem Ervits]

+1 on the plans, I won't get to the jiras I'm assigned to until next week.
Don't want to be a blocker.

On Tue, Jun 19, 2018, 4:48 AM Andras Piros  wrote:

> Good idea Gp!
>
> Thanks for volunteering as the release manager for 5.1.0. I can see *a few
> blockers
> <
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20OOZIE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%205.1.0%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20%20%20priority%20DESC%2C%20updated%20DESC
> >*
> for 5.1.0 now, of which *OOZIE-3178
> * seems to be a real
> one.
>
> Regards,
>
> Andras
>
> On Tue, Jun 19, 2018 at 12:50 PM Gézapeti Cseh 
> wrote:
>
> > Hi everyone,
> >
> > Now, that Oozie-2339 is in (with other fixes as well), I think it would
> be
> > nice to release it as part of Oozie 5.1.0 so more people will try it out
> > before we jump in to do the coordinator/bunlde part as well.
> >
> > Also, we're planning to pick up on new action types with Andras and Peter
> > and probably will do a bunch of new releases as they are ready later this
> > year. The git action is the closest one, but there are others in the
> > pipeline like Maven and callback.
> >
> > Is anyone aware of issues in which we should wait for to start working on
> > 5.1.0?
> > If nobody else does, I can volunteer to be the release manager for it and
> > start the branching process in a week or so.
> >
> > thanks
> > gp
> >
>

Re: Oozie for spark jobs without Hadoop

[Artem Ervits]

Here's some related info

https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-trouble/index.html

https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md


On Wed, May 16, 2018, 3:45 PM purna pradeep  wrote:

> Peter,
>
> I got rid of this error by adding
> hadoop-aws-2.8.3.jar and jets3t-0.9.4.jar
>
> But I’m getting below error now
>
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key
> must be specified by setting the fs.s3.awsAccessKeyId and
> fs.s3.awsSecretAccessKey properties (respectively)
>
> I have tried adding AWS access ,secret keys in
>
> oozie-site.xml and hadoop core-site.xml , and hadoop-config.xml
>
>
>
>
> On Wed, May 16, 2018 at 2:30 PM purna pradeep 
> wrote:
>
> >
> > I have tried this ,just added s3 instead of *
> >
> > 
> >
> >
>  oozie.service.HadoopAccessorService.supported.filesystems
> >
> > hdfs,hftp,webhdfs,s3
> >
> > 
> >
> >
> > Getting below error
> >
> > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> > org.apache.hadoop.fs.s3a.S3AFileSystem not found
> >
> > at
> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)
> >
> > at
> > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2793)
> >
> > at
> > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> >
> > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> >
> > at
> > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> >
> > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> >
> > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> >
> > at
> >
> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:625)
> >
> > at
> >
> org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623
> >
> >
> > On Wed, May 16, 2018 at 2:19 PM purna pradeep 
> > wrote:
> >
> >> This is what is in the logs
> >>
> >> 2018-05-16 14:06:13,500  INFO URIHandlerService:520 - SERVER[localhost]
> >> Loaded urihandlers [org.apache.oozie.dependency.FSURIHandler]
> >>
> >> 2018-05-16 14:06:13,501  INFO URIHandlerService:520 - SERVER[localhost]
> >> Loaded default urihandler org.apache.oozie.dependency.FSURIHandler
> >>
> >>
> >> On Wed, May 16, 2018 at 12:27 PM Peter Cseh 
> >> wrote:
> >>
> >>> That's strange, this exception should not happen in that case.
> >>> Can you check the server logs for messages like this?
> >>> LOG.info("Loaded urihandlers {0}", Arrays.toString(classes));
> >>> LOG.info("Loaded default urihandler {0}",
> >>> defaultHandler.getClass().getName());
> >>> Thanks
> >>>
> >>> On Wed, May 16, 2018 at 5:47 PM, purna pradeep <
> purna2prad...@gmail.com>
> >>> wrote:
> >>>
>  This is what I already have in my oozie-site.xml
> 
>  
> 
> 
>  oozie.service.HadoopAccessorService.supported.filesystems
> 
>  *
> 
>  
> 
>  On Wed, May 16, 2018 at 11:37 AM Peter Cseh 
>  wrote:
> 
> > You'll have to configure
> > oozie.service.HadoopAccessorService.supported.filesystems
> > hdfs,hftp,webhdfs Enlist
> > the different filesystems supported for federation. If wildcard "*"
> is
> > specified, then ALL file schemes will be allowed.properly.
> >
> > For testing purposes it's ok to put * in there in oozie-site.xml
> >
> > On Wed, May 16, 2018 at 5:29 PM, purna pradeep <
> > purna2prad...@gmail.com>
> > wrote:
> >
> > > Peter,
> > >
> > > I have tried to specify dataset with uri starting with s3://,
> s3a://
> > and
> > > s3n:// and I am getting exception
> > >
> > >
> > >
> > > Exception occurred:E0904: Scheme [s3] not supported in uri
> > > [s3://mybucket/input.data] Making the job failed
> > >
> > > org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3]
> > not
> > > supported in uri [s3:// mybucket /input.data]
> > >
> > > at
> > > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > > URIHandlerService.java:185)
> > >
> > > at
> > > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > > URIHandlerService.java:168)
> > >
> > > at
> > > org.apache.oozie.service.URIHandlerService.getURIHandler(
> > > URIHandlerService.java:160)
> > >
> > > at
> > > org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(
> > > CoordCommandUtils.java:465)
> > >
> > > at
> > > org.apache.oozie.command.coord.CoordCommandUtils.
> > > separateResolvedAndUnresolved(CoordCommandUtils.java:404)
> > >
> > > at
> > > org.apache.oozie.command.coord.CoordCommandUtils.

Re: Spark 2.3 in oozie

[Artem Ervits]

Did you run
mvn clean install first on the parent directory?

On Tue, May 15, 2018, 11:35 AM purna pradeep 
wrote:

> Thanks peter,
>
> I have tried changing Dspark.version to 2.3.0 and compiled oozie I’m
> getting below error from oozie examples
>
>
> *ERROR] Failed to execute goal on project oozie-examples: Could not resolve
> dependencies for project org.apache.oozie:oozie-examples:jar:5.0.0: Could
> not find artifact org.apache.spark:spark-core_2.10:jar:2.3.0 in resolution
> *
>
> On Tue, May 15, 2018 at 11:14 AM Peter Cseh  wrote:
>
> > Oozie has a spark-2 profile that is currently hard-coded to Spark 2.1:
> > https://github.com/apache/oozie/blob/master/pom.xml#L1983
> > I'm sure if you overwrite the -Dspark.version and compile Oozie that way
> it
> > will work.
> > gp
> >
> >
> > On Tue, May 15, 2018 at 5:07 PM, purna pradeep 
> > wrote:
> >
> > > Hello,
> > >
> > > Does oozie supports spark 2.3? Or will it even care of the spark
> version
> > >
> > > I want to use spark action
> > >
> > >
> > >
> > > Thanks,
> > > Purna
> > >
> >
> >
> >
> > --
> > *Peter Cseh *| Software Engineer
> > cloudera.com 
> >
> > [image: Cloudera] 
> >
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> Cloudera
> > on LinkedIn] 
> > --
> >
>

Re: oozie job action fails with JA017

[Artem Ervits]

What version of Spark is it? Why not take advantage of the Spark action
directly? I highly recommend asking vendor specific questions in the
associated forums.

On Feb 16, 2018 4:54 AM, "Malathi Thangarajan" 
wrote:

> Hi,
>
> I have a oozie shell action that executes a spark program. The action works
> fine for small data. However when the dataset becomes large the spark
> program runs longer than 2 hours. But in the oozie web console, I see that
> the job failed with code JA017 after an hour.
>
> This is the error message from oozie web console:
>
>
> *JA017: Could not lookup launched hadoop Job ID [job_1515381588900_7387]
> which was associated with  action
> [751-180104034149429-oozie-oozi-W@taste-vector-affinity].  Failing
> this
> action!*
> But actually the job runs in YARN and completes successfully after 2 hours.
> So why does oozie fails the shell action? Could there be any configuration
> that has gone wrong?
>
> *version deatils:*
> Ambari - Version 2.4.0.4
> oozie - 4.2.0.2.5
> HDP-2.5.6.2
>
>
> [image: Crayon Data]
>
> [image: Twitter]  [image: Facebook]
>  [image: LinkedIn]
>  [image: Youtube]
>  [image: Youtube]
> 
>
>
> Malathi T / Software Developer
> +91 9677211473
>
> Crayon Data
> www.crayondata.com
> Top 5 in IBM's Watson Mobile Developer Challenge 2014
>  challenge.html/?utm_source=email>
> Top 3 at Orange Fab 2015 ,
> Tokyo
> Finalist at Code_n 2014 @ CEBIT
> , Hanover
>

Re: Welcoming new Oozie PMC members: Attila Sasvari, Peter Cseh, and Peter Bacsko

[Artem Ervits]

Well deserved, congratulations guys!!

On Jan 31, 2018 6:27 PM, "Robert Kanter"  wrote:

> Hello Oozie community,
>
> It's my pleasure to announce that Attila Sasvari, Peter Cseh, and Peter
> Bascko
> have become the newest additions to the Oozie PMC.
>
> Please join me congratulating them.
> Congrats!
>
>
> - Robert, on behalf of the Oozie PMC
>

Re: Oozie coordinator and kafka

[Artem Ervits]

like I said wrong mailing list and I don't want to steal attention from
Oozie. Let's move along.

On Tue, Dec 19, 2017 at 10:40 AM, Ilya Karpov <i.kar...@cleverdata.ru>
wrote:

> Why not?
>
> > 19 дек. 2017 г., в 18:38, Serega Sheypak <serega.shey...@gmail.com>
> написал(а):
> >
> > Don't push NiFi :) Please don't :)
> >
> > 2017-12-19 16:16 GMT+01:00 Artem Ervits <artemerv...@gmail.com>:
> >
> >> you're probably better off using Apache Nifi at that point, Kafka Nifi
> is
> >> better integrated today than Oozie and Kafka. Probably wrong mailing
> list
> >> to push Nifi but it's a common use case in that realm.
> >>
> >> Thanks
> >>
> >> On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak <
> serega.shey...@gmail.com>
> >> wrote:
> >>
> >>> Hi, I know default coordinator functionality, but it's limited (almost)
> >> to
> >>> HDFS.
> >>> Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes
> >>> integration contract much more flexible.
> >>> I could have traceability, debuggability, transparency, throttling,
> >>> concurrency of oozie and push coordinator job on demand. And I'm not
> >>> limited to strict HDFS path pattern.
> >>>
> >>>
> >>> 2017-12-18 18:14 GMT+01:00 Andras Piros <andras.pi...@cloudera.com>:
> >>>
> >>>> Hi Serega,
> >>>>
> >>>> not to my knowledge. Would be interested on your use case, though.
> >>>>
> >>>> Would start w/ *Coordinator Input Events / Datasets
> >>>> <https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.
> >>>> html#a5._Dataset>*
> >>>> .
> >>>>
> >>>> Andras
> >>>>
> >>>> On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak <
> >>> serega.shey...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi, did anyone try to integrate oozie coordinator with kafka?
> >>>>> use case:
> >>>>>
> >>>>> System publishes message to kafka topic (sample message)
> >>>>> - cluster: hdfs://prod-cluster
> >>>>> - path: /my/input/data
> >>>>> - format: avro
> >>>>>
> >>>>> Oozie coordinator listens to kafka topic, consumes message and starts
> >>>>> workflow.
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: Oozie coordinator and kafka

[Artem Ervits]

you're probably better off using Apache Nifi at that point, Kafka Nifi is
better integrated today than Oozie and Kafka. Probably wrong mailing list
to push Nifi but it's a common use case in that realm.

Thanks

On Mon, Dec 18, 2017 at 7:04 PM, Serega Sheypak 
wrote:

> Hi, I know default coordinator functionality, but it's limited (almost) to
> HDFS.
> Kafka (any other pub/sub or queue like rabbitMQ, whatever-MQ) makes
> integration contract much more flexible.
> I could have traceability, debuggability, transparency, throttling,
> concurrency of oozie and push coordinator job on demand. And I'm not
> limited to strict HDFS path pattern.
>
>
> 2017-12-18 18:14 GMT+01:00 Andras Piros :
>
> > Hi Serega,
> >
> > not to my knowledge. Would be interested on your use case, though.
> >
> > Would start w/ *Coordinator Input Events / Datasets
> >  > html#a5._Dataset>*
> > .
> >
> > Andras
> >
> > On Sat, Dec 16, 2017 at 2:54 PM, Serega Sheypak <
> serega.shey...@gmail.com>
> > wrote:
> >
> > > Hi, did anyone try to integrate oozie coordinator with kafka?
> > > use case:
> > >
> > > System publishes message to kafka topic (sample message)
> > > - cluster: hdfs://prod-cluster
> > > - path: /my/input/data
> > > - format: avro
> > >
> > > Oozie coordinator listens to kafka topic, consumes message and starts
> > > workflow.
> > >
> >
>

Re: Welcoming new Oozie PMC member Satish Saley

[Artem Ervits]

Well deserved, congratulations Satish!

On Dec 11, 2017 2:49 PM, "Robert Kanter"  wrote:

> Hello Oozie community,
>
> It's my pleasure to announce that Satish Saley has become the newest
> addition to the Oozie PMC.
>
> Please join me congratulating him.
> Congrats!
>
>
> - Robert, on behalf of the Oozie PMC
>

Re: Welcome new Oozie Committer - Attila Sasvari

[Artem Ervits]

congratulations Attila!

On Thu, Jul 20, 2017 at 12:03 PM, Andras Piros 
wrote:

> Yy! Great job, well done, Sasi!
>
> Congrats!
>
> Andras
>
> On Thu, Jul 20, 2017 at 6:01 PM, Robert Kanter  wrote:
>
> > Hi everyone,
> >
> > It is my pleasure to announce that the Oozie PMC has invited
> > Attila Sasvari to become an Oozie committer and he has
> > accepted our invitation.
> >
> > Please join me in congratulating him.
> > Congrats!
> >
> >
> > - Robert, on behalf of the Oozie PMC
> >
>

Re: SLA alerts on running actions

[Artem Ervits]

You can access SLA information via JMS or REST in case you need that info
prior to job completion. Also check the following page, there is some
tuning available to make processing of SLA events faster
https://oozie.apache.org/docs/4.0.0/AG_Install.html#Notifications_Configuration


On May 24, 2017 11:14 AM, "Suresh V"  wrote:

> Hello Oozie users - we want Oozie to send an email notification when a
> workflow action is running more than a set duration.
>
> We experimented with the SLA feature however the message is sent only after
> the action is completed, not when the action is in running state and the
> duration is passed.
>
> Is there a way to achieve notification while action is running?
>
> Thnak you
> Suresh.
>

Re: Immutable vs mutable parameters in Oozie 4.2.0

[Artem Ervits]

Toby, your 2.5 release doesn't seem right. Latest 2.5 release is 2.5.3.
here are the changes for Oozie in 2.5
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/patch_oozie.html

And

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_release-notes/content/fixed_issues.html

I'm at Summit, stop by our booth and say Hi.

On Apr 5, 2017 4:28 PM, "Venkat Ranganathan" 
wrote:

> Toby
>
> For HDP specific questions, please visit https://community.hortonworks.com
>
> Thanks
>
> Venkat
> 
> From: Toby Evans 
> Sent: Wednesday, April 05, 2017 6:00 AM
> To: user@oozie.apache.org
> Subject: Re: Immutable vs mutable parameters in Oozie 4.2.0
>
> Ok, will do, currently at Munich for DataSummit, on it when i get back
>
> Cheers
>
> Toby
>
> On Wed, 5 Apr 2017, 09:27 Andras Piros,  wrote:
>
> Hi Toby,
>
> thanks for your interest on using Oozie!
>
> In order I can help you further, can you please:
>
>- add *Apache GitHub * git hashes of
>Oozie versions for each HDP version
>- add a small example of two workflows, one sets a parameter, the other
>reads the changed parameter - that was working in HDP versions 2.3 and
> 2.4,
>but not anymore in 2.5?
>
> Regards,
>
> Andras
>
> On Tue, Apr 4, 2017 at 4:38 PM, Toby Evans  wrote:
>
> > Hi there,
> >
> > We've got a big Oozie pipeline, which I'm having a little difficulty
> > upgrading from Hortonworks 2.3 > 2.4 > 2.5
> >
> > HDP Oozie
> > 2.3 4.2.0.2.3.2.0-2950
> > 2.4 4.2.0.2.4.2.0-258
> > 2.5 4.2.0.2.5.0.0-1245
> >
> >
> > So, we ingest and process data from a range of sources, eg databases,
> FTP,
> > SQL dbs etc.   These are all different, but share so many common features
> > that we have grown various heavily templated workflows. Each of our
> > workflows has a set of parameters (eg date range, source type,
> > hasCustomStep) all that sort of thing, all stored in "profile" workflows.
> > We first call our "profile" workflow, which sets up the various
> parameters
> > and then calls the template workflow, which then fills in the values
> > supplied from the profile.
> >
> > This has run fine for a couple of years in HDP 2.3, seems OK in 2.4, but
> > definitely doesn't in 2.5. The problem is that once a parameter has been
> > set in a workflow chain, then it stays set. I understand that you don't
> > want to be changing a parameter inside the same workflow, and we've
> > refactored so that we don't have a parameter value passed to a workflow,
> > which then changes the parameter value then calls another sub-workflow.
> > However, we're getting caught out by parameters being set by a workflow
> and
> > then not being reset much further down the pipeline when used by a
> > different "profile", which should be isolated but clearly isn't
> >
> > Has there been a code change between the versions, and is there a config
> > setting that allows us to switch off cross-workflow parameter
> immutability?
> >
> > thanks
> >
> > Toby
> >
>

Re: Extended attributes

[Artem Ervits]

Thanks Peter, I created https://issues.apache.org/jira/browse/OOZIE-2780
and submitted a patch. Not sure if there's anything else I need to add?

On Tue, Jan 17, 2017 at 12:13 PM, Peter Cseh <gezap...@cloudera.com> wrote:

> I don't know how long it will take to review and submit oozie 1770.
> Feel free to open an issue to bump the default hadoop version and add a
> patch to it.
> I'm sure it will be submitted earlier.
> Go
>
>
> On Jan 17, 2017 17:07, "Artem Ervits" <artemerv...@gmail.com> wrote:
>
> > do you know if it's a long wait? I'd like to contribute a patch soon.
> >
> > On Mon, Jan 16, 2017 at 3:00 AM, Peter Cseh <gezap...@cloudera.com>
> wrote:
> >
> > > Hi,
> > >
> > > the patch for OOZIE-1770
> > > <https://issues.apache.org/jira/browse/OOZIE-1770> changes
> > > the hadoop version to 2.6.0.
> > > We're working on getting it to the master branch.
> > >
> > > gp
> > >
> > >
> > > On Mon, Jan 16, 2017 at 2:37 AM, Artem Ervits <dbis...@gmail.com>
> wrote:
> > >
> > > > Hello all, I'm trying to extend FS action functionality to extended
> > > > attributes and I realized that this functionality is not exposed in
> > > Hadoop
> > > > 2.4. All major distros moved to 2.6.1 at the least, what are the
> plans
> > to
> > > > migrate these libraries?
> > > >
> > >
> > >
> > >
> > > --
> > > Peter Cseh
> > > Software Engineer
> > > <http://www.cloudera.com>
> > >
> >
>

Re: Extended attributes

[Artem Ervits]

do you know if it's a long wait? I'd like to contribute a patch soon.

On Mon, Jan 16, 2017 at 3:00 AM, Peter Cseh <gezap...@cloudera.com> wrote:

> Hi,
>
> the patch for OOZIE-1770
> <https://issues.apache.org/jira/browse/OOZIE-1770> changes
> the hadoop version to 2.6.0.
> We're working on getting it to the master branch.
>
> gp
>
>
> On Mon, Jan 16, 2017 at 2:37 AM, Artem Ervits <dbis...@gmail.com> wrote:
>
> > Hello all, I'm trying to extend FS action functionality to extended
> > attributes and I realized that this functionality is not exposed in
> Hadoop
> > 2.4. All major distros moved to 2.6.1 at the least, what are the plans to
> > migrate these libraries?
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>

Extended attributes

[Artem Ervits]

Hello all, I'm trying to extend FS action functionality to extended
attributes and I realized that this functionality is not exposed in Hadoop
2.4. All major distros moved to 2.6.1 at the least, what are the plans to
migrate these libraries?

RE: create a dir with timestamp for sqoop import

[Artem Ervits]

Thank you, I will look into that tomorrow.

-Original Message-
From: Paul Chavez [mailto:pcha...@verticalsearchworks.com] 
Sent: Tuesday, March 05, 2013 6:55 PM
To: user@oozie.apache.org
Subject: RE: create a dir with timestamp for sqoop import

I have a nightly Sqoop job that runs off a coordinator and uses a combination 
of output datasets and EL functions to provide the workflow parameters. Nice 
thing about that is you can 'backfill' with the coordinator by putting past 
days in the start/end attributes of the coordinator.

Here's a piece of my workflow, using a sqoop action:

sqoop xmlns=uri:oozie:sqoop-action:0.2
job-tracker${jobTracker}/job-tracker
name-node${nameNode}/name-node
  argimport/arg
  arg--table/arg
  argMyTable/arg
  arg--target-dir/arg
  arg${InputPath}/arg
  arg--where/arg
  argcreateddate between '${SDateMDY}' and '${SDateMDY} 
23:59:59.997'/arg /sqoop 

You can see I use two workflow properties, one for the path and one for the 
date format I want.

Then the coordinator:

coordinator-app name=NightlySqoop_coord
 frequency=${coord:days(1)}
 start=${coord_start} end=${coord_end} timezone=America/Los_Angeles
 xmlns=uri:oozie:coordinator:0.1
 datasets
  dataset name=ExportDaily frequency=${coord:days(1)}
   initial-instance=2013-01-01T11:00Z timezone=America/Los_Angeles
   
uri-template${nameNode}/ExportDir/Export${YEAR}${MONTH}${DAY}/uri-template
  /dataset
 /datasets
 output-events
  data-out name=ExportPath dataset=ExportDaily
   instance${coord:current(-1)}/instance
  /data-out
 /output-events
 action
  workflow
   ...
   configuration
property
 nameInputPath/name
 value${coord:dataOut('ExportPath')}/value
/property
property
 nameSDateMDY/name
 value${coord:formatTime(coord:dateOffset(coord:nominalTime(), -1, 
'DAY'), 'MM/dd/')}/value
/property
   /configuration
  /workflow
 /action
/coordinator-app

The -1 offset for the properties are because the job triggers every morning at 
3am local time but I want it to pull the previous day's data. The output 
dataset instance provides the path I need based on the uri-template, and the EL 
functions coord:formatTime(), coord:dateOffset() and cood:nominalTime() build 
the formatted date string.

Hope that helps.

Paul


-Original Message-
From: Artem Ervits [mailto:are9...@nyp.org]
Sent: Tuesday, March 05, 2013 1:49 PM
To: user@oozie.apache.org
Subject: create a dir with timestamp for sqoop import

Hello all,

I am trying to do the following in Oozie workflow:

sqoop job --exec jobName -- -target-dir /test/$(date +%Y%m%d%H%M%S)

I can think of a couple of options, one is to create a sqoop action but passing 
$(date +%Y%m%d%H%M%S) as argument does not work. The other option I tried was 
create a shell action TIMESTAMP=$(date +%Y%m%d%H%M%S) and then in sqoop 
outputdir do this:  ${outputDir}/$TIMESTAMP. Unfortunately that did not work 
either. Another option I tried was ${outputDir}/${timestamp()} but that format 
is not recognized by the OS. Is there a replace method for ${timestamp()} 
function? That would be the easiest implementation then.






This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.






This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.







This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.






This electronic message

RE: Oozie uses different M/R than the one specified in jar

[Artem Ervits]

Thank you will try that.

-Original Message-
From: Rohini Palaniswamy [mailto:rohini.adi...@gmail.com] 
Sent: Monday, March 04, 2013 5:41 AM
To: user@oozie.apache.org
Subject: Re: Oozie uses different M/R than the one specified in jar

Do you have some other jar containing file with same package name and class 
name somewhere in the classpath? You should try removing the duplicate

Regards,
Rohini


On Sat, Mar 2, 2013 at 6:05 AM, Artem Ervits are9...@nyp.org wrote:


 Hello all,

 I'm trying to write a M/R workflow. It simply counts the records in a 
 SequenceFile. When I execute the job, I get completely different 
 output than what I expect. Seems as if Oozie picks it's own jar to 
 execute against the dataset. I am using MR2 for API. How do I 
 explicitly force Oozie to use a certain jar?


 

 This electronic message is intended to be for the use only of the 
 named recipient, and may contain information that is confidential or 
 privileged.
  If you are not the intended recipient, you are hereby notified that 
 any disclosure, copying, distribution or use of the contents of this 
 message is strictly prohibited.  If you have received this message in 
 error or are not the named recipient, please notify us immediately by 
 contacting the sender at the electronic mail address noted above, and 
 delete and destroy all copies of this message.  Thank you.




 

 This electronic message is intended to be for the use only of the 
 named recipient, and may contain information that is confidential or 
 privileged.
  If you are not the intended recipient, you are hereby notified that 
 any disclosure, copying, distribution or use of the contents of this 
 message is strictly prohibited.  If you have received this message in 
 error or are not the named recipient, please notify us immediately by 
 contacting the sender at the electronic mail address noted above, and 
 delete and destroy all copies of this message.  Thank you.








This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.






This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.

default MapReduce class

[Artem Ervits]

Hello all,

Where does Oozie grab the default mapreduce mapper/reducer classes to execute? 
I am trying to specify explicitly the mapper and reducer classes, as well as 
input and output format classes and Oozie completely ignores them.





This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.






This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.

RE: Oozie uses different M/R than the one specified in jar

[Artem Ervits]

I solved the issue by copying an xml from a regular Hadoop jar run.

-Original Message-
From: Rohini Palaniswamy [mailto:rohini.adi...@gmail.com] 
Sent: Monday, March 04, 2013 5:41 AM
To: user@oozie.apache.org
Subject: Re: Oozie uses different M/R than the one specified in jar

Do you have some other jar containing file with same package name and class 
name somewhere in the classpath? You should try removing the duplicate

Regards,
Rohini


On Sat, Mar 2, 2013 at 6:05 AM, Artem Ervits are9...@nyp.org wrote:


 Hello all,

 I'm trying to write a M/R workflow. It simply counts the records in a 
 SequenceFile. When I execute the job, I get completely different 
 output than what I expect. Seems as if Oozie picks it's own jar to 
 execute against the dataset. I am using MR2 for API. How do I 
 explicitly force Oozie to use a certain jar?


 

 This electronic message is intended to be for the use only of the 
 named recipient, and may contain information that is confidential or 
 privileged.
  If you are not the intended recipient, you are hereby notified that 
 any disclosure, copying, distribution or use of the contents of this 
 message is strictly prohibited.  If you have received this message in 
 error or are not the named recipient, please notify us immediately by 
 contacting the sender at the electronic mail address noted above, and 
 delete and destroy all copies of this message.  Thank you.




 

 This electronic message is intended to be for the use only of the 
 named recipient, and may contain information that is confidential or 
 privileged.
  If you are not the intended recipient, you are hereby notified that 
 any disclosure, copying, distribution or use of the contents of this 
 message is strictly prohibited.  If you have received this message in 
 error or are not the named recipient, please notify us immediately by 
 contacting the sender at the electronic mail address noted above, and 
 delete and destroy all copies of this message.  Thank you.








This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.






This electronic message is intended to be for the use only of the named 
recipient, and may contain information that is confidential or privileged.  If 
you are not the intended recipient, you are hereby notified that any 
disclosure, copying, distribution or use of the contents of this message is 
strictly prohibited.  If you have received this message in error or are not the 
named recipient, please notify us immediately by contacting the sender at the 
electronic mail address noted above, and delete and destroy all copies of this 
message.  Thank you.