Hi Peter,
I am working with Purna. I have tried to specify dataset with uri starting with
s3://, s3a:// and s3n:// and I am getting exception
Exception occurred:E0904: Scheme [s3] not supported in uri
[s3://cmsegmentation-qa/oozie-test/input.data] Making the job failed
org.apache.oozie.dependency.URIHandlerException: E0904: Scheme [s3] not
supported in uri [s3://cmsegmentation-qa/oozie-test/input.data]
at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:185)
at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:168)
at
org.apache.oozie.service.URIHandlerService.getURIHandler(URIHandlerService.java:160)
at
org.apache.oozie.command.coord.CoordCommandUtils.createEarlyURIs(CoordCommandUtils.java:465)
at
org.apache.oozie.command.coord.CoordCommandUtils.separateResolvedAndUnresolved(CoordCommandUtils.java:404)
at
org.apache.oozie.command.coord.CoordCommandUtils.materializeInputDataEvents(CoordCommandUtils.java:731)
at
org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:546)
at
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:492)
at
org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materialize(CoordMaterializeTransitionXCommand.java:362)
at
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:73)
at
org.apache.oozie.command.MaterializeTransitionXCommand.execute(MaterializeTransitionXCommand.java:29)
at org.apache.oozie.command.XCommand.call(XCommand.java:290)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:181)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is S3 support specific to CDH distribution or should in work in Apache Oozie as
well?
Thanks!
On 5/16/18, 10:29 AM, "Peter Cseh" <[email protected]> wrote:
I think it should be possible for Oozie to poll S3. Check out this
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cloudera.com_documentation_enterprise_5-2D9-2Dx_topics_admin-5Foozie-5Fs3.html&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=dGiatnV6rgwui2UT5_IPwJR0CLUiRosukYp1v9Wa6ig&e=>
description on how to make it work in jobs, something similar should work
on the server side as well
On Tue, May 15, 2018 at 4:43 PM, purna pradeep <[email protected]>
wrote:
> Thanks Andras,
>
> Also I also would like to know if oozie supports Aws S3 as input events to
> poll for a dependency file before kicking off a spark action
>
>
> For example: I don’t want to kick off a spark action until a file is
> arrived on a given AWS s3 location
>
> On Tue, May 15, 2018 at 10:17 AM Andras Piros <[email protected]>
> wrote:
>
> > Hi,
> >
> > Oozie needs HDFS to store workflow, coordinator, or bundle definitions,
> as
> > well as sharelib files in a safe, distributed and scalable way. Oozie
> needs
> > YARN to run almost all of its actions, Spark action being no exception.
> >
> > At the moment it's not feasible to install Oozie without those Hadoop
> > components. How to install Oozie please *find here
> >
<https://urldefense.proofpoint.com/v2/url?u=https-3A__oozie.apache.org_docs_5.0.0_AG-5FInstall.html&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=YRCcmhnMovHP2GiPKZ59gzGrZjKvw_rfPJ_hqeOu_qY&e=>*.
> >
> > Regards,
> >
> > Andras
> >
> > On Tue, May 15, 2018 at 4:11 PM, purna pradeep <[email protected]>
> > wrote:
> >
> > > Hi,
> > >
> > > Would like to know if I can use sparkaction in oozie without having
> > Hadoop
> > > cluster?
> > >
> > > I want to use oozie to schedule spark jobs on Kubernetes cluster
> > >
> > > I’m a beginner in oozie
> > >
> > > Thanks
> > >
> >
>
--
*Peter Cseh *| Software Engineer
cloudera.com
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cloudera.com&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=jHSKCL-Qn2Rqhd4R10JbyR84_tOemXIG7k7DXYFx_nY&e=>
[image: Cloudera]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cloudera.com_&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=yyz_zjLclExig_qrlg5ACyK8h0FuBcO1ueVlTbj0-QY&e=>
[image: Cloudera on Twitter]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_cloudera&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=dBGBZEcTe7Dk0WoYS4uXo2iL-EcyctZdhD5JLqKH0NM&e=>
[image:
Cloudera on Facebook]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_cloudera&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=W1Prrd87Wvqmug5DhHrb4yeaqZ_sd-HWeyDQonNf3b8&e=>
[image: Cloudera
on LinkedIn]
<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_cloudera&d=DwIFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=cWp0qG1SHE-BjGVXxc3DDdovhT3_gnV52lKkc9n1jXXo0m3nwaDOWVTuvrRxMuTU&m=Y7B2ffRtI5fOx8cSBGPHfKBiVS_t88GFcDvpDY19xl0&s=NDwrLruguxGxGEV1u-MqV5_PzYWh6Uet16GkSZIvl1M&e=>
------------------------------
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates and may only be used solely in performance of
work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
