Hi,
Hi,
I’m very new to oozie ,actually I would like to run Spark 2.3 jobs on oozie
based on file arrival on aws s3 which is a dependency for the job
I see some examples which uses s3 as input event datasets as below
s3n://mybucket/a/b/${YEAR}/${MONTH}/${DAY}
So my question is does oozie
on.
>
> At the moment it's not feasible to install Oozie without those Hadoop
> components. How to install Oozie please *find here
> <https://oozie.apache.org/docs/5.0.0/AG_Install.html>*.
>
> Regards,
>
> Andras
>
> On Tue, May 15, 2018 at 4:11 PM, purna pradeep &
Hello,
Does oozie supports spark 2.3? Or will it even care of the spark version
I want to use spark action
Thanks,
Purna
purna pradeep <purna2prad...@gmail.com>
wrote:
> Ok I have tried this
>
> It appears that s3a support requires httpclient 4.4.x and oozie is bundled
> with httpclient 4.3.6. When httpclient is upgraded, the ext UI stops
> loading.
>
>
>
> On Thu, May 17, 2
ames are slightly different so you'll have to change
> the example I've given.
>
>
>
> On Thu, May 17, 2018 at 4:16 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
>
>> Peter,
>>
>> I’m using latest oozie 5.0.0 and I have tried below changes but no luck
tion on how to make it work in jobs, something similar should work
> on the server side as well
>
> On Tue, May 15, 2018 at 4:43 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
>
> > Thanks Andras,
> >
> > Also I also would like to know if oozie supports
+Peter
On Wed, May 16, 2018 at 11:29 AM purna pradeep <purna2prad...@gmail.com>
wrote:
> Peter,
>
> I have tried to specify dataset with uri starting with s3://, s3a:// and
> s3n:// and I am getting exception
>
>
>
> Exception occurred:E0904: Scheme [s3] not sup
stems
> hdfs,hftp,webhdfs Enlist
> the different filesystems supported for federation. If wildcard "*" is
> specified, then ALL file schemes will be allowed.properly.
>
> For testing purposes it's ok to put * in there in oozie-site.xml
>
> On Wed, May 16, 2018 at 5:29
f you overwrite the -Dspark.version and compile Oozie that way it
> will work.
> gp
>
>
> On Tue, May 15, 2018 at 5:07 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
>
> > Hello,
> >
> > Does oozie supports spark 2.3? Or will it even care of the spark versi
ses));
> LOG.info("Loaded default urihandler {0}",
> defaultHandler.getClass().getName());
> Thanks
>
> On Wed, May 16, 2018 at 5:47 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
>
>> This is what I already have in my oozie-site.xml
>&
)
at
org.apache.oozie.service.HadoopAccessorService$5.run(HadoopAccessorService.java:623
On Wed, May 16, 2018 at 2:19 PM purna pradeep <purna2prad...@gmail.com>
wrote:
> This is what is in the logs
>
> 2018-05-16 14:06:13,500 INFO URIHandlerService:520 - SERVER[loca
artemerv...@gmail.com> wrote:
>
> > Did you run
> > mvn clean install first on the parent directory?
> >
> > On Tue, May 15, 2018, 11:35 AM purna pradeep <purna2prad...@gmail.com>
> > wrote:
> >
> > > Thanks peter,
> >
Cseh <gezap...@cloudera.com> wrote:
> Wow, great work!
> Can you please summarize the required steps? This would be useful for
> others so we probably should add it to our documentation.
> Thanks in advance!
> Peter
>
> On Fri, May 18, 2018 at 11:33 PM, purna prade
.sh
But now I’m getting this error
On Thu, May 17, 2018 at 2:53 PM purna pradeep <purna2prad...@gmail.com>
wrote:
> Ok I got passed this error
>
> By rebuilding oozie with Dhttpclient.version=4.5.5 -Dhttpcore.version=4.4.9
>
> now getting this error
>
>
credentials from service
endpoint]
On Thu, May 17, 2018 at 12:24 PM purna pradeep <purna2prad...@gmail.com>
wrote:
>
> Peter,
>
> Also When I submit a job with new http client jar, I get
>
> ```Error: IO_ERROR : java.io.IOException: Error while connecting Oozie
> server. N
https://search.maven.org/#artifactdetails%7Corg.apache.spark%7Cspark-kubernetes_2.11%7C2.3.0%7Cjar
> >
> in
> the sharelib/spark/pom.xml as a compile-time dependency.
>
> gp
>
> On Tue, May 15, 2018 at 9:04 PM, purna pradeep <purna2prad...@gmail.com>
> wrote:
&g
(respectively)
I have tried adding AWS access ,secret keys in
oozie-site.xml and hadoop core-site.xml , and hadoop-config.xml
On Wed, May 16, 2018 at 2:30 PM purna pradeep <purna2prad...@gmail.com>
wrote:
>
> I have tried this ,just adde
Hello ,
Event trigger Oozie datasets
1) Does oozie supports event trigger?
Trigger Workflow based on a file arrival on AWS s3
As per my understanding based on start date mentioned on coordinator it
can poll for a file on s3 and once dependency is met it can execute an
action/SparkAction
18 matches
Mail list logo
