Thanks for you inputs Alejandro, Mona! That definitely sounds plausible, I'll give it a try.
-Som On Wed, Jun 19, 2013 at 2:32 PM, Alejandro Abdelnur <[email protected]>wrote: > or if you don't want to go into writing a custom action executor for now, > you could use the Oozie shell or ssh actions to start your mesos job. > > > On Wed, Jun 19, 2013 at 1:58 PM, Mona Chitnis <[email protected]> > wrote: > > > That is an interesting use case Som! Oozie is tightly coupled with Hadoop > > right now. You might be able to use custom ActionExecutor class and plug > > it in to perform your Step2 to a Spark job in S3, the first step to > > "launching" that action is still going to be a Hadoop Map task by Oozie. > > And there would be problem to fetch secure delegation tokens from S3, in > > order for launcher to launch that action. > > > > --Mona > > > > On 6/19/13 1:44 PM, "Som Satpathy" <[email protected]> wrote: > > > > >Any body has anything to share here? > > > > > >Thanks, > > >Som > > > > > >On Mon, Jun 17, 2013 at 3:25 PM, Som Satpathy <[email protected]> > > >wrote: > > > > > >> Hi, > > >> > > >> I had a question regarding a use case where a user might want to > design > > >> and execute workflows on YARN/Mesos which runs hadoop and spark in > > >> parallel. Does Oozie currently support or will it support in future > > >>actions > > >> that can execute in a framework agnostic manner? > > >> > > >> For simplicity let's consider an example - a user defines a workflow > as > > >> follows: > > >> > > >> Step1: Mapreduce over hadoop job to do some initial data processing > > >> (Input: data from s3, output: data into s3) > > >> Step2: Spark job to further process the data (Input: data from s3, > > >>output: > > >> data into s3) > > >> > > >> Here assumption is both hadoop and spark are running in the same > cluster > > >> managed by YARN/Mesos. > > >> > > >> Can oozie support such a usecase today? > > >> > > >> Thanks, > > >> Som > > >> > > > > > > > -- > Alejandro >
