Hi Divya,
Please find below code to invoke spark from oozie.
Oozie file:
+++++++++++++
<workflow-app xmlns="uri:oozie:workflow:0.2" name="DE_2_workflow_US">
<start to="spark-scala"/>
<action name="mysparkscala">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>maprfs:///</job-tracker>
<name-node>maprfs:///</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>dileep</value>
</property>
</configuration>
<exec>sparkshell.sh</exec>
<file>/ggg/gms/gmsrffr/dev/dileep/sparkshell.sh</file>
<file>/axp/gms/gmsrffr/dev/dileep/sparkshell.scala</file>
</shell>
<ok to="end" />
<error to="fail" />
</action>
<kill name="fail">
<message>Java failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end" />
</workflow-app>
#############
SparkShell.sh
##############
/opt/mapr/spark/spark-1.2.1/bin/spark-shell --name perf108pret1
--num-executors 1 --executor-cores 1 --executor-memory 1G --driver-memory
2G -i sparkshell.scala
exit 0
On Mon, Mar 7, 2016 at 9:32 PM, Chandeep Singh <[email protected]> wrote:
> As a work around you could put your spark-submit statement in a shell
> script and then use Oozie’s SSH action to execute that script.
>
> On Mar 7, 2016, at 3:58 PM, Neelesh Salian <[email protected]> wrote:
>
> Hi Divya,
>
> This link should have the details that you need to begin using the Spark
> Action on Oozie:
> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html
>
> Thanks.
>
> On Mon, Mar 7, 2016 at 7:52 AM, Benjamin Kim <[email protected]> wrote:
>
>> To comment…
>>
>> At my company, we have not gotten it to work in any other mode than
>> local. If we try any of the yarn modes, it fails with a “file does not
>> exist” error when trying to locate the executable jar. I mentioned this to
>> the Hue users group, which we used for this, and they replied that the
>> Spark Action is very basic implementation and that they will be writing
>> their own for production use.
>>
>> That’s all I know...
>>
>> On Mar 7, 2016, at 1:18 AM, Deepak Sharma <[email protected]> wrote:
>>
>> There is Spark action defined for oozie workflows.
>> Though I am not sure if it supports only Java SPARK jobs or Scala jobs as
>> well.
>> https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html
>> Thanks
>> Deepak
>>
>> On Mon, Mar 7, 2016 at 2:44 PM, Divya Gehlot <[email protected]>
>> wrote:
>>
>>> Hi,
>>>
>>> Could somebody help me by providing the steps /redirect me to
>>> blog/documentation on how to run Spark job written in scala through Oozie.
>>>
>>> Would really appreciate the help.
>>>
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>>
>> --
>> Thanks
>> Deepak
>> www.bigdatabig.com
>> www.keosha.net
>>
>>
>>
>
>
> --
> Neelesh Srinivas Salian
> Customer Operations Engineer
>
>
>
>
>
--
Regards
Dileep Kumar
+91 9742443302
