---------- Forwarded message ---------- From: Ahmed Kamal <[email protected]> Date: Mon, Feb 29, 2016 at 4:18 PM Subject: Spark Node with Mesos and Spark Standalone To: [email protected]
Hi guys, I'm investigating the Oozie Spark Node and I came across this sentence "The master element indicates the url of the Spark Master. Ex: spark://host:port, <https://oozie.apache.org/docs/4.2.0/spark://host:port,.html> mesos://host:port, <https://oozie.apache.org/docs/4.2.0/mesos://host:port,.html> yarn-cluster, yarn-master, or local. " in the documentation of Oozie https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html from what I understood oozie uses Spark Submit script to either run spark jobs in local mode or to run on yarn and it depend on the Yarn API to get the job status but it isn't possible to use it with spark standalone or spark mesos as the current implementation doesn't have a valid way to check the status. Am I right ? If so , I'm interested in extending the current spark node to support them and contributing this to the project. I'm preparing a design document for the change. Should I open an issue on JIRA ? -- Regards , Ahmed Kamal Junior Big Data Engineer , BadrIT -- JAK , Ahmed Kamal Junior Big Data Engineer , BadrIT
