Hi,
How to set --master, --deploy-mode, --driver-class-path and
--driver-java-options through Apache Livy?
I want to set the master, spark deploy-mode, driver-class-path and
driver-java-options for the Spark job when the job is triggered through Apache
Livy without having to restart the Livy server when these settings change. How
to do this since there are no direct options to do this in Livy?
I understand these are used to set the parameters for Client mode. I want to
set these when I spawn a job in client mode.
Also when I try to set "spark.master" through REST through "conf" param, it is
not being set.
This is the JSON I'm sending to the Livy API.
{
"file": "/user/livy/spark-examples.jar",
"conf" : {
"spark.master": "yarn",
"spark.submit.deployMode": "cluster"
},
"args": ["2"],
"className": "org.apache.spark.examples.SparkPi"
}
Thanks and Regards,
Sarthak
Privileged/Confidential Information may be contained in this message and is
intended only for the use of the addressee.
If you are not the addressee, or person responsible for delivering it to the
addressee, you should not copy or deliver
this to anyone else. If you receive this message by mistake, please delete the
message from any/all computer/s and
notify the sender immediately by reply Email. We appreciate your assistance in
preserving the confidentiality of our
correspondence. Any information in this message that does not relate to the
official business of the organization
shall be understood as neither given nor endorsed by it. Please advise
immediately if you or your employer
does not consent to Internet Email for messages of this kind. Thank you.
(HCID0411)
