[
https://issues.apache.org/jira/browse/LIVY-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyorgy Gal updated LIVY-430:
----------------------------
Fix Version/s: 0.10.0
(was: 0.9.0)
This issue has been moved to the 0.10.0 release as part of a bulk update. If
you feel this is moved out inappropriately, feel free to provide justification
and reset the Fix Version to 0.9.0.
> Livy always run on local mode
> -----------------------------
>
> Key: LIVY-430
> URL: https://issues.apache.org/jira/browse/LIVY-430
> Project: Livy
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: Spark 2.2.0
> Hadoop 2.7.4
> Reporter: Bohdan Demydov
> Priority: Major
> Fix For: 0.10.0
>
>
> I am trying to run Pyspark (or Spark) job via Livy server with
> "spark.master=yarn".
> *What I have done:*
> 1) In *spark-defaults.conf:*
> spark.master yarn
> spark.submit.deployMode client
> 2) In *livy.conf*:
> livy.spark.master = yarn
> livy.spark.deployMode = client
> 3) I send request via CURL *with "conf": {"spark.master": "yarn"}
> *
> Example:
> curl -X POST -H "Content-Type: application/json" localhost:8998/batches
> --data '{"file": "hdfs:///user/grzegorz/hello-world.py", "name": "MY",
> "conf": {"spark.master": "yarn"} }'
> {"id":3,"state":"running","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout:
> ","\nstderr: "]}
> *And what I am always getting in logs:*
> 18/01/02 14:45:07.880 qtp1758624236-28 INFO BatchSession$: Creating batch
> session 3: [owner: null, request: [proxyUser: None, file:
> hdfs:///user/grzegorz/hello-world.py, name: MY, conf: spark.master -> yarn]]
> 18/01/02 14:45:07.883 qtp1758624236-28 INFO SparkProcessBuilder: Running
> '/usr/local/share/spark/spark-2.0.2/bin/spark-submit' '--name' 'MY' '--conf'*
> 'spark.master=local' *'hdfs:///user/grzegorz/hello-world.py'
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
