[ZEPPELIN-572] pyspark interpreter doesn't work on yarn-client ### What is this PR for? Set `spark.yarn.isPython` to be `true` to distribute pyspark libraries to workers when master is `yarn-client`
### What type of PR is it? Bug Fix ### Is there a relevant Jira issue? [ZEPPELIN-572](https://issues.apache.org/jira/browse/ZEPPELIN-572) ### How should this be tested? You need yarn cluster to test this PR. Simple way to test this PR would be running below code in paragraph and see if it throws error. ``` %pyspark print(sc.parallelize([1, 2]).count()) ``` And you should be able to see that `spark.yarn.isPython` is set to `true` in **Spark UI > Environment > Spark Properties** only when you set spark.master as `yarn-client`. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Mina Lee <[email protected]> Closes #605 from minahlee/ZEPPELIN-572 and squashes the following commits: 8c99de5 [Mina Lee] Set spark.yarn.isPython to be true to distribute needed pyspark libraries to workers when master is yarn-client Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/ed1cc023 Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/ed1cc023 Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/ed1cc023 Branch: refs/heads/branch-0.5.6 Commit: ed1cc023953834ff29d0bc212d83f0fe8d000eff Parents: dedd6a6 Author: Mina Lee <[email protected]> Authored: Wed Jan 6 21:23:18 2016 -0800 Committer: Alexander Bezzubov <[email protected]> Committed: Mon Jan 11 22:13:14 2016 +0900 ---------------------------------------------------------------------- .../main/java/org/apache/zeppelin/spark/SparkInterpreter.java | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/ed1cc023/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java ---------------------------------------------------------------------- diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java index 2f2829e..b967c29 100644 --- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java +++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java @@ -335,6 +335,10 @@ public class SparkInterpreter extends Interpreter { conf.set("spark.submit.pyArchives", Joiner.on(":").join(pythonLibs)); } + // Distributes needed libraries to workers. + if (getProperty("master").equals("yarn-client")) { + conf.set("spark.yarn.isPython", "true"); + } SparkContext sparkContext = new SparkContext(conf); return sparkContext;
