Repository: zeppelin
Updated Branches:
refs/heads/master 5afd7cacd -> 7cc908b3a
[MINOR] Enable pyspark test in local mode
### What is this PR for?
Enabling test for pyspark in local mode
### What type of PR is it?
[Improvement]
### Todos
* [x] - Add spark configuration and
* [x] - Edit a logic for finding spark home
### What is the Jira issue?
N/A
### How should this be tested?
1. Download spark under {ZEPPELIN_HOME}
1. run `testing/startSparkCluster.sh`
1. `mvn clean package -Pspark-1.6 -Ppyspark -Pyarn -DskipTests`
1. Run `ZeppelinSparkClusterTest`
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee <[email protected]>
Closes #1201 from jongyoul/minor/enable-pyspark-test-in-local-mode and squashes
the following commits:
67ccc2d [Jongyoul Lee] Added spark configuration while not using CI
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7cc908b3
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7cc908b3
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7cc908b3
Branch: refs/heads/master
Commit: 7cc908b3a1824d05ad47f9ffd152540f3777c237
Parents: 5afd7ca
Author: Jongyoul Lee <[email protected]>
Authored: Mon Jul 18 18:09:42 2016 +0900
Committer: Jongyoul Lee <[email protected]>
Committed: Fri Jul 29 00:56:06 2016 +0900
----------------------------------------------------------------------
.../test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7cc908b3/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
----------------------------------------------------------------------
diff --git
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
index 58af796..580e5a0 100644
---
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
+++
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/AbstractTestRestApi.java
@@ -171,6 +171,8 @@ public abstract class AbstractTestRestApi {
String sparkHome = getSparkHome();
if (sparkHome != null) {
+ sparkIntpSetting.getProperties().setProperty("master", "spark://" +
getHostname() + ":7071");
+ sparkIntpSetting.getProperties().setProperty("spark.cores.max", "2");
// set spark home for pyspark
sparkIntpSetting.getProperties().setProperty("spark.home",
sparkHome);
pySpark = true;
@@ -192,7 +194,7 @@ public abstract class AbstractTestRestApi {
}
private static String getSparkHome() {
- String sparkHome = getSparkHomeRecursively(new
File(System.getProperty("user.dir")));
+ String sparkHome = getSparkHomeRecursively(new
File(System.getProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_HOME.getVarName())));
System.out.println("SPARK HOME detected " + sparkHome);
return sparkHome;
}