GitHub user Leemoonsoo opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/244
ZEPPELIN-160 Working with provided Spark, Hadoop.
Zeppelin currently embeds all spark dependencies under interpreter/spark
and loading them on runtime.
Which is useful because of user can try Zeppelin + Spark with local mode
without installation and configuration of spark.
However, when user has existing spark and hadoop installation, it'll be
really helpful to just pointing them instead of build zeppelin with specific
version of spark and hadoop combination.
This PR implements ability to use external spark and hadoop installation,
by doing
* spark-dependencies module packages spark/hadoop dependencies under
interpreter/spark/dep, to support local mode (current behavior)
* When SPARK_HOME and HADOOP_HOME is defined, bin/interpreter.sh exclude
interpreter/spark/dep from classpath and include system installed spark and
hadoop into the classpath.
This patch makes Zeppelin binary independent from spark version. Once
Zeppelin is been built, SPARK_HOME can point any version of spark.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leemoonsoo/incubator-zeppelin spark_provided
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/244.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #244
----
commit c92f18534d85e318e41fe53c0cf684b3805be719
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T17:56:43Z
Separate spark-dependency into submodule
commit c49c154bc1ca6ebd783022b774e4ff92c37c8d4c
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T17:59:47Z
Load interpreter/spark/dep only when SPARK_HOME is undefined
commit 3cd1d4b123fd90620c30bb363d7d73986c5e4cb0
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T18:38:16Z
Add hadoop jar and spark jar from HADOOP_HOME, SPARK_HOME when they are
defined
commit fee333f317092e1450faf56d70ca4fe62811b546
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T18:52:24Z
Add scala-library in test scope
commit ea3312a36867438afbd986a686fa59f654fc8e28
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T23:26:30Z
Add dependency for compile and test
commit a026d7898f71bd83c070628d71aaa615a6f3527c
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T23:26:50Z
py4j version as a property
commit d579754ad0699c75829f7955652b8daa84b1fe44
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-21T23:27:11Z
Handle ZEPPELIN_CLASSPATH proper way
commit 454efab853df1484a527cb8a298a7fc3eb476376
Author: Lee moon soo <m...@apache.org>
Date: 2015-08-22T01:36:44Z
Use reflection not to use import org.apache.spark.scheduler.Stage
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
