GitHub user AhyoungRyu reopened a pull request:
https://github.com/apache/zeppelin/pull/1339
[ZEPPELIN-1332] Remove spark-dependencies & suggest new way
### What is this PR for?
Currently, Zeppelin's embedded Spark is located under `interpreter/spark/`.
For whom **builds Zeppelin from source**, this Spark is downloaded when
they build the source with [build
profiles](https://github.com/apache/zeppelin#spark-interpreter). I think this
various build profiles are useful to customize the embedded Spark, but many
Spark users are using their own Spark not Zeppelin's embedded one. Nowadays
only Spark&Zeppelin beginners use this embedded Spark. For them, there are too
many build profiles(it's so complicated i think).
In case of **Zeppelin binary package**, it's included by default under
`interpreter/spark/`. That's why Zeppelin package size is so huge.
#### New suggestions
This PR will change the embedded Spark binary downloading mechanism like
below.
The below text will be saved in `spark-2.0.0-hadoop2.7/README.txt` if the
user answers "N/n"
```
Please note that you answered 'No' when we asked whether you want to
download local Spark binary under ZEPPELIN_HOME/local-spark/ or not.
If you want to use Spark interpreter in Apache Zeppelin, you need to set
your own SPARK_HOME.
See
http://zeppelin.apache.org/docs/ZEPPELIN_VERSION/interpreter/spark.html#configuration
for the further details about Spark configuration in Zeppelin.
```
### What type of PR is it?
Improvement
### Todos
- [x] - trap `ctrl+c` & `ctrl+z` key interruption during downloading Spark
- [x] - test in the different OS
- [ ] - update related document pages again after get feedbacks
### What is the Jira issue?
[ZEPPELIN-1332](https://issues.apache.org/jira/browse/ZEPPELIN-1332)
### How should this be tested?
1. `rm -r spark-dependencies`
2. Apply this patch and build with `mvn clean package -DskipTests`
3. Start Zeppelin with `bin/zeppelin-daemon.sh` or `bin/zeppelin.sh`
### Screenshots (if appropriate)
- Without `ZEPPELIN_HOME/local-spark/spark-2.0.0-hadoop2.7`
- Do you want to download local Spark? "Yes"
<img width="939" alt="screen shot 2016-09-23 at 1 33 03 am"
src="https://cloud.githubusercontent.com/assets/10060731/18757222/ea30add2-812d-11e6-97e8-b31199b15283.png";>
- "No"
<img width="953" alt="screen shot 2016-09-23 at 1 34 12 am"
src="https://cloud.githubusercontent.com/assets/10060731/18757229/ee6f330a-812d-11e6-84c9-db5fe4d5a35b.png";>
- With `ZEPPELIN_HOME/local-spark/spark-2.0.0-hadoop2.7`
Nothing happened. Zeppelin will be started like before.
### Questions:
- Does the licenses files need update? no
- Is there breaking changes for older versions? no
- Does this needs documentation? Need to update some related documents
(e.g. README.md, spark.md and install.md ?)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/AhyoungRyu/zeppelin ZEPPELIN-1332
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/1339.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1339
----
commit aaabb9a274810b9bbc903587c715d2589b8ecc0a
Author: AhyoungRyu <[email protected]>
Date: 2016-08-16T15:08:19Z
Fix typo comment in interpreter.sh
commit 9b5e7eacc72c613c1dc66502df6d54f82e51d937
Author: AhyoungRyu <[email protected]>
Date: 2016-08-17T01:52:06Z
Remove spark-dependencies
commit cb65e7e5b56dab01412c2cbe8a17e36335f6e4eb
Author: AhyoungRyu <[email protected]>
Date: 2016-08-17T07:14:35Z
Add spark-2.*-bin-hadoop* to .gitignore
commit 126a7470c40518f857db85fc5a003bd8ff5d209e
Author: AhyoungRyu <[email protected]>
Date: 2016-08-17T15:22:25Z
Add download-spark.sh file
commit 40276e19b3cd394301f0d13869f45c53e0408024
Author: AhyoungRyu <[email protected]>
Date: 2016-08-17T15:28:51Z
Remove useless comment line in common.sh
commit 8e827577538fc406ddc1c02aa5f618981fc840b8
Author: AhyoungRyu <[email protected]>
Date: 2016-08-18T03:32:11Z
Remove zeppelin-spark-dependencies from r/pom.xml
commit 33b9dce0c4cdf23056eb128a35ba65cbb1021b28
Author: AhyoungRyu <[email protected]>
Date: 2016-08-21T05:38:55Z
Change SPARK_HOME with proper message
commit 050877c60a9ee9d320134939decf0b3cd8e9c4a3
Author: AhyoungRyu <[email protected]>
Date: 2016-09-06T08:55:20Z
Check interpreter/spark/ instead of SPARK_HOME
commit 7990c5aae5da4f7234ebf26e913f2cf7b434d1fb
Author: AhyoungRyu <[email protected]>
Date: 2016-09-06T08:55:40Z
Refactor download-spark.sh
commit db53a9e63edb82d417593f5373a9652dc065fcbd
Author: AhyoungRyu <[email protected]>
Date: 2016-09-07T07:48:15Z
Revert: remove spark-dependencies
commit f7c5a23199a289bf3941978860ace489e8dff1fe
Author: AhyoungRyu <[email protected]>
Date: 2016-09-07T13:23:11Z
Remove useless ZEPPELIN_HOME
commit 455417450fcfa85c71fb0c3d965ffcaded289f4a
Author: AhyoungRyu <[email protected]>
Date: 2016-09-08T05:51:40Z
Change dir of Spark bin to 'local-spark'
commit cc4012eb2a664ac79425cdb0bf6e849ffd87b83b
Author: AhyoungRyu <[email protected]>
Date: 2016-09-08T06:07:20Z
Set timeout for travis test
commit f3ab4756b749841ffe034a5c57f3494889ae87f2
Author: AhyoungRyu <[email protected]>
Date: 2016-09-08T06:16:54Z
Add license header to download-spark.cmd
commit 7cce923097ed48a0bd4873c4de77f56b656fe44f
Author: AhyoungRyu <[email protected]>
Date: 2016-09-08T11:48:43Z
Fix wrong check condition in common.sh
commit ce7766775dd98cf9d7a76984ef5d8cc93977dfef
Author: AhyoungRyu <[email protected]>
Date: 2016-09-08T13:14:29Z
Add travis condition to download-spark.sh
commit a5ef077e339b6e45d3d9908ac0102aba3a7f65f1
Author: AhyoungRyu <[email protected]>
Date: 2016-09-12T05:42:29Z
Remove bin/download-spark.cmd again
commit 1edd5fb67486c2b51e51f74ce6fa3fb6962abb41
Author: AhyoungRyu <[email protected]>
Date: 2016-09-12T16:25:31Z
Remove spark-dependency profiles & reorganize some titles in README.md
commit 132d24b35d47dc384999405f90b62706307dc0c4
Author: AhyoungRyu <[email protected]>
Date: 2016-09-12T18:30:41Z
Update spark.md to add a guide for local-spark mode
commit 8e4a256036b8a067114b3b40d6434e6bb478caaa
Author: AhyoungRyu <[email protected]>
Date: 2016-09-12T18:49:49Z
Remove '-Ppyspark' build options
commit 117c52d26bcb55dda15b36e416597b6b73803ef9
Author: AhyoungRyu <[email protected]>
Date: 2016-09-13T08:09:18Z
Remove useless creating .bak file process
commit 5ba99ea598f091e668fdc3b0f3005a3b2ceb6ea5
Author: AhyoungRyu <[email protected]>
Date: 2016-09-13T11:21:10Z
Update install.md & spark.md
commit 075195ddc43bbc5f3798d4ac73e7b5365c0881cb
Author: AhyoungRyu <[email protected]>
Date: 2016-09-14T09:35:37Z
Resolve 'sed' command issue between OSX & Linux
commit b4ef1f54730374e269a405a4f7a9b306cbcdbc24
Author: AhyoungRyu <[email protected]>
Date: 2016-09-14T11:20:31Z
Trap ctrl+c during downloading Spark
commit b21188b3f4b7c5c6c60aa80bd44c061438462914
Author: AhyoungRyu <[email protected]>
Date: 2016-09-14T11:26:56Z
Remove useless condition
commit ce1d0c44d4755cd9007443ad0178093b691665cd
Author: AhyoungRyu <[email protected]>
Date: 2016-09-20T17:05:16Z
Make local spark mode with zero-configuration as @moon suggested
commit 4440554d63ff17ce4e34d9472dc1932a04ca917f
Author: AhyoungRyu <[email protected]>
Date: 2016-09-22T06:47:05Z
Put 'autodetect HADOOP_CONF_HOME by heuristic' back code blocks
commit fb27690d98616156c0a3059b6c748860a7d64788
Author: AhyoungRyu <[email protected]>
Date: 2016-09-22T14:20:31Z
Modify SparkRInterpreter.java to enable SparkR without SPARK_HOME
commit fea61077273e62778eef2f1a637a7dd9b3df5f6d
Author: AhyoungRyu <[email protected]>
Date: 2016-09-22T17:01:40Z
Remove duplicated variable declaration
commit 972fb3e6bab4b41164f70c2abe5b06fc7d09aa6b
Author: AhyoungRyu <[email protected]>
Date: 2016-09-22T17:02:01Z
Update related docs again
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
![https://cloud.githubusercontent.com/assets/10060731/18757222/ea30add2-812d-11e6-97e8-b31199b15283.png"](https://cloud.githubusercontent.com/assets/10060731/18757222/ea30add2-812d-11e6-97e8-b31199b15283.png"){kind=link}
![https://cloud.githubusercontent.com/assets/10060731/18757229/ee6f330a-812d-11e6-84c9-db5fe4d5a35b.png"](https://cloud.githubusercontent.com/assets/10060731/18757229/ee6f330a-812d-11e6-84c9-db5fe4d5a35b.png"){kind=link}