GitHub user ribamar-santarosa opened a pull request:
https://github.com/apache/incubator-toree/pull/141
fix kernel generation for Spark Yarn // TOREE-97
It looks like the TOREE-97 issue -- support for Spark Yarn was closed
without definitive solution (or something went wrong on the way). Toree does
support it, but it won't work if a user don't add manually in their kernel.json
definition, the env vars for `HADOOP_CONF_DIR`. Without that env var, Spark
doesn't know what to do with the option `--master=yarn` (set in
`__TOREE_SPARK_OPTS__`). It would be desirable to have it by default, and this
patch provides this functionality.
Probably this is not the nicest way to solve the problem, because it just
hard codes more vars into the JSON file -- ideally it would be nice to have an
interface to add or remove env vars from those files, however,
`HADOOP_CONF_DIR` and `SPARK_CONF_DIR` look basic to be exported. Even for an
Spark Standalone deployment, `HADOOP_CONF_DIR` won't hurt. So, here it goes
our 2 cents to improve a bit the situation.
I cloned the TOREE-97 into TOREE-438 to sign this issue.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Bright-Computing/incubator-toree master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-toree/pull/141.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #141
----
commit 688e96e75e306bc9172fdc49bb143f8cf67ceb11
Author: Ribamar Santarosa <[email protected]>
Date: 2017-09-11T12:35:08Z
fix kernel generation for Spark Yarn // TOREE-97
----
---
