[ https://issues.apache.org/jira/browse/TOREE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ribamar Santarosa updated TOREE-438: ------------------------------------ Description: It looks like the TOREE-97 issue -- support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user don't add manually in their kernel.json definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark doesn't know what to do with the option {{--master=yarn}} (set in {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default, and this patch provides this functionality. Probably this is not the nicest way to solve the problem, because it just hard codes more vars into the JSON file -- ideally it would be nice to have an interface to add or remove env vars from those files, however, {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here it goes our 2 cents to improve a bit the situation. I cloned the TOREE-97 into TOREE-438 to sign this issue. was: It looks like the TOREE-97 issue -- support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user don't add manually in their kernel.json definition, the env vars for `HADOOP_CONF_DIR`. Without that env var, Spark doesn't know what to do with the option `--master=yarn` (set in `__TOREE_SPARK_OPTS__`). It would be desirable to have it by default, and this patch provides this functionality. Probably this is not the nicest way to solve the problem, because it just hard codes more vars into the JSON file -- ideally it would be nice to have an interface to add or remove env vars from those files, however, `HADOOP_CONF_DIR` and `SPARK_CONF_DIR` look basic to be exported. Even for an Spark Standalone deployment, `HADOOP_CONF_DIR` won't hurt. So, here it goes our 2 cents to improve a bit the situation. I cloned the TOREE-97 into TOREE-438 to sign this issue. > CLONE - How to support Spark on Yarn model? > ------------------------------------------- > > Key: TOREE-438 > URL: https://issues.apache.org/jira/browse/TOREE-438 > Project: TOREE > Issue Type: Bug > Reporter: Ribamar Santarosa > > It looks like the TOREE-97 issue -- support for Spark Yarn was closed without > definitive solution (or something went wrong on the way). Toree does support > it, but it won't work if a user don't add manually in their kernel.json > definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark > doesn't know what to do with the option {{--master=yarn}} (set in > {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default, and > this patch provides this functionality. > Probably this is not the nicest way to solve the problem, because it just > hard codes more vars into the JSON file -- ideally it would be nice to have > an interface to add or remove env vars from those files, however, > {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even > for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here > it goes our 2 cents to improve a bit the situation. > I cloned the TOREE-97 into TOREE-438 to sign this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)