[ https://issues.apache.org/jira/browse/TOREE-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ribamar Santarosa updated TOREE-438: ------------------------------------ Description: It looks like the TOREE-97 issue -- support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user doesn't add manually in their kernel.json definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark doesn't know what to do with the option {{--master=yarn}} (set in {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default. Probably this is not the nicest way to solve the problem, because it just hard codes more vars into the JSON file -- ideally it would be nice to have an interface to add or remove env vars from those files, however, {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here it goes our 2 cents to improve a bit the situation. I cloned the TOREE-97 into TOREE-438 to sign this issue. was: It looks like the TOREE-97 issue -- support for Spark Yarn was closed without definitive solution (or something went wrong on the way). Toree does support it, but it won't work if a user don't add manually in their kernel.json definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark doesn't know what to do with the option {{--master=yarn}} (set in {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default. Probably this is not the nicest way to solve the problem, because it just hard codes more vars into the JSON file -- ideally it would be nice to have an interface to add or remove env vars from those files, however, {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here it goes our 2 cents to improve a bit the situation. I cloned the TOREE-97 into TOREE-438 to sign this issue. > CLONE - How to support Spark on Yarn model? > ------------------------------------------- > > Key: TOREE-438 > URL: https://issues.apache.org/jira/browse/TOREE-438 > Project: TOREE > Issue Type: Bug > Reporter: Ribamar Santarosa > > It looks like the TOREE-97 issue -- support for Spark Yarn was closed without > definitive solution (or something went wrong on the way). Toree does support > it, but it won't work if a user doesn't add manually in their kernel.json > definition, the env vars for {{HADOOP_CONF_DIR}}. Without that env var, Spark > doesn't know what to do with the option {{--master=yarn}} (set in > {{__TOREE_SPARK_OPTS__}}). It would be desirable to have it by default. > Probably this is not the nicest way to solve the problem, because it just > hard codes more vars into the JSON file -- ideally it would be nice to have > an interface to add or remove env vars from those files, however, > {{HADOOP_CONF_DIR}} and {{SPARK_CONF_DIR}} look basic to be exported. Even > for an Spark Standalone deployment, {{HADOOP_CONF_DIR}} won't hurt. So, here > it goes our 2 cents to improve a bit the situation. > I cloned the TOREE-97 into TOREE-438 to sign this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)