Badger created TOREE-404:
----------------------------
Summary: Issues with SPARK_OPTS
Key: TOREE-404
URL: https://issues.apache.org/jira/browse/TOREE-404
Project: TOREE
Issue Type: Bug
Affects Versions: 0.2.0
Reporter: Badger
Hi
I’m using Toree 0.2.0.dev1 and I’ve spent the last day or so trying to
configure the kernel to talk to a cluster in yarn-client mode. I’m putting
together a docker image for our analysts so that they can talk directly to our
cluster. This is largely based around the jupyter docker-stacks docker images.
When configuring toree with :
jupyter toree install --spark_opts='--master yarn --deploy-mode client
--driver-memory 512m --executor-memory 512m --executor-cores 4
--driver-java-options -Dhdp.version=2.5.3.0-37 --conf
spark.hadoop.yarn.timeline-service.enabled=false'
I see the following in the kernel start up logs
17/04/13 12:34:37 [INFO] o.a.t.b.l.StandardComponentInitialization$$anon$1 -
Connecting to spark.master local[*]
I also tried the following:
jupyter toree install --spark_opts='--master yarn --deploy-mode client
--driver-memory 512m --executor-memory 512m --executor-cores 4
--driver-java-options -Dhdp.version=2.5.3.0-37 --conf
spark.hadoop.yarn.timeline-service.enabled=false spark.master=yarn-client'
After lots of digging I added the following to my spark-defaults.conf
spark.master=yarn-client
spark.hadoop.yarn.timeline-service.enabled=false
It seems that the options set by “conf” in the “spark_opts” are not used.
After setting these options in the spark-defaults.conf, I can see an
application registered as running in the hadoop node manager.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
