Hi,
I am trying to setup my windows machine as a remote client by installing
Jupyter+toree to use Spark cluster which is on cloud(IBM Bluemix).
I have installed jupyter+toree. I have also extracted Spark 1.6 on my
windows machine…
When I run Jupyter, its works with Python3 kernel, however the Toree Scala
Kernel fails to work.
My objective is to make this Toree-Scala kernel to work with the Spark
Cloud instance.
Where do I need to change the configuration?(my assumption is local
spark-default.conf)
Note: My Spark Cloud instance is configured to use Yarn
My local spark-defaul.conf looks like
# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.
# Example:
# spark.master spark://master:7077
# spark.eventLog.enabled true
# spark.eventLog.dir hdfs://namenode:8021/directory
# spark.serializer
org.apache.spark.serializer.KryoSerializer
# spark.driver.memory 5g
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value
-Dnumbers="one two three"
Spark Cloud instance spark-default.conf looks like :
spark.yarn.applicationMaster.waitTries 10
spark.yarn.submit.file.replication 3
spark.history.kerberos.principal none
spark.yarn.historyServer.address xxxxxxxxxxxxx:18080
spark.yarn.queue default
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.history.kerberos.keytab none
spark.driver.extraJavaOptions -Diop.version={{iop_full_version}}
spark.eventLog.dir hdfs://xxxxxxxxxxxx/iop/apps/4.1.0.0/spark/logs/
spark.history.ui.port 18080
spark.yarn.preserve.staging.files false
spark.yarn.driver.memoryOverhead 384
spark.yarn.containerLauncherMaxThreads 25
spark.eventLog.enabled true
spark.yarn.max.executor.failures 3
spark.yarn.jar hdfs://xxxxxxxxxxxx:8020/iop/apps/
4.1.0.0/spark/jars/spar
spark.yarn.am.extraJavaOptions -Diop.version={{iop_full_version}}
spark.yarn.executor.memoryOverhead 384
where xxxxxxxxx was the actual IP address.
--
Vinayak Agrawal
"To Strive, To Seek, To Find and Not to Yield!"
~Lord Alfred Tennyson
