thanks Saisai for the reply, can you point to something looking at these
logs?. Am stuck at this problem for quit a long now.I think it's related to rpc
server not being able to talk to livy (or the other way around). am not sure how
that works tbh.
On Wed, Oct 25, 2017 6:09 PM, Saisai Shao [email protected] wrote:
Unfortunately Mesos cluster manager is not supported currently for Livy, we only
tested on local and yarn mode.
On Wed, Oct 25, 2017 at 8:47 PM, Junaid Nasir <[email protected]> wrote:
livy version 0.4trying to create interactive session. may be it's not supported
with mesos?
On Wed, Oct 25, 2017 3:55 PM, Junaid Nasir [email protected] wrote:
Hi,
I am trying to run Livy with DC/OS Mesos.it tries to create driver but that
fails with error blow. Is there a way i can provide it with spark-internal file?
I am unable to find what this spark-internal file does and i am unable to find
it anywhere. only thing i found related to this was in ContextLauncher.java line
232 ` launcher.setAppResource("spark-internal");`
Failed to fetch spark-internal I1025 10:41:45.202145 27901 fetcher.cpp:531] Fetcher Info:
{"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"spark-internal"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14\/frameworks\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-0003\/executors\/driver-20171025104145-0034\/runs\/9976b5de-0ed7-428e-a23c-59056c6f29ed"}I1025
10:41:45.204314 27901 fetcher.cpp:442] Fetching URI 'spark-internal'I1025 10:41:45.204329 27901 fetcher.cpp:283] Fetching directly into the sandbox directoryI1025 10:41:45.204346 27901
fetcher.cpp:220] Fetching URI 'spark-internal'Failed to fetch 'spark-internal': A relative path was passed for the resource but the Mesos framework home was not specified. Please either provide this
config option or avoid using a relative path
Not using Mixmax yet?
Spark-submit works fine I am unable to run a job with following command
bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://
10.128.0.23:22401 --deploy-mode cluster --supervise --executor-memory 1G
--total-executor-cores 1 --conf spark.mesos.executor.docker.
image=mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6 --conf spark.executor.home=/opt/
spark/dist https://downloads.mesosphere.com/spark/examples/pi.py 30
I have set spark master to the same ip and set cluster mode. sending post
request to session with these parametersdata = {'kind':
'spark','conf':{'spark.mesos.executor.docker.image':'
mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6 ','spark.executor.home':'/opt/
spark/dist'}}
Any help would be highlyappreciated
