Unfortunately Mesos cluster manager is not supported currently for Livy, we only tested on local and yarn mode.
On Wed, Oct 25, 2017 at 8:47 PM, Junaid Nasir <[email protected]> wrote: > livy version 0.4 > trying to create interactive session. may be it's not supported with mesos? > > > > On Wed, Oct 25, 2017 3:55 PM, Junaid Nasir [email protected] wrote: > >> Hi, >> >> I am trying to run Livy with DC/OS Mesos. >> it tries to create driver but that fails with error blow. Is there a way >> i can provide it with spark-internal file? I am unable to find what this >> spark-internal file does and i am unable to find it anywhere. only thing i >> found related to this was in ContextLauncher.java line 232 ` >> launcher.setAppResource("spark-internal");` >> >> >> Failed to fetch spark-internal >> >> I1025 10:41:45.202145 27901 fetcher.cpp:531] Fetcher Info: >> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"spark-internal"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14\/frameworks\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-0003\/executors\/driver-20171025104145-0034\/runs\/9976b5de-0ed7-428e-a23c-59056c6f29ed"} >> I1025 10:41:45.204314 27901 fetcher.cpp:442] Fetching URI 'spark-internal' >> I1025 10:41:45.204329 27901 fetcher.cpp:283] Fetching directly into the >> sandbox directory >> I1025 10:41:45.204346 27901 fetcher.cpp:220] Fetching URI 'spark-internal' >> Failed to fetch 'spark-internal': A relative path was passed for the >> resource but the Mesos framework home was not specified. Please either >> provide this config option or avoid using a relative path >> >> [image: Mixmax] >> <https://mixmax.com/r/592fe3b429b79365389d2354> Not using Mixmax yet? >> <https://mixmax.com/r/592fe3b429b79365389d2354> >> >> >> Spark-submit works fine I am unable to run a job with following command >> >> bin/spark-submit --class org.apache.spark.examples.SparkPi --master >> mesos://10.128.0.23:22401 --deploy-mode cluster --supervise >> --executor-memory 1G --total-executor-cores 1 --conf >> spark.mesos.executor.docker.image=mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6 >> --conf spark.executor.home=/opt/spark/dist https://downloads.mesosphere. >> com/spark/examples/pi.py 30 >> >> >> I have set spark master to the same ip and set cluster mode. sending post >> request to session with these parameters >> >> data = {'kind': 'spark','conf':{'spark.mesos.executor.docker.image':' >> mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6 >> ','spark.executor.home':'/opt/spark/dist'}} >> >> >> >> Any help would be highly appreciated >> >
