I am able to run spark shell from command line using command
/spark/dist/bin/pyspark --master mesos://leader.mesos:5050 --conf
spark.mesos.executor.docker.image=mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6
--conf spark.executor.home=/opt/spark/dist
I have changed livy's configuration accordingly i.e set master to leader.mesos
when i send post request to /sessions with input data.{'kind': 'spark', 'conf':
{'spark.mesos.executor.docker.image':
'mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6', 'spark.executor.home':
'/opt/spark/dist'}}it produces these errors on livy console

                  17/10/25 13:00:00 INFO InteractiveSession$: Creating 
Interactive session 0: [owner: null, request: [kind: spark, proxyUser: None, 
conf: spark.mesos.executor.docker.image -> 
mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6,spark.executor.home -> 
/opt/spark/dist, heartbeatTimeoutInSecond: 0]]17/10/25 13:00:00 INFO RpcServer: 
Connected to the port 1000017/10/25 13:00:00 WARN RSCConf: Your hostname, 
agent0005.c.calm-airship-162807.internal, resolves to a loopback address, but 
we couldn't find any external IP address!17/10/25 13:00:00 WARN RSCConf: Set 
livy.rsc.rpc.server.address if you need to bind to another address.17/10/25 
13:00:00 INFO LineBufferedStream: stdout: spark-env: User: root17/10/25 
13:00:00 INFO LineBufferedStream: stdout: Skipping bootstrap IP 
detection17/10/25 13:00:00 INFO LineBufferedStream: stdout: spark-env: No 
kerberos KDC config found17/10/25 13:00:00 INFO InteractiveSessionManager: 
Registering new session 017/10/25 13:00:01 INFO LineBufferedStream: stdout: 
17/10/25 13:00:01 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable17/10/25 
13:00:01 INFO LineBufferedStream: stdout: 17/10/25 13:00:01 INFO RSCDriver: 
Connecting to: agent0005.c.calm-airship-162807.internal:1000017/10/25 13:00:01 
INFO LineBufferedStream: stdout: 17/10/25 13:00:01 INFO RSCDriver: Starting RPC 
server...17/10/25 13:00:01 INFO LineBufferedStream: stdout: 17/10/25 13:00:01 
INFO RpcServer: Connected to the port 1000117/10/25 13:00:01 INFO 
LineBufferedStream: stdout: 17/10/25 13:00:01 WARN RSCConf: Your hostname, 
agent0005.c.calm-airship-162807.internal, resolves to a loopback address, but 
we couldn't find any external IP address!17/10/25 13:00:01 INFO 
LineBufferedStream: stdout: 17/10/25 13:00:01 WARN RSCConf: Set 
livy.rsc.rpc.server.address if you need to bind to another address.17/10/25 
13:00:02 INFO LineBufferedStream: stdout: 17/10/25 13:00:02 INFO RSCDriver: 
Received job request fcbdd93f-fcd7-453d-9d0f-c303a687c7ae17/10/25 13:00:02 INFO 
LineBufferedStream: stdout: 17/10/25 13:00:02 INFO RSCDriver: SparkContext not 
yet up, queueing job request.17/10/25 13:00:04 WARN RSCClient: Client RPC 
channel closed unexpectedly.17/10/25 13:00:04 WARN ContextLauncher: Child 
process exited with code 137.17/10/25 13:00:04 WARN RSCClient: Error stopping 
RPC.io.netty.util.concurrent.BlockingOperationException: 
DefaultChannelPromise@1831bad3(uncancellable)        at 
io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)  
      at 
io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
        at 
io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)        
at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129) 
       at 
io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)     
   at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218)     
   at 
io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)     
   at 
io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)      
  at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:307)        at 
org.apache.livy.rsc.RSCClient.stop(RSCClient.java:232)        at 
org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:129)        at 
org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:123)        at 
org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108)        at 
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
        at 
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
        at 
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)     
   at 
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at 
io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956)
        at 
io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608)
        at 
io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586) 
       at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
        at 
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
        at 
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)     
   at 
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at 
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)    
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)        at 
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        at java.lang.Thread.run(Thread.java:745)17/10/25 13:00:04 ERROR 
SparkProcApp: spark-submit exited with code 13717/10/25 13:00:04 INFO 
RSCClient: Failing pending job fcbdd93f-fcd7-453d-9d0f-c303a687c7ae due to 
shutdown.17/10/25 13:00:04 INFO InteractiveSession: Failed to ping RSC driver 
for session 0. Killing application.17/10/25 13:00:04 INFO InteractiveSession: 
Stopping InteractiveSession 0...17/10/25 13:00:04 INFO InteractiveSession: 
Stopped InteractiveSession 0.
                

Not using Mixmax yet?  
i noticed the line Set livy.rsc.rpc.server.address if you need to bind to
another address. have tried setting this in conf/livy-client with livy's server
local ip address. but error remains.
any help would be great
Thanks for your time  





On Wed, Oct 25, 2017 5:47 PM, Junaid Nasir jna...@an10.io  wrote:
livy version 0.4trying to create interactive session. may be it's not supported
with mesos?  





On Wed, Oct 25, 2017 3:55 PM, Junaid Nasir jna...@an10.io  wrote:
Hi,
I am trying to run Livy with DC/OS Mesos.it tries to create driver but that
fails with error blow. Is there a way i can provide it with spark-internal file?
I am unable to find what this spark-internal file does and i am unable to find
it anywhere. only thing i found related to this was in ContextLauncher.java line
232 ` launcher.setAppResource("spark-internal");`

Failed to fetch spark-internal                   I1025 10:41:45.202145 27901 
fetcher.cpp:531] Fetcher Info: 
{"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14","items":[{"action":"BYPASS_CACHE","uri":{"cache":false,"extract":true,"value":"spark-internal"}}],"sandbox_directory":"\/var\/lib\/mesos\/slave\/slaves\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-S14\/frameworks\/babc8653-5a13-4fca-9fd9-fe2870f9ad50-0003\/executors\/driver-20171025104145-0034\/runs\/9976b5de-0ed7-428e-a23c-59056c6f29ed"}I1025
 10:41:45.204314 27901 fetcher.cpp:442] Fetching URI 'spark-internal'I1025 
10:41:45.204329 27901 fetcher.cpp:283] Fetching directly into the sandbox 
directoryI1025 10:41:45.204346 27901 fetcher.cpp:220] Fetching URI 
'spark-internal'Failed to fetch 'spark-internal': A relative path was passed 
for the resource but the Mesos framework home was not specified. Please either 
provide this config option or avoid using a relative path
                

Not using Mixmax yet?  

Spark-submit works fine I am unable to run a job with following command
bin/spark-submit --class org.apache.spark.examples.SparkPi --master
mesos://10.128.0.23:22401 --deploy-mode cluster --supervise --executor-memory 1G
--total-executor-cores 1 --conf
spark.mesos.executor.docker.image=mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6
--conf spark.executor.home=/opt/spark/dist
https://downloads.mesosphere.com/spark/examples/pi.py 30
I have set spark master to the same ip and set cluster mode. sending post
request to session with these parametersdata = {'kind':
'spark','conf':{'spark.mesos.executor.docker.image':'mesosphere/spark:2.0.1-2.2.0-1-hadoop-2.6
','spark.executor.home':'/opt/spark/dist'}} 
Any help would be highlyappreciated

Reply via email to