Options include:use 'spark.driver.host' and 'spark.driver.port' setting to
stabilize the driver-side endpoint. (ref)use host networking for your
container, i.e. "docker run --net=host ..."use yarn-cluster mode (see
SPARK-5162)
Hope this helps,Eron
Date: Wed, 10 Jun 2015 13:43:04 -0700
Subject: Problem with pyspark on Docker talking to YARN cluster
From: ashwinshanka...@gmail.com
To: dev@spark.apache.org; u...@spark.apache.org
All,I was wondering if any of you have solved this problem :
I have pyspark(ipython mode) running on docker talking toa yarn
cluster(AM/executors are NOT running on docker).
When I start pyspark in the docker container, it binds to port 49460.
Once the app is submitted to YARN, the app(AM) on the cluster side fails with
the following error message :ERROR yarn.ApplicationMaster: Failed to connect to
driver at :49460
This makes sense because AM is trying to talk to container directly andit
cannot, it should be talking to the docker host instead.
Question :How do we make Spark AM talk to host1:port1 of the docker host(not
the container), which would thenroute it to container which is running pyspark
on host2:port2 ?
One solution I could think of is : after starting the driver(say on
hostA:portA), and before submitting the app to yarn, we could reset driver's
host/port to hostmachine's ip/port. So the AM can then talk hostmachine's
ip/port, which would be mappedto the container.
Thoughts ? --
Thanks,
Ashwin
