All,
I was wondering if any of you have solved this problem :
I have pyspark(ipython mode) running on docker talking to
a yarn cluster(AM/executors are NOT running on docker).
When I start pyspark in the docker container, it binds to port *49460.*
Once the app is submitted to YARN, the app(AM)
Hi Eron, Thanks for your reply, but none of these options works for us.
1. use 'spark.driver.host' and 'spark.driver.port' setting to
stabilize the driver-side endpoint. (ref
https://spark.apache.org/docs/latest/configuration.html#networking)
This unfortunately won't help since if
Trying spark-dev mailing list to see if anyone knows.
-- Forwarded message --
From: Ashwin Shankar ashwinshanka...@gmail.com
Date: Wed, Jun 3, 2015 at 5:38 PM
Subject: How to pass system properties in spark ?
To: u...@spark.apache.org u...@spark.apache.org
Hi,
I'm trying to use
Hi Spark devs/users,
One of the things we are investigating here at Netflix is if Spark would
suit us for our ETL needs, and one of requirements is multi tenancy.
I did read the official doc
http://spark.apache.org/docs/latest/job-scheduling.html and the book, but
I'm still not clear on certain
, Oct 22, 2014 at 11:47 AM, Ashwin Shankar
ashwinshanka...@gmail.com wrote:
Here are my questions :
1. Sharing spark context : How exactly multiple users can share the
cluster
using same spark
context ?
That's not something you might want to do usually. In general, a
SparkContext
