Re: Spark in docker over EC2

[Teng Qiu]

Hi, you can take a look at this project, it is a distributed HA Spark
cluster for AWS environment using Docker, we put the spark ec2
instances in an ELB, and using this code snippet to get the instance
IPs: 
https://github.com/zalando-incubator/spark-appliance/blob/master/utils.py#L49-L56

Dockerfil: 
https://github.com/zalando-incubator/spark-appliance/blob/master/Dockerfile

2017-01-11 0:49 GMT+01:00 Darren Govoni :
> Anyone got a good guide for getting spark master to talk to remote workers
> inside dockers? I followed the tips found by searching but doesn't work
> still. Spark 1.6.2.
>
> I exposed all the ports and tried to set local IP inside container to the
> host IP but spark complains it can't bind ui ports.
>
> Thanks in advance!
>
>
>
> Sent from my Verizon, Samsung Galaxy smartphone

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Spark in docker over EC2

[Darren Govoni]

Anyone got a good guide for getting spark master to talk to remote workers 
inside dockers? I followed the tips found by searching but doesn't work still. 
Spark 1.6.2.
I exposed all the ports and tried to set local IP inside container to the host 
IP but spark complains it can't bind ui ports.
Thanks in advance!


Sent from my Verizon, Samsung Galaxy smartphone