Yup, it’s also important to have low latency between the drivers and the workers. If you plan to expose this to the outside (e.g. offer a shell interface), it would be better to write something on top.
Matei On Nov 24, 2013, at 3:17 PM, Patrick Wendell <[email protected]> wrote: > Or more generally, it should be fully visible to the slave nodes > inside of the cluster. > > On Sun, Nov 24, 2013 at 3:16 PM, Patrick Wendell <[email protected]> wrote: >> Currently, no. The driver is a stateful component that is heavy-weight >> and should be run inside of the cluster. >> >> On Fri, Nov 22, 2013 at 4:49 PM, Dmitriy Lyubimov <[email protected]> wrote: >>> Hello, >>> >>> as far as i can tell, spark executors use akka to connect back to the >>> driver. >>> >>> However, if driver is behind NAT, it becomes impossible since the tcp >>> connections are flowing from workers to the driver. >>> >>> Is there any known way to set up spark clients behind NAT? >>> >>> thanks. -Dmitriy
