This is neat. With some code cleanup and as long as users can still use custom driver/executor/init-container images if they want to, I think this is great to have. I don't think there's a particular reason why having a single image wouldn't work. Thanks for doing this!
On Fri, Jan 5, 2018 at 5:06 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > Hey all, especially those working on the k8s stuff. > > Currently we have 3 docker images that need to be built and provided > by the user when starting a Spark app: driver, executor, and init > container. > > When the initial review went by, I asked why do we need 3, and I was > told that's because they have different entry points. That never > really convinced me, but well, everybody wanted to get things in to > get the ball rolling. > > But I still think that's not the best way to go. I did some pretty > simple hacking and got things to work with a single image: > > https://github.com/vanzin/spark/commit/k8s-img > > Is there a reason why that approach would not work? You could still > create separate images for driver and executor if wanted, but there's > no reason I can see why we should need 3 images for the simple case. > > Note that the code there can be cleaned up still, and I don't love the > idea of using env variables to propagate arguments to the container, > but that works for now. > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
