mccheah commented on issue #23347: allow building spark gpu docker images URL: https://github.com/apache/spark/pull/23347#issuecomment-453642925 Actually we can take a step back here and consider the premise of the change... I'm not entirely sure if we want to be opinionated towards supporting GPU features in a first-class way in Spark. If we do that, then what's to stop asks for more features for their custom images, to be built into the Spark ones? For example, how do we decide whether or not we want to include support for particular instrumentation tools? And then as @vanzin has pointed out as well, what if someone wants to use GPU support but doesn't want to use this distribution of Linux? What about other kinds of distributions of Linux that use a different package manager? That requires a different set of Dockerfiles altogether. To me, the docker image tool and the dockerfile we provide out of the box should be used for the more basic use cases. But the flexibility of Kubernetes allows any user to build their own image however they wish. I think if we optimize for the more general case of building custom images, we'll better support these kinds of features in images without having to make huge Dockerfiles with lots of options.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
