rongou commented on issue #23347: allow building spark gpu docker images URL: https://github.com/apache/spark/pull/23347#issuecomment-453706000 @mccheah with the latest commit this now uses debian slim as the base image for CPUs. The total size is within striking distance of alpine-based image. I think it was decided that Spark is not going to publish images to docker hub because of licensing constraints. So the shell script and dockerfiles are more of an example than a complete deployment solution. The scenario is different from those projects that support different distributions of linux with the explicit goal to provide ready-to-pull base images. GPU support is a little trickier because the driver cannot be easily added to all base images, for example, Alpine doesn't support it at all. The easiest approach is to use one of the `nvidia/cuda` images as base, which only support Ubuntu and CentOS. In comparison, other packages (e.g. instrumentation tools) can be easily added as a new layer on top of the base Spark image.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
