> 1. The size regression: `apache/spark:3.4.0` tag which is claimed to be a replacement of the existing `apache/spark:v3.4.0`. However, 3.4.0 is 500MB while the original v3.4.0 is 405MB. 25% is huge in terms of the size.
> 2. Accidental overwrite: `apache/spark:latest` was accidentally overwritten by `apache/spark:python3` image which has a bigger size due to the additional python binary. This is a breaking change to enforce the downstream users to change to something like `apache/spark:scala`. Just FYI, we also had a discussion about tag policy (latest/3.4.0) and also rough size estimation [1] in "SPIP: Support Docker Official Image for Spark". [1] https://docs.google.com/document/d/1nN-pKuvt-amUcrkTvYAQ-bJBgtsWb9nAkNoVNRM2S2o/edit?disco=AAAAf2TyFr0 Regards, Yikun On Tue, May 9, 2023 at 5:03 AM Dongjoon Hyun <dongj...@apache.org> wrote: > Thank you for initiating the discussion in the community. Yes, we need to > give more context in the dev mailing list. > > This root cause is not about SPARK-40941 or SPARK-40513. Technically, this > situation started 16 days ago due to SPARK-43148 because it made some > breaking changes. > > https://github.com/apache/spark-docker/pull/33 > SPARK-43148 Add Apache Spark 3.4.0 Dockerfiles > > 1. The size regression: `apache/spark:3.4.0` tag which is claimed to be a > replacement of the existing `apache/spark:v3.4.0`. However, 3.4.0 is 500MB > while the original v3.4.0 is 405MB. 25% is huge in terms of the size. > > 2. Accidental overwrite: `apache/spark:latest` was accidentally > overwritten by `apache/spark:python3` image which has a bigger size due to > the additional python binary. This is a breaking change to enforce the > downstream users to change to something like `apache/spark:scala`. > > I believe (1) and (2) were our mistakes. We had better recover them ASAP. > For Java questions, I prefer to be consistent with Apache Spark repo's > default. > > Dongjoon. > > On 2023/05/08 08:56:26 Yikun Jiang wrote: > > This is a call for discussion for how we can unified Apache Spark Docker > > image tag fluently. > > > > As you might know, there is an apache/spark-docker > > <https://github.com/apache/spark-docker> repo to store the dockerfiles > and > > help to publish the docker images, also intended to replace the original > > manually publish workflow. > > > > The scope of new images is to cover previous image cases (K8s / docker > run) > > and also cover base image, standalone, Docker Official Image. > > > > - (Previous) apache/spark:v3.4.0, apache/spark-py:v3.4.0, > > apache/spark-r:v3.4.0 > > > > * The image build from apache/spark spark on k8s dockerfiles > > < > https://github.com/apache/spark/tree/branch-3.4/resource-managers/kubernetes/docker/src/main/dockerfiles/spark > > > > > > * Java version: Java 17 (It was Java 11 before v3.4.0, such as > > v3.3.0/v3.3.1/v3.3.2), set Java 17 by default in SPARK-40941 > > <https://github.com/apache/spark/pull/38417>. > > > > * Support: K8s / docker run > > > > * See also: Time to start publishing Spark Docker Images > > <https://lists.apache.org/thread/h729bxrf1o803l4wz7g8bngkjd56y6x8> > > > > * Link: https://hub.docker.com/r/apache/spark-py, > > https://hub.docker.com/r/apache/spark-r, > > https://hub.docker.com/r/apache/spark > > > > - (New) apache/spark:3.4.0-python3(3.4.0/latest), apache/spark:3.4.0-r, > > apache/spark:3.4.0-scala, and also a all in one image: > > apache/spark:3.4.0-scala2.12-java11-python3-r-ubuntu > > > > * The image build from apache/spark-docker dockerfiles > > <https://github.com/apache/spark-docker/tree/master/3.4.0> > > > > * Java version: Java 11, Java17 is supported by SPARK-40513 > > <https://github.com/apache/spark-docker/pull/35> (under review) > > > > * Support: K8s / docker run / base image / standalone / Docker > Official > > Image > > > > * See detail in: Support Docker Official Image for Spark > > <https://issues.apache.org/jira/browse/SPARK-40513> > > > > * About dropping prefix `v`: > > https://github.com/docker-library/official-images/issues/14506 > > > > * Link: https://hub.docker.com/r/apache/spark > > > > We had some initial discuss on spark-website#458 > > < > https://github.com/apache/spark-website/pull/458#issuecomment-1522426236>, > > the mainly discussion is around version tag and default Java version > > behavior changes, so we’d like to hear your idea in here about below > > questions: > > > > *#1.Which Java version should be used by default (latest tag)? Java8 or > > Java 11 or Java 17 or Any* > > > > *#2.Which tag should be used in apache/spark? v3.4.0 (with prefix v) or > > 3.4.0 (dropping prefix v) or Both or Any* > > > > Starts with my prefer: > > > > 1. Java8 or Java17 are also ok to me (mainly considering the Java > > maintenance cycle). BTW, other apache projects: flink (8/11, 11 as > default > > < > https://github.com/docker-library/official-images/blob/93270eb07fb448fe7756b28af5495428242dcd6b/library/flink#L10 > >), > > solr (11 as default > > < > https://github.com/apache/solr-docker/blob/989825ee6dce2f6bf7b31051f1ba053b6c4426f2/8.11/Dockerfile#L4 > > > > for 8.x, 17 as default > > < > https://github.com/apache/solr-docker/blob/989825ee6dce2f6bf7b31051f1ba053b6c4426f2/9.2/Dockerfile#L17 > > > > since solr9), zookeeper (11 as default > > < > https://github.com/31z4/zookeeper-docker/blob/181e5862c85b517e4599d79eb5c2c7339e60a4aa/3.8.1/Dockerfile#L1 > > > > ) > > > > 2. Only 3.4.0 (dropping prefix v). It will help us transition to the new > > tags with less confusion and also consider DOI suggestions > > <https://github.com/docker-library/official-images/issues/14506>. > > > > Please feel free to share your ideas. > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
