thomasrebele commented on code in PR #6825: URL: https://github.com/apache/hive/pull/6825#discussion_r4183521156
########## standalone-metastore/metastore-server/docker/hive-postgres-tpcds-metastore/Dockerfile: ########## @@ -14,9 +14,16 @@ # See the License for the specific language governing permissions and # limitations under the License. # -FROM postgres:12.3 +FROM postgres:18-alpine + +ADD https://github.com/thomasrebele/hive-postgres-metastore/releases/download/tpcds-30tb-histogram-1.0/metastore_tpcds30tb_with_histograms.raw_db.zstd /tmp/metastore_db.zstd Review Comment: Nice catch. I had in mind that Docker only stores the changes between the commands. The chown changes the owner, but the command that stores the file (tar?) will store the whole file again. I tried Option 2, and it reduces the image quite a lot: ``` $ podman image tree localhost/postgres-tpcds-metastore:1.5-no-entrypoint Image ID: 20b9602f63e2 Tags: [localhost/postgres-tpcds-metastore:1.5-no-entrypoint] Size: 306.7MB Image Layers ├── ID: 74d97c428c51 Size: 8.704MB ├── ID: 93ceee9c2e8d Size: 11.26kB ├── ID: 9398d68c8e63 Size: 1.99MB ├── ID: 98333e9d0aac Size: 1.536kB ├── ID: 93efeacdff4d Size: 295.8MB ├── ID: 2bfbe60cf6cc Size: 69.63kB ├── ID: ecbda7552d59 Size: 2.048kB ├── ID: 455094a58b81 Size: 20.99kB ├── ID: 2ba5c14f2a6e Size: 3.072kB Top Layer of: [docker.io/library/postgres:18-alpine] ├── ID: fb554da95cc0 Size: 7.168kB ├── ID: 83b315da97c7 Size: 5.12kB └── ID: 06a1693b8fa6 Size: 8.704kB Top Layer of: [localhost/postgres-tpcds-metastore:1.5-no-entrypoint] ``` The difference between postgres:18-alpine and the final image is just a few MB, which is suspiciously low. I logged into the DB, and I could not find the metastore DB. I used podman. Have you used docker? Could you check whether the container created by option 2 contains the DB? Has Docker switched to zstd? I was curious how big the image would be, so I copied it to another dir. <details><summary>Option 2 workaround dockerfile</summary> <p> ``` FROM postgres:18-alpine RUN wget -O /tmp/metastore_db.zstd https://github.com/thomasrebele/hive-postgres-metastore/releases/download/tpcds-30tb-histogram-1.0/metastore_tpcds30tb_with_histograms.raw_db.zstd \ && chown postgres:postgres /tmp/metastore_db.zstd \ && mkdir -p /tmp/extracted \ && zstdcat /tmp/metastore_db.zstd | tar -C /tmp/extracted/ -x \ && rm /tmp/metastore_db.zstd COPY postgresql.conf /etc/postgresql/postgresql.conf RUN chown postgres:postgres /etc/postgresql/postgresql.conf CMD ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"] ``` </p> </details> Image sizes: ``` separate-steps 1.44 GB (Dockerfile as in the PR) combined-download-chown 875 MB (option 1) no-entrypoint 307 MB (option 2 as-is, but broken) no-entrypoint-cp-db 1.54 GB (option 2 workaround to get image size) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
