shahar1 commented on issue #59625:
URL: https://github.com/apache/airflow/issues/59625#issuecomment-3707309256
> r: cannot execute: required file not found
This
> I tested the Debian-based hardened image with Python 3.12
(dhi.io/python:3.12-debian12-dev) locally by switching the base image in
Dockerfile.ci.
>
> The CI build starts successfully but fails during
install_os_dependencies.sh while configuring `usrmerge`.
>
> ```
> docker build -f Dockerfile.ci -t airflow-ci-hardened-test .
> [+] Building 75.3s (32/53)
docker:default
> => [internal] load build definition from Dockerfile.ci
0.4s
> => => transferring dockerfile: 75.89kB
0.3s
> => resolve image config for
docker-image://docker.io/docker/dockerfile:1.4
2.9s
> => [auth] docker/dockerfile:pull token for registry-1.docker.io
0.0s
> => CACHED
docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84
0.1s
> => => resolve
docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1
0.0s
> => [internal] load .dockerignore
0.2s
> => => transferring context: 3.89kB
0.1s
> => [internal] load metadata for dhi.io/python:3.12-debian12-dev
5.5s
> => [auth] python:pull token for dhi.io
0.0s
> => CACHED [internal] preparing inline document
0.0s
> => CACHED [internal] preparing inline document
0.0s
> => CACHED [internal] preparing inline document
0.1s
> => CACHED [internal] preparing inline document
0.0s
> => CACHED [internal] preparing inline document
0.1s
> => CANCELED [internal] load build context
65.5s
> => => transferring context: 121.58MB
65.3s
> => CACHED [internal] preparing inline document
0.0s
> => [main 1/25] FROM
dhi.io/python:3.12-debian12-dev@sha256:e900ff32403cd549308c26b86483cdbc01aeb98a94d8e84f3a7
16.4s
> => => resolve
dhi.io/python:3.12-debian12-dev@sha256:e900ff32403cd549308c26b86483cdbc01aeb98a94d8e84f3a7f3580de9
0.0s
> => =>
sha256:af1b037a09005a11d133c859b40eaf8c0d6be6170f63c415bbc828aec87bfb5b 6.41kB
/ 6.41kB 1.4s
> => =>
sha256:9be530e8229a70fed5cb8cbea4ed0809482bea28b076f891f75fa88fd0d1020a 19.07kB
/ 19.07kB 2.1s
> => =>
sha256:08ae8e6b4b480889462ac05cc0da081fc5011a236279bebc562c2f84996feac6 3.62MB
/ 3.62MB 2.5s
> => =>
sha256:b60d3afc1edd3461aaeaacd5dcb1dc333326f7f29d4d206d8b8f44f63b4b35e3 9.62MB
/ 9.62MB 10.3s
> => =>
sha256:ed0e31a07806c0754ff48a7f1986653e19ed3e0ac2e2c986804e363970f2044d 454B /
454B 2.2s
> => =>
sha256:059b1d91f61528c5f8f36516ba9d602d641ef23d544536b80a3c300a1ed7ad5e
181.09kB / 181.09kB 1.2s
> => =>
sha256:c7ea417bd788fb4fafe9128b8c1b47546cacd2bdf8290137d81cea05cee503bd 26.13MB
/ 26.13MB 9.5s
> => => extracting
sha256:059b1d91f61528c5f8f36516ba9d602d641ef23d544536b80a3c300a1ed7ad5e
0.2s
> => => extracting
sha256:c7ea417bd788fb4fafe9128b8c1b47546cacd2bdf8290137d81cea05cee503bd
2.5s
> => => extracting
sha256:ed0e31a07806c0754ff48a7f1986653e19ed3e0ac2e2c986804e363970f2044d
0.1s
> => => extracting
sha256:b60d3afc1edd3461aaeaacd5dcb1dc333326f7f29d4d206d8b8f44f63b4b35e3
1.0s
> => => extracting
sha256:08ae8e6b4b480889462ac05cc0da081fc5011a236279bebc562c2f84996feac6
0.2s
> => => extracting
sha256:9be530e8229a70fed5cb8cbea4ed0809482bea28b076f891f75fa88fd0d1020a
0.1s
> => => extracting
sha256:af1b037a09005a11d133c859b40eaf8c0d6be6170f63c415bbc828aec87bfb5b
0.1s
> => CACHED [internal] preparing inline document
0.0s
> => CACHED [internal] preparing inline document
0.0s
> => CACHED [internal] preparing inline document
0.0s
> => CACHED [internal] preparing inline document
0.0s
> => [scripts 2/11] COPY <<EOF /install_os_dependencies.sh
1.4s
> => [main 2/25] RUN echo "Base image version:
dhi.io/python:3.12-debian12-dev" 2.6s
> => [scripts 3/11] COPY <<EOF /install_mysql.sh
0.1s
> => [scripts 4/11] COPY <<EOF /install_mssql.sh
0.1s
> => [scripts 5/11] COPY <<EOF /install_postgres.sh
0.2s
> => [scripts 6/11] COPY <<EOF /install_packaging_tools.sh
0.2s
> => [scripts 7/11] COPY <<EOF /common.sh
0.2s
> => [scripts 8/11] COPY <<EOF /install_airflow_when_building_images.sh
0.2s
> => [scripts 9/11] COPY <<EOF /install_additional_dependencies.sh
0.2s
> => [scripts 10/11] COPY <<EOF /entrypoint_ci.sh
0.2s
> => [scripts 11/11] COPY <<EOF /entrypoint_exec.sh
0.2s
> => [main 3/25] COPY --from=scripts install_os_dependencies.sh
/scripts/docker/ 0.5s
> => ERROR [main 4/25] RUN PYTHON_LTO=true bash
/scripts/docker/install_os_dependencies.sh ci 44.8s
> ------
> > [main 4/25] RUN PYTHON_LTO=true bash
/scripts/docker/install_os_dependencies.sh ci:
> 0.995 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
> 1.404 Get:2 http://deb.debian.org/debian bookworm-updates InRelease [55.4
kB]
> 1.746 Get:3 http://deb.debian.org/debian-security bookworm-security
InRelease [48.0 kB]
> 2.075 Get:4 http://deb.debian.org/debian bookworm/main Sources [9494 kB]
> 4.550 Get:5 http://deb.debian.org/debian bookworm/main amd64 Packages
[8791 kB]
> 6.788 Get:6 http://deb.debian.org/debian bookworm-updates/main Sources
[3288 B]
> 6.801 Get:7 http://deb.debian.org/debian bookworm-updates/main amd64
Packages [6924 B]
> 6.811 Get:8 http://deb.debian.org/debian-security bookworm-security/main
Sources [195 kB]
> 6.832 Get:9 http://deb.debian.org/debian-security bookworm-security/main
amd64 Packages [290 kB]
> 8.133 Fetched 19.0 MB in 8s (2519 kB/s)
> 8.133 Reading package lists...
> 14.32 Reading package lists...
> 15.14 Building dependency tree...
> 15.36 Reading state information...
> 15.61 ca-certificates is already the newest version (20230311+deb12u1).
> 15.61 The following additional packages will be installed:
> 15.61 dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client
> 15.61 gpg-wks-server gpgconf gpgsm init-system-helpers libassuan0
libbrotli1
> 15.61 libcurl4 libfile-find-rule-perl libgdbm-compat4 libgdbm6
libgssapi-krb5-2
> 15.61 libk5crypto3 libkeyutils1 libkrb5-3 libkrb5support0 libksba8
libldap-2.5-0
> 15.61 libnghttp2-14 libnpth0 libnumber-compare-perl libperl5.36 libpsl5
librtmp1
> 15.61 libsasl2-2 libsasl2-modules-db libssh2-1 libtext-glob-perl
lsb-base perl
> 15.62 perl-modules-5.36 pinentry-curses sysvinit-utils usrmerge
> 15.62 Suggested packages:
> 15.62 dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie
xloadimage
> 15.62 scdaemon gdbm-l10n krb5-doc krb5-user sensible-utils perl-doc
> 15.62 libterm-readline-gnu-perl | libterm-readline-perl-perl make
> 15.62 libtap-harness-archive-perl pinentry-doc
> 15.62 Recommended packages:
> 15.62 krb5-locales libldap-common publicsuffix libsasl2-modules
> 15.96 The following NEW packages will be installed:
> 15.96 curl dirmngr gnupg gnupg-l10n gnupg-utils gnupg2 gpg gpg-agent
> 15.96 gpg-wks-client gpg-wks-server gpgconf gpgsm init-system-helpers
libassuan0
> 15.96 libbrotli1 libcurl4 libfile-find-rule-perl libgdbm-compat4 libgdbm6
> 15.97 libgssapi-krb5-2 libk5crypto3 libkeyutils1 libkrb5-3
libkrb5support0
> 15.97 libksba8 libldap-2.5-0 libnghttp2-14 libnpth0
libnumber-compare-perl
> 15.97 libperl5.36 libpsl5 librtmp1 libsasl2-2 libsasl2-modules-db
libssh2-1
> 15.97 libtext-glob-perl lsb-base lsb-release perl perl-modules-5.36
> 15.97 pinentry-curses sysvinit-utils usrmerge wget
> 16.05 0 upgraded, 44 newly installed, 0 to remove and 0 not upgraded.
> 16.05 Need to get 19.0 MB of archives.
> 16.05 After this operation, 75.0 MB of additional disk space will be used.
> 16.05 Get:1 http://deb.debian.org/debian bookworm/main amd64
perl-modules-5.36 all 5.36.0-7+deb12u3 [2815 kB]
> 16.71 Get:2 http://deb.debian.org/debian bookworm/main amd64 libgdbm6
amd64 1.23-3 [72.2 kB]
> 16.73 Get:3 http://deb.debian.org/debian bookworm/main amd64
libgdbm-compat4 amd64 1.23-3 [48.2 kB]
> 16.74 Get:4 http://deb.debian.org/debian bookworm/main amd64 libperl5.36
amd64 5.36.0-7+deb12u3 [4196 kB]
> 17.78 Get:5 http://deb.debian.org/debian bookworm/main amd64 perl amd64
5.36.0-7+deb12u3 [239 kB]
> 17.84 Get:6 http://deb.debian.org/debian bookworm/main amd64
libnumber-compare-perl all 0.03-3 [6332 B]
> 17.84 Get:7 http://deb.debian.org/debian bookworm/main amd64
libtext-glob-perl all 0.11-3 [7676 B]
> 17.84 Get:8 http://deb.debian.org/debian bookworm/main amd64
libfile-find-rule-perl all 0.34-4~deb12u1 [26.7 kB]
> 17.85 Get:9 http://deb.debian.org/debian bookworm/main amd64 usrmerge all
37~deb12u1 [13.3 kB]
> 17.86 Get:10 http://deb.debian.org/debian bookworm/main amd64
init-system-helpers all 1.65.2+deb12u1 [39.0 kB]
> 18.17 Get:11 http://deb.debian.org/debian bookworm/main amd64
sysvinit-utils amd64 3.06-4 [31.0 kB]
> 18.19 Get:12 http://deb.debian.org/debian bookworm/main amd64 libpsl5
amd64 0.21.2-1 [58.7 kB]
> 18.21 Get:13 http://deb.debian.org/debian bookworm/main amd64 wget amd64
1.21.3-1+deb12u1 [937 kB]
> 18.35 Get:14 http://deb.debian.org/debian bookworm/main amd64 libbrotli1
amd64 1.0.9-2+b6 [275 kB]
> 18.41 Get:15 http://deb.debian.org/debian bookworm/main amd64
libkrb5support0 amd64 1.20.1-2+deb12u4 [33.2 kB]
> 18.42 Get:16 http://deb.debian.org/debian bookworm/main amd64 libk5crypto3
amd64 1.20.1-2+deb12u4 [79.8 kB]
> 18.45 Get:17 http://deb.debian.org/debian bookworm/main amd64 libkeyutils1
amd64 1.6.3-2 [8808 B]
> 18.45 Get:18 http://deb.debian.org/debian bookworm/main amd64 libkrb5-3
amd64 1.20.1-2+deb12u4 [334 kB]
> 18.53 Get:19 http://deb.debian.org/debian bookworm/main amd64
libgssapi-krb5-2 amd64 1.20.1-2+deb12u4 [135 kB]
> 18.56 Get:20 http://deb.debian.org/debian bookworm/main amd64
libsasl2-modules-db amd64 2.1.28+dfsg-10 [20.3 kB]
> 18.58 Get:21 http://deb.debian.org/debian bookworm/main amd64 libsasl2-2
amd64 2.1.28+dfsg-10 [59.7 kB]
> 18.59 Get:22 http://deb.debian.org/debian bookworm/main amd64
libldap-2.5-0 amd64 2.5.13+dfsg-5 [183 kB]
> 18.63 Get:23 http://deb.debian.org/debian bookworm/main amd64
libnghttp2-14 amd64 1.52.0-1+deb12u2 [73.0 kB]
> 18.66 Get:24 http://deb.debian.org/debian bookworm/main amd64 librtmp1
amd64 2.4+20151223.gitfa8646d.1-2+b2 [60.8 kB]
> 18.68 Get:25 http://deb.debian.org/debian bookworm/main amd64 libssh2-1
amd64 1.10.0-3+b1 [179 kB]
> 18.71 Get:26 http://deb.debian.org/debian bookworm/main amd64 libcurl4
amd64 7.88.1-10+deb12u14 [392 kB]
> 18.82 Get:27 http://deb.debian.org/debian bookworm/main amd64 curl amd64
7.88.1-10+deb12u14 [316 kB]
> 18.90 Get:28 http://deb.debian.org/debian bookworm/main amd64 libassuan0
amd64 2.5.5-5 [48.5 kB]
> 18.91 Get:29 http://deb.debian.org/debian bookworm/main amd64 gpgconf
amd64 2.2.40-1.1+deb12u1 [564 kB]
> 19.06 Get:30 http://deb.debian.org/debian bookworm/main amd64 lsb-base all
11.6 [4584 B]
> 19.06 Get:31 http://deb.debian.org/debian bookworm/main amd64 libksba8
amd64 1.6.3-2 [128 kB]
> 19.10 Get:32 http://deb.debian.org/debian bookworm/main amd64 libnpth0
amd64 1.6-3 [19.0 kB]
> 19.12 Get:33 http://deb.debian.org/debian bookworm/main amd64 dirmngr
amd64 2.2.40-1.1+deb12u1 [792 kB]
> 19.34 Get:34 http://deb.debian.org/debian bookworm/main amd64 gnupg-l10n
all 2.2.40-1.1+deb12u1 [1093 kB]
> 19.62 Get:35 http://deb.debian.org/debian bookworm/main amd64 gnupg-utils
amd64 2.2.40-1.1+deb12u1 [927 kB]
> 19.86 Get:36 http://deb.debian.org/debian bookworm/main amd64 gpg amd64
2.2.40-1.1+deb12u1 [949 kB]
> 20.11 Get:37 http://deb.debian.org/debian bookworm/main amd64
pinentry-curses amd64 1.2.1-1 [77.4 kB]
> 20.12 Get:38 http://deb.debian.org/debian bookworm/main amd64 gpg-agent
amd64 2.2.40-1.1+deb12u1 [694 kB]
> 20.30 Get:39 http://deb.debian.org/debian bookworm/main amd64
gpg-wks-client amd64 2.2.40-1.1+deb12u1 [540 kB]
> 20.44 Get:40 http://deb.debian.org/debian bookworm/main amd64
gpg-wks-server amd64 2.2.40-1.1+deb12u1 [531 kB]
> 20.58 Get:41 http://deb.debian.org/debian bookworm/main amd64 gpgsm amd64
2.2.40-1.1+deb12u1 [671 kB]
> 20.93 Get:42 http://deb.debian.org/debian bookworm/main amd64 gnupg all
2.2.40-1.1+deb12u1 [846 kB]
> 22.63 Get:43 http://deb.debian.org/debian bookworm/main amd64 gnupg2 all
2.2.40-1.1+deb12u1 [446 kB]
> 22.66 Get:44 http://deb.debian.org/debian bookworm/main amd64 lsb-release
all 12.0-1 [6416 B]
> 24.76 Fetched 19.0 MB in 7s (2846 kB/s)
> 24.81 Selecting previously unselected package perl-modules-5.36.
> (Reading database ... 5576 files and directories currently installed.)
> 24.82 Preparing to unpack .../0-perl-modules-5.36_5.36.0-7+deb12u3_all.deb
...
> 24.83 Unpacking perl-modules-5.36 (5.36.0-7+deb12u3) ...
> 36.31 Selecting previously unselected package libgdbm6:amd64.
> 36.31 Preparing to unpack .../1-libgdbm6_1.23-3_amd64.deb ...
> 36.32 Unpacking libgdbm6:amd64 (1.23-3) ...
> 36.50 Selecting previously unselected package libgdbm-compat4:amd64.
> 36.50 Preparing to unpack .../2-libgdbm-compat4_1.23-3_amd64.deb ...
> 36.51 Unpacking libgdbm-compat4:amd64 (1.23-3) ...
> 36.66 Selecting previously unselected package libperl5.36:amd64.
> 36.67 Preparing to unpack .../3-libperl5.36_5.36.0-7+deb12u3_amd64.deb ...
> 36.68 Unpacking libperl5.36:amd64 (5.36.0-7+deb12u3) ...
> 40.96 Selecting previously unselected package perl.
> 40.97 Preparing to unpack .../4-perl_5.36.0-7+deb12u3_amd64.deb ...
> 41.00 Unpacking perl (5.36.0-7+deb12u3) ...
> 41.67 Selecting previously unselected package libnumber-compare-perl.
> 41.68 Preparing to unpack .../5-libnumber-compare-perl_0.03-3_all.deb ...
> 41.69 Unpacking libnumber-compare-perl (0.03-3) ...
> 41.84 Setting up libnumber-compare-perl (0.03-3) ...
> 41.96 Selecting previously unselected package libtext-glob-perl.
> (Reading database ... 7594 files and directories currently installed.)
> 41.97 Preparing to unpack .../libtext-glob-perl_0.11-3_all.deb ...
> 41.99 Unpacking libtext-glob-perl (0.11-3) ...
> 42.14 Setting up libtext-glob-perl (0.11-3) ...
> 42.25 Selecting previously unselected package libfile-find-rule-perl.
> (Reading database ... 7601 files and directories currently installed.)
> 42.26 Preparing to unpack
.../libfile-find-rule-perl_0.34-4~deb12u1_all.deb ...
> 42.28 Unpacking libfile-find-rule-perl (0.34-4~deb12u1) ...
> 42.46 Setting up perl-modules-5.36 (5.36.0-7+deb12u3) ...
> 42.49 Setting up libgdbm6:amd64 (1.23-3) ...
> 42.53 Setting up libgdbm-compat4:amd64 (1.23-3) ...
> 42.58 Setting up libperl5.36:amd64 (5.36.0-7+deb12u3) ...
> 42.62 Setting up perl (5.36.0-7+deb12u3) ...
> 43.47 Setting up libfile-find-rule-perl (0.34-4~deb12u1) ...
> 43.60 Selecting previously unselected package usrmerge.
> (Reading database ... 7616 files and directories currently installed.)
> 43.61 Preparing to unpack .../usrmerge_37~deb12u1_all.deb ...
> 43.65 Unpacking usrmerge (37~deb12u1) ...
> 43.79 Setting up usrmerge (37~deb12u1) ...
> 43.86 removed '/lib64'
> 43.86 /var/lib/dpkg/info/usrmerge.postinst: line 70: /usr/bin/rmdir:
cannot execute: required file not found
> 43.87 dpkg: error processing package usrmerge (--configure):
> 43.87 installed usrmerge package post-installation script subprocess
returned error exit status 127
> 43.89 Errors were encountered while processing:
> 43.89 usrmerge
> 43.92 E: Sub-process /usr/bin/dpkg returned an error code (1)
> 43.92 E: Problem executing scripts DPkg::Post-Invoke 'rm -f
/var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb
/var/cache/apt/*.bin || true'
> 43.92 E: Sub-process returned an error code
> ------
> ERROR: failed to build: failed to solve: process "/bin/bash -o pipefail -o
errexit -o nounset -o nolog -c PYTHON_LTO=${PYTHON_LTO} bash
/scripts/docker/install_os_dependencies.sh ci" did not complete successfully:
exit code: 100
> ```
>
> waiting for further guidance.
This is the part of "trial and error" - we need to understand what parts
should be adjusted, as some dependencies **will** be missing due to the
hardened nature of the Docker image.
For each failure, try to figure out:
1. What dependencies are missing.
2. Why they are missing - this affects how we treat it
According to the above, try to figure out how to resolve the failure:
1. We can just install the missing dependency (in some cases it might not be
possible, or recommended to do so).
2. Work around it.
3. Ignore the step, if it's not really required.
This will be an iterative process, maybe a daunting and long one - but it
should be possible. You may use LLMs to try making it a bit easier, but please
carefully review any of its suggestions.
Also, you could create a draft PR that shows the changes you implement, and
we could comment on it along the way.
Good luck!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
