janhoy opened a new issue, #859: URL: https://github.com/apache/solr-operator/issues/859
`hack/release/smoke_test/test_cluster.sh` and `tests/scripts/manage_e2e_tests.sh` load images into Kind with `kind load docker-image`. On Docker with the **containerd image store** — the default for new installs since Docker Engine 29, and on by default in Docker Desktop — this fails for any **multi-arch image pulled from a registry**: ``` ERROR: failed to load image: command "docker exec --privileged -i ... ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1 Command Output: ctr: content digest sha256:...: not found ``` Cause: the containerd store keeps the full multi-platform index but only the host platform's blobs. `kind load docker-image` runs `docker save | ctr images import --all-platforms`, and `ctr` refuses an index whose referenced content (other platforms, buildx attestation manifests) is not present. Tracked upstream as kubernetes-sigs/kind#3795 since 2024-11, no fix; kind maintainers consider it a containerd/Docker issue. **Locally built single-platform images load fine**, which is why CI's `make smoke-test` still passes — it builds the operator image on the runner. `make e2e-tests` is affected: `manage_e2e_tests.sh:179-181` pulls then loads `solr:9.10.0` and `pravega/zookeeper:0.2.15`. **This is already live in CI.** `ubuntu-latest` is migrating to Ubuntu 26.04 (Oct 19 – Nov 19, 2026), whose image ships Docker 29.4.2 via a plain apt install with no `daemon.json`. Our runs on 2026-10-23 were already on `ubuntu-26.04`. The e2e workflow only triggers on code paths, so it has not hit this yet — it will on the next code PR. Any developer on a fresh Docker 29 or current Docker Desktop hits it locally today. ### Proposed fix Only `kind load` images that exist nowhere but the local Docker (a fresh `make docker-build`). Let the Kind nodes pull registry images themselves, with `pullPolicy: IfNotPresent` instead of `Never`. - `test_cluster.sh`: when `LOCATION` is a URL the image was just `docker pull`ed from a registry — skip the load, use `IfNotPresent`. When `LOCATION` is a local path keep the load and `Never`. Never load `SOLR_IMAGE`; it is always a registry image and the Solr install does not force `Never`. - `manage_e2e_tests.sh`: keep the load for `OPERATOR_IMAGE` (local build), stop pull-then-load for `SOLR_IMAGE` and `ADDITIONAL_IMAGES`. The e2e Go code forces `Never` only for the operator (`tests/e2e/test_utils_test.go:85`), so no Go change. - Wrap the remaining `kind load` with a clear error pointing at kind#3795. Besides being store- and OS-agnostic, this makes release validation more faithful: loading a host copy with `pullPolicy: Never` never verified that the *published* image is actually pullable. Trade-off: a fresh Kind cluster pulls `solr:9.10.0` in-node instead of from the host cache (`REUSE_KIND_CLUSTER_IF_EXISTS` mitigates for e2e). Docker Hub anonymous limit is 100 pulls / 6h / IP; CI already pulls anonymously on the host, so this shifts rather than adds pulls. ### Rejected alternatives - kind's documented workaround (`docker save --platform` + `kind load image-archive`): fails on our own published image on arm64 (the arm64 manifest has no `variant`, so Docker's platform matcher rejects `linux/arm64`), and exits 1 on official images because an in-toto attestation layer is still bundled. Would pass in CI (amd64) and fail for every Mac developer. - Disabling the containerd store: per-machine config, cannot be enforced, and Docker is deprecating the classic store. - Pinning CI to `ubuntu-24.04`: buys time, fixes nothing for developers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
