Hi,
you could try to disable use_repo_digest:
ceph config help mgr/cephadm/use_repo_digest
mgr/cephadm/use_repo_digest - Automatically convert image tags to
image digest. Make sure all daemons use the same image
(bool, advanced)
Default: true
Can update at runtime: false
You would need to restart the MGRs to apply that config update.
Regards,
Eugen
Zitat von Adam Prycki via ceph-users <[email protected]>:
Hello,
we are having issues upgrading from 19.2.2 to 19.2.3
We have mixed ARM64 AMD64 ceph deployment.
For performance reason we had to host our MON, MGR and MDS daemons
on intel server.
It didn't cause any issues for the past 3 years but today we
discovered that we are not able to perform an upgrade.
Before starting the upgrade, I manually pulled ceph image on all of
our servers with podman pull. (podman pull
quay.io/ceph/ceph:v19.2.3) Every server has the new image.
We start upgrade with normal cephadm command:
ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3
Cephadm always fails on the first host with "UPGRADE_FAILED_PULL:
Upgrade: failed to pull target image"
ceph orch upgrade status says that ceph performs upgrade to
"target_image":
"quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717",
not to specified quay.io/ceph/ceph:v19.2.3
865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717
appears to be ARM64 version of the container.
https://quay.io/repository/ceph/ceph/manifest/sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee
ARM64 container fails to start on AMD64 machines and upgrade can never start.
2026-03-25 12:27:15,337 7f98685251c0 DEBUG
--------------------------------------------------------------------------------
cephadm ['--image',
'quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717', '--timeout', '1495',
'pull']
2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image
quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717...
2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout
23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75,[quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717]
2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from
/usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host
--entrypoint ceph --init -e
CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 -e NODE_NAME=k02r02nvm01 quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717
--version
2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR
(catatonit:2): failed to exec pid1: Exec format error
2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG
--------------------------------------------------------------------------------
Is this a known issue? Has there been a change to how cephadm
handles image version?
We have made multiple upgrade of this cluster in the past and this
is the first time we are seeing this issue.
( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 )
Is there a way to force cephadm to use image tags instead if digests.
Best regards
Adam Prycki
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
