Hello,
we are having issues upgrading from 19.2.2 to 19.2.3
We have mixed ARM64 AMD64 ceph deployment.
For performance reason we had to host our MON, MGR and MDS daemons on
intel server.
It didn't cause any issues for the past 3 years but today we discovered
that we are not able to perform an upgrade.
Before starting the upgrade, I manually pulled ceph image on all of our
servers with podman pull. (podman pull quay.io/ceph/ceph:v19.2.3) Every
server has the new image.
We start upgrade with normal cephadm command:
ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3
Cephadm always fails on the first host with "UPGRADE_FAILED_PULL:
Upgrade: failed to pull target image"
ceph orch upgrade status says that ceph performs upgrade to
"target_image":
"quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717",
not to specified quay.io/ceph/ceph:v19.2.3
865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717 appears
to be ARM64 version of the container.
https://quay.io/repository/ceph/ceph/manifest/sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee
ARM64 container fails to start on AMD64 machines and upgrade can never
start.
2026-03-25 12:27:15,337 7f98685251c0 DEBUG
--------------------------------------------------------------------------------
cephadm ['--image',
'quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717',
'--timeout', '1495', 'pull']
2026-03-25 12:27:15,466 7f98685251c0 INFO Pulling container image
quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717...
2026-03-25 12:27:17,429 7f98685251c0 DEBUG /usr/bin/podman: stdout
23b591a142203efef0aeb561b81712350869becf57e9d42ddc36ab8a0124fb75,[quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717]
2026-03-25 12:27:17,656 7f98685251c0 INFO Non-zero exit code 1 from
/usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host
--entrypoint ceph --init -e
CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717
-e NODE_NAME=k02r02nvm01
quay.io/ceph/ceph@sha256:865adb13cc6c0c0fbea23e7217ecf1555975816a8b999792b54d1f5c3fea0717
--version
2026-03-25 12:27:17,657 7f98685251c0 INFO ceph: stderr ERROR
(catatonit:2): failed to exec pid1: Exec format error
2026-03-25 12:27:43,188 7f769f0cc1c0 DEBUG
--------------------------------------------------------------------------------
Is this a known issue? Has there been a change to how cephadm handles
image version?
We have made multiple upgrade of this cluster in the past and this is
the first time we are seeing this issue.
( list of upgrades 18.2.2 -> 19.2.0, 19.2.0 -> 19.2.2 )
Is there a way to force cephadm to use image tags instead if digests.
Best regards
Adam Prycki
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
