Hi Dev, Tips, real quick:
- stop zapping devices :-) - I've never heard of any 'back' option on the 'ceph orch upgrade' command [1], but 'stop' could be used to allow other operations to proceed, would it be needed at some point. BTW, I will assume that you started the upgrade prior to OSDs failing to start, and not after. - I would advise that you set 'osd_op_queue' to 'wpq', to help with the current situation. Start with a single OSD (ceph osd config set osd.0 osd_op_queue wpq) and keep going with other OSDs if it started and joined the cluster without issues. - If using Podman, SSH to every hosts and run 'df -i' to check for inode exhaustion on /run partition. There's a podman bug [2] in EL 9.5 that was fixed in EL 9.6 by podman-5:5.4.0-9.el9_6.x86_64 . Could also affect other distros. In case of inode exhaustion, no containers would start. Inode exhaustion can happen really fast as the orchestrator is regularly running containers on every hosts for inventory but not only. - Choose one of the '8 OSD(s) experiencing slow operations in BlueStore' and share 'ceph tell osd.X bluefs stats' and 'ceph tell osd.0 bluestore allocator score block'. - Please share the manufacturer/model of SSDs and NVMes. Regards, Frédéric. [1] [ https://docs.ceph.com/en/latest/cephadm/upgrade/ | https://docs.ceph.com/en/latest/cephadm/upgrade/ ] [2] [ https://github.com/containers/podman/issues/25291 | https://github.com/containers/podman/issues/25291 ] ----- Le 20 Juin 25, à 5:36, Devender Singh <deven...@netskrt.io> a écrit : > Thanks for your reply.. > There were multiple OSD’s failure on different hosts and had to zap and read. > Which lead us to this issue… and upgrade was running too.. > Now I see three options for upgrade “Back, Pause & Stop” — is back for > rollback? > And which one I should use the best option…. > Tried restating osd, but it seems dam slow and few are not coming up….and then > need zap and add again… > What to do with incomplete pigs? > Here are the details/… > Sorry for the wrong detail in previous emails… correct are below > root@phl-host03n:~# ceph -s > cluster: > id: a0bd51e8-4dfc-11ee-b5a9-3b06e501a0ac > health: HEALTH_WARN > 8 OSD(s) experiencing slow operations in BlueStore > Failed to apply 2 service(s): osd.all-available-devices,osd.iops_optimized > 1 failed cephadm daemon(s) > failed to probe daemons or devices > noscrub,nodeep-scrub flag(s) set > Degraded data redundancy: 1739710/39361366 objects degraded (4.420%), 92 pgs > degraded, 103 pgs undersized > 127 pgs not deep-scrubbed in time > services: > mon: 5 daemons, quorum > phl-host04n,phl-host02n,phl-host01n,phl-host05n,phl-host03n (age 32h) > mgr: phl-host01n.pkkztj(active, since 110m), standbys: phl-host02n.wokbnx > mds: 3/3 daemons up, 2 standby > osd: 88 osds: 88 up (since 2h), 88 in (since 34m); 182 remapped pgs > flags noscrub,nodeep-scrub > rbd-mirror: 2 daemons active (2 hosts) > rgw: 2 daemons active (2 hosts, 1 zones) > data: > volumes: 3/3 healthy > pools: 23 pools, 1169 pgs > objects: 9.22M objects, 32 TiB > usage: 68 TiB used, 239 TiB / 307 TiB avail > pgs: 1739710/39361366 objects degraded (4.420%) > 3592355/39361366 objects misplaced (9.127%) > 982 active+clean > 85 active+undersized+degraded+remapped+backfill_wait > 78 active+remapped+backfill_wait > 10 active+undersized+remapped+backfill_wait > 7 active+undersized+degraded+remapped+backfilling > 5 active+clean+scrubbing+deep > 1 active+remapped+backfilling > 1 active+undersized+remapped+backfilling > io: > client: 140 MiB/s rd, 9.6 MiB/s wr, 528 op/s rd, 379 op/s wr > recovery: 104 MiB/s, 27 objects/s > root@phl-host03n:~# ceph orch upgrade status > { > "target_image": > "quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6", > "in_progress": true, > "which": "Upgrading all daemon types on all hosts", > "services_complete": [ > "crash", > "mgr", > "mon" > ], > "progress": "74/112 daemons upgraded", > "message": "", > "is_paused": false > } > # ceph config dump > WHO MASK LEVEL OPTION VALUE RO > global advanced client_reconnect_stale true > global advanced cluster_network 10.104.5.0/24 * > global basic container_image > quay.io/ceph/ceph@sha256:1b9158ce28975f95def6a0ad459fa19f1336506074267a4b47c1bd914a00fec0 > * > global basic log_to_file true > global basic mds_cache_memory_limit 10589934592 > global advanced mgr/devicehealth/self_heal true > global advanced mon_cluster_log_to_file true > global advanced osd_pool_default_pg_autoscale_mode on > global dev rbd_mirror_memory_base 1005306368 > global advanced rbd_skip_partial_discard false > mon advanced auth_allow_insecure_global_id_reclaim false > mon basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > mon advanced mon_allow_pool_delete true > mon advanced mon_mds_skip_sanity true > mon advanced public_network 10.104.1.0/24 * > mgr basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > mgr advanced mgr/cephadm/container_init True * > mgr advanced mgr/cephadm/migration_current 7 * > mgr advanced mgr/cephadm/warn_on_stray_daemons/warn_on_stray_hosts false * > mgr advanced mgr/cephadm/warn_on_stray_hosts false * > mgr advanced mgr/cephadm/warn_stray_daemons/warn_on_stray_hosts false * > mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST > http://pl-host04n.phl.example.com.org:9093 * > mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * > mgr advanced mgr/dashboard/GRAFANA_API_URL > https://pl-host02n.phl.example.com.org:3000 * > mgr advanced mgr/dashboard/PROMETHEUS_API_HOST > http://pl-host04n.phl.example.com.org:9095 * > mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY * > mgr advanced mgr/dashboard/RGW_API_SECRET_KEY * > mgr advanced mgr/dashboard/ssl_server_port 8443 * > mgr advanced mgr/mds_autoscaler/log_level warning > mgr advanced mgr/mds_autoscaler/log_to_cluster true > mgr advanced mgr/mds_autoscaler/log_to_file true > mgr advanced mgr/orchestrator/orchestrator cephadm > mgr advanced mgr/selftest/pl-host04n.tvzwes/testkey foo * > mgr advanced mgr/selftest/rwoption1 1234 * > mgr advanced mgr/selftest/rwoption2 10 * > mgr advanced mgr/selftest/rwoption3 1.500000 * > mgr advanced mgr/selftest/rwoption4 foo * > mgr advanced mgr/selftest/rwoption5 false * > mgr advanced mgr/selftest/testkey test * > mgr advanced mgr/telemetry/channel_basic false * > mgr advanced mgr/telemetry/channel_crash false * > mgr advanced mgr/telemetry/channel_device false * > mgr advanced mgr/telemetry/contact t...@test.com * > mgr advanced mgr/telemetry/interval 60 * > mgr advanced mgr/telemetry/leaderboard true * > mgr advanced mgr/zabbix/identifier ceph * > mgr advanced mgr/zabbix/log_to_cluster false > mgr advanced mgr/zabbix/log_to_file false > mgr advanced mgr/zabbix/zabbix_host 10.0.2.110 * > osd advanced bdev_aio_poll_ms 200 > osd advanced bdev_ioring false > osd advanced bluestore_throttle_bytes 134217728 > osd advanced ms_async_op_threads 4 > osd advanced osd_heartbeat_grace 70 > osd advanced osd_max_backfills 1 > osd advanced osd_mclock_override_recovery_settings true > osd host:phl-converged01n basic osd_memory_target 19352618016 > osd host:phl-converged02n basic osd_memory_target 31346288264 > osd host:phl-converged03n basic osd_memory_target 19804699313 > osd host:phl-converged04n basic osd_memory_target 18593027461 > osd host:phl-converged05n basic osd_memory_target 20606720591 > osd host:pl-host01n basic osd_memory_target 15058518075 > osd host:pl-host02n basic osd_memory_target 14086545280 > osd host:pl-host03n basic osd_memory_target 33342959038 > osd host:pl-host04n basic osd_memory_target 53122886860 > osd host:pl-host05n basic osd_memory_target 16313721223 > osd advanced osd_memory_target_autotune true > osd advanced osd_recovery_max_active_ssd 1 > osd advanced osd_recovery_max_single_start 1 > osd advanced osd_recovery_sleep 0.000000 > osd advanced osd_requested_scrub_priority 1 > osd advanced osd_scrub_disable_reservation_queuing true > osd advanced osd_scrub_sleep 0.000000 > osd advanced osd_snap_trim_cost 262144 > osd.0 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.0 basic osd_mclock_max_capacity_iops_ssd 37869.100179 > osd.1 basic osd_mclock_max_capacity_iops_ssd 38610.482854 > osd.10 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.10 basic osd_mclock_max_capacity_iops_ssd 35297.387846 > osd.100 basic osd_mclock_max_capacity_iops_ssd 29091.307131 > osd.101 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.101 basic osd_mclock_max_capacity_iops_ssd 27963.050594 > osd.102 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.102 basic osd_mclock_max_capacity_iops_ssd 31172.711483 > osd.103 basic osd_mclock_max_capacity_iops_ssd 28664.146986 > osd.104 basic osd_mclock_max_capacity_iops_ssd 14405.473296 > osd.105 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.105 basic osd_mclock_max_capacity_iops_ssd 28291.779421 > osd.106 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.106 basic osd_mclock_max_capacity_iops_ssd 30678.497618 > osd.107 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.107 basic osd_mclock_max_capacity_iops_ssd 26799.726954 > osd.108 basic osd_mclock_max_capacity_iops_ssd 27523.880430 > osd.109 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.109 basic osd_mclock_max_capacity_iops_ssd 27819.225848 > osd.11 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.11 basic osd_mclock_max_capacity_iops_ssd 35852.830870 > osd.110 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.110 basic osd_mclock_max_capacity_iops_ssd 28671.008445 > osd.111 basic osd_mclock_max_capacity_iops_ssd 28610.088976 > osd.112 basic osd_mclock_max_capacity_iops_ssd 27462.279164 > osd.113 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.113 basic osd_mclock_max_capacity_iops_ssd 25652.864986 > osd.114 basic osd_mclock_max_capacity_iops_ssd 27469.642657 > osd.115 basic osd_mclock_max_capacity_iops_ssd 26599.443843 > osd.116 basic osd_mclock_max_capacity_iops_ssd 28159.154679 > osd.117 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.117 basic osd_mclock_max_capacity_iops_ssd 30631.799864 > osd.118 basic osd_mclock_max_capacity_iops_ssd 26888.868629 > osd.119 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.119 basic osd_mclock_max_capacity_iops_ssd 27837.170185 > osd.12 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.12 basic osd_mclock_max_capacity_iops_ssd 36523.110430 > osd.120 basic osd_mclock_max_capacity_iops_ssd 25156.699403 > osd.121 basic osd_mclock_max_capacity_iops_ssd 16854.236813 > osd.122 basic osd_mclock_max_capacity_iops_ssd 29425.668046 > osd.123 basic osd_mclock_max_capacity_iops_ssd 28424.967707 > osd.124 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.124 basic osd_mclock_max_capacity_iops_ssd 30490.214968 > osd.125 basic osd_mclock_max_capacity_iops_ssd 29875.742604 > osd.126 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.126 basic osd_mclock_max_capacity_iops_ssd 29827.105097 > osd.127 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.127 basic osd_mclock_max_capacity_iops_ssd 28899.535735 > osd.128 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.128 basic osd_mclock_max_capacity_iops_ssd 27569.775057 > osd.129 basic osd_mclock_max_capacity_iops_ssd 32820.522874 > osd.13 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.13 basic osd_mclock_max_capacity_iops_ssd 38260.450852 > osd.130 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.130 basic osd_mclock_max_capacity_iops_ssd 33740.656917 > osd.131 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.131 basic osd_mclock_max_capacity_iops_ssd 30837.814068 > osd.132 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.132 basic osd_mclock_max_capacity_iops_ssd 32357.136152 > osd.133 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.133 basic osd_mclock_max_capacity_iops_ssd 33199.681376 > osd.134 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.134 basic osd_mclock_max_capacity_iops_ssd 32701.919940 > osd.135 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.135 basic osd_mclock_max_capacity_iops_ssd 31471.381829 > osd.136 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.136 basic osd_mclock_max_capacity_iops_ssd 36040.644043 > osd.137 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.137 basic osd_mclock_max_capacity_iops_ssd 34135.109403 > osd.138 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.138 basic osd_mclock_max_capacity_iops_ssd 28739.646099 > osd.139 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.139 basic osd_mclock_max_capacity_iops_ssd 28611.027050 > osd.14 basic osd_mclock_max_capacity_iops_ssd 32587.786337 > osd.15 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.15 basic osd_mclock_max_capacity_iops_ssd 33930.129246 > osd.16 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.16 basic osd_mclock_max_capacity_iops_ssd 37610.362437 > osd.17 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.17 basic osd_mclock_max_capacity_iops_ssd 22609.652689 > osd.18 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.18 basic osd_mclock_max_capacity_iops_ssd 29116.630263 > osd.19 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.19 basic osd_mclock_max_capacity_iops_ssd 34639.127606 > osd.2 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.2 basic osd_mclock_max_capacity_iops_ssd 32777.963192 > osd.20 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.20 basic osd_mclock_max_capacity_iops_ssd 33042.683283 > osd.21 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.21 basic osd_mclock_max_capacity_iops_ssd 34608.069667 > osd.22 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.22 basic osd_mclock_max_capacity_iops_ssd 33672.786827 > osd.23 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.23 basic osd_mclock_max_capacity_iops_ssd 37253.942367 > osd.24 basic osd_mclock_max_capacity_iops_ssd 32010.520279 > osd.25 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.25 basic osd_mclock_max_capacity_iops_ssd 18731.103127 > osd.26 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.26 basic osd_mclock_max_capacity_iops_ssd 33844.524796 > osd.27 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.27 basic osd_mclock_max_capacity_iops_ssd 33592.404677 > osd.28 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.28 basic osd_mclock_max_capacity_iops_ssd 34945.380022 > osd.29 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.29 basic osd_mclock_max_capacity_iops_ssd 33041.812034 > osd.3 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.3 basic osd_mclock_max_capacity_iops_ssd 30526.239088 > osd.30 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.30 basic osd_mclock_max_capacity_iops_ssd 33849.746190 > osd.31 basic osd_mclock_max_capacity_iops_ssd 31073.672124 > osd.32 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.32 basic osd_mclock_max_capacity_iops_ssd 34475.463692 > osd.33 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.33 basic osd_mclock_max_capacity_iops_ssd 35294.129689 > osd.34 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.34 basic osd_mclock_max_capacity_iops_ssd 33500.582011 > osd.35 basic osd_mclock_max_capacity_iops_ssd 33937.027809 > osd.36 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.36 basic osd_mclock_max_capacity_iops_ssd 33363.104343 > osd.37 basic osd_mclock_max_capacity_iops_ssd 33136.270826 > osd.38 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.38 basic osd_mclock_max_capacity_iops_ssd 34327.772673 > osd.39 basic osd_mclock_max_capacity_iops_ssd 36197.080652 > osd.4 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.4 basic osd_mclock_max_capacity_iops_ssd 31292.480847 > osd.40 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.40 basic osd_mclock_max_capacity_iops_ssd 30200.043883 > osd.41 basic osd_mclock_max_capacity_iops_ssd 28660.094990 > osd.42 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.42 basic osd_mclock_max_capacity_iops_ssd 29042.271559 > osd.43 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.43 basic osd_mclock_max_capacity_iops_ssd 29802.985872 > osd.44 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.44 basic osd_mclock_max_capacity_iops_ssd 29872.248343 > osd.45 basic osd_mclock_max_capacity_iops_ssd 27048.306734 > osd.46 basic osd_mclock_max_capacity_iops_ssd 27445.922772 > osd.47 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.47 basic osd_mclock_max_capacity_iops_ssd 29463.719353 > osd.48 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.48 basic osd_mclock_max_capacity_iops_ssd 27648.479917 > osd.49 basic osd_mclock_max_capacity_iops_ssd 26156.443046 > osd.5 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.5 basic osd_mclock_max_capacity_iops_ssd 31042.919796 > osd.50 basic osd_mclock_max_capacity_iops_ssd 27126.914926 > osd.51 basic osd_mclock_max_capacity_iops_ssd 24087.063752 > osd.52 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.52 basic osd_mclock_max_capacity_iops_ssd 29446.685299 > osd.53 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.53 basic osd_mclock_max_capacity_iops_ssd 30223.321026 > osd.54 basic osd_mclock_max_capacity_iops_ssd 29041.708424 > osd.55 basic osd_mclock_max_capacity_iops_ssd 31812.937889 > osd.56 basic osd_mclock_max_capacity_iops_ssd 28114.742336 > osd.57 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.57 basic osd_mclock_max_capacity_iops_ssd 28757.116835 > osd.58 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.58 basic osd_mclock_max_capacity_iops_ssd 29569.415786 > osd.59 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.59 basic osd_mclock_max_capacity_iops_ssd 29009.310857 > osd.6 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.6 basic osd_mclock_max_capacity_iops_ssd 35515.197243 > osd.60 basic osd_mclock_max_capacity_iops_ssd 27353.095641 > osd.61 basic osd_mclock_max_capacity_iops_ssd 28033.751890 > osd.62 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.62 basic osd_mclock_max_capacity_iops_ssd 28270.316050 > osd.63 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.63 basic osd_mclock_max_capacity_iops_ssd 29194.558053 > osd.64 basic osd_mclock_max_capacity_iops_ssd 25543.077545 > osd.65 basic osd_mclock_max_capacity_iops_ssd 29257.957631 > osd.66 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.66 basic osd_mclock_max_capacity_iops_ssd 27251.592177 > osd.67 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.67 basic osd_mclock_max_capacity_iops_ssd 28428.290504 > osd.68 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.68 basic osd_mclock_max_capacity_iops_ssd 30541.378615 > osd.69 basic osd_mclock_max_capacity_iops_ssd 29164.568681 > osd.7 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.7 basic osd_mclock_max_capacity_iops_ssd 35788.435091 > osd.70 basic osd_mclock_max_capacity_iops_ssd 27050.432719 > osd.71 basic osd_mclock_max_capacity_iops_ssd 27615.938915 > osd.72 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.72 basic osd_mclock_max_capacity_iops_ssd 29912.870688 > osd.73 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.73 basic osd_mclock_max_capacity_iops_ssd 30750.315068 > osd.74 basic osd_mclock_max_capacity_iops_hdd 274.216582 > osd.74 basic osd_mclock_max_capacity_iops_ssd 31296.317898 > osd.75 basic osd_mclock_max_capacity_iops_ssd 31330.904882 > osd.76 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.76 basic osd_mclock_max_capacity_iops_ssd 29541.036281 > osd.77 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.77 basic osd_mclock_max_capacity_iops_ssd 27755.161947 > osd.78 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.78 basic osd_mclock_max_capacity_iops_ssd 30181.440559 > osd.79 basic osd_mclock_max_capacity_iops_ssd 28820.911162 > osd.8 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.8 basic osd_mclock_max_capacity_iops_ssd 32704.310615 > osd.80 basic osd_mclock_max_capacity_iops_ssd 30604.883487 > osd.81 basic osd_mclock_max_capacity_iops_ssd 26188.393507 > osd.82 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.82 basic osd_mclock_max_capacity_iops_ssd 27407.424799 > osd.83 basic osd_mclock_max_capacity_iops_hdd 271.607604 > osd.83 basic osd_mclock_max_capacity_iops_ssd 29270.002337 > osd.84 basic osd_mclock_max_capacity_iops_ssd 29874.802768 > osd.85 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.85 basic osd_mclock_max_capacity_iops_ssd 33267.445310 > osd.86 basic osd_mclock_max_capacity_iops_ssd 26647.075973 > osd.87 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.87 basic osd_mclock_max_capacity_iops_ssd 23232.299984 > osd.88 basic osd_mclock_max_capacity_iops_ssd 30225.432154 > osd.89 basic osd_mclock_max_capacity_iops_ssd 26223.229524 > osd.9 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.9 basic osd_mclock_max_capacity_iops_hdd 265.536539 > osd.9 basic osd_mclock_max_capacity_iops_ssd 35082.146366 > osd.90 basic osd_mclock_max_capacity_iops_ssd 28870.253846 > osd.91 basic osd_mclock_max_capacity_iops_ssd 28370.754122 > osd.92 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.92 basic osd_mclock_max_capacity_iops_ssd 25124.799906 > osd.93 basic osd_mclock_max_capacity_iops_ssd 26721.773615 > osd.94 basic osd_mclock_max_capacity_iops_ssd 28325.510268 > osd.95 basic osd_mclock_max_capacity_iops_ssd 27777.045030 > osd.96 basic osd_mclock_max_capacity_iops_ssd 28368.353309 > osd.97 basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > osd.97 basic osd_mclock_max_capacity_iops_ssd 22786.649038 > osd.98 basic osd_mclock_max_capacity_iops_ssd 29639.368028 > osd.99 basic osd_mclock_max_capacity_iops_ssd 27038.486473 > mds.k8s-it-infra-cephfs basic mds_join_fs k8s-it-infra-cephfs > mds.k8s-prod-cephfs basic mds_join_fs k8s-prod-cephfs > mds.phl.prod basic mds_join_fs phl.prod > client advanced client_asio_thread_count 4 > client advanced rbd_op_threads 2 > client.crash basic container_image > quay.io/ceph/ceph@sha256:8214ebff6133ac27d20659038df6962dbf9d77da21c9438a296b2e2059a56af6 > * > client.rgw.phl.pl-host01n.titvzo basic rgw_frontends beast ssl_port=443 > ssl_certificate=config://rgw/cert/rgw.phl * > client.rgw.phl.pl-host05n.frnpbx basic rgw_frontends beast ssl_port=443 > ssl_certificate=config://rgw/cert/rgw.phl * > # ceph osd tree > ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF > -1 307.39172 root default > -19 83.83411 host pl-host01n > 130 nvme 3.49309 osd.130 up 1.00000 1.00000 > 131 nvme 3.49309 osd.131 up 1.00000 1.00000 > 132 nvme 3.49309 osd.132 up 1.00000 1.00000 > 133 nvme 3.49309 osd.133 up 1.00000 1.00000 > 134 nvme 3.49309 osd.134 up 1.00000 1.00000 > 135 nvme 3.49309 osd.135 up 1.00000 1.00000 > 136 nvme 3.49309 osd.136 up 1.00000 1.00000 > 137 nvme 3.49309 osd.137 up 1.00000 1.00000 > 15 ssd 3.49309 osd.15 up 1.00000 1.00000 > 32 ssd 3.49309 osd.32 up 1.00000 1.00000 > 34 ssd 3.49309 osd.34 up 1.00000 1.00000 > 38 ssd 3.49309 osd.38 up 1.00000 1.00000 > 42 ssd 3.49309 osd.42 up 1.00000 1.00000 > 76 ssd 3.49309 osd.76 up 1.00000 1.00000 > 101 ssd 3.49309 osd.101 up 1.00000 1.00000 > 105 ssd 3.49309 osd.105 up 1.00000 1.00000 > 109 ssd 3.49309 osd.109 up 1.00000 1.00000 > 113 ssd 3.49309 osd.113 up 1.00000 1.00000 > 117 ssd 3.49309 osd.117 up 1.00000 1.00000 > 119 ssd 3.49309 osd.119 up 1.00000 1.00000 > 124 ssd 3.49309 osd.124 up 1.00000 1.00000 > 127 ssd 3.49309 osd.127 up 1.00000 1.00000 > 138 ssd 3.49309 osd.138 up 1.00000 1.00000 > 139 ssd 3.49309 osd.139 up 1.00000 1.00000 > -25 83.83411 host pl-host02n > 0 nvme 3.49309 osd.0 up 1.00000 1.00000 > 40 nvme 3.49309 osd.40 up 1.00000 1.00000 > 47 nvme 3.49309 osd.47 up 1.00000 1.00000 > 52 nvme 3.49309 osd.52 up 1.00000 1.00000 > 57 nvme 3.49309 osd.57 up 1.00000 1.00000 > 62 nvme 3.49309 osd.62 up 1.00000 1.00000 > 67 nvme 3.49309 osd.67 up 1.00000 1.00000 > 72 nvme 3.49309 osd.72 up 1.00000 1.00000 > 92 nvme 3.49309 osd.92 up 1.00000 1.00000 > 7 ssd 3.49309 osd.7 up 1.00000 1.00000 > 10 ssd 3.49309 osd.10 up 1.00000 1.00000 > 11 ssd 3.49309 osd.11 up 1.00000 1.00000 > 17 ssd 3.49309 osd.17 up 1.00000 1.00000 > 20 ssd 3.49309 osd.20 up 1.00000 1.00000 > 23 ssd 3.49309 osd.23 up 1.00000 1.00000 > 25 ssd 3.49309 osd.25 up 1.00000 1.00000 > 26 ssd 3.49309 osd.26 up 1.00000 1.00000 > 27 ssd 3.49309 osd.27 up 1.00000 1.00000 > 28 ssd 3.49309 osd.28 up 1.00000 1.00000 > 30 ssd 3.49309 osd.30 up 1.00000 1.00000 > 77 ssd 3.49309 osd.77 up 1.00000 1.00000 > 82 ssd 3.49309 osd.82 up 1.00000 1.00000 > 87 ssd 3.49309 osd.87 up 1.00000 1.00000 > 97 ssd 3.49309 osd.97 up 1.00000 1.00000 > -22 38.42397 host pl-host03n > 3 ssd 3.49309 osd.3 up 1.00000 1.00000 > 4 ssd 3.49309 osd.4 up 1.00000 1.00000 > 6 ssd 3.49309 osd.6 up 1.00000 1.00000 > 8 ssd 3.49309 osd.8 up 1.00000 1.00000 > 13 ssd 3.49309 osd.13 up 1.00000 1.00000 > 18 ssd 3.49309 osd.18 up 1.00000 1.00000 > 59 ssd 3.49309 osd.59 up 1.00000 1.00000 > 66 ssd 3.49309 osd.66 up 1.00000 1.00000 > 102 ssd 3.49309 osd.102 up 1.00000 1.00000 > 106 ssd 3.49309 osd.106 up 1.00000 1.00000 > 110 ssd 3.49309 osd.110 up 1.00000 1.00000 > -28 20.95853 host pl-host04n > 48 nvme 3.49309 osd.48 up 1.00000 1.00000 > 5 ssd 3.49309 osd.5 up 1.00000 1.00000 > 9 ssd 3.49309 osd.9 up 1.00000 1.00000 > 21 ssd 3.49309 osd.21 up 1.00000 1.00000 > 36 ssd 3.49309 osd.36 up 1.00000 1.00000 > 43 ssd 3.49309 osd.43 up 1.00000 1.00000 > -31 80.34102 host pl-host05n > 24 nvme 3.49309 osd.24 up 1.00000 1.00000 > 45 nvme 3.49309 osd.45 up 1.00000 1.00000 > 49 nvme 3.49309 osd.49 up 1.00000 1.00000 > 55 nvme 3.49309 osd.55 up 1.00000 1.00000 > 60 nvme 3.49309 osd.60 up 1.00000 1.00000 > 65 nvme 3.49309 osd.65 up 1.00000 1.00000 > 70 nvme 3.49309 osd.70 up 1.00000 1.00000 > 85 nvme 3.49309 osd.85 up 1.00000 1.00000 > 90 nvme 3.49309 osd.90 up 1.00000 1.00000 > 1 ssd 3.49309 osd.1 up 1.00000 1.00000 > 14 ssd 3.49309 osd.14 up 1.00000 1.00000 > 31 ssd 3.49309 osd.31 up 1.00000 1.00000 > 35 ssd 3.49309 osd.35 up 1.00000 1.00000 > 37 ssd 3.49309 osd.37 up 1.00000 1.00000 > 39 ssd 3.49309 osd.39 up 1.00000 1.00000 > 41 ssd 3.49309 osd.41 up 1.00000 1.00000 > 80 ssd 3.49309 osd.80 up 1.00000 1.00000 > 95 ssd 3.49309 osd.95 up 1.00000 1.00000 > 100 ssd 3.49309 osd.100 up 1.00000 1.00000 > 108 ssd 3.49309 osd.108 up 1.00000 1.00000 > 112 ssd 3.49309 osd.112 up 1.00000 1.00000 > 116 ssd 3.49309 osd.116 up 1.00000 1.00000 > 125 ssd 3.49309 osd.125 up 1.00000 1.00000 >> On Jun 19, 2025, at 3:25 AM, Frédéric Nass <frederic.n...@univ-lorraine.fr> >> wrote: >> hare a 'ceph config dump' (remove any passwords / confidential information >> from >> the output) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
