Hi Eugen,
Thanks a lot for chiming in — even a partial recollection helps narrow
the search space.
A few notes from our side, in case it helps differentiate the cases:
- No rbd-mirroring at the customer; single cluster, 39 nodes on Reef 18.2.x.
- Pattern affects 4 OSDs on 7 out of 39 nodes, persistent over days
(15M+ log lines in 3 days), no clock skew, quorum stable.
- We reproduced a very similar log signature on a small test cluster
(Reef 18.2.7) by isolating a node hosting MON leader + OSDs for >3h,
matching the original xiaowenhao111 scenario. On our test cluster the
affected OSDs eventually self-recovered after a couple of key-rotation
cycles.
- At the customer, the same signature is NOT clearing on its own.
Two questions on your prior case, if anything comes back to you:
1. Was zap-and-readd actually required, or was it more of a defensive
cleanup after another action (e.g. a MON failover, key rotation, or
simply waiting)? In other words: did you ever observe the OSDs unstick
on their own, or were they truly stuck?
2. Do you remember whether a MON leader failover alone was tried before
going to OSD rebuild, and whether it had any effect?
The fact that our reproducer self-recovered while the production case
doesn't suggests there's something cluster-specific keeping the rotation
desynchronised — possibly stale state on the affected nodes rather than
a pure timing/quorum effect, which would line up with your "remainder
from previous tests" intuition.
If nobody points to an existing tracker, we're prepared to open one with
logs from both the reproduction and the customer environment.
Thanks again,
Andrea
Il 24/04/26 22:06, Eugen Block via ceph-users ha scritto:
Hi,
how did you end up in that situation (did something happen in the
cluster)? Is it affecting only OSDs or other daemons as well?
We did investigate such an issue one and a half years ago for a
customer. They added hosts to one of the two clusters, at that time
the customer had two sites, with rbd mirroring enabled. And
I don't recall the exact details, we did solve it in an online meeting
and I don't think anyone wrote a protocoll. And the session definitely
wasn't recored. :-) It might have some remainder on the new host from
previous tests, or some misconfiguration, I really can't tell anymore.
I think in the end we ended up removing the OSDs, zapping them and
adding them back. Of course, it could be a different root cause, not
sure which role the mon leader can play in this. But I thought I'd
share this tiny bit at least. :-)
Regards,
Eugen
Zitat von Andrea Martra via ceph-users <[email protected]>:
Hi all,
I'm reaching back to this old thread from December 2023 because
we're investigating a very similar pattern at a customer site
running Ceph Reef 18.2.x. Logs show identical shape:
AuthRegistry get_handler peer_type 4 method 2 ...
cephx: verify_authorizer decrypted service osd secret_id=3524
auth: could not find secret_id=3524
auth: dump_rotating: (local range ~971-973)
cephx: verify_authorizer could not get service secret ...
Affecting 4 OSDs on 7 out of 39 nodes, persistent over days
(15M+ log lines in 3 days). No clock skew, quorum stable.
@xiaowenhao111 or anyone else who hit this: did you ever
find a root cause or a reliable fix?
The original thread
didn't seem to reach a conclusion. Specifically:
- Did restarting the affected OSDs clear it, or did the
pattern come back?
- Was a MON leader failover enough, or did you need to
rebuild anything?
- Any tracker issue filed for this?
We're trying to reproduce on a test cluster (Reef 18.2.7)
by isolating a node hosting MON leader + OSDs via network
for 3+ hours, to match the scenario described in the
original thread. Happy to share findings back to the list.
Thanks,
Andrea
Il 21/12/23 02:42, xiaowenhao111 ha scritto:
Hello ceph-users:
First ,sorry my english ...
I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why
test steps:
1、config all hosts contain chronyd service
2、cephadm --image quay.io/ceph/ceph:v16.2.10
<http://quay.io/ceph/ceph:v16.2.10> bootstrap
--dashboard-password-noupdate --mon-ip 10.40.10.200
<http://10.40.10.200> --cluster-network=10.40.10.0/24
<http://10.40.10.0/24> --skip-pull --allow-overwrite
--skip-monitoring-stack --allow-mismatched-release
3、add hosts
ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2
ceph orch host add node2 10.40.10.202
<tel:2104010202> --labels=_admin
ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1
ceph orch host add node-1 10.40.10.202
<tel:1104010202> --labels=_admin
4、add osds:
ceph orch daemon add osd node1:/dev/sda
ceph orch daemon add osd node1:/dev/sdb
ceph orch daemon add osd node1:/dev/sdc
ceph orch daemon add osd node1:/dev/sdd
ceph orch daemon add osd node2:/dev/sda
ceph orch daemon add osd node2:/dev/sdb
ceph orch daemon add osd node2:/dev/sdc
ceph orch daemon add osd node2:/dev/sdd
5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3
6、disconnect node1 (which is monitor leader) cluster and public
network
ifdown enp35s0f0
ifdown enp35s0f1
7、stay overnight（more than three hours）
8、connect node1 cluster and public network
ifup enp35s0f0
ifup enp35s0f1
then, the osds of node1 call get_auth_session_key from mon.node1
which is not new leader，get the wrong secret_id=4.
the others report error "could not find secret_id=4" more than 40
minutes
the ceph.log content of node1:
2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800
mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800
mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s))
2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800
mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800
mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall
HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm daemon(s);
1 pool(s) do not have an application enabled
2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800
mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s))
2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800
mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0
(osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it
is still running
2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800
mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 calling
monitor election
2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800
mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2
calling monitor election
2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800
mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor
election
2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800
mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 calling
monitor election
2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800
mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new
leader, mons node1,node2,node-1 in quorum (ranks 0,1,2)
2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800
mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check
cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1)
the ceph.log content of node2:
2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800
mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800
mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s))
2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800
mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800
mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s))
2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800
mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800
mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked
itself dead as of e91
2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800
mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked
itself dead as of e91
2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800
mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked
itself dead as of e91
2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800 osd.5
(osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down, but it
is still running
2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800 osd.4
(osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, but it
is still running
2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800
mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check
update: Degraded data redundancy: 1/8 objects degraded (12.500
<tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED)
2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800
mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked
itself dead as of e94
2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3
(osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but it
is still running
2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800
mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR] Failed
to load ceph-mgr modules: k8sevents
2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800 osd.0
(osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down, but it
is still running
2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800
mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1 calling
monitor election
2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800
mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2
calling monitor election
2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800
mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor
election
2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800
mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1 calling
monitor election
2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800
mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is new
leader, mons node1,node2,node-1 in quorum (ranks 0,1,2)
2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800
mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check
cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1)
the osd log of node1:
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6804
<http://10.40.10.202:6804> osd.6 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6812
<http://10.40.10.202:6812> osd.7 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6820
<http://10.40.10.202:6820> osd.8 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.202:6828
<http://10.40.10.202:6828> osd.9 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: reset
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: 0x562066 <tel:09562066>bbc000
handle_response ret = 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: got initial server challenge
5fd5d9eca256ae69
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: validate_tickets: want=53 need=0 have=53
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: validate_tickets want 53 have 0 need 53
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: want=53 need=53 have=0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: build_request
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
20 <tel:864020> cephx client: old ticket len=96
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: get auth session key: client_challenge
7ca3f7c3ef30b2bf
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: 0x562066 <tel:09562066>bbc000
handle_response ret = 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: get_auth_session_key
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply got 1 keys
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: got key for service_id auth
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply got encrypted ticket
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=2
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply service auth
secret_id 2 session_key AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg==
validity=259200.000000 <tel:259200000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket
expires=2023-12-24 <tel:20231224>T01:13:44.682217
<tel:44682217>+0800 renew_after=2023-12-23
<tel:20231223>T07:13:44.682217 <tel:44682217>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: want=53 need=53 have=0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: got connection bl 84 and extra
tickets 550
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: got connection_secret 64 bytes
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply got 3 keys
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: got key for service_id mon
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply service mon
secret_id 4 session_key AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ==
validity=3600.000000 <tel:3600000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket
expires=2023-12-21 <tel:20231221>T02:13:44.682256
<tel:44682256>+0800 renew_after=2023-12-21
<tel:20231221>T01:58:44.682256 <tel:44682256>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: got key for service_id osd
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply service osd
secret_id 4 session_key AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA==
validity=3600.000000 <tel:3600000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket
expires=2023-12-21 <tel:20231221>T02:13:44.682270
<tel:44682270>+0800 renew_after=2023-12-21
<tel:20231221>T01:58:44.682270 <tel:44682270>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: got key for service_id mgr
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket.secret_id=4
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply service mgr
secret_id 4 session_key AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg==
validity=3600.000000 <tel:3600000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: verify_service_ticket_reply ticket
expires=2023-12-21 <tel:20231221>T02:13:44.682286
<tel:44682286>+0800 renew_after=2023-12-21
<tel:20231221>T01:58:44.682286 <tel:44682286>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: got extra service_tickets
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: validate_tickets want 53 have 53 need 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx: validate_tickets want 53 have 53 need 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
20 <tel:864020> cephx client: need_tickets: want=53 have=53 need=0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
-1 <tel:86401> monclient: _check_auth_rotating possible clock skew,
rotating keys expired way too early (before 2023-12-21
<tel:20231221>T00:13:44.682376 <tel:44682376>+0800)
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800 7fcd2bed8640
10 <tel:864010> cephx client: build_rotating_request
2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.201:6804
<http://10.40.10.201:6804> osd.1 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.201:6812
<http://10.40.10.201:6812> osd.2 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800 7fcd27c2e640
-1 osd.0 83 heartbeat_check: no reply from 10.40.10.200:6814
<http://10.40.10.200:6814> osd.3 ever on either front or back, first
ping sent 2023-12-21 <tel:20231221>T01:13:23.148457
<tel:23148457>+0800 (oldest deadline 2023-12-21
<tel:20231221>T01:13:43.148457 <tel:43148457>+0800)
the osd log of node2:
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640
10 <tel:64010> cephx: validate_tickets want 53 have 53 need 0
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640
20 <tel:64020> cephx client: need_tickets: want=53 have=53 need=0
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640
10 <tel:64010> auth: dump_rotating:
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800 7fa573a6d640
10 <tel:64010> auth: id 17 AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg==
expires 2023-12-21 <tel:20231221>T00:34:21.152105 <tel:21152105>+0800
2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 7fa573a6d640
10 <tel:64010> auth: id 18 AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA==
expires 2023-12-21 <tel:20231221>T01:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800 7fa573a6d640
10 <tel:64010> auth: id 19 AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g==
expires 2023-12-21 <tel:20231221>T02:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640
20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0) get_handler
peer_type 4 method 2 cluster_methods [2] service_methods [2]
client_methods [2]
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640
10 <tel:58129364010> cephx: verify_authorizer decrypted service osd
secret_id=4
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640
10 <tel:58129364010> auth: dump_rotating:
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640
10 <tel:58129364010> auth: id 17
AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21
<tel:20231221>T00:34:21.152105 <tel:21152105>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640
10 <tel:58129364010> auth: id 18
AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21
<tel:20231221>T01:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800 7fa581293640
10 <tel:58129364010> auth: id 19
AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21
<tel:20231221>T02:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could not
get service secret for service osd secret_id=4
2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 7fa573a6d640
10 <tel:64010> cephx: validate_tickets want 53 have 53 need 0
2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800 7fa573a6d640
20 <tel:64020> cephx client: need_tickets: want=53 have=53 need=0
the mon log of node1:
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> start_session entity_name=mgr.node1.apderh
global_id=14184 <tel:14184> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: start_session
server_challenge 524ce5eea3e4dd75
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: handle_request
get_auth_session_key for mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
20 <tel:964020> cephx server mgr.node1.apderh: checking key:
req.key=f41952 <tel:41952>c0f02e6c0f expected_key=f41952
<tel:41952>c0f02e6c0f
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
20 <tel:964020> cephx server mgr.node1.apderh: checking old_ticket:
secret_id=2 len=112, old_ticket_may_be_omitted=0
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
20 <tel:964020> cephx server mgr.node1.apderh: decoded old_ticket:
global_id=14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: allowing reclaim of
global_id 14184 <tel:14184> (valid ticket presented, will encrypt
new ticket)
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket_reply encoding 1 tickets
with secret AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg==
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket service auth secret_id 2
ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id
14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: adding key for
service mon
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: adding key for
service mds
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: adding key for
service osd
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx server mgr.node1.apderh: adding key for
service mgr
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_caps: name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx keyserverdata: get_secret: num of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket_reply encoding 4 tickets
with secret AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A==
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket service mon secret_id 4
ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id
14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket service mds secret_id 4
ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id
14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket service osd secret_id 4
ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id
14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800 7fa7976e9640
10 <tel:964010> cephx: build_service_ticket service mgr secret_id 4
ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh ticket.global_id
14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800
7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command
mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}
v 0) v1
2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800
7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] :
from='mgr.14184 10.40.10.200:0/2514043697
<http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]:
dispatch
2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800
7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3 handle_command
mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}
v 0) v1
2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800
7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] :
from='mgr.14184 10.40.10.200:0/2514043697
<http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]:
dispatch
the mon log of node2:
2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800
7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
{"time_micros": 1703092390293629, "job": 1145, "event":
"table_file_deletion", "file_number": 2272}
2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800
7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
{"time_micros": 1703092390296385, "job": 1145, "event":
"table_file_deletion", "file_number": 2270}
2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800
7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
{"time_micros": 1703092390304703, "job": 1145, "event":
"table_file_deletion", "file_number": 2269}
2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800
7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}
v 0) v1
2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]:
dispatch
2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}
v 0) v1
2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]:
dispatch
2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix": "config dump", "format": "json"} v 0) v1
2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix": "config generate-minimal-conf"} v 0) v1
2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix": "auth get", "entity": "client.admin"} v 0) v1
2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}] v
0) v1
2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800
7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800 7f88bd6c9640
20 <tel:964020> cephx keyserver: prepare_rotating_update before:
data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640
10 <tel:64010> start_session entity_name=osd.0 global_id=14205
<tel:14205> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640
10 <tel:64010> cephx server osd.0: start_session server_challenge
dba982c2bced072d
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640
10 <tel:64010> start_session entity_name=osd.3 global_id=14379
<tel:14379> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640
10 <tel:64010> cephx server osd.3: start_session server_challenge
e9db92f3f5af188c
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640
10 <tel:64010> start_session entity_name=mgr.node1.apderh
global_id=14190 <tel:14190> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88beecc640
10 <tel:64010> cephx server mgr.node1.apderh: start_session
server_challenge b9809292 <tel:9809292>ea2e62f6
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640
10 <tel:64010> start_session entity_name=osd.5 global_id=24253
<tel:24253> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800 7f88bf6cd640
10 <tel:64010> cephx server osd.5: start_session server_challenge
1be9f89b4bd9cdbc
2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88b9ec2640
10 <tel:264010> start_session entity_name=osd.4 global_id=14406
<tel:14406> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88b9ec2640
10 <tel:264010> cephx server osd.4: start_session server_challenge
cca4694d496c39a8
2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800 7f88beecc640
10 <tel:64010> cephx server osd.0: handle_request
get_auth_session_key for osd.0
_______________________________________________
ceph-users mailing list [email protected]
To unsubscribe send an email [email protected]
--
--
Andrea Martra
Senior Manager
+39 393 9048451
Open Technologies Sas
Via Michele Buniva, 26 - 10064 Pinerolo (TO)
Web:http://www.oscct.it
** Open Source Cloud Computing Technologies **
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
--
Andrea Martra
Senior Manager
+39 393 9048451
Open Technologies Sas
Via Michele Buniva, 26 - 10064 Pinerolo (TO)
Web: http://www.oscct.it
** Open Source Cloud Computing Technologies **
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
