Hi,
I'm quite sure that we didn't investige MONs at that time, so no, we
haven't tried a leader failover. But it might be possible that they
rebooted nodes, so that would have eventually led to a failover. I
remember that we also did try to rotate keys, that I do have in my
email history, but it didn't have any impact. Waiting also didn't
help, sorry for not having more memories of that incident. :-D
But it's interesting that you could reproduce it with the mon leader,
I wouldn't have expected that.
Zitat von Andrea Martra via ceph-users <[email protected]>:
Hi Eugen,
Thanks a lot for chiming in — even a partial recollection helps
narrow the search space.
A few notes from our side, in case it helps differentiate the cases:
- No rbd-mirroring at the customer; single cluster, 39 nodes on Reef 18.2.x.
- Pattern affects 4 OSDs on 7 out of 39 nodes, persistent over days
(15M+ log lines in 3 days), no clock skew, quorum stable.
- We reproduced a very similar log signature on a small test cluster
(Reef 18.2.7) by isolating a node hosting MON leader + OSDs for >3h,
matching the original xiaowenhao111 scenario. On our test cluster
the affected OSDs eventually self-recovered after a couple of
key-rotation cycles.
- At the customer, the same signature is NOT clearing on its own.
Two questions on your prior case, if anything comes back to you:
1. Was zap-and-readd actually required, or was it more of a
defensive cleanup after another action (e.g. a MON failover, key
rotation, or simply waiting)? In other words: did you ever observe
the OSDs unstick on their own, or were they truly stuck?
2. Do you remember whether a MON leader failover alone was tried
before going to OSD rebuild, and whether it had any effect?
The fact that our reproducer self-recovered while the production
case doesn't suggests there's something cluster-specific keeping the
rotation desynchronised — possibly stale state on the affected nodes
rather than a pure timing/quorum effect, which would line up with
your "remainder from previous tests" intuition.
If nobody points to an existing tracker, we're prepared to open one
with logs from both the reproduction and the customer environment.
Thanks again,
Andrea
Il 24/04/26 22:06, Eugen Block via ceph-users ha scritto:
Hi,
how did you end up in that situation (did something happen in the
cluster)? Is it affecting only OSDs or other daemons as well?
We did investigate such an issue one and a half years ago for a
customer. They added hosts to one of the two clusters, at that time
the customer had two sites, with rbd mirroring enabled. And
I don't recall the exact details, we did solve it in an online
meeting and I don't think anyone wrote a protocoll. And the session
definitely wasn't recored. :-) It might have some remainder on the
new host from previous tests, or some misconfiguration, I really
can't tell anymore. I think in the end we ended up removing the
OSDs, zapping them and adding them back. Of course, it could be a
different root cause, not sure which role the mon leader can play
in this. But I thought I'd share this tiny bit at least. :-)
Regards,
Eugen
Zitat von Andrea Martra via ceph-users <[email protected]>:
Hi all,
I'm reaching back to this old thread from December 2023 because
we're investigating a very similar pattern at a customer site
running Ceph Reef 18.2.x. Logs show identical shape:
AuthRegistry get_handler peer_type 4 method 2 ...
cephx: verify_authorizer decrypted service osd secret_id=3524
auth: could not find secret_id=3524
auth: dump_rotating: (local range ~971-973)
cephx: verify_authorizer could not get service secret ...
Affecting 4 OSDs on 7 out of 39 nodes, persistent over days
(15M+ log lines in 3 days). No clock skew, quorum stable.
@xiaowenhao111 or anyone else who hit this: did you ever
find a root cause or a reliable fix?
The original thread
didn't seem to reach a conclusion. Specifically:
- Did restarting the affected OSDs clear it, or did the
pattern come back?
- Was a MON leader failover enough, or did you need to
rebuild anything?
- Any tracker issue filed for this?
We're trying to reproduce on a test cluster (Reef 18.2.7)
by isolating a node hosting MON leader + OSDs via network
for 3+ hours, to match the scenario described in the
original thread. Happy to share findings back to the list.
Thanks,
Andrea
Il 21/12/23 02:42, xiaowenhao111 ha scritto:
Hello ceph-users:
First ,sorry my english ...
I find a bug (16.2.10 <tel:16210> and 18.2.1), but i do not know why
test steps:
1、config all hosts contain chronyd service
2、cephadm --image quay.io/ceph/ceph:v16.2.10
<http://quay.io/ceph/ceph:v16.2.10> bootstrap
--dashboard-password-noupdate --mon-ip 10.40.10.200
<http://10.40.10.200> --cluster-network=10.40.10.0/24
<http://10.40.10.0/24> --skip-pull --allow-overwrite
--skip-monitoring-stack --allow-mismatched-release
3、add hosts
ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node2
ceph orch host add node2 10.40.10.202 <tel:2104010202> --labels=_admin
ssh-copy-id -f -i /etc/ceph/ceph.pub <http://ceph.pub> root@node-1
ceph orch host add node-1 10.40.10.202
<tel:1104010202> --labels=_admin
4、add osds:
ceph orch daemon add osd node1:/dev/sda
ceph orch daemon add osd node1:/dev/sdb
ceph orch daemon add osd node1:/dev/sdc
ceph orch daemon add osd node1:/dev/sdd
ceph orch daemon add osd node2:/dev/sda
ceph orch daemon add osd node2:/dev/sdb
ceph orch daemon add osd node2:/dev/sdc
ceph orch daemon add osd node2:/dev/sdd
5、ceph osd pool create test_pool 512 512 <tel:512512> --size=3
6、disconnect node1 (which is monitor leader) cluster and public network
ifdown enp35s0f0
ifdown enp35s0f1
7、stay overnight（more than three hours）
8、connect node1 cluster and public network
ifup enp35s0f0
ifup enp35s0f1
then, the osds of node1 call get_auth_session_key from mon.node1
which is not new leader，get the wrong secret_id=4.
the others report error "could not find secret_id=4" more than 40 minutes
the ceph.log content of node1:
2023-12-20 <tel:20231220>T09:48:30.199224 <tel:30199224>+0800
mon.node1 (mon.0) 1891 <tel:01891> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-20 <tel:20231220>T09:49:37.816260 <tel:37816260>+0800
mon.node1 (mon.0) 1901 <tel:01901> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
daemon(s))
2023-12-20 <tel:20231220>T09:49:44.222046 <tel:44222046>+0800
mon.node1 (mon.0) 1902 <tel:01902> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-20 <tel:20231220>T09:50:00.000090 <tel:00000090>+0800
mon.node1 (mon.0) 1905 <tel:01905> : cluster [WRN] overall
HEALTH_WARN Failed to place 1 daemon(s); 3 failed cephadm
daemon(s); 1 pool(s) do not have an application enabled
2023-12-20 <tel:20231220>T09:50:51.916464 <tel:51916464>+0800
mon.node1 (mon.0) 1912 <tel:01912> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
daemon(s))
2023-12-20 <tel:20231220>T09:51:07.127982 <tel:07127982>+0800
mon.node1 (mon.0) 1916 <tel:01916> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800
osd.0 (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down,
but it is still running
2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800
mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1
calling monitor election
2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800
mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2
calling monitor election
2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800
mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor
election
2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800
mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1
calling monitor election
2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800
mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is
new leader, mons node1,node2,node-1 in quorum (ranks 0,1,2)
2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800
mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check
cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1)
the ceph.log content of node2:
2023-12-21 <tel:20231221>T01:10:01.065051 <tel:01065051>+0800
mon.node2 (mon.1) 11522 <tel:111522> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:11:11.092640 <tel:11092640>+0800
mon.node2 (mon.1) 11531 <tel:111531> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
daemon(s))
2023-12-21 <tel:20231221>T01:11:19.197355 <tel:19197355>+0800
mon.node2 (mon.1) 11532 <tel:111532> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:12:34.469403 <tel:34469403>+0800
mon.node2 (mon.1) 11547 <tel:111547> : cluster [INF] Health check
cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1
daemon(s))
2023-12-21 <tel:20231221>T01:12:42.476793 <tel:42476793>+0800
mon.node2 (mon.1) 11549 <tel:111549> : cluster [WRN] Health check
failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
2023-12-21 <tel:20231221>T01:13:55.234163 <tel:55234163>+0800
mon.node2 (mon.1) 11572 <tel:111572> : cluster [INF] osd.5 marked
itself dead as of e91
2023-12-21 <tel:20231221>T01:13:55.236348 <tel:55236348>+0800
mon.node2 (mon.1) 11573 <tel:111573> : cluster [INF] osd.4 marked
itself dead as of e91
2023-12-21 <tel:20231221>T01:13:55.242209 <tel:55242209>+0800
mon.node2 (mon.1) 11574 <tel:111574> : cluster [INF] osd.0 marked
itself dead as of e91
2023-12-21 <tel:20231221>T01:13:55.233774 <tel:55233774>+0800
osd.5 (osd.5) 3 : cluster [WRN] Monitor daemon marked osd.5 down,
but it is still running
2023-12-21 <tel:20231221>T01:13:55.236038 <tel:55236038>+0800
osd.4 (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down,
but it is still running
2023-12-21 <tel:20231221>T01:14:00.109136 <tel:109136>+0800
mon.node2 (mon.1) 11589 <tel:111589> : cluster [WRN] Health check
update: Degraded data redundancy: 1/8 objects degraded (12.500
<tel:12500>%), 1 pg degraded, 188 pgs undersized (PG_DEGRADED)
2023-12-21 <tel:20231221>T01:14:00.311644 <tel:311644>+0800
mon.node2 (mon.1) 11590 <tel:111590> : cluster [INF] osd.3 marked
itself dead as of e94
2023-12-21 <tel:20231221>T01:14:00.311410 <tel:311410>+0800 osd.3
(osd.3) 3 : cluster [WRN] Monitor daemon marked osd.3 down, but
it is still running
2023-12-21 <tel:20231221>T01:14:01.922553 <tel:01922553>+0800
mgr.node1.apderh (mgr.34343) 1 <tel:343431> : cluster [ERR]
Failed to load ceph-mgr modules: k8sevents
2023-12-21 <tel:20231221>T01:13:55.241891 <tel:55241891>+0800
osd.0 (osd.0) 3 : cluster [WRN] Monitor daemon marked osd.0 down,
but it is still running
2023-12-21 <tel:20231221>T01:14:05.739585 <tel:05739585>+0800
mon.node1 (mon.0) 1969 <tel:01969> : cluster [INF] mon.node1
calling monitor election
2023-12-21 <tel:20231221>T01:14:05.852877 <tel:05852877>+0800
mon.node2 (mon.1) 11595 <tel:111595> : cluster [INF] mon.node2
calling monitor election
2023-12-21 <tel:20231221>T01:14:05.853054 <tel:05853054>+0800
mon.node-1 (mon.2) 36 : cluster [INF] mon.node-1 calling monitor
election
2023-12-21 <tel:20231221>T01:14:05.964608 <tel:05964608>+0800
mon.node1 (mon.0) 1970 <tel:01970> : cluster [INF] mon.node1
calling monitor election
2023-12-21 <tel:20231221>T01:14:06.013934 <tel:06013934>+0800
mon.node1 (mon.0) 1971 <tel:01971> : cluster [INF] mon.node1 is
new leader, mons node1,node2,node-1 in quorum (ranks 0,1,2)
2023-12-21 <tel:20231221>T01:14:06.041158 <tel:06041158>+0800
mon.node1 (mon.0) 1976 <tel:01976> : cluster [INF] Health check
cleared: MON_DOWN (was: 1/3 mons down, quorum node2,node-1)
the osd log of node1:
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.202:6804 <http://10.40.10.202:6804> osd.6 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.202:6812 <http://10.40.10.202:6812> osd.7 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.202:6820 <http://10.40.10.202:6820> osd.8 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:43.720 <tel:43720>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.202:6828 <http://10.40.10.202:6828> osd.9 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: reset
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: 0x562066
<tel:09562066>bbc000 handle_response ret = 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: got initial server
challenge 5fd5d9eca256ae69
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: validate_tickets:
want=53 need=0 have=53
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have
0 need 53
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: build_request
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 20 <tel:864020> cephx client: old ticket len=96
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: get auth session key:
client_challenge 7ca3f7c3ef30b2bf
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: 0x562066
<tel:09562066>bbc000 handle_response ret = 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: get_auth_session_key
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
got 1 keys
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: got key for service_id auth
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
got encrypted ticket
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket.secret_id=2
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
service auth secret_id 2 session_key
AQDIIINlUlSmKBAAvL5rKyyIb6vt89rxUGlsxg== validity=259200.000000
<tel:259200000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket expires=2023-12-24 <tel:20231224>T01:13:44.682217
<tel:44682217>+0800 renew_after=2023-12-23
<tel:20231223>T07:13:44.682217 <tel:44682217>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: want=53 need=53 have=0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: got connection bl 84
and extra tickets 550
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: got connection_secret
64 bytes
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
got 3 keys
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mon
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket.secret_id=4
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
service mon secret_id 4 session_key
AQDIIINlWvamKBAAxOaZsuCMdgutax1n2ZRbZQ== validity=3600.000000
<tel:3600000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket expires=2023-12-21 <tel:20231221>T02:13:44.682256
<tel:44682256>+0800 renew_after=2023-12-21
<tel:20231221>T01:58:44.682256 <tel:44682256>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: got key for service_id osd
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket.secret_id=4
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
service osd secret_id 4 session_key
AQDIIINlXAinKBAAOwTlViH3CJRgSvj5Kn9qtA== validity=3600.000000
<tel:3600000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket expires=2023-12-21 <tel:20231221>T02:13:44.682270
<tel:44682270>+0800 renew_after=2023-12-21
<tel:20231221>T01:58:44.682270 <tel:44682270>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: got key for service_id mgr
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket.secret_id=4
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
service mgr secret_id 4 session_key
AQDIIINlUiGnKBAAr6WBaSawzVyDoqEs8cENdg== validity=3600.000000
<tel:3600000000>
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: verify_service_ticket_reply
ticket expires=2023-12-21 <tel:20231221>T02:13:44.682286
<tel:44682286>+0800 renew_after=2023-12-21
<tel:20231221>T01:58:44.682286 <tel:44682286>+0800
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: got extra
service_tickets
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have
53 need 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx: validate_tickets want 53 have
53 need 0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 20 <tel:864020> cephx client: need_tickets: want=53
have=53 need=0
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 -1 <tel:86401> monclient: _check_auth_rotating
possible clock skew, rotating keys expired way too early (before
2023-12-21 <tel:20231221>T00:13:44.682376 <tel:44682376>+0800)
2023-12-21 <tel:20231221>T01:13:44.680 <tel:44680>+0800
7fcd2bed8640 10 <tel:864010> cephx client: build_rotating_request
2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.201:6804 <http://10.40.10.201:6804> osd.1 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.201:6812 <http://10.40.10.201:6812> osd.2 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
2023-12-21 <tel:20231221>T01:13:44.719 <tel:44719>+0800
7fcd27c2e640 -1 osd.0 83 heartbeat_check: no reply from
10.40.10.200:6814 <http://10.40.10.200:6814> osd.3 ever on either
front or back, first ping sent 2023-12-21
<tel:20231221>T01:13:23.148457 <tel:23148457>+0800 (oldest
deadline 2023-12-21 <tel:20231221>T01:13:43.148457
<tel:43148457>+0800)
the osd log of node2:
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have
53 need 0
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53
have=53 need=0
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
7fa573a6d640 10 <tel:64010> auth: dump_rotating:
2023-12-21 <tel:20231221>T01:13:51.255 <tel:51255>+0800
7fa573a6d640 10 <tel:64010> auth: id 17
AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21
<tel:20231221>T00:34:21.152105 <tel:21152105>+0800
2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800
7fa573a6d640 10 <tel:64010> auth: id 18
AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21
<tel:20231221>T01:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:51.256 <tel:51256>+0800
7fa573a6d640 10 <tel:64010> auth: id 19
AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21
<tel:20231221>T02:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 20 <tel:58129364020> AuthRegistry(0x7ffc86c1e7b0)
get_handler peer_type 4 method 2 cluster_methods [2]
service_methods [2] client_methods [2]
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 10 <tel:58129364010> cephx: verify_authorizer
decrypted service osd secret_id=4
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 0 <tel:5812936400> auth: could not find secret_id=4
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 10 <tel:58129364010> auth: dump_rotating:
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 10 <tel:58129364010> auth: id 17
AQBr+4Jlx7z6GRAASZPpy2tmX0O0YI+hwmtuHg== expires 2023-12-21
<tel:20231221>T00:34:21.152105 <tel:21152105>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 10 <tel:58129364010> auth: id 18
AQCBCYNlibFfLxAAHEf4/1kldHQcXRqZE7fkcA== expires 2023-12-21
<tel:20231221>T01:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 10 <tel:58129364010> auth: id 19
AQCOF4NlI5MZCxAAKOfv6wvo7f/HZbDIs20I/g== expires 2023-12-21
<tel:20231221>T02:34:25.794799 <tel:25794799>+0800
2023-12-21 <tel:20231221>T01:13:52.170 <tel:52170>+0800
7fa581293640 0 <tel:5812936400> cephx: verify_authorizer could
not get service secret for service osd secret_id=4
2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800
7fa573a6d640 10 <tel:64010> cephx: validate_tickets want 53 have
53 need 0
2023-12-21 <tel:20231221>T01:13:52.256 <tel:52256>+0800
7fa573a6d640 20 <tel:64020> cephx client: need_tickets: want=53
have=53 need=0
the mon log of node1:
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> start_session
entity_name=mgr.node1.apderh global_id=14184
<tel:14184> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
start_session server_challenge 524ce5eea3e4dd75
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
handle_request get_auth_session_key for mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh:
checking key: req.key=f41952 <tel:41952>c0f02e6c0f
expected_key=f41952 <tel:41952>c0f02e6c0f
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh:
checking old_ticket: secret_id=2 len=112,
old_ticket_may_be_omitted=0
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 20 <tel:964020> cephx server mgr.node1.apderh:
decoded old_ticket: global_id=14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
allowing reclaim of global_id 14184 <tel:14184> (valid ticket
presented, will encrypt new ticket)
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply
encoding 1 tickets with secret
AQBmKIJlvHeYJRAAf5h+auT4c9Ilo/nlUqRJlg==
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
auth secret_id 2 ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh
ticket.global_id 14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
adding key for service mon
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
adding key for service mds
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
adding key for service osd
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx server mgr.node1.apderh:
adding key for service mgr
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_caps:
name=mgr.node1.apderh
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx keyserverdata: get_secret: num
of caps=3
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket_reply
encoding 4 tickets with secret
AQDIIINldY1mGRAApH04rNRZtuyAUrQn5PNG7A==
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
mon secret_id 4 ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh
ticket.global_id 14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
mds secret_id 4 ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh
ticket.global_id 14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
osd secret_id 4 ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh
ticket.global_id 14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.424 <tel:44424>+0800
7fa7976e9640 10 <tel:964010> cephx: build_service_ticket service
mgr secret_id 4 ticket_info.ticket.name
<http://ticket_info.ticket.name>=mgr.node1.apderh
ticket.global_id 14184 <tel:14184>
2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800
7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3
handle_command mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"} v 0)
v1
2023-12-21 <tel:20231221>T01:13:44.433 <tel:44433>+0800
7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] :
from='mgr.14184 10.40.10.200:0/2514043697
<http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/mirror_snapshot_schedule"}]:
dispatch
2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800
7fa7936e1640 0 <tel:16400> mon.node1@0(probing) e3
handle_command mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"} v 0)
v1
2023-12-21 <tel:20231221>T01:13:44.434 <tel:44434>+0800
7fa7936e1640 0 <tel:16400> log_channel(audit) log [INF] :
from='mgr.14184 10.40.10.200:0/2514043697
<http://10.40.10.200:0/2514043697>' entity='mgr.node1.apderh'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node1.apderh/trash_purge_schedule"}]:
dispatch
the mon log of node2:
2023-12-21 <tel:20231221>T01:13:10.292 <tel:10292>+0800
7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
{"time_micros": 1703092390293629, "job": 1145, "event":
"table_file_deletion", "file_number": 2272}
2023-12-21 <tel:20231221>T01:13:10.295 <tel:10295>+0800
7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
{"time_micros": 1703092390296385, "job": 1145, "event":
"table_file_deletion", "file_number": 2270}
2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800
7f88c0ed0640 4 <tel:06404> rocksdb: EVENT_LOG_v1
{"time_micros": 1703092390304703, "job": 1145, "event":
"table_file_deletion", "file_number": 2269}
2023-12-21 <tel:20231221>T01:13:10.303 <tel:10303>+0800
7f88b66bb640 4 rocksdb:
[db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb:
[db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb:
[db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb:
[db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:10.304 <tel:10304>+0800
7f88b66bb640 4 rocksdb:
[db_impl/db_impl_compaction_flush.cc:1442
<http://db_impl_compaction_flush.cc:1442>] [default] Manual
compaction starting
2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:13.434 <tel:13434>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:18.435 <tel:18435>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:23.436 <tel:23436>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:28.437 <tel:28437>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"} v 0)
v1
2023-12-21 <tel:20231221>T01:13:32.438 <tel:32438>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/trash_purge_schedule"}]:
dispatch
2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"} v 0)
v1
2023-12-21 <tel:20231221>T01:13:32.440 <tel:32440>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix":"config
rm","who":"mgr","name":"mgr/rbd_support/node2.rnirud/mirror_snapshot_schedule"}]:
dispatch
2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:33.438 <tel:33438>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:38.438 <tel:38438>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:43.439 <tel:43439>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:48.440 <tel:48440>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:51.142 <tel:51142>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix": "config dump", "format": "json"} v 0) v1
2023-12-21 <tel:20231221>T01:13:51.143 <tel:51143>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix": "config dump", "format": "json"}]: dispatch
2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix": "config generate-minimal-conf"} v 0) v1
2023-12-21 <tel:20231221>T01:13:51.144 <tel:51144>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [DBG] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch
2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command({"prefix": "auth get", "entity": "client.admin"} v 0)
v1
2023-12-21 <tel:20231221>T01:13:51.145 <tel:51145>+0800
7f88baec4640 0 <tel:46400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch
2023-12-21 <tel:20231221>T01:13:52.247 <tel:52247>+0800
7f88baec4640 0 <tel:46400> mon.node2@1(leader) e3 handle_command
mon_command([{prefix=config-key set, key=mgr/cephadm/host.node1}]
v 0) v1
2023-12-21 <tel:20231221>T01:13:52.260 <tel:52260>+0800
7f88b96c1640 0 <tel:16400> log_channel(audit) log [INF] :
from='mgr.14252 10.40.10.201:0/357060750
<http://10.40.10.201:0/357060750>' entity='mgr.node2.rnirud'
2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800
7f88bd6c9640 1 <tel:96401> mon.node2@1(leader).osd e91
_set_new_cache_sizes cache_size:1020054731
<tel:1020054731> inc_alloc: 71303168 <tel:71303168> full_alloc:
71303168 <tel:71303168> kv_alloc: 876609536 <tel:876609536>
2023-12-21 <tel:20231221>T01:13:53.440 <tel:53440>+0800
7f88bd6c9640 20 <tel:964020> cephx keyserver:
prepare_rotating_update before: data.rotating_ver=17
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88beecc640 10 <tel:64010> start_session entity_name=osd.0
global_id=14205 <tel:14205> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88beecc640 10 <tel:64010> cephx server osd.0: start_session
server_challenge dba982c2bced072d
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.3
global_id=14379 <tel:14379> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88bf6cd640 10 <tel:64010> cephx server osd.3: start_session
server_challenge e9db92f3f5af188c
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88beecc640 10 <tel:64010> start_session
entity_name=mgr.node1.apderh global_id=14190
<tel:14190> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88beecc640 10 <tel:64010> cephx server mgr.node1.apderh:
start_session server_challenge b9809292 <tel:9809292>ea2e62f6
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88bf6cd640 10 <tel:64010> start_session entity_name=osd.5
global_id=24253 <tel:24253> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.187 <tel:55187>+0800
7f88bf6cd640 10 <tel:64010> cephx server osd.5: start_session
server_challenge 1be9f89b4bd9cdbc
2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800
7f88b9ec2640 10 <tel:264010> start_session entity_name=osd.4
global_id=14406 <tel:14406> is_new_global_id=0
2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800
7f88b9ec2640 10 <tel:264010> cephx server osd.4: start_session
server_challenge cca4694d496c39a8
2023-12-21 <tel:20231221>T01:13:55.188 <tel:55188>+0800
7f88beecc640 10 <tel:64010> cephx server osd.0: handle_request
get_auth_session_key for osd.0
_______________________________________________
ceph-users mailing list [email protected]
To unsubscribe send an email [email protected]
--
--
Andrea Martra
Senior Manager
+39 393 9048451
Open Technologies Sas
Via Michele Buniva, 26 - 10064 Pinerolo (TO)
Web:http://www.oscct.it
** Open Source Cloud Computing Technologies **
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--
--
Andrea Martra
Senior Manager
+39 393 9048451
Open Technologies Sas
Via Michele Buniva, 26 - 10064 Pinerolo (TO)
Web: http://www.oscct.it
** Open Source Cloud Computing Technologies **
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
