Package: ceph
Version: 12.2.11+dfsg1-2.1
Severity: grave
Justification: renders package unusable
Dear Maintainer,
I'm trying to deploy a 2 monitor ceph cluster with 2 arm64 server nodes.
root@ceph-node1:~# ceph -v
ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous
(stable)
ceph-node1 192.168.2.233
ceph-node2 192.168.2.177
root@ceph-node1:~# ceph-deploy --version
2.0.1
ceph.conf file:
[global]
fsid = 76018d7b-2258-425b-be7d-f76ae5192486
mon_initial_members = ceph-node1, ceph-node2
mon_host = 192.168.2.233,192.168.2.177
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
public_network = 192.168.2.0/24
when I do mon-initial opertion: ceph-deploy mon create-inital, I get some
failure info
[ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries
left: 5
[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying
[ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries
left: 4
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries
left: 3
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries
left: 2
[ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying
[ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries
left: 1
[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying
[ceph_deploy.mon][INFO ] processing monitor mon.ceph-node2
[ceph-node2][DEBUG ] connected to host: ceph-node2
[ceph-node2][DEBUG ] detect platform information from remote host
[ceph-node2][DEBUG ] detect machine type
[ceph-node2][DEBUG ] find the location of an executable
[ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node2.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries
left: 5
[ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying
[ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node2.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries
left: 4
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node2.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries
left: 3
[ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying
[ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node2.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries
left: 2
[ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying
[ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node2.asok mon_status
[ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries
left: 1
[ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying
[ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum:
[ceph_deploy.mon][ERROR ] ceph-node1
[ceph_deploy.mon][ERROR ] ceph-node2
monmap:mons info: ceph-node2's addr is incorrect
[ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][DEBUG ]
********************************************************************************
[ceph-node1][DEBUG ] status for monitor: mon.ceph-node1
[ceph-node1][DEBUG ] {
[ceph-node1][DEBUG ] "election_epoch": 1,
[ceph-node1][DEBUG ] "extra_probe_peers": [
[ceph-node1][DEBUG ] "192.168.2.177:6789/0"
[ceph-node1][DEBUG ] ],
[ceph-node1][DEBUG ] "feature_map": {
[ceph-node1][DEBUG ] "mon": {
[ceph-node1][DEBUG ] "group": {
[ceph-node1][DEBUG ] "features": "0x3ffddff8eeacfffb",
[ceph-node1][DEBUG ] "num": 1,
[ceph-node1][DEBUG ] "release": "luminous"
[ceph-node1][DEBUG ] }
[ceph-node1][DEBUG ] }
[ceph-node1][DEBUG ] },
[ceph-node1][DEBUG ] "features": {
[ceph-node1][DEBUG ] "quorum_con": "0",
[ceph-node1][DEBUG ] "quorum_mon": [],
[ceph-node1][DEBUG ] "required_con": "0",
[ceph-node1][DEBUG ] "required_mon": []
[ceph-node1][DEBUG ] },
[ceph-node1][DEBUG ] "monmap": {
[ceph-node1][DEBUG ] "created": "2019-08-19 15:36:56.080981",
[ceph-node1][DEBUG ] "epoch": 0,
[ceph-node1][DEBUG ] "features": {
[ceph-node1][DEBUG ] "optional": [],
[ceph-node1][DEBUG ] "persistent": []
[ceph-node1][DEBUG ] },
[ceph-node1][DEBUG ] "fsid": "76018d7b-2258-425b-be7d-f76ae5192486",
[ceph-node1][DEBUG ] "modified": "2019-08-19 15:36:56.080981",
[ceph-node1][DEBUG ] "mons": [
[ceph-node1][DEBUG ] {
[ceph-node1][DEBUG ] "addr": "192.168.2.233:6789/0",
[ceph-node1][DEBUG ] "name": "ceph-node1",
[ceph-node1][DEBUG ] "public_addr": "192.168.2.233:6789/0",
[ceph-node1][DEBUG ] "rank": 0
[ceph-node1][DEBUG ] },
[ceph-node1][DEBUG ] {
[ceph-node1][DEBUG ] "addr": "0.0.0.0:0/1",
[ceph-node1][DEBUG ] "name": "ceph-node2",
[ceph-node1][DEBUG ] "public_addr": "0.0.0.0:0/1",
[ceph-node1][DEBUG ] "rank": 1
[ceph-node1][DEBUG ] }
[ceph-node1][DEBUG ] ]
[ceph-node1][DEBUG ] },
[ceph-node1][DEBUG ] "name": "ceph-node1",
[ceph-node1][DEBUG ] "outside_quorum": [
[ceph-node1][DEBUG ] "ceph-node1"
[ceph-node1][DEBUG ] ],
[ceph-node1][DEBUG ] "quorum": [],
[ceph-node1][DEBUG ] "rank": 1,
[ceph-node1][DEBUG ] "state": "probing",
[ceph-node1][DEBUG ] "sync_provider": []
[ceph-node1][DEBUG ] }
When I insert an iptables-rules as follow, this problem is fixed. But mon-init
operation is always unsuccessful.
[ceph-node1][DEBUG ] "mons": [
[ceph-node1][DEBUG ] {
[ceph-node1][DEBUG ] "addr": "192.168.2.177:6789/0",
[ceph-node1][DEBUG ] "name": "ceph-node2",
[ceph-node1][DEBUG ] "public_addr": "192.168.2.177:6789/0",
[ceph-node1][DEBUG ] "rank": 0
[ceph-node1][DEBUG ] },
[ceph-node1][DEBUG ] {
[ceph-node1][DEBUG ] "addr": "192.168.2.233:6789/0",
[ceph-node1][DEBUG ] "name": "ceph-node1",
[ceph-node1][DEBUG ] "public_addr": "192.168.2.233:6789/0",
[ceph-node1][DEBUG ] "rank": 1
[ceph-node1][DEBUG ] }
[ceph-node1][DEBUG ] ]
I check ceph mon log: /var/log/ceph/ceph-mon.ceph-node1.log and some valuable
error print is found.
2019-08-19 20:59:31.371409 ffff94d8a7c0 0 mon.ceph-node1@1(probing) e0
ms_verify_authorizer bad authorizer
from mon 192.168.2.177:6789/0
2019-08-19 20:59:31.371501 ffff94d8a7c0 0 bad crc in front 947773956 != exp
2766898612
2019-08-19 20:59:31.371548 ffff94d8a7c0 0 -- 192.168.2.233:6789/0 >>
192.168.2.177:6789/0 conn(0xaaab28ce4800
:6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0
l=0).handle_connect_msg accept connect_seq 11843403 vs
existing csq=11843403 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY
2019-08-19 20:59:31.371792 ffff94d8a7c0 0 mon.ceph-node1@1(probing) e0
ms_verify_authorizer bad authorizer
from mon 192.168.2.177:6789/0
2019-08-19 20:59:31.372688 ffff94d8a7c0 0 -- 192.168.2.233:6789/0 >>
192.168.2.177:6789/0 conn(0xaaab277e9800
:-1 s=STATE_OPEN pgs=7648759 cs=11843404 l=0).fault initiating reconnect
That looks like crc check failure leading to bad authorizer problem.
-- System Information:
Debian Release: 9.9
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: arm64 (aarch64)
Kernel: Linux 4.19.0-9-arm64 (SMP w/64 CPU cores)
Locale: LANG=en_HK.UTF-8, LC_CTYPE=en_HK.UTF-8 (charmap=UTF-8),
LANGUAGE=en_HK.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages ceph depends on:
ii ceph-mgr 12.2.11+dfsg1-2.1
ii ceph-mon 12.2.11+dfsg1-2.1
ii ceph-osd 12.2.11+dfsg1-2.1
ceph recommends no packages.
Versions of packages ceph suggests:
pn ceph-mds <none>
-- no debconf information
