Your message dated Fri, 8 Jan 2021 09:17:36 +0100 with message-id <[email protected]> and subject line Ceph-deploy is not to be used has caused the Debian Bug report #935156, regarding ceph: Multiple mon deployment failure on arm64: ms_verfity_authorizer bad authorizer and crc check failure to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact [email protected] immediately.) -- 935156: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935156 Debian Bug Tracking System Contact [email protected] with problems
--- Begin Message ---Package: ceph Version: 12.2.11+dfsg1-2.1 Severity: grave Justification: renders package unusable Dear Maintainer, I'm trying to deploy a 2 monitor ceph cluster with 2 arm64 server nodes. root@ceph-node1:~# ceph -v ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) ceph-node1 192.168.2.233 ceph-node2 192.168.2.177 root@ceph-node1:~# ceph-deploy --version 2.0.1 ceph.conf file: [global] fsid = 76018d7b-2258-425b-be7d-f76ae5192486 mon_initial_members = ceph-node1, ceph-node2 mon_host = 192.168.2.233,192.168.2.177 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 192.168.2.0/24 when I do mon-initial opertion: ceph-deploy mon create-inital, I get some failure info [ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries left: 5 [ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying [ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries left: 4 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries left: 3 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries left: 2 [ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying [ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node1 monitor is not yet in quorum, tries left: 1 [ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying [ceph_deploy.mon][INFO ] processing monitor mon.ceph-node2 [ceph-node2][DEBUG ] connected to host: ceph-node2 [ceph-node2][DEBUG ] detect platform information from remote host [ceph-node2][DEBUG ] detect machine type [ceph-node2][DEBUG ] find the location of an executable [ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node2.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries left: 5 [ceph_deploy.mon][WARNIN] waiting 5 seconds before retrying [ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node2.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries left: 4 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node2.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries left: 3 [ceph_deploy.mon][WARNIN] waiting 10 seconds before retrying [ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node2.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries left: 2 [ceph_deploy.mon][WARNIN] waiting 15 seconds before retrying [ceph-node2][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node2.asok mon_status [ceph_deploy.mon][WARNIN] mon.ceph-node2 monitor is not yet in quorum, tries left: 1 [ceph_deploy.mon][WARNIN] waiting 20 seconds before retrying [ceph_deploy.mon][ERROR ] Some monitors have still not reached quorum: [ceph_deploy.mon][ERROR ] ceph-node1 [ceph_deploy.mon][ERROR ] ceph-node2 monmap:mons info: ceph-node2's addr is incorrect [ceph-node1][INFO ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status [ceph-node1][DEBUG ] ******************************************************************************** [ceph-node1][DEBUG ] status for monitor: mon.ceph-node1 [ceph-node1][DEBUG ] { [ceph-node1][DEBUG ] "election_epoch": 1, [ceph-node1][DEBUG ] "extra_probe_peers": [ [ceph-node1][DEBUG ] "192.168.2.177:6789/0" [ceph-node1][DEBUG ] ], [ceph-node1][DEBUG ] "feature_map": { [ceph-node1][DEBUG ] "mon": { [ceph-node1][DEBUG ] "group": { [ceph-node1][DEBUG ] "features": "0x3ffddff8eeacfffb", [ceph-node1][DEBUG ] "num": 1, [ceph-node1][DEBUG ] "release": "luminous" [ceph-node1][DEBUG ] } [ceph-node1][DEBUG ] } [ceph-node1][DEBUG ] }, [ceph-node1][DEBUG ] "features": { [ceph-node1][DEBUG ] "quorum_con": "0", [ceph-node1][DEBUG ] "quorum_mon": [], [ceph-node1][DEBUG ] "required_con": "0", [ceph-node1][DEBUG ] "required_mon": [] [ceph-node1][DEBUG ] }, [ceph-node1][DEBUG ] "monmap": { [ceph-node1][DEBUG ] "created": "2019-08-19 15:36:56.080981", [ceph-node1][DEBUG ] "epoch": 0, [ceph-node1][DEBUG ] "features": { [ceph-node1][DEBUG ] "optional": [], [ceph-node1][DEBUG ] "persistent": [] [ceph-node1][DEBUG ] }, [ceph-node1][DEBUG ] "fsid": "76018d7b-2258-425b-be7d-f76ae5192486", [ceph-node1][DEBUG ] "modified": "2019-08-19 15:36:56.080981", [ceph-node1][DEBUG ] "mons": [ [ceph-node1][DEBUG ] { [ceph-node1][DEBUG ] "addr": "192.168.2.233:6789/0", [ceph-node1][DEBUG ] "name": "ceph-node1", [ceph-node1][DEBUG ] "public_addr": "192.168.2.233:6789/0", [ceph-node1][DEBUG ] "rank": 0 [ceph-node1][DEBUG ] }, [ceph-node1][DEBUG ] { [ceph-node1][DEBUG ] "addr": "0.0.0.0:0/1", [ceph-node1][DEBUG ] "name": "ceph-node2", [ceph-node1][DEBUG ] "public_addr": "0.0.0.0:0/1", [ceph-node1][DEBUG ] "rank": 1 [ceph-node1][DEBUG ] } [ceph-node1][DEBUG ] ] [ceph-node1][DEBUG ] }, [ceph-node1][DEBUG ] "name": "ceph-node1", [ceph-node1][DEBUG ] "outside_quorum": [ [ceph-node1][DEBUG ] "ceph-node1" [ceph-node1][DEBUG ] ], [ceph-node1][DEBUG ] "quorum": [], [ceph-node1][DEBUG ] "rank": 1, [ceph-node1][DEBUG ] "state": "probing", [ceph-node1][DEBUG ] "sync_provider": [] [ceph-node1][DEBUG ] } When I insert an iptables-rules as follow, this problem is fixed. But mon-init operation is always unsuccessful. [ceph-node1][DEBUG ] "mons": [ [ceph-node1][DEBUG ] { [ceph-node1][DEBUG ] "addr": "192.168.2.177:6789/0", [ceph-node1][DEBUG ] "name": "ceph-node2", [ceph-node1][DEBUG ] "public_addr": "192.168.2.177:6789/0", [ceph-node1][DEBUG ] "rank": 0 [ceph-node1][DEBUG ] }, [ceph-node1][DEBUG ] { [ceph-node1][DEBUG ] "addr": "192.168.2.233:6789/0", [ceph-node1][DEBUG ] "name": "ceph-node1", [ceph-node1][DEBUG ] "public_addr": "192.168.2.233:6789/0", [ceph-node1][DEBUG ] "rank": 1 [ceph-node1][DEBUG ] } [ceph-node1][DEBUG ] ] I check ceph mon log: /var/log/ceph/ceph-mon.ceph-node1.log and some valuable error print is found. 2019-08-19 20:59:31.371409 ffff94d8a7c0 0 mon.ceph-node1@1(probing) e0 ms_verify_authorizer bad authorizer from mon 192.168.2.177:6789/0 2019-08-19 20:59:31.371501 ffff94d8a7c0 0 bad crc in front 947773956 != exp 2766898612 2019-08-19 20:59:31.371548 ffff94d8a7c0 0 -- 192.168.2.233:6789/0 >> 192.168.2.177:6789/0 conn(0xaaab28ce4800 :6789 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_msg accept connect_seq 11843403 vs existing csq=11843403 existing_state=STATE_CONNECTING_WAIT_CONNECT_REPLY 2019-08-19 20:59:31.371792 ffff94d8a7c0 0 mon.ceph-node1@1(probing) e0 ms_verify_authorizer bad authorizer from mon 192.168.2.177:6789/0 2019-08-19 20:59:31.372688 ffff94d8a7c0 0 -- 192.168.2.233:6789/0 >> 192.168.2.177:6789/0 conn(0xaaab277e9800 :-1 s=STATE_OPEN pgs=7648759 cs=11843404 l=0).fault initiating reconnect That looks like crc check failure leading to bad authorizer problem. -- System Information: Debian Release: 9.9 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: arm64 (aarch64) Kernel: Linux 4.19.0-9-arm64 (SMP w/64 CPU cores) Locale: LANG=en_HK.UTF-8, LC_CTYPE=en_HK.UTF-8 (charmap=UTF-8), LANGUAGE=en_HK.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages ceph depends on: ii ceph-mgr 12.2.11+dfsg1-2.1 ii ceph-mon 12.2.11+dfsg1-2.1 ii ceph-osd 12.2.11+dfsg1-2.1 ceph recommends no packages. Versions of packages ceph suggests: pn ceph-mds <none> -- no debconf information
--- End Message ---
--- Begin Message ---Hi, I'm not sure if this is related, but ceph-deploy is deprecated and not to be used. I very much doubt that Ceph 12.2.11+dfsg1-2.1 is just not usable at all as you wrote. Cheers, Thomas Goirand (zigo)
--- End Message ---
