Some test results: 1.If using custom admin_secret for MONs, can see /etc/ceph/ceph.client.admin.keyring which was missing in previous attempts. 2.stop firewalld on all MONs.
Both attempts gave same result as before: got stuck at collect admin and bootstrap keys as before. Appreciate any clue. Instead of deploying mimic, should we deploy luminous with latest ansible and ceph-ansible 4.0 or master, then upgrade to mimic ? Thanks a lot. From: ceph-users <[email protected]> On Behalf Of ST Wong (ITSC) Sent: Tuesday, May 7, 2019 11:48 AM To: solarflow99 <[email protected]> Cc: [email protected] Subject: Re: [ceph-users] ceph-create-keys loops Some more information: Rpm installed on MONS: python-cephfs-13.2.5-0.el7.x86_64 ceph-base-13.2.5-0.el7.x86_64 libcephfs2-13.2.5-0.el7.x86_64 ceph-common-13.2.5-0.el7.x86_64 ceph-selinux-13.2.5-0.el7.x86_64 ceph-mon-13.2.5-0.el7.x86_64 Doing a mon_status gives following. Only the local host running the command can see addr/pub_addr correctly while all other MONs have addr/pub_addr “0.0.0.0”. Besides, the mon feature shows it’s luminous while we’re installing mimic. ====================== [root@cphmon2a ~]# ceph --admin-daemon /var/run/ceph/ceph-mon.cphmon2a.asok mon_status { "name": "cphmon2a", "rank": 0, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "0", "required_mon": [], "quorum_con": "0", "quorum_mon": [] }, "outside_quorum": [ "cphmon2a" ], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 0, "fsid": "cf746667-3132-476d-a287-19143404cf39", "modified": "0.000000", "created": "0.000000", "features": { "persistent": [], "optional": [] }, "mons": [ { "rank": 0, "name": "cphmon2a", "addr": "123.123.7.93:6789/0", "public_addr": "123.123.7.93:6789/0" }, { "rank": 1, "name": "cphmon1a", "addr": "0.0.0.0:0/1", "public_addr": "0.0.0.0:0/1" }, { "rank": 2, "name": "cphmon3a", "addr": "0.0.0.0:0/2", "public_addr": "0.0.0.0:0/2" }, { "rank": 3, "name": "cphmon4b", "addr": "0.0.0.0:0/3", "public_addr": "0.0.0.0:0/3" }, { "rank": 4, "name": "cphmon5b", "addr": "0.0.0.0:0/4", "public_addr": "0.0.0.0:0/4" }, { "rank": 5, "name": "cphmon6b", "addr": "0.0.0.0:0/5", "public_addr": "0.0.0.0:0/5" } ] }, "feature_map": { "mon": [ { "features": "0x3ffddff8ffacfffb", "release": "luminous", "num": 1 } ] } } ====================== Thanks a lot. From: ceph-users <[email protected]<mailto:[email protected]>> On Behalf Of ST Wong (ITSC) Sent: Tuesday, May 7, 2019 7:59 AM To: solarflow99 <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: Re: [ceph-users] ceph-create-keys loops yes, we’re using 3.2 stable, on RHEL 7. Thanks. From: solarflow99 <[email protected]<mailto:[email protected]>> Sent: Tuesday, May 7, 2019 1:40 AM To: ST Wong (ITSC) <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: Re: [ceph-users] ceph-create-keys loops you mention the version of ansible, that is right. How about the branch of ceph-ansible? should be 3.2-stable, what OS? I haven't come across this problem myself, a lot of other ones. On Mon, May 6, 2019 at 3:47 AM ST Wong (ITSC) <[email protected]<mailto:[email protected]>> wrote: Hi all, I’ve problem in deploying mimic using ceph-ansible at following step: ---------- cut here ----------- TASK [ceph-mon : collect admin and bootstrap keys] ***************************** Monday 06 May 2019 17:01:23 +0800 (0:00:00.854) 0:05:38.899 ************ fatal: [cphmon3a]: FAILED! => {"changed": false, "cmd": ["ceph-create-keys", "--cluster", "ceph", "-i", "cphmon3a", "-t", "600"], "delta": "0:11:24.675833", "end": "2019-05-06 17:12:48.500996", "msg": "non-zero return code", "rc": 1, "start": "2019-05-06 17:01:23.825163", "stderr": "INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\n INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\nINFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'\n ---------- cut here ----------- There are 2 NIC on all MONs. The site.yml contains some basic configuration: ----------- site.yml -------- dummy: ceph_origin: repository ceph_repository: community ceph_stable_release: mimic public_network: "123.123.7.0/24<http://123.123.7.0/24>" <-- fake ip range of our public network cluster_network: "192.168.77.0/24<http://192.168.77.0/24>" monitor_interface: p2p1 ----------- site.yml -------- And the ceph.conf created on MONs : -------- ceph.conf ------- # Please do not change this file directly since it is managed by Ansible and will be overwritten [global] fsid = 17db45c6-b5ac-47e8-b5cb-b3e5215f4af4 mon initial members = cphmon1a,cphmon2a,cphmon3a,cphmon4b,cphmon5b,cphmon6b osd pool default crush rule = -1 mon host = 123.123.7.92,123.123.7.93,123.123.7.94,123.123.7.95,123.123.7.96,123.123.7.97 public network = 123.123.7.0/24<http://123.123.7.0/24> cluster network = 192.168.77.0/24<http://192.168.77.0/24> -------- ceph.conf ------- Also get same error if running following on one of the MONs: #ceph-create-keys --cluster ceph --id cphmon1a INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing' INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing' We’re using Ansible 2.6 (for deploying Mimic using ceph-ansible). Would anyone please help? Thanks a lot. /st wong _______________________________________________ ceph-users mailing list [email protected]<mailto:[email protected]> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
