Since this was a test lab, I totally purged the whole cluster and re-deployed.. working good now, thank you.
Alex F. Evonosky <https://twitter.com/alexevon> <https://www.linkedin.com/in/alexevonosky> On Sat, Jan 7, 2017 at 9:14 PM, Alex Evonosky <[email protected]> wrote: > Thank you.. > > > After sending the post, I totally removed the mon and issued the build > with ceph-deploy: > > > In the logs now: > > 2017-01-07 21:12:38.113534 7fa9613fd700 0 cephx: verify_reply couldn't > decrypt with error: error decoding block for decryption > 2017-01-07 21:12:38.113546 7fa9613fd700 0 -- 10.10.10.138:6789/0 >> > 10.10.10.103:6789/0 pipe(0x55feb2e90000 sd=12 :50266 s=1 pgs=0 cs=0 l=0 > c=0x55feb2ca0a80).failed verifying authorize reply > 2017-01-07 21:12:38.114529 7fa95787b700 0 cephx: verify_reply couldn't > decrypt with error: error decoding block for decryption > 2017-01-07 21:12:38.114567 7fa95787b700 0 -- 10.10.10.138:6789/0 >> > 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=11 :38690 s=1 pgs=0 cs=0 l=0 > c=0x55feb2ca0c00).failed verifying authorize reply > 2017-01-07 21:12:40.114522 7fa9613fd700 0 cephx: verify_reply couldn't > decrypt with error: error decoding block for decryption > 2017-01-07 21:12:40.114542 7fa9613fd700 0 -- 10.10.10.138:6789/0 >> > 10.10.10.103:6789/0 pipe(0x55feb2e90000 sd=11 :50278 s=1 pgs=0 cs=0 l=0 > c=0x55feb2ca0a80).failed verifying authorize reply > 2017-01-07 21:12:40.115706 7fa95787b700 0 cephx: verify_reply couldn't > decrypt with error: error decoding block for decryption > 2017-01-07 21:12:40.115721 7fa95787b700 0 -- 10.10.10.138:6789/0 >> > 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=12 :38702 s=1 pgs=0 cs=0 l=0 > c=0x55feb2ca0c00).failed verifying authorize reply > 2017-01-07 21:12:41.621916 7fa956f79700 0 cephx: verify_authorizer could > not decrypt ticket info: error: NSS AES final round failed: -8190 > 2017-01-07 21:12:41.621929 7fa956f79700 0 mon.alex-desktop@1(probing) e0 > ms_verify_authorizer bad authorizer from mon 10.10.10.103:6789/0 > 2017-01-07 21:12:41.621944 7fa956f79700 0 -- 10.10.10.138:6789/0 >> > 10.10.10.103:6789/0 pipe(0x55feb2fb5400 sd=21 :6789 s=0 pgs=0 cs=0 l=0 > c=0x55feb2ca1500).accept: got bad authorizer > > > > $ sudo ceph -s > cluster f5aba719-4856-4ae2-a5d4-f9ff0f614b60 > health HEALTH_WARN > 512 pgs degraded > 348 pgs stale > 512 pgs stuck unclean > 512 pgs undersized > 6 requests are blocked > 32 sec > recovery 25013/50026 objects degraded (50.000%) > mds cluster is degraded > 1 mons down, quorum 0,2 alpha,toshiba-laptop > monmap e17: 3 mons at {alex-desktop=10.10.10.138: > 6789/0,alpha=10.10.10.103:6789/0,toshiba-laptop=10.10.10.252:6789/0} > election epoch 806, quorum 0,2 alpha,toshiba-laptop > fsmap e201858: 1/1/1 up {0=1=up:replay} > osdmap e200229: 3 osds: 2 up, 2 in; 85 remapped pgs > flags sortbitwise > pgmap v4088774: 512 pgs, 4 pools, 50883 MB data, 25013 objects > 59662 MB used, 476 GB / 563 GB avail > 25013/50026 objects degraded (50.000%) > 348 stale+active+undersized+degraded > 164 active+undersized+degraded > > > > root@alex-desktop:/var/lib/ceph/mon/ceph-alex-desktop# ls -ls > total 8 > 0 -rw-r--r-- 1 ceph ceph 0 Jan 7 21:11 done > 4 -rw------- 1 ceph ceph 77 Jan 7 21:05 keyring > 4 drwxr-xr-x 2 ceph ceph 4096 Jan 7 21:10 store.db > 0 -rw-r--r-- 1 ceph ceph 0 Jan 7 21:05 systemd > > > > > Very odd... never seen this issue on the other monitor deployments... > > > > > > > > > Alex F. Evonosky > > <https://twitter.com/alexevon> <https://www.linkedin.com/in/alexevonosky> > > On Sat, Jan 7, 2017 at 8:54 PM, Shinobu Kinjo <[email protected]> wrote: > >> Using ``ceph-deploy`` will save your life: >> >> # https://github.com/ceph/ceph/blob/master/doc/start/quick-cep >> h-deploy.rst >> * Please look at: Adding Monitors >> >> If you are using centos or similar, the latest package is available here: >> >> # http://download.ceph.com/rpm-jewel/el7/noarch/ceph-deploy-1. >> 5.37-0.noarch.rpm >> >> Regards, >> >> >> On Sun, Jan 8, 2017 at 9:53 AM, Alex Evonosky <[email protected]> >> wrote: >> >>> Thank you for the reply! >>> >>> I followed this article: >>> >>> http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/ >>> >>> >>> Under the section: ADDING A MONITOR (MANUAL) >>> >>> >>> >>> Alex F. Evonosky >>> >>> <https://twitter.com/alexevon> >>> <https://www.linkedin.com/in/alexevonosky> >>> >>> On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo <[email protected]> wrote: >>> >>>> How did you add a third MON? >>>> >>>> Regards, >>>> >>>> On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky <[email protected]> >>>> wrote: >>>> > Anyone see this before? >>>> > >>>> > >>>> > 2017-01-07 16:55:11.406047 7f095b379700 0 cephx: verify_reply >>>> couldn't >>>> > decrypt with error: error decoding block for decryption >>>> > 2017-01-07 16:55:11.406053 7f095b379700 0 -- 10.10.10.138:6789/0 >> >>>> > 10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0 >>>> l=0 >>>> > c=0x55cf8ce28f00).failed verifying authorize reply >>>> > >>>> > >>>> > >>>> > Two monitors are up just fine, just trying to add a third and a quorum >>>> > cannot be met. NTP is running and no iptables running at all on >>>> internal >>>> > cluster. >>>> > >>>> > >>>> > Thank you. >>>> > -Alex >>>> > >>>> > >>>> > >>>> > >>>> > _______________________________________________ >>>> > ceph-users mailing list >>>> > [email protected] >>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> > >>>> >>> >>> >> >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
