subscribe > 在 2018年4月16日，04:01，[email protected] 写道： > > Send ceph-users mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > > Today's Topics: > > 1. Re: Error Creating OSD (Rhian Resnick) > 2. ZeroDivisionError: float division by zero in > /usr/lib/ceph/mgr/dashboard/module.py (12.2.4) (Nicolas Huillard) > 3. High TCP retransmission rates, only with Ceph (Robert Stanford) > 4. Re: High TCP retransmission rates, only with Ceph (Pawe? Sadowski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 14 Apr 2018 21:17:32 +0000 > From: Rhian Resnick <[email protected]> > To: Alfredo Deza <[email protected]> > Cc: "[email protected]" <[email protected]> > Subject: Re: [ceph-users] Error Creating OSD > Message-ID: > > <dm5pr0401mb362486ba8476827e92a0ed25cb...@dm5pr0401mb3624.namprd04.prod.outlook.com> > > Content-Type: text/plain; charset="iso-8859-1" > > Afternoon, > > > Happily, I resolved this issue. > > > Running vgdisplay showed that ceph-volume tried to create a disk on failed > disk. (We didn't know we had a bad did so this is information that was new to > us) and when the command failed it left three bad volume groups. Since you > cannot rename them you need to use the following command to delete them. > > > vgdisplay to find the bad volume groups > > vgremove --select vg_uuid=your uuid -f # -f forces it to be removed > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > > ________________________________ > From: Rhian Resnick > Sent: Saturday, April 14, 2018 12:47 PM > To: Alfredo Deza > Cc: [email protected] > Subject: Re: [ceph-users] Error Creating OSD > > > Thanks all, > > > Here is a link to our our command being executed: > https://pastebin.com/iy8iSaKH > > > > Here are the results from the command > > > Executed with debug enabled (after a zap with destroy) > > > [root@ceph-storage3 ~]# ceph-volume lvm create --bluestore --data /dev/sdu > Running command: ceph-authtool --gen-print-key > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > 664894a8-530a-4557-b2f4-1af5b391f2b7 > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > Running command: ceph osd purge osd.140 --yes-i-really-mean-it > stderr: purged osd.140 > Traceback (most recent call last): > File "/sbin/ceph-volume", line 6, in <module> > main.Volume() > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 37, in > __init__ > self.main(self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, > in newfunc > return f(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 153, in > main > terminal.dispatch(self.mapper, subcommand_args) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py", > line 38, in main > terminal.dispatch(self.mapper, self.argv) > File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, > in dispatch > instance.main() > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", > line 74, in main > self.create(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, > in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/create.py", > line 26, in create > prepare_step.safe_prepare(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", > line 217, in safe_prepare > self.prepare(args) > File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, > in is_root > return func(*a, **kw) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", > line 283, in prepare > block_lv = self.prepare_device(args.data, 'block', cluster_fsid, osd_fsid) > File "/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/prepare.py", > line 193, in prepare_device > if api.get_vg(vg_name=vg_name): > File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 334, in > get_vg > return vgs.get(vg_name=vg_name, vg_tags=vg_tags) > File "/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py", line 429, in > get > raise MultipleVGsError(vg_name) > ceph_volume.exceptions.MultipleVGsError: Got more than 1 result looking for > volume group: ceph-6a2e8f21-bca2-492b-8869-eecc995216cc > > > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > > ________________________________ > From: Alfredo Deza <[email protected]> > Sent: Saturday, April 14, 2018 8:45 AM > To: Rhian Resnick > Cc: [email protected] > Subject: Re: [ceph-users] Error Creating OSD > > > > On Fri, Apr 13, 2018 at 8:20 PM, Rhian Resnick > <[email protected]<mailto:[email protected]>> wrote: > > Evening, > > When attempting to create an OSD we receive the following error. > > [ceph-admin@ceph-storage3 ~]$ sudo ceph-volume lvm create --bluestore --data > /dev/sdu > Running command: ceph-authtool --gen-print-key > Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new > c8cb8cff-dad9-48b8-8d77-6f130a4b629d > --> Was unable to complete a new OSD, will rollback changes > --> OSD will be fully purged from the cluster, because the ID was generated > Running command: ceph osd purge osd.140 --yes-i-really-mean-it > stderr: purged osd.140 > --> MultipleVGsError: Got more than 1 result looking for volume group: > ceph-6a2e8f21-bca2-492b-8869-eecc995216cc > > Any hints on what to do? This occurs when we attempt to create osd's on this > node. > > Can you use a paste site and get the /var/log/ceph/ceph-volume.log contents? > Also, if you could try the same command but with: > > CEPH_VOLUME_DEBUG=1 > > I think you are hitting two issues here: > > 1) Somehow `osd new` is not completing and failing > 2) The `purge` command to wipe out the LV is getting multiple LV's and cannot > make sure to match the one it used. > > #2 definitely looks like something we are doing wrong, and #1 can have a lot > of different causes. The logs would be tremendously helpful! > > > Rhian Resnick > > Associate Director Middleware and HPC > > Office of Information Technology > > > Florida Atlantic University > > 777 Glades Road, CM22, Rm 173B > > Boca Raton, FL 33431 > > Phone 561.297.2647 > > Fax 561.297.0222 > > [image] <https://hpc.fau.edu/wp-content/uploads/2015/01/image.jpg> > > _______________________________________________ > ceph-users mailing list > [email protected]<mailto:[email protected]> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20180414/ea2b5e6c/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Sun, 15 Apr 2018 14:46:46 +0200 > From: Nicolas Huillard <[email protected]> > To: [email protected] > Subject: [ceph-users] ZeroDivisionError: float division by zero in > /usr/lib/ceph/mgr/dashboard/module.py (12.2.4) > Message-ID: <[email protected]> > Content-Type: text/plain; charset="UTF-8" > > Hi, > > I'm not sure if this have been solved since 12.2.4. The same code > occurs in a different file in Github https://github.com/ceph/ceph/blob/ > 50412f7e9c2691ec10132c8bf9310a05a40e9f9d/src/pybind/mgr/status/module.p > y > The ZeroDivisionError occurs when the dashboard is open, and there is a > network outage (link between 2 datacenters is broken). I'm not sure > about the behaviour of the actual UI in the dashboard at the same time. > > Syslog trace: > > ceph-mgr[1324]: [15/Apr/2018:09:47:12] HTTP Traceback (most recent call last): > ceph-mgr[1324]: File > "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 670, in > respond > ceph-mgr[1324]: response.body = self.handler() > ceph-mgr[1324]: File > "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 217, in > __call__ > ceph-mgr[1324]: self.body = self.oldhandler(*args, **kwargs) > ceph-mgr[1324]: File > "/usr/lib/python2.7/dist-packages/cherrypy/lib/jsontools.py", line 63, in > json_handler > ceph-mgr[1324]: value = > cherrypy.serving.request._json_inner_handler(*args, **kwargs) > ceph-mgr[1324]: File > "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 61, in > __call__ > ceph-mgr[1324]: return self.callable(*self.args, **self.kwargs) > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 991, in > list_data > ceph-mgr[1324]: return self._osds_by_server() > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 1040, in > _osds_by_server > ceph-mgr[1324]: osd_map.osds_by_id[osd_id]) > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 1007, in > _osd_summary > ceph-mgr[1324]: result['stats'][s.split(".")[1]] = > global_instance().get_rate('osd', osd_spec, s) > ceph-mgr[1324]: File "/usr/lib/ceph/mgr/dashboard/module.py", line 268, in > get_rate > ceph-mgr[1324]: return (data[-1][1] - data[-2][1]) / float(data[-1][0] - > data[-2][0]) > ceph-mgr[1324]: ZeroDivisionError: float division by zero > > HTH, > > -- > Nicolas Huillard > > > ------------------------------ > > Message: 3 > Date: Sun, 15 Apr 2018 13:18:37 -0500 > From: Robert Stanford <[email protected]> > To: [email protected] > Subject: [ceph-users] High TCP retransmission rates, only with Ceph > Message-ID: > <caaca39mo+840q2jskn-gklofcwwr_gvcdpnzfmxvt44+9sm...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Iperf gives about 7Gb/s between a radosgw host and one of my OSD hosts (8 > disks, 8 OSD daemons, one of 3 OSD hosts). When I benchmark radosgw with > cosbench I see high TCP retransmission rates (from sar -n ETCP 1). I don't > see this with iperf. Why would Ceph, but not iperf, cause high TCP > retransmission rates? > > Thank you > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20180415/07b492bd/attachment-0001.html> > > ------------------------------ > > Message: 4 > Date: Sun, 15 Apr 2018 20:48:40 +0200 > From: Pawe? Sadowski <[email protected]> > To: [email protected] > Subject: Re: [ceph-users] High TCP retransmission rates, only with > Ceph > Message-ID: <[email protected]> > Content-Type: text/plain; charset=utf-8; format=flowed > > On 04/15/2018 08:18 PM, Robert Stanford wrote: >> >> ?Iperf gives about 7Gb/s between a radosgw host and one of my OSD >> hosts (8 disks, 8 OSD daemons, one of 3 OSD hosts).? When I benchmark >> radosgw with cosbench I see high TCP retransmission rates (from sar -n >> ETCP 1).? I don't see this with iperf.? Why would Ceph, but not iperf, >> cause high TCP retransmission rates? > > Most probably your application (radosgw in this case) is not able to > process requests fast enough and some packets are dropped. > > -- > PS > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list > [email protected] > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ------------------------------ > > End of ceph-users Digest, Vol 63, Issue 15 > ******************************************
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
