Re: [ceph-users] SLOW SSD's after moving to Bluestore

On Tue, Dec 11, 2018 at 7:28 PM Tyler Bishop wrote: > > Now I'm just trying to figure out how to create filestore in Luminous. > I've read every doc and tried every flag but I keep ending up with > either a data LV of 100% on the VG or a bunch fo random errors for > unsupported flags... An LV wit

Re: [ceph-users] SLOW SSD's after moving to Bluestore

On Tue, Dec 11, 2018 at 8:16 PM Mark Kirkwood wrote: > > Looks like the 'delaylog' option for xfs is the problem - no longer supported > in later kernels. See > https://github.com/torvalds/linux/commit/444a702231412e82fb1c09679adc159301e9242c > > Offhand I'm not sure where that option is being a

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Looks like the 'delaylog' option for xfs is the problem - no longer supported in later kernels. See https://github.com/torvalds/linux/commit/444a702231412e82fb1c09679adc159301e9242c Offhand I'm not sure where that option is being added (whether ceph-deploy or ceph-volume), but you could just do su

Re: [ceph-users] SLOW SSD's after moving to Bluestore

> > [root] osci-1001.infra.cin1.corp:~/cephdeploy # ceph-deploy osd create >> --filestore --fs-type xfs --data /dev/sdb2 --journal /dev/sdb1 osci-1001 > > [ceph_deploy.conf][DEBUG ] found configuration file at: >> /root/.cephdeploy.conf > > [ceph_deploy.cli][INFO ] Invoked (2.0.1): /usr/bin/ceph-d

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Now I'm just trying to figure out how to create filestore in Luminous. I've read every doc and tried every flag but I keep ending up with either a data LV of 100% on the VG or a bunch fo random errors for unsupported flags... # ceph-disk prepare --filestore --fs-type xfs --data-dev /dev/sdb1 --jou

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Hello, On Tue, 11 Dec 2018 23:22:40 +0300 Igor Fedotov wrote: > Hi Tyler, > > I suspect you have BlueStore DB/WAL at these drives as well, don't you? > > Then perhaps you have performance issues with f[data]sync requests which > DB/WAL invoke pretty frequently. > Since he explicitly mentioned

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Hi Tyler, I suspect you have BlueStore DB/WAL at these drives as well, don't you? Then perhaps you have performance issues with f[data]sync requests which DB/WAL invoke pretty frequently. See the following links for details: https://www.percona.com/blog/2018/02/08/fsync-performance-storage-d

Re: [ceph-users] SLOW SSD's after moving to Bluestore

On Mon, 10 Dec 2018 21:42:46 -0500 Tyler Bishop wrote: > All 4 of these SSD that i've converted to Bluestore are behaving this > way. I have around 300 of these drives in a very large production > cluster and do not have this type of behavior with Filestore. > And the same/similar loading (as i

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Do they have a form of disk write cache? Have seen really bad performance on HD with disk write cache enabled, same thing disk showing 100% high latency. Disabling via hdparm -W0 has solved my and a few others issues. On Tue, 11 Dec 2018 at 10:43 AM, Tyler Bishop < tyler.bis...@beyondhosting.net

Re: [ceph-users] SLOW SSD's after moving to Bluestore

All 4 of these SSD that i've converted to Bluestore are behaving this way. I have around 300 of these drives in a very large production cluster and do not have this type of behavior with Filestore. On the filestore setup these SSD are partitioned 20GB for journal and 800GB for data. The systems

Re: [ceph-users] SLOW SSD's after moving to Bluestore

On Mon, 10 Dec 2018 21:01:31 -0500 Tyler Bishop wrote: > LVM | dm-0 | busy101% | read 137 | write 1761 | > KiB/r 4 | KiB/w 30 | MBr/s0.1 | MBw/s5.3 | avq > 185.42 | avio 5.31 ms | > DSK | sdb | busy100% | read 127 | write 1208

Re: [ceph-users] SLOW SSD's after moving to Bluestore

LVM | dm-0 | busy101% | read 137 | write 1761 | KiB/r 4 | KiB/w 30 | MBr/s0.1 | MBw/s5.3 | avq 185.42 | avio 5.31 ms | DSK | sdb | busy100% | read 127 | write 1208 | KiB/r 4 | KiB/w 32 | MBr/s0.1 | MBw/s3.9 | a

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Older Crucial/Micron M500/M600 _ *Tyler Bishop* EST 2007 O: 513-299-7108 x1000 M: 513-646-5809 http://BeyondHosting.net This email is intended only for the recipient(s) above and/or otherwise authorized personnel. The info

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Hello, On Mon, 10 Dec 2018 20:43:40 -0500 Tyler Bishop wrote: > I don't think thats my issue here because I don't see any IO to justify the > latency. Unless the IO is minimal and its ceph issuing a bunch of discards > to the ssd and its causing it to slow down while doing that. > What does at

Re: [ceph-users] SLOW SSD's after moving to Bluestore

I don't think thats my issue here because I don't see any IO to justify the latency. Unless the IO is minimal and its ceph issuing a bunch of discards to the ssd and its causing it to slow down while doing that. Log isn't showing anything useful and I have most debugging disabled. On Mon, Dec

Re: [ceph-users] SLOW SSD's after moving to Bluestore

Hi Tyler, I think we had a user a while back that reported they had background deletion work going on after upgrading their OSDs from filestore to bluestore due to PGs having been moved around.  Is it possible that your cluster is doing a bunch of work (deletion or otherwise) beyond the regul

[ceph-users] SLOW SSD's after moving to Bluestore

Hi, I have an SSD only cluster that I recently converted from filestore to bluestore and performance has totally tanked. It was fairly decent before, only having a little additional latency than expected. Now since converting to bluestore the latency is extremely high, SECONDS. I am trying to d