Re: [ceph-users] Increase queue_depth in KVM
Konstantin, Thanks for explanation. But unfortunately, upgrading qemu is nearly impossible in my case. So is there something else I can do, or I have to agree with fact that write IOPS had to be 8x smaller inside KVM rather than outside KVM? :| pt., 13 lip 2018 o 04:22 Konstantin Shalygin napisał(a): > > I've seen some people using 'num_queues' but I don't have this parameter > > in my schemas(libvirt version = 1.3.1, qemu version = 2.5.0 > > > num-queues is available from qemu 2.7 [1] > > > [1] https://wiki.qemu.org/ChangeLog/2.7 > > > > > k > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Increase queue_depth in KVM
Hello, Steffen, Thanks for Your reply. Sorry but I was on holidays, now I'm back and still digging into my problem.. :( I've read thousands of google links but can't find anything which could help me. - tried all qemu drive IO(io=) and cache(cache=) modes, nothing could come even close to the results I'm getting outside KVM. - enabling blk-mq inside KVM guest didn't help - enabling iothreads didn't help - 'queues' parameter in my libvirt schemas can be only applied to 'virtio-serial', can't use it in virtio-scsi nor virtio-blk) - I've seen some people using 'num_queues' but I don't have this parameter in my schemas(libvirt version = 1.3.1, qemu version = 2.5.0) So, is there really no way to increase queue_depth to rbd device in kvm domain or any other way to achieve results similar to those obtained outside KVM? :/ wt., 26 cze 2018 o 15:19 Steffen Winther Sørensen napisał(a): > > > > On 26 Jun 2018, at 14.04, Damian Dabrowski wrote: > > > > Hi Stefan, thanks for reply. > > > > Unfortunately it didn't work. > > > > disk config: > > > >discard='unmap'/> > > > > > > > >name='volumes-nvme/volume-ce247187-a625-49f1-bacd-fc03df215395'> > > > > > > > > > > > > ce247187-a625-49f1-bacd-fc03df215395 > > > > > > > > > > Controller config: > > > > > >function='0x0'/> > > > > > > > > benchmark command: fio --randrepeat=1 --ioengine=libaio --direct=1 > --name=test --filename=test --bs=4k --iodepth=64 --size=1G > --readwrite=randwrite --time_based --runtime=60 > --write_iops_log=write_results --numjobs=8 > > > > And I'm still getting very low random write IOPS inside KVM instance > with 8vcores(3-5k compared to 20k+ outside KVM) > > > > Maybe do You have any idea how to deal with it? > What about trying with io=‘threads’ and/or maybe cache=’none’ or swap from > virtio-scsi to blk-mq? > > Other people had similar issues, try asking ‘G’ > > https://serverfault.com/questions/425607/kvm-guest-io-is-much-slower-than-host-io-is-that-normal > https://wiki.mikejung.biz/KVM_/_Xen > > /Steffen > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Increase queue_depth in KVM
Hello, When I mount rbd image with -o queue_depth=1024 I can see much improvement, generally on writes(random write improvement from 3k IOPS on standard queue_depth to 24k IOPS on queue_depth=1024). But is there any way to attach rbd disk to KVM instance with custom queue_depth? I can't find any information about it. Thanks for any informations. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph recovery kill VM's even with the smallest priority
ok now I understand, thanks for all this helpful answers!
On Sat, Apr 7, 2018, 15:26 David Turner <drakonst...@gmail.com> wrote:
> I'm seconding what Greg is saying There is no reason to set nobackfill
> and norecover just for restarting OSDs. That will only cause the problems
> you're seeing without giving you any benefit. There are reasons to use
> norecover and nobackfill but unless you're manually editing the crush map,
> having osds consistently segfault, or for any other reason you really just
> need to stop the io from recovery, then they aren't the flags for you. Even
> at that, nobackfill is most likely what you need and norecover is still
> probably not helpful.
>
> On Wed, Apr 4, 2018, 6:59 PM Gregory Farnum <gfar...@redhat.com> wrote:
>
>> On Thu, Mar 29, 2018 at 3:17 PM Damian Dabrowski <scoot...@gmail.com>
>> wrote:
>>
>>> Greg, thanks for Your reply!
>>>
>>> I think Your idea makes sense, I've did tests and its quite hard to
>>> understand for me. I'll try to explain my situation in few steps
>>> below.
>>> I think that ceph showing progress in recovery but it can only solve
>>> objects which doesn't really changed. It won't try to repair objects
>>> which are really degraded because of norecovery flag. Am i right?
>>> After a while I see blocked requests(as You can see below).
>>>
>>
>> Yeah, so the implementation of this is a bit funky. Basically, when the
>> OSD gets a map specifying norecovery, it will prevent any new recovery ops
>> from starting once it processes that map. But it doesn't change the state
>> of the PGs out of recovery; they just won't queue up more work.
>>
>> So probably the existing recovery IO was from OSDs that weren't
>> up-to-date yet. Or maybe there's a bug in the norecover implementation; it
>> definitely looks a bit fragile.
>>
>> But really I just wouldn't use that command. It's an expert flag that you
>> shouldn't use except in some extreme wonky cluster situations (and even
>> those may no longer exist in modern Ceph). For the use case you shared in
>> your first email, I'd just stick with noout.
>> -Greg
>>
>>
>>>
>>> - FEW SEC AFTER START OSD -
>>> # ceph status
>>> cluster 848b340a-be27-45cb-ab66-3151d877a5a0
>>> health HEALTH_WARN
>>> 140 pgs degraded
>>> 1 pgs recovering
>>> 92 pgs recovery_wait
>>> 140 pgs stuck unclean
>>> recovery 942/5772119 objects degraded (0.016%)
>>> noout,nobackfill,norecover flag(s) set
>>> monmap e10: 3 mons at
>>> {node-19=
>>> 172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
>>> election epoch 724, quorum 0,1,2 node-19,node-21,node-20
>>> osdmap e18727: 36 osds: 36 up, 30 in
>>> flags noout,nobackfill,norecover
>>> pgmap v20851644: 1472 pgs, 7 pools, 8510 GB data, 1880 kobjects
>>> 25204 GB used, 17124 GB / 42329 GB avail
>>> 942/5772119 objects degraded (0.016%)
>>> 1332 active+clean
>>> 92 active+recovery_wait+degraded
>>> 47 active+degraded
>>>1 active+recovering+degraded
>>> recovery io 31608 kB/s, 4 objects/s
>>> client io 73399 kB/s rd, 80233 kB/s wr, 1218 op/s
>>>
>>> - 1 MIN AFTER OSD START, RECOVERY STUCK, BLOCKED REQUESTS -
>>> # ceph status
>>> cluster 848b340a-be27-45cb-ab66-3151d877a5a0
>>> health HEALTH_WARN
>>> 140 pgs degraded
>>> 1 pgs recovering
>>> 109 pgs recovery_wait
>>> 140 pgs stuck unclean
>>> 80 requests are blocked > 32 sec
>>> recovery 847/5775929 <(847)%20577-5929> objects degraded
>>> (0.015%)
>>> noout,nobackfill,norecover flag(s) set
>>> monmap e10: 3 mons at
>>> {node-19=
>>> 172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
>>> election epoch 724, quorum 0,1,2 node-19,node-21,node-20
>>> osdmap e18727: 36 osds: 36 up, 30 in
>>> flags noout,nobackfill,norecover
>>> pgmap v20851812: 1472 pgs, 7 pools, 8520 GB data, 1881 kobjects
>>> 25234 GB used, 17094 GB / 42329 GB avail
>>> 847/5775929 <(847)%20577-5929> objects degraded (0.015%)
>>>
Re: [ceph-users] Use trimfs on already mounted RBD image
Ok, thanks for Your reply. 2018-04-05 8:15 GMT+02:00 Wido den Hollander <w...@42on.com>: > > > On 04/04/2018 07:30 PM, Damian Dabrowski wrote: > > Hello, > > > > I wonder if it is any way to run `trimfs` on rbd image which is > > currently used by the KVM process? (when I don't have access to VM) > > > > I know that I can do this by qemu-guest-agent but not all VMs have it > > installed. > > > > I can't use rbdmap too, because most images don't have distributed > > filesystems. It's mostly ext4/xfs so I can't mount them in two places at > > same time. > > > > I would be grateful for any help. > > > > No, you can not. If you map and mount a filesystem twice you will > corrupt the data on there. > > Same goes when you are going to fstrim, you will probably break/corrupt > data on the system. > > Do not attempt it. Shut down the VM and then do the fstrim. > > Wido > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Use trimfs on already mounted RBD image
Hello, I wonder if it is any way to run `trimfs` on rbd image which is currently used by the KVM process? (when I don't have access to VM) I know that I can do this by qemu-guest-agent but not all VMs have it installed. I can't use rbdmap too, because most images don't have distributed filesystems. It's mostly ext4/xfs so I can't mount them in two places at same time. I would be grateful for any help. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph recovery kill VM's even with the smallest priority
Greg, thanks for Your reply!
I think Your idea makes sense, I've did tests and its quite hard to
understand for me. I'll try to explain my situation in few steps
below.
I think that ceph showing progress in recovery but it can only solve
objects which doesn't really changed. It won't try to repair objects
which are really degraded because of norecovery flag. Am i right?
After a while I see blocked requests(as You can see below).
- FEW SEC AFTER START OSD -
# ceph status
cluster 848b340a-be27-45cb-ab66-3151d877a5a0
health HEALTH_WARN
140 pgs degraded
1 pgs recovering
92 pgs recovery_wait
140 pgs stuck unclean
recovery 942/5772119 objects degraded (0.016%)
noout,nobackfill,norecover flag(s) set
monmap e10: 3 mons at
{node-19=172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
election epoch 724, quorum 0,1,2 node-19,node-21,node-20
osdmap e18727: 36 osds: 36 up, 30 in
flags noout,nobackfill,norecover
pgmap v20851644: 1472 pgs, 7 pools, 8510 GB data, 1880 kobjects
25204 GB used, 17124 GB / 42329 GB avail
942/5772119 objects degraded (0.016%)
1332 active+clean
92 active+recovery_wait+degraded
47 active+degraded
1 active+recovering+degraded
recovery io 31608 kB/s, 4 objects/s
client io 73399 kB/s rd, 80233 kB/s wr, 1218 op/s
- 1 MIN AFTER OSD START, RECOVERY STUCK, BLOCKED REQUESTS -
# ceph status
cluster 848b340a-be27-45cb-ab66-3151d877a5a0
health HEALTH_WARN
140 pgs degraded
1 pgs recovering
109 pgs recovery_wait
140 pgs stuck unclean
80 requests are blocked > 32 sec
recovery 847/5775929 objects degraded (0.015%)
noout,nobackfill,norecover flag(s) set
monmap e10: 3 mons at
{node-19=172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
election epoch 724, quorum 0,1,2 node-19,node-21,node-20
osdmap e18727: 36 osds: 36 up, 30 in
flags noout,nobackfill,norecover
pgmap v20851812: 1472 pgs, 7 pools, 8520 GB data, 1881 kobjects
25234 GB used, 17094 GB / 42329 GB avail
847/5775929 objects degraded (0.015%)
1332 active+clean
109 active+recovery_wait+degraded
30 active+degraded < degraded objects count got stuck
1 active+recovering+degraded
recovery io 3743 kB/s, 0 objects/s < depend on command execution
this line showing 0 objects/s or doesn't exists
client io 26521 kB/s rd, 64211 kB/s wr, 1212 op/s
- FEW SECONDS AFTER UNSETTING FLAGS NOOUT, NORECOVERY, NOBACKFILL -
# ceph status
cluster 848b340a-be27-45cb-ab66-3151d877a5a0
health HEALTH_WARN
134 pgs degraded
134 pgs recovery_wait
134 pgs stuck degraded
134 pgs stuck unclean
recovery 591/5778179 objects degraded (0.010%)
monmap e10: 3 mons at
{node-19=172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
election epoch 724, quorum 0,1,2 node-19,node-21,node-20
osdmap e18730: 36 osds: 36 up, 30 in
pgmap v20851909: 1472 pgs, 7 pools, 8526 GB data, 1881 kobjects
25252 GB used, 17076 GB / 42329 GB avail
591/5778179 objects degraded (0.010%)
1338 active+clean
134 active+recovery_wait+degraded
recovery io 191 MB/s, 26 objects/s
client io 100654 kB/s rd, 184 MB/s wr, 6303 op/s
2018-03-29 18:22 GMT+02:00 Gregory Farnum <gfar...@redhat.com>:
>
> On Thu, Mar 29, 2018 at 7:27 AM Damian Dabrowski <scoot...@gmail.com> wrote:
>>
>> Hello,
>>
>> Few days ago I had very strange situation.
>>
>> I had to turn off few OSDs for a while. So I've set flags:noout,
>> nobackfill, norecover and then turned off selected OSDs.
>> All was ok, but when I started these OSDs again all VMs went down due
>> to recovery process(even when recovery priority was very low).
>
>
> So you forbade the OSDs from doing any recovery work, but then you turned on
> old ones that required recovery work to function properly?
>
> And your cluster stopped functioning?
>
>
>>
>>
>> There's more important config values:
>> "osd_recovery_threads": "1",
>> "osd_recovery_thread_timeout": "30",
>> "osd_recovery_thread_suicide_timeout": "300",
>> "osd_recovery_delay_start": "0",
>> "osd_recovery_max_active": "1",
>> "osd_recovery_max_single_start": "5",
>> "osd_recovery_max_chunk": "8388608",
>> &qu
[ceph-users] Ceph recovery kill VM's even with the smallest priority
Hello, Few days ago I had very strange situation. I had to turn off few OSDs for a while. So I've set flags:noout, nobackfill, norecover and then turned off selected OSDs. All was ok, but when I started these OSDs again all VMs went down due to recovery process(even when recovery priority was very low). There's more important config values: "osd_recovery_threads": "1", "osd_recovery_thread_timeout": "30", "osd_recovery_thread_suicide_timeout": "300", "osd_recovery_delay_start": "0", "osd_recovery_max_active": "1", "osd_recovery_max_single_start": "5", "osd_recovery_max_chunk": "8388608", "osd_client_op_priority": "63", "osd_recovery_op_priority": "1", "osd_recovery_op_warn_multiple": "16", "osd_backfill_full_ratio": "0.85", "osd_backfill_retry_interval": "10", "osd_backfill_scan_min": "64", "osd_backfill_scan_max": "512", "osd_kill_backfill_at": "0", "osd_max_backfills": "1", I don't know why ceph started recovery process if there was norecovery flags enabled but the fact is that it killed all VMs. Next, I've turned off noout, nobackfill, norecover flags and it started to look better. VM's went back online and recovery process was still going. I didn't saw performance impact on SSD disks but there was huge impact on spinners. Normally %util is about 25%, but during recovery it was nearly 100%. CPU Load increased on HDD based VMs by ~400%. iostat fragment(during recovery): Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdh 0.30 1.00 150.90 36.00 13665.60 954.60 156.4510.63 56.88 25.60 188.02 5.34 99.80 Now, I'm little lost, I don't know answers for few questions. 1. Why ceph started recovery even if nobackfill option was enabled? 2. Why recovery caused much more performance impact when norecovery options was enabled? 3. Why when norecovery was turned off, cluster started to look better but %util on HDD disks was so big(while recovery_op_priority=1 and client_op_priority=63)? 25% is normal, increased to 100% during recovery? Cluster information: ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90) 3x nodes(CPU E5-2630, 32GB RAM, 6xHDD 2TB with SSD journal, 3x SSD 1TB with NVMe journal), triple replication I would be very grateful If somebody can help me. Sorry if I've done something in wrong way - this is my first time writing on mailing list. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
