[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
[Expired for qemu (Ubuntu) because there has been no activity for 60 days.] ** Changed in: qemu (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
My contact is satisfied with the current workaround to disable os- prober, which apparently significantly help on this side. I'm still convince that os-prober is not the issue, but just make the situation less worst which is acceptable for my contact. I can't reproduce the situation on my side using the same exact parameter as his, so again I'm convince that his problem is probably related to app competing for writes, but my contact is not amenable to push the investigation further, Thanks Ryan and Christian. - Eric -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
I can't reproduce the situation using 2 disks + cache='none' + io='native +
bus='virtio'.
So my 1xDisk scenario was because I was exhausting the disk.
Disk 1 -> vg01 -> Host disk ("/")
Disk 2 -> vg02 -> Pool for KVM Guest
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:00 465.8G 0 disk
└─sda1 8:10 465.8G 0 part
└─vg01-root 252:00 372.5G 0 lvm /
sdb 8:16 0 465.8G 0 disk
└─vg02-kvmguest01 252:10 18.6G 0 lvm
# Idle
$ time update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.0-64-generic
Found initrd image: /boot/initrd.img-4.4.0-64-generic
done
real0m0.782s
user0m0.272s
sys 0m0.120s
# With my contact protocol to reproduce the situation :
$ time update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.0-64-generic
Found initrd image: /boot/initrd.img-4.4.0-64-generic
done
real0m0.782s
user0m0.272s
sys 0m0.120s
So the more I look at this, and the more I think it's not a bug but
something in my contact's system that maybe is competing for writes, I
agree that having some iostat date from my contact's system will help
identifying how the I/O looks like with and without his protocol.
- Eric
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1666555
Title:
update-grub slow with raw LVM source dev + VirtIO bus
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
On Wed, Feb 22, 2017 at 9:15 AM, Eric Desrochers < eric.desroch...@canonical.com> wrote: > I can't attach a sosreport since it may reveal data that my contact > doesn't want to be public on the bug, but I can check details you would > inquire. > I think an lshw output would be useful; mostly CPUs, RAM, disks data Something like sudo lshw -sanitize -xml > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1666555 > > Title: > update-grub slow with raw LVM source dev + VirtIO bus > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
I can't attach a sosreport since it may reveal data that my contact doesn't want to be public on the bug, but I can check details you would inquire. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
I can't attach a sosreport since it may reveal data that my contact doesn't want to be public on the bug, but I can check details you would inquire them. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
On Wed, Feb 22, 2017 at 3:53 PM, Ryan Harper <1666...@bugs.launchpad.net> wrote: > What sort of disks and connections are we talking about on the host? How > are they attached? Is sda in any of the LVM groups? > Any way we can get a sosreport from the host? > https://pastebin.canonical.com/180330/ Eric already took the todo to ask for the types to reproduce on the right setup. We all know there can be a huge diff between spinning and nvme and all in between I just heard he has sosreport and can attach it. I wonder if we can find the disk type in there. > > - In the guest it writes to Guest Disk. > > - This can only fill up sdb directly (the queue of sda should be > untouched > > by the guests activity) > > - Update-Grub is in the Host, so should be sda only > > > > Could still be locking up CPU/IRQ. > > > > Yes, 'virsh capabilities' from the host and 'virsh dumpxml ' would > be > helpful to understand the hypervisor/guest topology better. > Yeah that will help as well to reproduce the same case -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
On Wed, Feb 22, 2017 at 8:30 AM, ChristianEhrhardt < 1666...@bugs.launchpad.net> wrote: > Discussed on IRC, > > TL;DR of that: > The real customer case is different than we assumed so far: > sda <- Host root > sdb <- VG <- Guest Disk > > What sort of disks and connections are we talking about on the host? How are they attached? Is sda in any of the LVM groups? Any way we can get a sosreport from the host? > - In the guest it writes to Guest Disk. > - This can only fill up sdb directly (the queue of sda should be untouched > by the guests activity) > - Update-Grub is in the Host, so should be sda only > > Could still be locking up CPU/IRQ. > Yes, 'virsh capabilities' from the host and 'virsh dumpxml ' would be helpful to understand the hypervisor/guest topology better. > > Eric check exact disk types and reproduce in a similar env. > He then will come back with iostat and blktrace data of good/bad case. > > ** Changed in: qemu (Ubuntu) >Status: New => Incomplete > > -- > You received this bug notification because you are subscribed to the bug > report. > https://bugs.launchpad.net/bugs/1666555 > > Title: > update-grub slow with raw LVM source dev + VirtIO bus > > To manage notifications about this bug go to: > https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
Discussed on IRC, TL;DR of that: The real customer case is different than we assumed so far: sda <- Host root sdb <- VG <- Guest Disk - In the guest it writes to Guest Disk. - This can only fill up sdb directly (the queue of sda should be untouched by the guests activity) - Update-Grub is in the Host, so should be sda only Could still be locking up CPU/IRQ. Eric check exact disk types and reproduce in a similar env. He then will come back with iostat and blktrace data of good/bad case. ** Changed in: qemu (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
Thanks for the Data Eric. It confirms Ryans assumptions. The ongoing I/O keeps the device queue with 50-70 Requests of 1MB size busy. Any request from update-grub that enters has to go through that queue. Which due to the load has 190+-20ms. If now also the assumption on the barriers is true that means it can not enqueue all its requests at once but instead has to wait for each to complete first. With that as a rule of thumb it becomes #requests*avg-latency. Even if we assume no other time spent the 14 seconds are 14.000 ms and /190 that means ~#74 requests which could be true for an update-grub to run. So I don't see it as a bug, but more as a tuning and performance/integrity tradeoff. Please let us know once you tried the other tunings Ryan recommended. There is more you could do (mostly I think on workload disk split in the case here), but that is all performance tuning which I'm not sure you want look into here - do you? One question I could not find answered in the bug - does the additional I/O to the disk come from "inside" the guest or from a 3rd source? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
busy_case_iostat ** Attachment added: "busy_case_iostat" https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+attachment/4823789/+files/busy_iostat.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
idle_case_iostat ** Attachment added: "idle_case_iostat.txt" https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+attachment/4823790/+files/idle_iostat.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
I don't see an improvement using io='threads' nor 'default', as I previously tested it. I didn't try the cache option as in general 'none' is the best choice, but after trying with 'writethrough' I can tell it helps, by cutting half of waiting time. # cache='none' $ time update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.4.0-62-generic Found initrd image: /boot/initrd.img-4.4.0-62-generic done real0m14.703s user0m0.280s sys 0m0.300s # cache=writethrough $ time update-grub Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.4.0-62-generic Found initrd image: /boot/initrd.img-4.4.0-62-generic done real0m7.123s user0m0.376s sys 0m0.304s - Eric -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
On Tue, Feb 21, 2017 at 12:07 PM, Eric Desrochers <
eric.desroch...@canonical.com> wrote:
> @Ryan,
>
> This situation has been brought to my attention by someone, and I was
> able to reproduce it myself.
>
> The main difference between my setup and his, is that my contact has
> "vg02-vg02--labmysql01" on a separate disk, thus /dev/sdb instead of
> /dev/sda like my case as I only have 1 disk available for testing.
>
> $ lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:00 119.2G 0 disk
> ├─sda1 8:1028G 0 part
> │ └─vg01-vg01--root 252:0028G 0 lvm /
> ├─sda2 8:20 1K 0 part
> ├─sda5 8:50 18.6G 0 part
> │ └─vg01-vg01--swap 252:10 18.6G 0 lvm [SWAP]
> └─sda6 8:60 18.6G 0 part
> └─vg02-vg02--labmysql01 252:20 18.6G 0 lvm
>
> $ lvs
> LV VG Attr LSize Pool Origin Data% Meta% Move
> Log Cpy%Sync Convert
> vg01-root vg01 -wi-ao 27.94g
> vg01-swap vg01 -wi-ao 18.62g
> vg02-labmysql01 vg02 -wi-ao 18.62g
>
> $ vgs
> VG #PV #LV #SN Attr VSize VFree
> vg01 2 2 0 wz--n- 46.56g0
> vg02 1 1 0 wz--n- 18.62g0
>
> $ pvs
> PV VG Fmt Attr PSize PFree
> /dev/sda1 vg01 lvm2 a-- 27.94g0
> /dev/sda5 vg01 lvm2 a-- 18.62g0
> /dev/sda6 vg02 lvm2 a-- 18.62g0
>
> I'm currently re-installing my system to Zesty and see if I can
> reproduce it with the devel release.
>
Since you have a single spindle in play here, this is what I suspect is
going on.
You're opening the underlying device with O_DIRECT (cache=none) and
specifying
the use of the "native" io driver, which is Linux AIO. Virtio-blk will
enable the disk
with WCE enabled ("write cache" in /sys/class/block/vdX/cache_type); The
guest is going to issue write barriers, aka, a write-cache flush operation
to ensure consistency of the data is on the virtual disk (and not in the
cache).
This results in qemu on the host flushing the AIO queue via fdatasync()
which is going to ask the hypervisor to wait until it's sure the data is on
the disk. This behavior varies based on the host OS and disk devices.
There are a couple of things to try here:
1) io=threads; for a single disk, I don't think linux aio is providing any
performance benefit but you will pay more per-io submitted due to Linux AIO
overhead;
2) cache=writethrough; This avoids telling the guest it has a write-cache,
it enables higher read performance as host can cache that data, and writes
are not returned until the underlying write syscall submitted by qemu does
which should avoid any costly guest-driven cache flush operations (The
opens the underlying disk on the host with O_DSYNC flag).
This is mostly a question of performance vs. integrity. If it's OK for the
data to become corrupt (it's replaceable or throw-away, or backed up) then
one may want to use cache=writeback or even cache=unsafe which enables more
caching and no barriers.
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1666555
>
> Title:
> update-grub slow with raw LVM source dev + VirtIO bus
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions
>
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1666555
Title:
update-grub slow with raw LVM source dev + VirtIO bus
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
@Ryan, This situation has been brought to my attention by someone, and I was able to reproduce it myself. The main difference between my setup and his, is that my contact has "vg02-vg02--labmysql01" on a separate disk, thus /dev/sdb instead of /dev/sda like my case as I only have 1 disk available for testing. $ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:00 119.2G 0 disk ├─sda1 8:1028G 0 part │ └─vg01-vg01--root 252:0028G 0 lvm / ├─sda2 8:20 1K 0 part ├─sda5 8:50 18.6G 0 part │ └─vg01-vg01--swap 252:10 18.6G 0 lvm [SWAP] └─sda6 8:60 18.6G 0 part └─vg02-vg02--labmysql01 252:20 18.6G 0 lvm $ lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert vg01-root vg01 -wi-ao 27.94g vg01-swap vg01 -wi-ao 18.62g vg02-labmysql01 vg02 -wi-ao 18.62g $ vgs VG #PV #LV #SN Attr VSize VFree vg01 2 2 0 wz--n- 46.56g0 vg02 1 1 0 wz--n- 18.62g0 $ pvs PV VG Fmt Attr PSize PFree /dev/sda1 vg01 lvm2 a-- 27.94g0 /dev/sda5 vg01 lvm2 a-- 18.62g0 /dev/sda6 vg02 lvm2 a-- 18.62g0 I'm currently re-installing my system to Zesty and see if I can reproduce it with the devel release. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1666555] Re: update-grub slow with raw LVM source dev + VirtIO bus
This opens the underlying device (lvm) via O_DIRECT and submits Linux AIO requests to the device. If the VMs are sharing the LVM pool it may be overwhelming the underlying devices. This feels like there are syncs after each write which is going to serialize access to the LVM (and underlying devices). IDE emulation allows only one outstanding IO request; that likely keeps things fair. Can you share how the LVM is configured (pvs, vgs, lvs)? Can you attach capture some iostat data in your idle and busy cases: iostat -x -k 2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1666555 Title: update-grub slow with raw LVM source dev + VirtIO bus To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1666555/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
