[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
** Tags added: cscc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
In my opinion I also agree to use deadline scheduler for server kernel which is also the same problem on the 4.15 kernel, if a server kernel would still exist. Under trusty we generally saw good performance with deadline scheduler and on a recent SSD based deployment with 4.15 I noticed a large performance regression for random IO until we reverted to deadline. You could argue that the scheduler can be changed at any time but on the contrary I would assume more general suitable parameters from a server distro vs a desktop distro. My larger problem is that this change was seemingly done via https://kernel.ubuntu.com/git/ubuntu/ubuntu-zesty.git/commit/?id=af80b83a2b6184fea27f050948146fcd9a28070d by a test based on kernel compiles etc, which to be honest, is not a standard server use case. Apart from the issue that I could not spot any documentation on the release notes that this has changed. If this has been documented and simply missed it, then please ignore this comment. But now I'm wondering what else has changed now. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
I observe substantial lag under heavy I/O (in my case I was unzipping a multigigabyte archive) with the deadline scheduler. Manually switching to the cfq scheduler improved responsiveness and eliminated the lag. Perhaps cfq is a better choice for the desktop version? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
BTW, that IBM article... it doesn't mention what versions they used, which I think is very important :-(. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
In my humble opinion, several kernel options should be tested before each new Ubuntu version to see what is the best one at the time. This is one such option. Isn't the CFQ scheduler in better shape in Ubuntu 14.04? Maybe not, I am just asking :-). Thanks for the info! :-) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
See also bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1064521 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Thanks for fixing this in Quantal! Deadline is the 'best' choice for servers and computers with SSD's. However there are situations where CFQ is preferred, like desktop PC's with a hard drive. But I can imagine it isn't easy to detect this in a reliable way. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Well the obvious downside of deadline is that priorities (ionice) are not supported. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
We (Ubuntu kernel team) have been evaluating CFQ vs deadline. For this specific bug report, we need some hard facts to make a decision about changing to deadline IO scheduler. For example, any test case or sequence we can reproduce easily. Here is comparison of CFQ vs Deadline http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=%2Fliaat%2Fliaatbppartitions.htm -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
We (Ubuntu kernel team) have been evaluating CFQ vs deadline. For this specific bug report, we need some hard facts to make a decision about changing to deadline IO scheduler. For example, any test case or sequence we can reproduce easily. Here is comparison of CFQ vs Deadline http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbpdeadline.htm -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Hark, We've changed default IO scheduler from CFQ to deadline for both server and desktop kernel. Please find our latest kernel from git tree for Quantal release, any test and bug report are welcome. Thanks, -Bryan ** Changed in: linux (Ubuntu) Status: Confirmed = In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Hi Bryan, Thanks for taking this seriously. I only have the testcase I described above. This server is now in production, so I cannot do any more tests on it. However it should be easy to test the performance of deadline vs CFQ on a server, and lots of people did this and wrote about it on their blogs/websites. But this is only the performance of course, for more serious problems you could resemble my scenario;start a KVM with ide emulation on a host with a normal (slow) disk, do regular writes on the KVM and do a write on the host that lasts at least several minutes on the same physical disk. I used a Dell R300 server with two WD Caviar Green disks, you should be able to get the same result on most other hardware configurations, as long as you don't use too fast disks, hardware raid controllers etc. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Hark, We (Ubuntu kernel team) have been evaluating CFQ vs deadline. For this specific bug report, we need some hard facts to make a decision about changing to deadline IO scheduler. For example, any test case or sequence we can reproduce easily. Do you have such simple test case sharing with us? We are also trying to collect some data right now. -Bryan -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
As Andy said the wrong scheduler decision should worst case result in worse performance. But getting so bad that a kvm guest sees disk failures is more than should be allowed. It would be good if we could reproduce this. For that it would be good to know more about the setup: For the host: - What kind of RAID (I assume md but which level) is used? - Which file system and is that on the RAID directly or on LVM-LVs? For the guest: - Is the virtual disk mapped to a image file or a partition/LV? - is it defined as a virtio disk or an emulated disk? Then the rsync: is that from inside the guest to the host? Or from another server to the host? Depending on the setup is the target on the same fs or not? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
For the host: -We use a md RAID indeed, raid 1 -The host and the guest both use ext4. On the host the file system is on a LVM LV indeed For the guest: -The virtual disk is in a qcow2 image file -At first we used an emulated IDE disk. We have now changed this to virtio, but this was done a few days after the problems had been solved by changing the scheduler to deadline. Maybe using IDE emulation helps to reproduce the problem, I don't know. The rsync was done from another server to the host, not with increased priority or anything like that. I have to admit that our setup was not optimal. The disks are modern but quite slow, and the RAID 1 config and IDE emulation doesn't help either. So I am not surprised this could be quite slow, however disk starvation inside the guest should not happen. As said, changing only the scheduler solved this problem completely. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Thanks for the detailed info. I agree that disk starvation should not happen. Though it could be that deadline just about covers some other problem. There are a few pieces that add to slow down the host io, the RAID1 and the qcow2 file image. Using the IDE emulation would actually slow down the guest. But, and I am speculating here a bit, having the rsync going on while the guest writes to the virtual disk may just get delayed a bit over the timeout that the IDE driver expects some answer from the emulated drive. And if that is the case then there could be some potential of this to happen regardless of the scheduler choice. It just happens to help in your case. I did not see the same problems on a local setup with cfq, but then I had a raw image and were using virtio. I will try to adapt this setup to at least having qcow2 and emulated disks to see whether that reproduces the problems. If possible I would like to keep my RAID5 the way it is. ;) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Thinking about it the IDE emulation may be the problem. If I remember it well the disk timeout messages on the guest had 'ata' in it. This is not really my cup of tea but I know that ATA and IDE are related. But please remember that my problem is only one example of what can happen when using CFQ on servers. For lots of common server tasks using CFQ is not recommended, so I still think this is a valid bug report. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
@Andy - The logs that could be interesting are the logs of the virtual server (kvm). However I don't have this log anymore, as I had to replace the kvm image with a backup because of this problem. But let me try to explain the situation, you'll see that unfairness is not the issue here. - I installed a new Ubuntu 12.04 server (host), with CFQ as scheduler (new default) - A kvm image was placed on the host (local disks, software raid), and it was started. I call this virtual server 'kvm'. - The kvm was running fine all day, no problems at all - At night some big files were rsync'd to the host, with normal priorities. - A few minutes after the rsync started the kvm started logging ata disk timeouts. After some time the disks were remounted read-only due to these errors. - The next day I tried to restart the kvm, but the image was completely corrupted. - After placing back a backup the kvm was started again - The next night exactly the same happened during the rsync. - The next day I placed back the backup again, and changed the scheduler to 'deadline' (old default). This solved the problem. But this is just one of many examples of what can go wrong with CFQ. Internet is full of stories about CFQ causing very slow i/o or worse for several common server tasks. I really think the default scheduler should be changed to deadline again for servers. It is not without reason that the old -server kernels did this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
@Hark -- though the default is changed we would not expect hangs from using CFQ, or are we just talking about gross unfairness. Can you share the logs of the failures. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Yes Leann and Bryan, I can boot with elevator=deadline and that is exactly what I did after I noticed that CFQ was being used. But this remains a bug that should be solved. Most people don't even know what a scheduler is, or how to change it. On a server it is almost always a bad idea to use CFQ. Brian, your suggestion to add a script in the server image sounds good to me, do you know how to get this to the right people? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
I believe you can boot with elevator=deadline to switch back to the Deadline I/O Scheduler. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
I tried install latest Ubuntu Server image on my machine and confirmed that we are using CFQ for default IO scheduler for both server and desktop now. Hark, we merged -server and -generic kernel image together for easier maintenance and most difference can be switched via kernel command line or user space interface. As Leann said, we can add elevator=deadline in kernel boot command line to use Deadline I/O scheduler by default. Or just change /sys/block/DEV/queue/scheduler from CFQ to deadline. If it's necessary, I'm think about add some script to do this in our server image. But that probably related to grub, upstart init script or other user space utils not kernel issue now. -Bryan -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
** Package changed: linux-meta (Ubuntu) = linux (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
Log files are completely irrelevant for this issue. ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
** Changed in: linux (Ubuntu) Assignee: (unassigned) = Bryan Wu (cooloney) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1008400] Re: Ubuntu server uses CFQ scheduler instead of deadline
** Changed in: linux (Ubuntu) Importance: Undecided = Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1008400 Title: Ubuntu server uses CFQ scheduler instead of deadline To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1008400/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
