[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
Hi, I am just figuring out the performance issues - thanks guys for this bug description! It looks like the problem remains unsolved: $iostat -cd Linux 4.4.0-63-generic (ruprecht) 21.2.2017 _x86_64_(8 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1,520,010,41 13,030,00 85,04 Device:tpskB_read/skB_wrtn/skB_readkB_wrtn sda 142,33 1832,54 541,552174234 642525 sdb 18,86 0,08 541,55 92 642525 ... dm-0158,76 1832,46 541,552174142 642525 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
It would be great if @lifeless could try attempting to contact dm-devel again. This is a very important patch that would provide large benefit across a variety of systems. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
it would be great if @lifeless could try attempting dm-devel again. This is a very important patch that will provide large benefit across a variety of systems. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
three years and this issue haven't been solved. (at least not as a proven release, rather unproved patches) what we need to do to make it happen? i'm using dual boot in my computer - windows 8 ubuntu 12.10. using intel rst (rapid storage technology) and unfortunately i see that reading passed only to the first device (using iostat to monitor). will be glad to hear if there are any news. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
I've forwarded it upstream http://www.redhat.com/archives/dm- devel/2012-June/msg00013.html but had no reply so far. Any advice on how to get upstream to look at it would be wonderful. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
@robert, If you can get that patch accepted upstream we can get it through a stable patch. However, it isn't SRU material and we're not going to carry it on our own. ** Tags added: kt-worked -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
I personally cannot run this command because I do not have an Ubuntu system in this configuration at this time. Also, this bug is easily confirmed by simple inspection of the Linux kernel source. It behaves exactly in accordance with how the code is written, although how it's written is decidedly suboptimal. ** Changed in: linux (Ubuntu) Status: Incomplete = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
Here is an implementation of load balancing; this chooses the closest drive for a read UI, which results in doing sequential IO from one drive (and this works a lot better than round-robin per-IO: dm mapping happens before IO request merging - doing round-robin results in no request merging and terrible performance). Random IO (e.g. kernel builds, regular use) shows up on both disks in my mirror set, often in similar amounts. Future iterations can build on this to do more advanced algorithms. ** Patch added: raid1-loadbalance.patch https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+attachment/2192144/+files/raid1-loadbalance.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
Nice patch! It should be pushed upstream. Also, is this design now superior to the mdraid implementation? Should similar changes be made over there? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
Thanks. Just followed my nose ;). So this implementation handles two sequential IO operations happening at once as-well or better than the md implementation - the md implementation tracks sequential IO at the array level and at the result-of-an-IO level, but due to tagged IOs the latter can be incorrect and the former fails if two different processes are both doing sequential reads. The implementation I've put together will either end up sticking on one disk for the IO's if both are closer together, or will no sequential IO on both disks if the starting position of each is closer to the relevant sectors. tl;dr - both implementations still have tradeoffs. merging of IO's happens separately; I think dm has space for a merge routine in the layer, and we probably would benefit from that - merging IO's sent to the raid1 driver would let us get bigger IO's we can sensibly dispatch round-robin style - basically have both disks reading and head moving roughly in sync. I don't know if I'll get time to fiddle with that,but who knows ;) -Rob -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
A while back, I changed my doctoral specialization to computer architecture. But prior to that, I was studying AI, and as I think about this, this problem screams for a reinforcement learning approach. For N drives, there are 2N+1 actions, those being read to each drive, write to each drive, and possibly do-nothing (even if there is work to do). If we can map past actions, knowledge of the consequences of those actions, and information about time since last action to some reasonable number of environment states, then it would be straightforward to implement a completely adaptive online optimization algorithm. It could also be tunable to favor throughput, latency, and even energy to different degrees. The general idea is to use measurements of actual performance to judge the effectiveness of past actions in achieving those goals and integrate that into future decisions. If we did it right, the algorithm would automatically adapt to differences in drive performance characteristics. Mind you, this might only get incremental improvements in performance over the more naive approaches. But I recall reading one paper about a DRAM controller scheduler that achieved a 22% throughput improvement over the best static method available at the time. (And as I continue to think about this, perhaps the best place to implement this first would be for reordering accesses to single drives.) Sorry about the rant. :) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 361733] Re: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads
I've retitled this bug - while its true that there being two implementations is an issue, the actual problem folk are running into is the load balancing issue; beyond that the way in which it is solved is orthogonal (merge drivers, implement the same balancing algorithm, ...) ** Summary changed: - fakeRAID and software RAID are two different things + dmraid(fakeRAID) raid1 driver doesn't loadbalance reads -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/361733 Title: dmraid(fakeRAID) raid1 driver doesn't loadbalance reads To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/361733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
