** Description changed: [SRU Justification] On a NovaLink system, the time drmgr takes to complete increases linearly with the number of virtual adapters. This is unreasonable. [Test case] To be completed by IBM, who have access to the hardware. 1. Add 200 virtual adapters to an LPAR 2. Run time /usr/sbin/drmgr -c slot -s <slot_name> -Q 3. Confirm that this takes multiple seconds to return. 4. Install powerpc-utils from -proposed. 5. Run time /usr/sbin/drmgr -c slot -s <slot_name> -Q again 6. Confirm that this takes less than a second to return. + + [Regression potential] + Any bugs introduced in this code could cause drmgr/lsslot to fail to correctly operate at all on the slots. However, the code is reasonably generic and the risk is low of this code failing intermittently: if it passes verification it's reasonable to expect it will work everywhere. == Comment: #0 - Jeremy A. Arnold ---Problem Description--- The time to run commands such as "drmgr -c slot -s U8247.22L.211E15A-V1-C210 -a -w 1" increases linearly with the number of slots on the system. For cloud environments with a large number of VMs hosted by a single NovaLink partition (or a small number of VIOS partitions), the number of virtual slots in the NovaLink partition can grow large, and the long drmgr time can be a major factor in the time to deploy new VMs. In one recent test, the call above took about 29 seconds to complete on a system with around 100 VMs. Earlier in the run (when there was less than 10 VMs) it only took about 2 seconds. I'm not at all an expert on this area, but it would appear that drmgr is iterating through all of the slots in order to find the one that was requested. Some evidence for this is provided by running: sudo time /usr/sbin/drmgr -c slot -s U8247.22L.211E15A-V1-C210 -Q This is taking about 14 seconds elapsed time (it may have been slower during the actual run due to concurrent executions of drmgr) on a system with about 232 virtual adapter slots. The time is similar if I make a request for a slot that does not exist (e.g. change C210 to C250), so it would appear that nearly all of the runtime is for looking up the slot and not for actually retrieving information about it. Adding -d20 to the above command provides additional debug data. This shows that the majority of the time is between these two lines of output: --- Could not find DRC property group in path: /proc/device-tree/pci@80000002000001f. DR nodes list --- For reference, the following command can identify the correct entry in the device tree in about 0.02 seconds. Obviously drmgr has more to do than just this, but this suggests that there is no fundamental reason the time has to scale with the number of slots: time (find /sys/firmware/devicetree/base/vdevice -name "ibm\,loc-code" | xargs grep "^U8247.22L.211E15A-V1-C210-T1$") ---uname output--- Linux cs-tul10-neo 4.4.21-customv1.29 #6 SMP Wed Apr 12 14:40:02 CDT 2017 ppc64le ppc64le ppc64le GNU/Linux Machine Type = 8247-22L ---Debugger--- A debugger is not configured ---Steps to Reproduce--- I used PowerVC to deploy 100 VMs to a NovaLink system and viewed /var/log/drmgr to observe how long the drmgr calls took during the test. I believe it would be sufficient to add 200 virtual adapters to an LPAR and then run "/usr/sbin/drmgr -c slot -s <slot_name> -Q" I'm happy to collect additional data in my environment if it would be helpful. Userspace tool common name: /usr/sbin/drmgr The userspace tool has the following bit modes: 64-bit Userspace rpm: powerpc-ibm-utils Userspace tool obtained from project website: na == Comment: #4 - Amartey S. Pearson I have a proposed fix for this in a github fork. In short, the algorithm used to populate the dr_nodes needs to be fixed. It currently walks the entire bus for every theoretical DRC (1000's of times). The fix is to walk the bus once. https://github.com/apearson-ibm/powerpc- utils/commit/6fefb6acb6fb302c97d71faef75a12674a50209a This addresses both drmgr and lsslot as the change is in common code. An example of the improvement we see: Here we have a system with 196 populated virtual slots. An lsslot takes 6.5 seconds. root@neo33-2:/usr/sbin# time /usr/sbin/lsslot -c slot | wc -l 196 real 0m6.495s user 0m1.108s sys 0m5.384s With the fix, the lsslot now takes 0.18 seconds, and scales well as more slots are added. root@neo33-2:~/powerpc-utils# time /usr/local/sbin/lsslot -c slot | wc -l 196 real 0m0.186s user 0m0.028s sys 0m0.156s == Comment: #7 - Anna A. Sortland We tried the patch in our test environment and it worked great. == Comment: #11 - Nathan D. Fontenot <nfont...@us.ibm.com> - 2017-05-18 13:30:42 == Patch submitted upstream. https://groups.google.com/forum/#!topic/powerpc-utils-devel/sd1gdvbQp0w
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1692837 Title: Ubuntu 16.04.02: powerpc-ibm-utils: drmgr does not scale with large number of virtual adapters To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1692837/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
