On 02/24/2014 02:57 PM, Alexander Duyck wrote:
>> >> The big(ger) problem here is that the ixgbe (and other drivers IIUC) do not >> do a >> good job of handling MSIs, making sure they are launched on the right cpus, >> and >> cleaning up during cpu hotplug operations. This code looks like it needs a >> bit >> of work so your advice is appreciated. >> > > I'm not seeing how this helps with hotplug since the driver doesn't get > notifications of any such event anyway, at least not currently. Are > there some changes coming in that regard? There have been some issues with CPU hotplug. The problem started a while ago with several different vendors systems being unable to perform cpu hotplug down without running into strange storage issues. I chased this down and came up with linux.git commit da6139e49c7cb0f4251265cb5243b8d220adb48d, x86: Add check for number of available vectors before CPU down [1]. What has caused that check to be necessary is that the ixgbe driver is now allocating so many interrupts that on large systems which full sockets are taken in and out of service, it is possible that there are not enough empty vectors for all the irqs on a down'd cpu. IMO what the ixgbe driver is effectively doing is starving the system of resources. If I rmmod the ixgbe driver (and free it's irqs of course) I have no problem in taking all cpus except 1 out of service. I've started coding a cpu notifier which would take a queue out of service when a cpu goes down but I hit these smaller issues which have to be fixed first before I come up with the notifier. P. [1] Please note that even this is incorrect. I don't properly take into account node-specific interrupts which can only be moved to specific cpus because of the affinity settings of MSI. I'm working on a patch right now to fix that issue. > >>> >>> ATR is supposed to map 1:1 queues to CPUs. The problem is RSS is also a >>> factor and not especially smart or NUMA aware. The ideal solution would >>> be to allocate the first N CPUs, where N is the number in the local node >>> for ATR/RSS. >> >> Okay ... I'll look into that. >> >> Then map all other queues as ATR with a 1:1 mapping to CPUs. >>> >> >> Hmmm ... but what if off-node CPUs cannot reach the device? Part of the >> puzzle >> here is that ACPI may be not only telling us that the device is on a specific >> node, but that the device is physically separated on a root bus. >> >> P. >> > > As far as I know there shouldn't be anything that explicitly prevents us > from reading/writing to any memory in the system. In the case of the > Intel NUMA configurations anyway we just can use the QPI bus if the CPU > isn't connected to the same root complex. > > Thanks, > > Alex ------------------------------------------------------------------------------ Flow-based real-time traffic analytics software. Cisco certified tool. Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer Customize your own dashboards, set traffic alerts and generate reports. Network behavioral analysis & security monitoring. All-in-one tool. http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired