On Mon, May 25, 2015 at 08:37:09AM +0000, Radu-Andrei Bulie wrote: > Hi, > > The patch mentioned in the title is somehow related with the idea of odp > threads isolation, I mean the odp data path threads. > I am not aware of any discussions regarding the cgroups to provide odp > threads isolation nor some updates in the odp > API that will provide an integration with the cgroup library and cpusets > usage. > As it is stated in the ODP documentation the odp data path threads are not > controlled by the Linux scheduler - instead there is > a HW SoC level scheduler that manages them (or a specialized SW). This > implies that there is not any strict dependency in isolating the odp threads > by the rest > of the linux processes. > The problem introduced by this patch, on our platform, is that it restricts > the usage of all cores available on the default scheduling domain provided by > Linux. > And when I say all cores I include here the cores which are isolated from > rest and do not participate in the load balancing scheme of the > scheduler.(this is achieved > with the kernel boot time isolcpus). This cores will be used for the odp data > path threads and will be assigned explicitly at threads' creation time. > Thus the Linux scheduler will load balance processes(threads) on the > non-isolated cores.(these will be the odp control path threads). > The exact problem of the patch is that - to determine the number of cores > available for threads' creation - it uses the pthread_getaffinity_np function > (and not > _SC_NPROCESSORS_ONLN as before) which will return all the cores on which a > threads is eligible to run - that is all the cores that participate in the > Linux > load balancing scheme. In this situation(with this patch), on our platform, > we are able to create, for example, just one thread - on core 0, which is > used for control, because > the other cores 1..n which are isolated, are not visible using the function > mentioned above.(and as I said they will be used for the fast path processing) > As a conclusion, an action should be taken regarding the mentioned patch. > > > Regards, > > Radu
You're right this does need to be fixed to cope with isolated cpus, sorry about that oversight. There were a number of problems with the previous implementation due to the assumptions; - there's a single non-dataplane cpu and it's cpu 0 - any offline cpus are at the top of the cpu number range - the process is able to run threads on all online cpus Which does actually work most of the time and happens to exactly match your use case, but isn't ideal as among other things it means you can't run two ODP instances with threads pinned to different cores. What we need is a way to accurately determine which cpus in the system are dataplane cpus available to the current ODP instance. I can't think of a way to do this reliably (though open to suggestions..) so it looks like that information is going to need to come from the user, probably by taking a cpu mask on the command line. I'll start looking into this unless anyone has a better idea. -- Stuart. _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
