> -----Original Message----- > From: lng-odp [mailto:[email protected]] On Behalf Of ext > Stuart Haslam > Sent: Wednesday, June 03, 2015 2:54 PM > To: Radu-Andrei Bulie > Cc: [email protected] > Subject: Re: [lng-odp] patch problem - "Support running with restricted > cpuset" > > On Mon, May 25, 2015 at 08:37:09AM +0000, Radu-Andrei Bulie wrote: > > Hi, > > > > The patch mentioned in the title is somehow related with the idea of odp > threads isolation, I mean the odp data path threads. > > I am not aware of any discussions regarding the cgroups to provide odp > threads isolation nor some updates in the odp > > API that will provide an integration with the cgroup library and cpusets > usage. > > As it is stated in the ODP documentation the odp data path threads are > not controlled by the Linux scheduler - instead there is > > a HW SoC level scheduler that manages them (or a specialized SW). This > implies that there is not any strict dependency in isolating the odp > threads by the rest > > of the linux processes. > > The problem introduced by this patch, on our platform, is that it > restricts the usage of all cores available on the default scheduling > domain provided by Linux. > > And when I say all cores I include here the cores which are isolated > from rest and do not participate in the load balancing scheme of the > scheduler.(this is achieved > > with the kernel boot time isolcpus). This cores will be used for the odp > data path threads and will be assigned explicitly at threads' creation > time. > > Thus the Linux scheduler will load balance processes(threads) on the > non-isolated cores.(these will be the odp control path threads). > > The exact problem of the patch is that - to determine the number of > cores available for threads' creation - it uses the pthread_getaffinity_np > function (and not > > _SC_NPROCESSORS_ONLN as before) which will return all the cores on which > a threads is eligible to run - that is all the cores that participate in > the Linux > > load balancing scheme. In this situation(with this patch), on our > platform, we are able to create, for example, just one thread - on core 0, > which is used for control, because > > the other cores 1..n which are isolated, are not visible using the > function mentioned above.(and as I said they will be used for the fast > path processing) > > As a conclusion, an action should be taken regarding the mentioned > patch. > > > > > > Regards, > > > > Radu > > You're right this does need to be fixed to cope with isolated cpus, > sorry about that oversight. > > There were a number of problems with the previous implementation due to > the assumptions; > > - there's a single non-dataplane cpu and it's cpu 0 > - any offline cpus are at the top of the cpu number range > - the process is able to run threads on all online cpus > > Which does actually work most of the time and happens to exactly match > your use case, but isn't ideal as among other things it means you can't > run two ODP instances with threads pinned to different cores. What we > need is a way to accurately determine which cpus in the system are > dataplane cpus available to the current ODP instance. I can't think of a > way to do this reliably (though open to suggestions..) so it looks like > that information is going to need to come from the user, probably by > taking a cpu mask on the command line. > > I'll start looking into this unless anyone has a better idea. > > -- > Stuart.
I'm just adding /** * Default cpumask for worker threads * * Creates cpumask based on starting count, actual value returned. * * @param[out] mask CPU mask to update * @param num Number of worker threads, zero for all available CPUs * @return Actual number of CPUs used to create the mask */ int odp_cpumask_def_worker(odp_cpumask_t *mask, int num); /** * Default cpumask for control threads * * Creates cpumask based on starting count, actual value returned. * * @param[out] mask CPU mask to update * @param num Number of control threads, zero for all available CPUs * @return Actual number of CPUs used to create the mask */ int odp_cpumask_def_control(odp_cpumask_t *mask, int num); ... an implementation instance can recommend usage of certain CPUs. User still decides (creates and pins) where the threads run. It's system specific from where the implementation gets information about available CPUs. These two would replace odph_linux_cpumask_default(). -Petri _______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
