On 18 August 2014 19:42, Stuart Haslam <[email protected]> wrote: > On Mon, Aug 18, 2014 at 12:32:36PM +0100, Santosh Shukla wrote: >> On 18 August 2014 16:30, Stuart Haslam <[email protected]> wrote: >> > On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote: >> >> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to >> >> create >> >> cpuset for dplane, cplane for odp. >> >> - Use odp-on-isolated-cpu.sh script to test this application. >> >> - test script location [1] >> >> >> >> - Refer readme for instruction to use. >> >> >> >> [1] >> >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2 >> >> >> >> Signed-off-by: Santosh Shukla <[email protected]> >> >> --- >> >> example/isolation/odp_isolation.c | 5 +++++ >> >> example/isolation/readme | 20 ++++++++++++++++++++ >> >> platform/linux-generic/odp_linux.c | 12 ++++++++++-- >> >> 3 files changed, 35 insertions(+), 2 deletions(-) >> >> create mode 100644 example/isolation/readme >> >> >> >> diff --git a/example/isolation/odp_isolation.c >> >> b/example/isolation/odp_isolation.c >> >> index 8d8fffc..1ade4c7 100644 >> >> --- a/example/isolation/odp_isolation.c >> >> +++ b/example/isolation/odp_isolation.c >> >> @@ -214,6 +214,11 @@ int main(int argc, char *argv[]) >> >> thr_id = odp_thread_create(0); >> >> odp_init_local(thr_id); >> >> >> >> + /* >> >> + * create dummy dp thread and don't pin them on any core, >> >> + * use odp-on-isolated-cpu.sh script to define dplane cpuset >> >> + * and to pin, run app on them, Refer readme. >> >> + */ >> >> for (i=0; i<num_workers; i++) { >> >> >> >> first_core = atoi(args.core_name[i]); >> >> diff --git a/example/isolation/readme b/example/isolation/readme >> >> new file mode 100644 >> >> index 0000000..d49d61a >> >> --- /dev/null >> >> +++ b/example/isolation/readme >> >> @@ -0,0 +1,20 @@ >> >> + >> >> +README on How-to run odp app on isolated core for no_hz_full kernel mode. >> >> + >> >> +- Disbale timer init in odp application. >> >> +- avoid using odp's cpuset. Let odp thread launch unpinned. >> >> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to >> >> + dplane cpuset. >> >> + >> >> +- download my local test-definition repo link [1] >> >> +- copy odp_isolation binary to /usr/local/bin >> >> +- run ./common/scripts/odp-on-isolated-cpu.sh >> >> +Or >> >> +- ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2" >> >> +- Above script shows isolation duration, before / after interrupt count. >> >> + >> >> +Known issue : >> >> +- x86 hw generates spurious ipi lead to break no_hz_full isolation >> >> (todo). >> >> + >> >> +[1] >> >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2 >> >> + >> >> diff --git a/platform/linux-generic/odp_linux.c >> >> b/platform/linux-generic/odp_linux.c >> >> index 6e2b448..7e853c7 100644 >> >> --- a/platform/linux-generic/odp_linux.c >> >> +++ b/platform/linux-generic/odp_linux.c >> >> @@ -22,6 +22,11 @@ >> >> #include <odp_debug.h> >> >> >> >> >> >> +#define NO_HZ_FULL_ISOL /* >> >> + * don't use odp cpuset. >> >> + * enable this for odp isolation mode >> >> + */ >> >> + >> >> typedef struct { >> >> int thr_id; >> >> void *(*start_routine) (void *); >> >> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t >> >> *thread_tbl, int num, >> >> int first_core, void *(*start_routine) (void *), void *arg) >> >> { >> >> int i; >> >> +#ifndef NO_HZ_FULL_ISOL >> >> cpu_set_t cpu_set; >> >> +#endif >> >> odp_start_args_t *start_args; >> >> int core_count; >> >> int cpu; >> >> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t >> >> *thread_tbl, int num, >> >> memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t)); >> >> >> >> for (i = 0; i < num; i++) { >> >> + cpu = (first_core + i) % core_count; >> >> +#ifndef NO_HZ_FULL_ISOL >> >> pthread_attr_init(&thread_tbl[i].attr); >> >> >> >> CPU_ZERO(&cpu_set); >> >> >> >> - cpu = (first_core + i) % core_count; >> >> CPU_SET(cpu, &cpu_set); >> >> >> >> pthread_attr_setaffinity_np(&thread_tbl[i].attr, >> >> sizeof(cpu_set_t), &cpu_set); >> >> - >> >> +#endif >> > >> > There's no need for this to be a compile time decision - this changes >> > the behaviour of all applications linked against this library. >> > >> >> Agree, There might be better way to avoid linkage issue. >> >> > I would rather leave the affinity setting in odp_linux_pthread_create >> > and let the script just ensure that the specified cores are isolated. Is >> > there anything actually preventing that from being possible? >> >> Yes, Isolation is a specific requirement to application. current odp >> pins thread to core in order which not necessarily be the case all the >> time. >> >> A Flexibility to pin any of odp thread to any core is desired choice. >> Therefore I had to break free from existing pinning assumption. > > Hence my suggestion in the other thread about changing > odp_linux_pthread_create. > If there's a limitation in the existing ODP API/implementation/apps then > it should be fixed rather than working around it. > >> Keeping two set of cpuset - One is default ODP whome thread pinned >> with then Second one created by isolation script - doesn't make sense >> to me. > > The two cpusets do different things (the cpusets created via /dev/cpuset > aren't related to the CPU_SET used with pthread_attr_setaffinity_np, > other than having a confusingly similar name). > > The cpuset created by the script specifies a group of cores the > application is permitted to run threads on (and ensures those cores are > isolated). The pthread_attr_setaffinity_np in the application specifies > a specific core within that group for a particular thread to run on. > > So I still don't see what the problem is with the problem is. > >> Also unpinning a thread from odp cpuset group to script cpuset >> group gives system error. > > Not sure what you mean exactly, but if the ODP application tries to > setaffinity to core that's not in the cpuset created by the script I > would expect that to fail. That's OK though as the list of cpus passed > to the script should be the same as that passed to the application, > anything else is an error.
Yes that was a problem which I wanted to avoid therefore choose to mask off odp thread affinity part. Also it doesn't make sense to move thread from odp cpuset to /dev/cpuset to me. However your concern is valid in favor of keeping odp generic so I'll revisit my changes. Thanks. > >> > >> > As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run >> > the application via the odp-on-isolated-cpu.sh script? and if you do run >> >> There is a readme, User must refer that in case he is looking for isolation. >> >> > via the script, what's happening during the time between the application >> > launching the threads and the script moving those threads onto the >> > correct cores? >> >> It is order, first application spwans a thread then script moves those >> thread to isol cores. Therefore it should work. > > Yes but there is a window in between where they could all be running on > the same core. yes, odp init time. > How long does it take for them to be moved to the correct > core? In case of l2fwd - I am migrating thread just after thread initialization.. no real IO started on those threads..So its fairly quick for me. Are you asking about a use-case where dpthread spawned, doing IO then migrated to other core by script? > how does the application know when it's been done?.. I verify this way (command based approach) ps -Leo pid,tid,psr,pcpu,comm | grep odp # 23025 23025 0 99.9 odp_l2fwd # 23025 23026 7 0.0 odp_l2fwd # 23025 23027 1 99.9 odp_l2fwd # 23025 23028 2 100 odp_l2fwd Last 2 threads (dpthread) moved to isolated core as per my script instruction. > there's no > need for the uncertainty when the application can set its own affinity. Right. > > -- > Stuart. > _______________________________________________ lng-odp mailing list [email protected] http://lists.linaro.org/mailman/listinfo/lng-odp
