On 18 August 2014 16:30, Stuart Haslam <[email protected]> wrote: > On Mon, Aug 18, 2014 at 08:41:45AM +0100, Santosh Shukla wrote: >> - Disable default odp thread cpuset. Use is-cpu-isolated.sh script to create >> cpuset for dplane, cplane for odp. >> - Use odp-on-isolated-cpu.sh script to test this application. >> - test script location [1] >> >> - Refer readme for instruction to use. >> >> [1] >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2 >> >> Signed-off-by: Santosh Shukla <[email protected]> >> --- >> example/isolation/odp_isolation.c | 5 +++++ >> example/isolation/readme | 20 ++++++++++++++++++++ >> platform/linux-generic/odp_linux.c | 12 ++++++++++-- >> 3 files changed, 35 insertions(+), 2 deletions(-) >> create mode 100644 example/isolation/readme >> >> diff --git a/example/isolation/odp_isolation.c >> b/example/isolation/odp_isolation.c >> index 8d8fffc..1ade4c7 100644 >> --- a/example/isolation/odp_isolation.c >> +++ b/example/isolation/odp_isolation.c >> @@ -214,6 +214,11 @@ int main(int argc, char *argv[]) >> thr_id = odp_thread_create(0); >> odp_init_local(thr_id); >> >> + /* >> + * create dummy dp thread and don't pin them on any core, >> + * use odp-on-isolated-cpu.sh script to define dplane cpuset >> + * and to pin, run app on them, Refer readme. >> + */ >> for (i=0; i<num_workers; i++) { >> >> first_core = atoi(args.core_name[i]); >> diff --git a/example/isolation/readme b/example/isolation/readme >> new file mode 100644 >> index 0000000..d49d61a >> --- /dev/null >> +++ b/example/isolation/readme >> @@ -0,0 +1,20 @@ >> + >> +README on How-to run odp app on isolated core for no_hz_full kernel mode. >> + >> +- Disbale timer init in odp application. >> +- avoid using odp's cpuset. Let odp thread launch unpinned. >> +- use odp-on-isolated-cpu.sh script to pin odp dp thread to >> + dplane cpuset. >> + >> +- download my local test-definition repo link [1] >> +- copy odp_isolation binary to /usr/local/bin >> +- run ./common/scripts/odp-on-isolated-cpu.sh >> +Or >> +- ./common/scripts/odp-on-isolated-cpu.sh 1,2 "odp_isolation -l 1,2" >> +- Above script shows isolation duration, before / after interrupt count. >> + >> +Known issue : >> +- x86 hw generates spurious ipi lead to break no_hz_full isolation (todo). >> + >> +[1] >> https://git.linaro.org/people/santosh.shukla/test-definitions.git/shortlog/refs/heads/isol-v2 >> + >> diff --git a/platform/linux-generic/odp_linux.c >> b/platform/linux-generic/odp_linux.c >> index 6e2b448..7e853c7 100644 >> --- a/platform/linux-generic/odp_linux.c >> +++ b/platform/linux-generic/odp_linux.c >> @@ -22,6 +22,11 @@ >> #include <odp_debug.h> >> >> >> +#define NO_HZ_FULL_ISOL /* >> + * don't use odp cpuset. >> + * enable this for odp isolation mode >> + */ >> + >> typedef struct { >> int thr_id; >> void *(*start_routine) (void *); >> @@ -49,7 +54,9 @@ void odp_linux_pthread_create(odp_linux_pthread_t >> *thread_tbl, int num, >> int first_core, void *(*start_routine) (void *), void *arg) >> { >> int i; >> +#ifndef NO_HZ_FULL_ISOL >> cpu_set_t cpu_set; >> +#endif >> odp_start_args_t *start_args; >> int core_count; >> int cpu; >> @@ -62,16 +69,17 @@ void odp_linux_pthread_create(odp_linux_pthread_t >> *thread_tbl, int num, >> memset(thread_tbl, 0, num * sizeof(odp_linux_pthread_t)); >> >> for (i = 0; i < num; i++) { >> + cpu = (first_core + i) % core_count; >> +#ifndef NO_HZ_FULL_ISOL >> pthread_attr_init(&thread_tbl[i].attr); >> >> CPU_ZERO(&cpu_set); >> >> - cpu = (first_core + i) % core_count; >> CPU_SET(cpu, &cpu_set); >> >> pthread_attr_setaffinity_np(&thread_tbl[i].attr, >> sizeof(cpu_set_t), &cpu_set); >> - >> +#endif > > There's no need for this to be a compile time decision - this changes > the behaviour of all applications linked against this library. >
Agree, There might be better way to avoid linkage issue. > I would rather leave the affinity setting in odp_linux_pthread_create > and let the script just ensure that the specified cores are isolated. Is > there anything actually preventing that from being possible? Yes, Isolation is a specific requirement to application. current odp pins thread to core in order which not necessarily be the case all the time. A Flexibility to pin any of odp thread to any core is desired choice. Therefore I had to break free from existing pinning assumption. Keeping two set of cpuset - One is default ODP whome thread pinned with then Second one created by isolation script - doesn't make sense to me. Also unpinning a thread from odp cpuset group to script cpuset group gives system error. > > As it is, what happens if you build with NO_HZ_FULL_ISOL but don't run > the application via the odp-on-isolated-cpu.sh script? and if you do run There is a readme, User must refer that in case he is looking for isolation. > via the script, what's happening during the time between the application > launching the threads and the script moving those threads onto the > correct cores? It is order, first application spwans a thread then script moves those thread to isol cores. Therefore it should work. > > -- > Stuart. > _______________________________________________ lng-odp mailing list [email protected] http://lists.linaro.org/mailman/listinfo/lng-odp
