From: Matthew Malcomson <[email protected]> This patch is a separate beneficial step of the work I'm doing towards PR119588. In that bug I describe how some internal workloads are hitting some performance problems w.r.t. the creation of many parallel regions one after each other.
I'm attempting to reduce overhead of parallel region creation when using a large number of threads. Earlier work adjusts the barrier design to something that performs better under heavy contention. This is a patch built on top of that earlier work, but with appropriate rebase could be independent. It addresses a different area where I've seen a performance impact. It's the area that I mentioned in the below mail to the GCC mailing list. https://gcc.gnu.org/pipermail/gcc/2025-September/246618.html The fundamental point is that the cost of teardown & setup of a parallel region for LLVM is very close to the cost of a single barrier. For us there is a significant chunk of extra work in the actual teardown and setup. When re-using a team with no changes in various ICVs there is very little work that is required to be performed on top of a plain barrier. Even when some ICV's have changed a large amount of the work that is currently in a loop performed by the primary thread while all threads are waiting on it can be moved into secondary threads. This patch makes those changes. There are three distinct but related cases: 1) When re-using a thread unchanged. 2) When re-using a thread with some changes (or when leaving a thread to exit). 3) When starting a new thread. In all of these cases `gomp_team_start` currently initialises the same value into each threads local value for all of the below: - *thr->task (initialisation) - thr->num_teams - thr->team_num - fn (either thr->fn or data->fn) - data (either thr->data or data->fn_data) - thr->ts.def_allocator Hence each of these can be handled by writing this information to a shared location on the relevant team and having each thread read the information from there. There is some data that is always set to the same value (at least in relation to the team we're running on): - thr->ts.work_share - thr->ts.last_work_share - thr->ts.single_count - thr->ts.static_trip - thr->task (where it is located) Initialisation of these is directly moved from the primary thread to the secondary threads without any new communication being added. There are some variables that are always the same for a non-nested team and always equal for all threads of a nested team: - thr->ts.active_level - thr->ts.level These are communicated to secondary threads when starting a new team via the `start_data` of the new pthread, and are left as-is for the re-use of threads in a non-nested team. Then finally there are some member variables that are kept the same as for the last team for all reused threads of a non-nested team as long as the places/affinity ICV's are not changed (and the team is reused): - thr->ts.place_partition_off - thr->ts.place_partition_len - thr->place - thr->ts.team_id - thr->team - Which thread's `release` member address is stored in which `team->ordered_release[i]`. We maintain a cache of the last used ICV's in the `team` structure. If these match then we can completely skip the loop where the primary initialises parts of each secondary threads local data. These must be provided to any new secondary threads when they are starting. The last part to allow the primary thread to avoid iterating over all secondary threads when nothing has changed is to invert the protocol used to communicate that a given thread must exit. The existing convention is that secondary threads set their own `thr->fn` to `NULL`, then the primary sets the `nthr->fn` of each thread it wants to continue to the function that said secondary thread must execute. This means the primary thread needs to perform a write on each thread that is continuing. Here I switch the convention so that the primary thread sets `thr->exit` on each of the threads it wants to exit. Those threads without this boolean flag set read the function that they are to execute from their team structure. This means the primary thread writes once per exiting thread instead of once per reused thread. In the case where the number of threads and affinity setup is not changing between parallel regions this means no work instead of O(N) work. There is still a clause in the main loop of `gomp_thread_start` checking for `thr->fn` in order to let `gomp_pause_host` and `gomp_free_thread` continue to work as intended. I could have changed those functions to adjust `pool->last_team->ti.fn` instead of setting `thr->fn` directly but it didn't seem any better either way. Only subtlety here is that in order to ensure memory safety the secondary thread must avoid performing this initialisation when `thr->fn == NULL`. If a secondary non-nested thread gets "left to exit" when a new team is started the primary thread could race ahead, cache the new team, and free the team that this secondary thread was using all before the secondary thread goes through the new initialisation code. That could lead to a use-after-free. Hence we have to early-exit when this thread is not going to perform any more work. There is a synchronisation side-benefits to putting the initialisation in each secondary thread: - gomp_init_task is now guaranteed to be called on the thread-local task *after* gomp_finish_task is called on it. This avoids any leak of `depend_hash` members. (When team_id has not changed these functions are called on the same thread, when they have changed we add a new synchronisation point between new threads and threads that have been "remapped"). ------- Testing done: - Bootstrap & regtest on aarch64 and x86_64. - With & without _LIBGOMP_CHECKING_. - Testsuite with & without OMP_WAIT_POLICY=passive for GOMP_BARRIER_TYPE=flat, GOMP_BARRIER_TYPE=tree, GOMP_BARRIER_TYPE=centralized, and unspecified barrier type. - With and without configure `--enable-linux-futex=no` for posix target. - With and without configure `--enable-linux-futex-waitv=no` to check we haven't broken original linux/ target. - Cross compilation & regtest on arm. - Built gcn, rtems, nvptx targets (just building -- not ran testsuite). - TSAN with checks against new/old "fingerprints" of warnings raised from the libgomp testsuite. Performance testing done: - SPEC 2026 speed (those benchmarks using OpenMP) - SPEC OMP 2012 - SPEC HPC 2021 (restricted to one socket) - EPCC OpenMP Microbenchmarks. - Microbenchmark mentioned in PR119588. EPCC OpenMP microbenchmarks ran with the cross-product of: - OMP_WAIT_POLICY={active,unset,passive} - OMP_PROC_BIND={close,unset} - OMP_NUM_THREADS={4,unset} - GOMP_BARRIER_TYPE={flat,tree,centralized,auto} libgomp/ChangeLog: PR libgomp/119588 * libgomp.h (struct gomp_team): Add fields for communication at team start time. New `ti` member, new `ppv` member, new `reused` boolean. * team.c (gomp_thread_start): Initialise thread-local data using members stored on team. (get_last_team): Set `team->reused` member when reusing. (gomp_new_team): Unset `team->reused` member when creating new team. (gomp_team_start): Primary thread stores data on team structure instead of copying that data to each secondary threads gomp_thread structure. Skip re-calculation of places on a reused team when inputs are the same. Signed-off-by: Matthew Malcomson <[email protected]> --- libgomp/libgomp.h | 22 ++++ libgomp/team.c | 301 ++++++++++++++++++++++++++++++++-------------- 2 files changed, 231 insertions(+), 92 deletions(-) diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h index 6a5d9a41209..62a81306dcc 100644 --- a/libgomp/libgomp.h +++ b/libgomp/libgomp.h @@ -941,7 +941,28 @@ struct gomp_team #ifdef __AMDGCN__ /* Used on AMD GCN to inform threads how to launch in a team. */ struct gomp_thread_start_data thr_start_data; +#elif LIBGOMP_USE_PTHREADS + /* Cache to help initialise each secondary threads `task` when re-using a + team very many times. This allows each secondary thread to perform the + re-initialisation instead of having the primary thread need to perform + that initialisation. */ + struct thread_init { + struct gomp_task_icv task_icv_init; + struct gomp_task *task_parent_init; + struct gomp_taskgroup *task_taskgroup; + uintptr_t def_allocator; + unsigned int num_teams; + unsigned int team_num; + void (*fn) (void *data); + void *data; + } ti; + struct prev_place_vars { + unsigned int place_partition_off; + unsigned int place_partition_len; + char bind; + } ppv; #endif + bool reused; /* This array contains structures for implicit tasks. */ struct gomp_task implicit_task[]; @@ -955,6 +976,7 @@ struct gomp_thread /* This is the function that the thread should run upon launch. */ void (*fn) (void *data); void *data; + bool exit; #ifdef __AMDGCN__ /* And these are the parameters it should set. */ diff --git a/libgomp/team.c b/libgomp/team.c index c61066481ad..5226be8497e 100644 --- a/libgomp/team.c +++ b/libgomp/team.c @@ -68,18 +68,37 @@ pthread_key_t gomp_tls_key; struct gomp_thread_start_data { - void (*fn) (void *); - void *fn_data; - struct gomp_team_state ts; - struct gomp_task *task; + struct gomp_team *team; struct gomp_thread_pool *thread_pool; unsigned int place; - unsigned int num_teams; - unsigned int team_num; + unsigned team_id; + unsigned active_level; + unsigned place_partition_off; + unsigned place_partition_len; bool nested; pthread_t handle; }; +static inline void +gomp_init_team_state (struct gomp_team_state *ts, struct gomp_team *team, + unsigned team_id, unsigned active_level, + unsigned place_partition_off, + unsigned place_partition_len) +{ + ts->team = team; + ts->work_share = &team->work_shares[0]; + ts->last_work_share = NULL; + ts->team_id = team_id; + ts->level = team->prev_ts.level + 1; + ts->active_level = active_level; + ts->place_partition_off = place_partition_off; + ts->place_partition_len = place_partition_len; + ts->def_allocator = team->ti.def_allocator; +#ifdef HAVE_SYNC_BUILTINS + ts->single_count = 0; +#endif + ts->static_trip = 0; +} /* This function is a pthread_create entry point. This contains the idle loop in which a thread waits to be called up to become part of a team. */ @@ -101,14 +120,17 @@ gomp_thread_start (void *xdata) gomp_sem_init (&thr->release, 0); /* Extract what we need from data. */ - local_fn = data->fn; - local_data = data->fn_data; + struct gomp_team *team = data->team; + gomp_init_team_state (&thr->ts, team, data->team_id, data->active_level, + data->place_partition_off, data->place_partition_len); + + local_fn = team->ti.fn; + local_data = team->ti.data; + thr->num_teams = team->ti.num_teams; + thr->team_num = team->ti.team_num; thr->thread_pool = data->thread_pool; - thr->ts = data->ts; - thr->task = data->task; thr->place = data->place; - thr->num_teams = data->num_teams; - thr->team_num = data->team_num; + thr->exit = false; #ifdef GOMP_NEEDS_THREAD_HANDLE thr->handle = data->handle; #endif @@ -116,7 +138,7 @@ gomp_thread_start (void *xdata) pthread_setspecific (gomp_tls_key, thr); #endif - thr->ts.team->ordered_release[thr->ts.team_id] = &thr->release; + team->ordered_release[thr->ts.team_id] = &thr->release; /* Make thread pool local. */ pool = thr->thread_pool; @@ -124,6 +146,10 @@ gomp_thread_start (void *xdata) if (data->nested) { struct gomp_team *team = thr->ts.team; + thr->task = &team->implicit_task[thr->ts.team_id]; + gomp_init_task (thr->task, team->ti.task_parent_init, + &team->ti.task_icv_init); + thr->task->taskgroup = team->ti.task_taskgroup; struct gomp_task *task = thr->task; gomp_barrier_wait (&team->barrier, thr->ts.team_id); @@ -138,11 +164,24 @@ gomp_thread_start (void *xdata) pool->threads[thr->ts.team_id] = thr; gomp_simple_barrier_wait (&pool->threads_dock); + + /* Ensure we initialise the task *after* the wait on threads_dock just in + case affinity changes have meant that this new thread has the same + team_id as a thread used to have. In this case we want to ensure the + `gomp_finish_task` below is certain to happen before the + `gomp_init_task` here. */ + thr->task = &team->implicit_task[thr->ts.team_id]; + gomp_init_task (thr->task, team->ti.task_parent_init, + &team->ti.task_icv_init); + thr->task->taskgroup = team->ti.task_taskgroup; + do { struct gomp_team *team = thr->ts.team; + unsigned team_id = thr->ts.team_id; struct gomp_task *task = thr->task; + thr->fn = NULL; local_fn (local_data); gomp_team_barrier_wait_final (&team->barrier, thr->ts.team_id); gomp_finish_task (task); @@ -154,15 +193,52 @@ gomp_thread_start (void *xdata) next iteration we know that it will synchronise with the primary thread before it frees this team because it will synchronise with the primary thread in the next iteration of this loop (in - `gomp_team_barrier_wait_final`). */ - if (!gomp_barrier_can_hold (&team->barrier) || thr->fn == NULL) + `gomp_team_barrier_wait_final`). + Also have to hold here if we adjusted our team_id since the above + `task` location is possibly going to be used by a newly started + thread so we have to synchronise with those. Use boolean + short-circuit to avoid data-race on thr->ts.team in case where the + final barrier can't hold threads. */ + if (!gomp_barrier_can_hold (&team->barrier) || thr->exit + || (thr->ts.team == team && thr->ts.team_id != team_id)) gomp_simple_barrier_wait (&pool->threads_dock); - local_fn = thr->fn; - local_data = thr->data; - thr->fn = NULL; + /* Early exit because if `thr->fn == NULL` then we can't guarantee + the `thr->ts.team` hasn't been freed by the primary thread racing + ahead. Hence we don't want to write to it. */ + if (thr->exit) + break; + + /* Usually `thr->fn` is not used or set. Is used when secondary + threads are "set going" without being assigned to any team. + See `gomp_pause_host` and `gomp_free_thread`. + In this case we don't want to bother reading the team + initialisation things because they've not been set up for us. */ + if (thr->fn) + { + local_fn = thr->fn; + local_data = thr->data; + continue; + } + + team = thr->ts.team; + thr->ts.work_share = &team->work_shares[0]; + thr->ts.last_work_share = NULL; + thr->ts.def_allocator = team->ti.def_allocator; +#ifdef HAVE_SYNC_BUILTINS + thr->ts.single_count = 0; +#endif + thr->ts.static_trip = 0; + local_fn = team->ti.fn; + local_data = team->ti.data; + thr->num_teams = team->ti.num_teams; + thr->team_num = team->ti.team_num; + thr->task = &team->implicit_task[thr->ts.team_id]; + gomp_init_task (thr->task, team->ti.task_parent_init, + &team->ti.task_icv_init); + thr->task->taskgroup = team->ti.task_taskgroup; } - while (local_fn); + while (true); } gomp_sem_destroy (&thr->release); @@ -191,6 +267,7 @@ get_last_team (unsigned nthreads) "prev_barrier not within cached team: %p != %p", pool->prev_barrier, &pool->last_team->barrier); pool->last_team = NULL; + last_team->reused = true; return last_team; } } @@ -249,6 +326,7 @@ gomp_new_team (unsigned nthreads) gomp_mutex_init (&team->task_lock); team->nthreads = nthreads; + team->reused = false; } team->work_share_chunk = 8; @@ -377,12 +455,14 @@ gomp_free_thread (void *arg __attribute__((unused))) #ifdef LIBGOMP_USE_PTHREADS static unsigned gomp_barrier_calc_wait (unsigned num_new_threads, unsigned num_exiting_threads, + unsigned num_remapped_threads, unsigned old_threads_used, unsigned nthreads, unsigned affinity_count, bool can_hold) { if (can_hold) { - unsigned tmp = num_new_threads + num_exiting_threads; + unsigned tmp + = num_new_threads + num_exiting_threads + num_remapped_threads; return tmp ? tmp + 1 : 0; } if (!affinity_count) @@ -404,6 +484,7 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, unsigned i, n, old_threads_used = 0; unsigned simple_barrier_n = 0; unsigned num_new_threads = 0; + unsigned num_remapped_threads = 0; unsigned spawned_new_threads __attribute__ ((unused)) = 0; unsigned num_exiting_threads = 0; pthread_attr_t thread_attr, *attr; @@ -536,6 +617,21 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, else bind = omp_proc_bind_false; + /* Copy entirely rather than copy pointer so that we have an immutable + copy for the secondary threads to take from for the time-frame that + said secondary threads are executing on. Can't copy from + `team->implicit_task[0].icv` because primary thread could have adjusted + its per-task ICV by the time the secondary thread gets around to + initialising things. */ + team->ti.task_icv_init = team->implicit_task[0].icv; + team->ti.task_parent_init = task; + team->ti.task_taskgroup = taskgroup; + team->ti.num_teams = thr->num_teams; + team->ti.team_num = thr->team_num; + team->ti.fn = fn; + team->ti.data = data; + team->ti.def_allocator = thr->ts.def_allocator; + /* We only allow the reuse of idle threads for non-nested PARALLEL regions. This appears to be implied by the semantics of threadprivate variables, but perhaps that's reading too much into @@ -548,6 +644,26 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, team->prev_ts.team_id); old_threads_used = pool->threads_used; + if (team->reused + && team->ppv.place_partition_off == thr->ts.place_partition_off + && team->ppv.place_partition_len == thr->ts.place_partition_len + && team->ppv.bind == bind) + { + gomp_assert ( + nthreads == old_threads_used, + "odd number of threads when re-using team: nthreads = %u " + "old_threads_used = %u\n", + nthreads, old_threads_used); + goto do_release; + } + else + { + team->ppv.place_partition_off = thr->ts.place_partition_off; + team->ppv.place_partition_len = thr->ts.place_partition_len; + team->ppv.bind = bind; + } + + if (nthreads == old_threads_used) { n = nthreads; @@ -555,8 +671,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, num_exiting_threads = 0; simple_barrier_n = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads, - old_threads_used, nthreads, - affinity_count, + num_remapped_threads, old_threads_used, + nthreads, affinity_count, gomp_barrier_can_hold ( pool->prev_barrier)); if (gomp_barrier_can_hold (pool->prev_barrier)) @@ -569,8 +685,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, num_exiting_threads = old_threads_used - nthreads; simple_barrier_n = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads, - old_threads_used, nthreads, - affinity_count, + num_remapped_threads, old_threads_used, + nthreads, affinity_count, gomp_barrier_can_hold ( pool->prev_barrier)); if (gomp_barrier_can_hold (pool->prev_barrier)) @@ -591,8 +707,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, num_exiting_threads = 0; simple_barrier_n = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads, - old_threads_used, nthreads, - affinity_count, + num_remapped_threads, old_threads_used, + nthreads, affinity_count, gomp_barrier_can_hold ( pool->prev_barrier)); /* Increase the barrier threshold to make sure all new @@ -720,6 +836,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, pool->threads[j]->data = affinity_thr[l]; affinity_thr[l] = pool->threads[j]; } + else + pool->threads[j]->exit = true; pool->threads[j] = NULL; } if (nthreads > old_threads_used) @@ -759,33 +877,37 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, else nthr = pool->threads[i]; __atomic_store_n (&nthr->ts.team, team, MEMMODEL_RELEASE); - nthr->ts.work_share = &team->work_shares[0]; - nthr->ts.last_work_share = NULL; + if (team->reused && nthr->ts.team_id != i) + num_remapped_threads++; nthr->ts.team_id = i; - nthr->ts.level = team->prev_ts.level + 1; - nthr->ts.active_level = thr->ts.active_level; + gomp_assert ( + nthr->ts.level == 1 && team->prev_ts.level == 0, + "Level indicators surprising: nthr: %u, team->prev_ts: %u\n", + nthr->ts.level, team->prev_ts.level); + gomp_assert ( + nthr->ts.active_level == 1 && thr->ts.active_level == 1, + "active_level indicators surprising: nthr: %u, thr: %u\n", + nthr->ts.active_level, thr->ts.active_level); nthr->ts.place_partition_off = place_partition_off; nthr->ts.place_partition_len = place_partition_len; - nthr->ts.def_allocator = thr->ts.def_allocator; -#ifdef HAVE_SYNC_BUILTINS - nthr->ts.single_count = 0; -#endif - nthr->ts.static_trip = 0; - nthr->num_teams = thr->num_teams; - nthr->team_num = thr->team_num; - nthr->task = &team->implicit_task[i]; nthr->place = place; - gomp_init_task (nthr->task, task, icv); - team->implicit_task[i].icv.nthreads_var = nthreads_var; - team->implicit_task[i].icv.bind_var = bind_var; - nthr->task->taskgroup = taskgroup; - nthr->fn = fn; - nthr->data = data; team->ordered_release[i] = &nthr->release; } if (__builtin_expect (affinity_thr != NULL, 0)) { + /* Mark each of the threads that are exiting as such. Hence in + `gomp_thread_start` they exit their main loop. */ + for (unsigned l = 0; l < team->prev_ts.place_partition_len; l++) + { + struct gomp_thread *t = affinity_thr[l]; + while (t) + { + struct gomp_thread *next = (struct gomp_thread *) t->data; + t->exit = true; + t = next; + } + } /* If AFFINITY_THR is non-NULL just because we had to permute some threads in the pool, but we've managed to find exactly as many old threads as we'd find @@ -868,18 +990,20 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, num_new_threads = (nthreads + affinity_count) - old_threads_used; num_exiting_threads = affinity_count; - simple_barrier_n - = gomp_barrier_calc_wait (num_new_threads, - num_exiting_threads, - old_threads_used, nthreads, - affinity_count, - gomp_barrier_can_hold ( - pool->prev_barrier)); - gomp_simple_barrier_reinit (&pool->threads_dock, - simple_barrier_n); } } } + else + for (unsigned j = n; j < old_threads_used; j++) + pool->threads[j]->exit = true; + + /* Account for any affinity changes & remapped threads. */ + simple_barrier_n + = gomp_barrier_calc_wait (num_new_threads, num_exiting_threads, + num_remapped_threads, old_threads_used, + nthreads, affinity_count, + gomp_barrier_can_hold (pool->prev_barrier)); + gomp_simple_barrier_reinit (&pool->threads_dock, simple_barrier_n); if (i == nthreads) { @@ -888,15 +1012,18 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, num_new_threads); gomp_assert (gomp_barrier_can_hold (pool->prev_barrier) ? (simple_barrier_n - == (num_exiting_threads ? num_exiting_threads + 1 - : 0)) + == ((num_exiting_threads || num_remapped_threads) + ? (num_exiting_threads + num_remapped_threads + + 1) + : 0)) : simple_barrier_n == old_threads_used, "Have calculated need simple_barrier_n == %u" " starting 0 threads, affinity_count == %u, " "old_threads_used == %u, calculated " - "num_exiting_threads == %u, nthreads == %u", + "num_exiting_threads == %u, nthreads == %u, " + "num_remapped_threads == %u", simple_barrier_n, affinity_count, old_threads_used, - num_exiting_threads, nthreads); + num_exiting_threads, nthreads, num_remapped_threads); goto do_release; } } @@ -937,8 +1064,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, { int err; - start_data->ts.place_partition_off = thr->ts.place_partition_off; - start_data->ts.place_partition_len = thr->ts.place_partition_len; + start_data->place_partition_off = thr->ts.place_partition_off; + start_data->place_partition_len = thr->ts.place_partition_len; start_data->place = 0; if (__builtin_expect (gomp_places_list != NULL, 0)) { @@ -972,11 +1099,11 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, if (p == (team->prev_ts.place_partition_off + team->prev_ts.place_partition_len)) p = team->prev_ts.place_partition_off; - start_data->ts.place_partition_off = p; + start_data->place_partition_off = p; if (p < rest) - start_data->ts.place_partition_len = s + 1; + start_data->place_partition_len = s + 1; else - start_data->ts.place_partition_len = s; + start_data->place_partition_len = s; } else { @@ -993,8 +1120,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, } else ++k; - start_data->ts.place_partition_off = p; - start_data->ts.place_partition_len = 1; + start_data->place_partition_off = p; + start_data->place_partition_len = 1; } break; } @@ -1004,29 +1131,12 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, gomp_init_thread_affinity (attr, p); } - start_data->fn = fn; - start_data->fn_data = data; - start_data->ts.team = team; - start_data->ts.work_share = &team->work_shares[0]; - start_data->ts.last_work_share = NULL; - start_data->ts.team_id = i; - start_data->ts.level = team->prev_ts.level + 1; - start_data->ts.active_level = thr->ts.active_level; - start_data->ts.def_allocator = thr->ts.def_allocator; -#ifdef HAVE_SYNC_BUILTINS - start_data->ts.single_count = 0; -#endif - start_data->ts.static_trip = 0; - start_data->num_teams = thr->num_teams; - start_data->team_num = thr->team_num; - start_data->task = &team->implicit_task[i]; - gomp_init_task (start_data->task, task, icv); - team->implicit_task[i].icv.nthreads_var = nthreads_var; - team->implicit_task[i].icv.bind_var = bind_var; - start_data->task->taskgroup = taskgroup; + start_data->team = team; + start_data->team_id = i; + + start_data->active_level = thr->ts.active_level; start_data->thread_pool = pool; start_data->nested = nested; - attr = gomp_adjust_thread_attr (attr, &thread_attr); err = pthread_create (&start_data->handle, attr, gomp_thread_start, start_data); @@ -1046,7 +1156,8 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, calc_check = (num_new_threads == (simple_barrier_n - 1)); else if (gomp_barrier_can_hold (pool->prev_barrier)) calc_check - = ((num_new_threads + num_exiting_threads) == (simple_barrier_n - 1)); + = ((num_new_threads + num_exiting_threads + num_remapped_threads) + == (simple_barrier_n - 1)); else calc_check = (num_new_threads == (simple_barrier_n - old_threads_used)); gomp_assert (calc_check, @@ -1055,9 +1166,11 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, ", nthreads: %u" ", num_new_threads: %u" ", num_exiting_threads: %u" + ", num_remapped_threads: %u" ", old_threads_used: %u", nested ? "nested" : "top-level", simple_barrier_n, nthreads, - num_new_threads, num_exiting_threads, old_threads_used); + num_new_threads, num_exiting_threads, num_remapped_threads, + old_threads_used); } if (__builtin_expect (attr == &thread_attr, 0)) @@ -1069,7 +1182,7 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, else { gomp_release_held_threads (pool, team, thr->ts.team_id); - if (num_new_threads || num_exiting_threads + if (num_new_threads || num_exiting_threads || num_remapped_threads || !gomp_barrier_can_hold (pool->prev_barrier)) gomp_simple_barrier_wait (&pool->threads_dock); } @@ -1119,16 +1232,20 @@ gomp_team_start (void (*fn) (void *), void *data, unsigned nthreads, if (nested) { start_data -= nthreads - 1; + struct gomp_team_state ts; for (i = 1; i < nthreads; ++i) { + gomp_init_team_state (&ts, team, start_data->team_id, + start_data->active_level, + start_data->place_partition_off, + start_data->place_partition_len); gomp_display_affinity_thread ( #ifdef LIBGOMP_USE_PTHREADS - start_data->handle, + start_data->handle, #else - gomp_thread_self (), + gomp_thread_self (), #endif - &start_data->ts, - start_data->place); + &ts, start_data->place); start_data++; } } -- 2.43.0
