Hi Thomas,
Thomas Schwinge wrote:
On 2024-01-23T10:55:16+0100, Tobias Burnus <tbur...@baylibre.com> wrote:
plugin/plugin-nvptx.c: Fix fini_device call when already shutdown [PR113513]
The following issue was found when running libgomp.c/target-52.c with
nvptx offloading when the dg-set-target-env-var was honored.
Curious, I've never seen this failure mode in my several different
configurations. :-|
I think we recently fixed a surprisingly high number of issues that we
didn't see before but were clearly preexisting for quite a while.
(Mostly for AMDGPU but still.)
But I concur that this one is a more tricky one.
For some unknown reasons, while this does not have an effect on the
order of the called plugin functions for initialization, it changes the
order of function calls for shutting down. Namely, when the two environment
variables are set, GOMP_offload_unregister_ver is called now before
gomp_target_fini.
Re "unknown reasons", isn't that indeed explained by the different
'atexit' function/'__attribute__((destructor))' sequencing, due to
different order of 'atexit'/'__attribute__((constructor))' calls?
Maybe or not. First, it does not seem to occur elsewhere but maybe
that's because remote setting of environment variables does not work
with DejaGNU and most code was run such a way. And secondly, I have no
idea how 'atexit' and destructors are implemented internally.
And it seems as if CUDA regards a call to cuModuleUnload
(or unloading the last module?) as indication that the device context should
be destroyed - or, at least, afterwards calling cuCtxGetDevice will return
CUDA_ERROR_DEINITIALIZED.
However, this I don't understand -- but would like to. Are you saying
that for:
--- libgomp/plugin/plugin-nvptx.c
+++ libgomp/plugin/plugin-nvptx.c
@@ -1556,8 +1556,16 @@ GOMP_OFFLOAD_unload_image (int ord, unsigned
version, const void *target_data)
if (image->target_data == target_data)
{
*prev_p = image->next;
- if (CUDA_CALL_NOCHECK (cuModuleUnload, image->module) != CUDA_SUCCESS)
+ CUresult r;
+ r = CUDA_CALL_NOCHECK (cuModuleUnload, image->module);
+ GOMP_PLUGIN_debug (0, "%s: cuModuleUnload: %s\n", __FUNCTION__,
cuda_error (r));
+ if (r != CUDA_SUCCESS)
ret = false;
+ CUdevice dev_;
+ r = CUDA_CALL_NOCHECK (cuCtxGetDevice, &dev_);
+ GOMP_PLUGIN_debug (0, "%s: cuCtxGetDevice: %s\n", __FUNCTION__,
cuda_error (r));
+ GOMP_PLUGIN_debug (0, "%s: dev_=%d, dev->dev=%d\n", __FUNCTION__, dev_,
dev->dev);
+ assert (dev_ == dev->dev);
free (image->fns);
free (image);
break;
..., you're seeing an error for 'libgomp.c/target-52.c' with
'env OMP_TARGET_OFFLOAD=mandatory OMP_DISPLAY_ENV=true'? I get:
GOMP_OFFLOAD_unload_image: cuModuleUnload: no error
GOMP_OFFLOAD_unload_image: cuCtxGetDevice: no error
GOMP_OFFLOAD_unload_image: dev_=0, dev->dev=0
Or, is something else happening in between the 'cuModuleUnload' and your
reportedly failing 'cuCtxGetDevice'?
I cluttered the plugin with "printf" debugging; hence, no other code
is calling *into* the run-time library as far as I can see.
But now I will try it with a vanilla code and your patch applied.
Result for the target-52.c with the env vars set:
DEBUG: GOMP_offload_unregister_ver dev=0; state=1
DEBUG: gomp_unload_image_from_device
DEBUG GOMP_OFFLOAD_unload_image, 0, 196609
GOMP_OFFLOAD_unload_image: cuModuleUnload: no error
GOMP_OFFLOAD_unload_image: cuCtxGetDevice: no error
GOMP_OFFLOAD_unload_image: dev_=0, dev->dev=0
DEBUG: gomp_target_fini; dev=0, state=1
DEBUG 0
DEBUG: nvptx_attach_host_thread_to_device - 0
DEBUG: ERROR nvptx_attach_host_thread_to_device - 0
libgomp: cuCtxGetDevice error: unknown cuda error
Hence: The immediately calling cuCtxGetDevice after
the device unloading does not fail.
But calling it soon late via gomp_target_fini
→ GOMP_OFFLOAD_fini_device → nvptx_attach_host_thread_to_device
does fail.
I have attached my printf patch for reference.
* * *
Re your PR113513 details, I don't see how your failure mode could be
related to (a) the PTX code ('--with-arch=sm_80'), or the GPU hardware
("NVIDIA RTX A1000 6GB") (..., unless the Nvidia Driver is doing "funny"
things, of course...), so could this possibly be due to a recent change
in the CUDA Driver/Nvidia Driver? You say "CUDA Version: 12.3", but
which which Nvidia Driver version? The latest I've now tested are:
Driver Version: 525.147.05 CUDA Version: 12.0
Driver Version: 535.154.05 CUDA Version: 12.2
My laptop has:
NVIDIA-SMI 545.29.06 Driver Version: 545.29.06 CUDA Version:
12.3
I'd like to please defer that one until we understand the actual origin
of the misbehavior.
(I think that patch makes still sense, but first finding out what goes
wrong is fine nonetheless.)
When reading the code, the following was observed in addition:
When gomp_fini_device is called, it invokes goacc_fini_asyncqueues
to ensure that the queue is emptied. It seems to make sense to do
likewise for GOMP_offload_unregister_ver, which this commit does in
addition.
I don't understand why offload image unregistration (a) should trigger
'goacc_fini_asyncqueues', and (b) how that relates to PR113513?
While there no direct relation and none to the testcase, this is
affected by the ordering of GOMP_offload_unregister_ver vs.before
gomp_target_fini, which is what the main issue is above.
Assume that by some reason GOMP_offload_unregister_ver gets called
before gomp_target_fini. In that case, the asynchronous queues can be
still running when the variables are removed and only when later
gomp_target_fini is called, it will invoke goacc_fini_asyncqueues.
Of course, when gomp_target_fini is called first, it will run
goacc_fini_asyncqueues first – and a later GOMP_offload_unregister_ver
is a no op as the device is already finalized.
Thus, this part of the patch adds a safeguard for something to be a
known issue for a related issue.
If we guarantee that gomp_target_fini is always called first, I suggest
to remove GOMP_offload_unregister_ver for good as that will then be
always unreachable ... (Well, that function itself not but it will not
do any actual work.)
If we don't think so and there might be an ordering issue, I very much
would like to see this safeguard in, which is very inexpensive if no
work remains to be completed.
Tobias
diff --git a/libgomp/plugin/plugin-nvptx.c b/libgomp/plugin/plugin-nvptx.c
index c04c3acd679..7fc7f4a5bbf 100644
--- a/libgomp/plugin/plugin-nvptx.c
+++ b/libgomp/plugin/plugin-nvptx.c
@@ -391,7 +391,7 @@ nvptx_attach_host_thread_to_device (int n)
CUresult r;
struct ptx_device *ptx_dev;
CUcontext thd_ctx;
-
+__builtin_fprintf (stderr, "DEBUG: nvptx_attach_host_thread_to_device - %d\n", n);
r = CUDA_CALL_NOCHECK (cuCtxGetDevice, &dev);
if (r == CUDA_ERROR_NOT_PERMITTED)
{
@@ -400,6 +400,7 @@ nvptx_attach_host_thread_to_device (int n)
}
if (r != CUDA_SUCCESS && r != CUDA_ERROR_INVALID_CONTEXT)
{
+__builtin_fprintf (stderr, "DEBUG: ERROR nvptx_attach_host_thread_to_device - %d\n", n);
GOMP_PLUGIN_error ("cuCtxGetDevice error: %s", cuda_error (r));
return false;
}
@@ -445,9 +446,11 @@ nvptx_open_device (int n)
ptx_dev->dev = dev;
ptx_dev->ctx_shared = false;
+__builtin_fprintf (stderr, "DEBUG: nvptx_open_device - %d\n", n);
r = CUDA_CALL_NOCHECK (cuCtxGetDevice, &ctx_dev);
if (r != CUDA_SUCCESS && r != CUDA_ERROR_INVALID_CONTEXT)
{
+__builtin_fprintf (stderr, "DEBUG: ERROR nvptx_open_device - %d\n", n);
GOMP_PLUGIN_error ("cuCtxGetDevice error: %s", cuda_error (r));
return NULL;
}
@@ -1174,24 +1177,28 @@ nvptx_get_current_cuda_context (void)
const char *
GOMP_OFFLOAD_get_name (void)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_name\n");
return "nvptx";
}
unsigned int
GOMP_OFFLOAD_get_caps (void)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_caps\n");
return GOMP_OFFLOAD_CAP_OPENACC_200 | GOMP_OFFLOAD_CAP_OPENMP_400;
}
int
GOMP_OFFLOAD_get_type (void)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_type\n");
return OFFLOAD_TARGET_TYPE_NVIDIA_PTX;
}
int
GOMP_OFFLOAD_get_num_devices (unsigned int omp_requires_mask)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_get_num_devices %u\n", omp_requires_mask);
int num_devices = nvptx_get_num_devices ();
/* Return -1 if no omp_requires_mask cannot be fulfilled but
devices were present. Unified-shared address: see comment in
@@ -1207,6 +1214,7 @@ GOMP_OFFLOAD_get_num_devices (unsigned int omp_requires_mask)
bool
GOMP_OFFLOAD_init_device (int n)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_init_device %u\n", n);
struct ptx_device *dev;
pthread_mutex_lock (&ptx_dev_lock);
@@ -1248,6 +1256,7 @@ GOMP_OFFLOAD_init_device (int n)
bool
GOMP_OFFLOAD_fini_device (int n)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_fini_device %u\n", n);
pthread_mutex_lock (&ptx_dev_lock);
if (ptx_devices[n] != NULL)
@@ -1278,6 +1287,7 @@ GOMP_OFFLOAD_fini_device (int n)
unsigned
GOMP_OFFLOAD_version (void)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_version\n");
return GOMP_VERSION;
}
@@ -1311,6 +1321,7 @@ GOMP_OFFLOAD_load_image (int ord, unsigned version, const void *target_data,
uint64_t **rev_fn_table,
uint64_t *host_ind_fn_table)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_load_image, %d, %u\n", ord, version);
CUmodule module;
const char *const *var_names;
const struct targ_fn_launch *fn_descs;
@@ -1538,6 +1549,7 @@ GOMP_OFFLOAD_load_image (int ord, unsigned version, const void *target_data,
bool
GOMP_OFFLOAD_unload_image (int ord, unsigned version, const void *target_data)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_unload_image, %d, %u\n", ord, version);
struct ptx_image_data *image, **prev_p;
struct ptx_device *dev = ptx_devices[ord];
@@ -1568,6 +1580,7 @@ GOMP_OFFLOAD_unload_image (int ord, unsigned version, const void *target_data)
void *
GOMP_OFFLOAD_alloc (int ord, size_t size)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_alloc, %d, %lu\n", ord, (long unsigned)size);
if (!nvptx_attach_host_thread_to_device (ord))
return NULL;
@@ -1604,6 +1617,7 @@ GOMP_OFFLOAD_alloc (int ord, size_t size)
bool
GOMP_OFFLOAD_free (int ord, void *ptr)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_free, %d\n", ord);
return (nvptx_attach_host_thread_to_device (ord)
&& nvptx_free (ptr, ptx_devices[ord]));
}
@@ -1615,6 +1629,7 @@ GOMP_OFFLOAD_openacc_exec (void (*fn) (void *),
void **devaddrs,
unsigned *dims, void *targ_mem_desc)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_exec\n");
GOMP_PLUGIN_debug (0, "nvptx %s\n", __FUNCTION__);
CUdeviceptr dp = (CUdeviceptr) devaddrs;
@@ -1637,6 +1652,7 @@ GOMP_OFFLOAD_openacc_async_exec (void (*fn) (void *),
unsigned *dims, void *targ_mem_desc,
struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_exec\n");
GOMP_PLUGIN_debug (0, "nvptx %s\n", __FUNCTION__);
CUdeviceptr dp = (CUdeviceptr) devaddrs;
@@ -1646,6 +1662,7 @@ GOMP_OFFLOAD_openacc_async_exec (void (*fn) (void *),
void *
GOMP_OFFLOAD_openacc_create_thread_data (int ord)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_create_thread_data\n");
struct ptx_device *ptx_dev;
struct nvptx_thread *nvthd
= GOMP_PLUGIN_malloc (sizeof (struct nvptx_thread));
@@ -1670,18 +1687,21 @@ GOMP_OFFLOAD_openacc_create_thread_data (int ord)
void
GOMP_OFFLOAD_openacc_destroy_thread_data (void *data)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_destroy_thread_data\n");
free (data);
}
void *
GOMP_OFFLOAD_openacc_cuda_get_current_device (void)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_get_current_device\n");
return nvptx_get_current_cuda_device ();
}
void *
GOMP_OFFLOAD_openacc_cuda_get_current_context (void)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_get_current_context\n");
return nvptx_get_current_cuda_context ();
}
@@ -1689,6 +1709,7 @@ GOMP_OFFLOAD_openacc_cuda_get_current_context (void)
void *
GOMP_OFFLOAD_openacc_cuda_get_stream (struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_get_stream\n");
return (void *) aq->cuda_stream;
}
@@ -1696,6 +1717,7 @@ GOMP_OFFLOAD_openacc_cuda_get_stream (struct goacc_asyncqueue *aq)
int
GOMP_OFFLOAD_openacc_cuda_set_stream (struct goacc_asyncqueue *aq, void *stream)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_cuda_set_stream\n");
if (aq->cuda_stream)
{
CUDA_CALL_ASSERT (cuStreamSynchronize, aq->cuda_stream);
@@ -1721,6 +1743,7 @@ nvptx_goacc_asyncqueue_construct (unsigned int flags)
struct goacc_asyncqueue *
GOMP_OFFLOAD_openacc_async_construct (int device __attribute__((unused)))
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_construct\n");
return nvptx_goacc_asyncqueue_construct (CU_STREAM_DEFAULT);
}
@@ -1735,12 +1758,14 @@ nvptx_goacc_asyncqueue_destruct (struct goacc_asyncqueue *aq)
bool
GOMP_OFFLOAD_openacc_async_destruct (struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_destruct\n");
return nvptx_goacc_asyncqueue_destruct (aq);
}
int
GOMP_OFFLOAD_openacc_async_test (struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_test\n");
CUresult r = CUDA_CALL_NOCHECK (cuStreamQuery, aq->cuda_stream);
if (r == CUDA_SUCCESS)
return 1;
@@ -1761,6 +1786,7 @@ nvptx_goacc_asyncqueue_synchronize (struct goacc_asyncqueue *aq)
bool
GOMP_OFFLOAD_openacc_async_synchronize (struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_synchronize\n");
return nvptx_goacc_asyncqueue_synchronize (aq);
}
@@ -1768,6 +1794,7 @@ bool
GOMP_OFFLOAD_openacc_async_serialize (struct goacc_asyncqueue *aq1,
struct goacc_asyncqueue *aq2)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_serialize\n");
CUevent e;
CUDA_CALL_ERET (false, cuEventCreate, &e, CU_EVENT_DISABLE_TIMING);
CUDA_CALL_ERET (false, cuEventRecord, e, aq1->cuda_stream);
@@ -1790,6 +1817,7 @@ GOMP_OFFLOAD_openacc_async_queue_callback (struct goacc_asyncqueue *aq,
void (*callback_fn)(void *),
void *userptr)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_queue_callback\n");
struct nvptx_callback *b = GOMP_PLUGIN_malloc (sizeof (*b));
b->fn = callback_fn;
b->ptr = userptr;
@@ -1837,6 +1865,7 @@ cuda_memcpy_sanity_check (const void *h, const void *d, size_t s)
bool
GOMP_OFFLOAD_host2dev (int ord, void *dst, const void *src, size_t n)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_host2dev\n");
if (!nvptx_attach_host_thread_to_device (ord)
|| !cuda_memcpy_sanity_check (src, dst, n))
return false;
@@ -1847,6 +1876,7 @@ GOMP_OFFLOAD_host2dev (int ord, void *dst, const void *src, size_t n)
bool
GOMP_OFFLOAD_dev2host (int ord, void *dst, const void *src, size_t n)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_dev2host\n");
if (!nvptx_attach_host_thread_to_device (ord)
|| !cuda_memcpy_sanity_check (dst, src, n))
return false;
@@ -1857,6 +1887,7 @@ GOMP_OFFLOAD_dev2host (int ord, void *dst, const void *src, size_t n)
bool
GOMP_OFFLOAD_dev2dev (int ord, void *dst, const void *src, size_t n)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_dev2dev\n");
CUDA_CALL (cuMemcpyDtoDAsync, (CUdeviceptr) dst, (CUdeviceptr) src, n, NULL);
return true;
}
@@ -1868,6 +1899,7 @@ GOMP_OFFLOAD_memcpy2d (int dst_ord, int src_ord, size_t dim1_size,
const void *src, size_t src_offset1_size,
size_t src_offset0_len, size_t src_dim1_size)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_memcpy2d\n");
if (!nvptx_attach_host_thread_to_device (src_ord != -1 ? src_ord : dst_ord))
return false;
@@ -1960,6 +1992,7 @@ GOMP_OFFLOAD_memcpy3d (int dst_ord, int src_ord, size_t dim2_size,
size_t src_offset0_len, size_t src_dim2_size,
size_t src_dim1_len)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_memcpy3d\n");
if (!nvptx_attach_host_thread_to_device (src_ord != -1 ? src_ord : dst_ord))
return false;
@@ -2050,6 +2083,7 @@ bool
GOMP_OFFLOAD_openacc_async_host2dev (int ord, void *dst, const void *src,
size_t n, struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_host2dev\n");
if (!nvptx_attach_host_thread_to_device (ord)
|| !cuda_memcpy_sanity_check (src, dst, n))
return false;
@@ -2061,6 +2095,7 @@ bool
GOMP_OFFLOAD_openacc_async_dev2host (int ord, void *dst, const void *src,
size_t n, struct goacc_asyncqueue *aq)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_async_dev2host\n");
if (!nvptx_attach_host_thread_to_device (ord)
|| !cuda_memcpy_sanity_check (dst, src, n))
return false;
@@ -2071,6 +2106,7 @@ GOMP_OFFLOAD_openacc_async_dev2host (int ord, void *dst, const void *src,
union goacc_property_value
GOMP_OFFLOAD_openacc_get_property (int n, enum goacc_property prop)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_openacc_get_property\n");
union goacc_property_value propval = { .val = 0 };
pthread_mutex_lock (&ptx_dev_lock);
@@ -2211,6 +2247,7 @@ nvptx_stacks_acquire (struct ptx_device *ptx_dev, size_t size, int num)
void
GOMP_OFFLOAD_run (int ord, void *tgt_fn, void *tgt_vars, void **args)
{
+__builtin_fprintf (stderr, "DEBUG GOMP_OFFLOAD_run\n");
struct targ_fn_descriptor *tgt_fn_desc
= (struct targ_fn_descriptor *) tgt_fn;
CUfunction function = tgt_fn_desc->fn;
diff --git a/libgomp/target.c b/libgomp/target.c
index 1367e9cce6c..f758b20ba4c 100644
--- a/libgomp/target.c
+++ b/libgomp/target.c
@@ -2524,6 +2524,8 @@ gomp_unload_image_from_device (struct gomp_device_descr *devicep,
node = splay_tree_lookup (&devicep->mem_map, &k);
}
+__builtin_fprintf(stderr, "DEBUG: gomp_unload_image_from_device\n");
+
if (!devicep->unload_image_func (devicep->target_id, version, target_data))
{
gomp_mutex_unlock (&devicep->lock);
@@ -2698,12 +2700,14 @@ GOMP_offload_unregister_ver (unsigned version, const void *host_table,
target_data = data;
gomp_mutex_lock (®ister_lock);
+__builtin_fprintf(stderr, "DEBUG: GOMP_offload_unregister_ver\n");
/* Unload image from all initialized devices. */
for (i = 0; i < num_devices; i++)
{
struct gomp_device_descr *devicep = &devices[i];
gomp_mutex_lock (&devicep->lock);
+__builtin_fprintf(stderr, "DEBUG: GOMP_offload_unregister_ver dev=%d; state=%d\n", i, devicep->state);
if (devicep->type == target_type
&& devicep->state == GOMP_DEVICE_INITIALIZED)
gomp_unload_image_from_device (devicep, version,
@@ -2775,6 +2779,7 @@ gomp_fini_device (struct gomp_device_descr *devicep)
attribute_hidden void
gomp_unload_device (struct gomp_device_descr *devicep)
{
+__builtin_fprintf(stderr, "DEBUG: gomp_unload_device; state=%d\n", devicep->state);
if (devicep->state == GOMP_DEVICE_INITIALIZED)
{
unsigned i;
@@ -5217,6 +5222,7 @@ gomp_target_fini (void)
bool ret = true;
struct gomp_device_descr *devicep = &devices[i];
gomp_mutex_lock (&devicep->lock);
+__builtin_fprintf(stderr, "DEBUG: gomp_target_fini; dev=%d, state=%d\n", i, devicep->state);
if (devicep->state == GOMP_DEVICE_INITIALIZED)
ret = gomp_fini_device (devicep);
gomp_mutex_unlock (&devicep->lock);
