PR #25615 opened by Kilian (lusoris) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25615 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25615.patch
A failed CUDA call during a Vulkan frame transfer currently destroys state the caller still owns, so the next transfer on the same frame crashes. With this series the error is returned and both frames stay usable. **The defect** `vulkan_transfer_data_to_cuda` and `vulkan_transfer_data_from_cuda` handle a failed `cuMemcpy2DAsync`, `cuWaitExternalSemaphoresAsync` or `cuSignalExternalSemaphoresAsync` by calling `vulkan_free_internal` and unreferencing `dst->buf[0]`. The Vulkan frame stays in its pool, so the next use locks the freed `update_mutex` in `ff_vk_exec_add_dep_frame`. A failed import in `vulkan_export_to_cuda` takes the same path. **The fix** - The transfer failure path waits for queued commands, pops the CUDA context, returns the image from the external queue and returns the error. The destination's buffer reference and the frame's internals stay in place. - A failed import releases only the partial CUDA import and clears its handles, so the next transfer imports again. **Testing** CUDA failures cannot be provoked from the API, and FATE has no CUDA test, so I used an out-of-tree harness that overrides the CUDA calls with dlsym interposition. It runs one failing call per case, then retries the transfer on the same frame and compares pixels. | Injected failure | Directions | Before | After | | :--- | :--- | :--- | :--- | | none | to and from Vulkan | ok | ok | | memcpy, 2D memcpy, semaphore wait, semaphore signal | to and from Vulkan | SIGSEGV on the retry | ok, pixels match | | semaphore import, memory map, level array | to and from Vulkan | SIGSEGV on the retry | ok, pixels match | - Hardware: NVIDIA GeForce RTX 4090, 615.78.08. - `make fate-libavutil`: 73 tests, none failed. - Vulkan validation layer loaded on the 12 injected cases it was run on: no VUID lines. I did not prove that the layer would have reported a planted violation, so read this as no reported errors rather than as a validation proof. The harness is not part of this series; I can share it if that helps review. Both this series and PR 25053 change `vulkan_export_to_cuda`, and both split the CUDA teardown out of `vulkan_free_internal` into a helper. Each applies to current master on its own, but `git merge-tree` of the two heads reports content conflicts in `libavutil/hwcontext_vulkan.c`, so whichever lands second needs a rebase. >From 282d6802141e801e906163a1ff9f2c3eece5913d Mon Sep 17 00:00:00 2001 From: lusoris <[email protected]> Date: Fri, 30 Oct 2026 14:37:41 +0100 Subject: [PATCH 1/2] avutil/hwcontext_vulkan: keep frames after failed CUDA transfers When cuWaitExternalSemaphoresAsync, cuMemcpy2DAsync or cuSignalExternalSemaphoresAsync failed, the transfer freed the internal state of the Vulkan frame and unreferenced the destination's first buffer. The caller still owns the destination, and the Vulkan frame stays in its pool, so the next transfer on that frame crashed locking the freed update_mutex in ff_vk_exec_add_dep_frame(). Return the error and leave both frames alone. Wait for commands queued before the failure and take the image back from the external queue, as a successful transfer does. Signed-off-by: lusoris <[email protected]> --- libavutil/hwcontext_vulkan.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 0ddfafb1b8e..43baef0cefa 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -4286,7 +4286,6 @@ static int vulkan_transfer_data_from_cuda(AVHWFramesContext *hwfc, CUcontext dummy; AVVkFrame *dst_f; AVVkFrameInternal *dst_int; - VulkanDevicePriv *p = hwfc->device_ctx->hwctx; VulkanFramesPriv *fp = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(hwfc->sw_format); @@ -4366,9 +4365,11 @@ static int vulkan_transfer_data_from_cuda(AVHWFramesContext *hwfc, return err = prepare_frame(hwfc, &fp->upload_exec, dst_f, PREP_MODE_EXTERNAL_IMPORT); fail: + /* Commands queued before the failure may still use the image, and the + * frame stays usable, so return it the way a successful transfer does. */ + CHECK_CU(cu->cuStreamSynchronize(cuda_dev->stream)); CHECK_CU(cu->cuCtxPopCurrent(&dummy)); - vulkan_free_internal(p, dst_f); - av_buffer_unref(&dst->buf[0]); + prepare_frame(hwfc, &fp->upload_exec, dst_f, PREP_MODE_EXTERNAL_IMPORT); return err; } #endif @@ -5214,7 +5215,6 @@ static int vulkan_transfer_data_to_cuda(AVHWFramesContext *hwfc, AVFrame *dst, CUcontext dummy; AVVkFrame *dst_f; AVVkFrameInternal *dst_int; - VulkanDevicePriv *p = hwfc->device_ctx->hwctx; VulkanFramesPriv *fp = hwfc->hwctx; const int planes = av_pix_fmt_count_planes(hwfc->sw_format); const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(hwfc->sw_format); @@ -5294,9 +5294,11 @@ static int vulkan_transfer_data_to_cuda(AVHWFramesContext *hwfc, AVFrame *dst, return prepare_frame(hwfc, &fp->upload_exec, dst_f, PREP_MODE_EXTERNAL_IMPORT); fail: + /* Commands queued before the failure may still use the image, and the + * frame stays usable, so return it the way a successful transfer does. */ + CHECK_CU(cu->cuStreamSynchronize(cuda_dev->stream)); CHECK_CU(cu->cuCtxPopCurrent(&dummy)); - vulkan_free_internal(p, dst_f); - av_buffer_unref(&dst->buf[0]); + prepare_frame(hwfc, &fp->upload_exec, dst_f, PREP_MODE_EXTERNAL_IMPORT); return err; } #endif -- 2.54.0 >From 6bbdd8752d145c620d0265de53bb9a06b970e56d Mon Sep 17 00:00:00 2001 From: lusoris <[email protected]> Date: Fri, 30 Oct 2026 14:38:31 +0100 Subject: [PATCH 2/2] avutil/hwcontext_vulkan: keep frame internals after failed CUDA imports A failure while importing a frame's memory or semaphores into CUDA freed all of its internal state. The frame stays in its pool, so the next transfer on it crashed locking the freed update_mutex, and every other use of the frame would have done the same. Release only the partial CUDA import, clear the handles so the next transfer imports again, and take the image back from the external queue. Signed-off-by: lusoris <[email protected]> --- libavutil/hwcontext_vulkan.c | 87 ++++++++++++++++++++++-------------- 1 file changed, 53 insertions(+), 34 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 43baef0cefa..023b11224ea 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -2403,6 +2403,51 @@ static int alloc_mem(AVHWDeviceContext *ctx, VkMemoryRequirements *req, return 0; } +#if CONFIG_CUDA +static void vulkan_free_internal_cuda(AVVkFrameInternal *internal) +{ + AVHWFramesContext *cuda_fc; + AVHWDeviceContext *cuda_cu; + AVCUDADeviceContext *cuda_dev; + AVCUDADeviceContextInternal *cu_internal; + CudaFunctions *cu; + int planes; + + if (!internal->cuda_fc_ref) + return; + + cuda_fc = (AVHWFramesContext *)internal->cuda_fc_ref->data; + planes = av_pix_fmt_count_planes(cuda_fc->sw_format); + cuda_cu = cuda_fc->device_ctx; + cuda_dev = cuda_cu->hwctx; + cu_internal = cuda_dev->internal; + cu = cu_internal->cuda_dl; + + for (int i = 0; i < planes; i++) { + if (internal->cu_sem[i]) + CHECK_CU(cu->cuDestroyExternalSemaphore(internal->cu_sem[i])); + if (internal->cu_mma[i]) + CHECK_CU(cu->cuMipmappedArrayDestroy(internal->cu_mma[i])); + if (internal->ext_mem[i]) + CHECK_CU(cu->cuDestroyExternalMemory(internal->ext_mem[i])); + internal->cu_sem[i] = NULL; + internal->cu_mma[i] = NULL; + internal->cu_array[i] = NULL; + internal->ext_mem[i] = NULL; +#ifdef _WIN32 + if (internal->ext_sem_handle[i]) + CloseHandle(internal->ext_sem_handle[i]); + if (internal->ext_mem_handle[i]) + CloseHandle(internal->ext_mem_handle[i]); + internal->ext_sem_handle[i] = NULL; + internal->ext_mem_handle[i] = NULL; +#endif + } + + av_buffer_unref(&internal->cuda_fc_ref); +} +#endif + static void vulkan_free_internal(VulkanDevicePriv *p, AVVkFrame *f) { av_unused AVVkFrameInternal *internal = f->internal; @@ -2412,31 +2457,7 @@ static void vulkan_free_internal(VulkanDevicePriv *p, AVVkFrame *f) return; #if CONFIG_CUDA - if (internal->cuda_fc_ref) { - AVHWFramesContext *cuda_fc = (AVHWFramesContext *)internal->cuda_fc_ref->data; - int planes = av_pix_fmt_count_planes(cuda_fc->sw_format); - AVHWDeviceContext *cuda_cu = cuda_fc->device_ctx; - AVCUDADeviceContext *cuda_dev = cuda_cu->hwctx; - AVCUDADeviceContextInternal *cu_internal = cuda_dev->internal; - CudaFunctions *cu = cu_internal->cuda_dl; - - for (int i = 0; i < planes; i++) { - if (internal->cu_sem[i]) - CHECK_CU(cu->cuDestroyExternalSemaphore(internal->cu_sem[i])); - if (internal->cu_mma[i]) - CHECK_CU(cu->cuMipmappedArrayDestroy(internal->cu_mma[i])); - if (internal->ext_mem[i]) - CHECK_CU(cu->cuDestroyExternalMemory(internal->ext_mem[i])); -#ifdef _WIN32 - if (internal->ext_sem_handle[i]) - CloseHandle(internal->ext_sem_handle[i]); - if (internal->ext_mem_handle[i]) - CloseHandle(internal->ext_mem_handle[i]); -#endif - } - - av_buffer_unref(&internal->cuda_fc_ref); - } + vulkan_free_internal_cuda(internal); #endif if (internal->drm_sync_sem != VK_NULL_HANDLE) @@ -4275,7 +4296,9 @@ static int vulkan_export_to_cuda(AVHWFramesContext *hwfc, return 0; fail: - vulkan_free_internal(p, dst_f); + /* Only the import failed: the frame is still valid and stays in its pool, + * and the next transfer starts the import over. */ + vulkan_free_internal_cuda(dst_int); return err; } @@ -4311,10 +4334,8 @@ static int vulkan_transfer_data_from_cuda(AVHWFramesContext *hwfc, return err; err = vulkan_export_to_cuda(hwfc, src->hw_frames_ctx, dst); - if (err < 0) { - CHECK_CU(cu->cuCtxPopCurrent(&dummy)); - return err; - } + if (err < 0) + goto fail; dst_int = dst_f->internal; @@ -5240,10 +5261,8 @@ static int vulkan_transfer_data_to_cuda(AVHWFramesContext *hwfc, AVFrame *dst, return err; err = vulkan_export_to_cuda(hwfc, dst->hw_frames_ctx, src); - if (err < 0) { - CHECK_CU(cu->cuCtxPopCurrent(&dummy)); - return err; - } + if (err < 0) + goto fail; dst_int = dst_f->internal; -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
