PR #25053 opened by Kilian (lusoris) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25053 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25053.patch
Fixes #25052. `vulkan_export_to_cuda()` kept the CUDA import of a Vulkan frame for the first CUDA context it met and reused it for every later context. When a frame meets two CUDA contexts, for example `hwupload_cuda` or NVDEC on one CUDA device and `hwupload=derive_device=cuda` deriving a second one from an independent Vulkan device, the second context received foreign handles and `cuWaitExternalSemaphoresAsync()` failed with `CUDA_ERROR_INVALID_VALUE`. The patch compares the CUDA context of the stored import with the one of the current export. When they differ, it waits for the stream of the old context, releases the old import and imports the frame again. The release code moves out of `vulkan_free_internal()` into `vulkan_free_cuda_import()`, which also clears the handles so that the import can be made again. ## Testing RTX 4090, driver 615.78.08, Linux; master `ebfc9497f7` built twice from the same tree, without and with this commit (`--enable-ffnvcodec --enable-nvenc --enable-vulkan`). Each case runs 120 frames. | Case | Without the patch | With the patch | | --- | --- | --- | | `hwupload_cuda`, independent Vulkan device, `hwupload=derive_device=cuda`, download (reproducer in the issue) | exit 187, 0 frames | 120 frames | | NVDEC, independent Vulkan device, `hwupload=derive_device=cuda`, download | exit 187, 0 frames | 120 frames | | same, encoded with `hevc_nvenc` | exit 187, 0 frames | 120 frames | | NVDEC, `gblur_vulkan,gblur_vulkan`, `hwupload=derive_device=cuda`, download | exit 187 after 2 frames | 120 frames | | controls: one CUDA context (Vulkan derived from CUDA, with `gblur_vulkan` once or twice, downloaded or encoded with `hevc_nvenc`); system memory to Vulkan to CUDA; CUDA or NVDEC to Vulkan, then download only; NVDEC copied to system memory | 120 frames | 120 frames | Output content, compared with `-f framemd5`: - the `hwupload_cuda` reproducer matches `format=nv12` in system memory; - the NVDEC case matches NVDEC with `hwdownload`; - the `gblur_vulkan` case matches the same filters on a Vulkan device derived from the CUDA device. A frame that alternates between two contexts is imported again on each switch. With 1200 frames at 1280x720, the reproducer took 3.0 to 5.0 s and the same hops in one CUDA context took 2.4 to 3.3 s (three runs each, on a host with load average about 50). At that load the extra cost cannot be separated from the run-to-run spread. Deriving the Vulkan device from the CUDA device avoids the re-import. `make fate-source` and `make fate-hwcontext-cuda` pass. No FATE test covers CUDA and Vulkan interop; `hwcontext-cuda` does not use Vulkan. >From ba3b134ffc0afa469ab4513f5aa948e937051b35 Mon Sep 17 00:00:00 2001 From: lusoris <[email protected]> Date: Wed, 14 Oct 2026 00:22:00 +0200 Subject: [PATCH] hwcontext_vulkan: import a frame again when the CUDA context changes vulkan_export_to_cuda() imports the memory and semaphores of a Vulkan frame into the current CUDA context on first use and keeps the handles on the frame. A later export to a different CUDA context reused those handles, which belong to the first context, and cuWaitExternalSemaphoresAsync() failed with CUDA_ERROR_INVALID_VALUE. Two CUDA contexts meet a frame when CUDA frames (hwupload_cuda, or NVDEC on its own device) are uploaded to an independent Vulkan device and then passed on with hwupload=derive_device=cuda, which derives a second CUDA context from the Vulkan device: ffmpeg -init_hw_device vulkan=vk:0,disable_multiplane=1 \ -filter_hw_device vk -f lavfi -i testsrc2=s=1280x720:r=30:d=4 \ -vf format=nv12,hwupload_cuda=device=0,hwupload,hwupload=derive_device=cuda,hwdownload,format=nv12 \ -f null - Compare the CUDA context of the stored import with the one of the current export. When they differ, wait for the stream of the old context, release the old import and import the frame again. A frame that alternates between two contexts is imported again on each switch; deriving the Vulkan device from the CUDA device keeps one context and avoids that cost. Signed-off-by: lusoris <[email protected]> --- libavutil/hwcontext_vulkan.c | 97 ++++++++++++++++++++++++++---------- 1 file changed, 72 insertions(+), 25 deletions(-) diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c index 63d2ab5c43..94b7b4a28d 100644 --- a/libavutil/hwcontext_vulkan.c +++ b/libavutil/hwcontext_vulkan.c @@ -2403,6 +2403,49 @@ static int alloc_mem(AVHWDeviceContext *ctx, VkMemoryRequirements *req, return 0; } +#if CONFIG_CUDA +static void vulkan_free_cuda_import(AVVkFrameInternal *internal) +{ + AVHWFramesContext *cuda_fc; + AVHWDeviceContext *cuda_cu; + AVCUDADeviceContext *cuda_dev; + CudaFunctions *cu; + int planes; + + if (!internal->cuda_fc_ref) + return; + + cuda_fc = (AVHWFramesContext *)internal->cuda_fc_ref->data; + planes = av_pix_fmt_count_planes(cuda_fc->sw_format); + cuda_cu = cuda_fc->device_ctx; + cuda_dev = cuda_cu->hwctx; + cu = cuda_dev->internal->cuda_dl; + + for (int i = 0; i < planes; i++) { + if (internal->cu_sem[i]) + CHECK_CU(cu->cuDestroyExternalSemaphore(internal->cu_sem[i])); + if (internal->cu_mma[i]) + CHECK_CU(cu->cuMipmappedArrayDestroy(internal->cu_mma[i])); + if (internal->ext_mem[i]) + CHECK_CU(cu->cuDestroyExternalMemory(internal->ext_mem[i])); + internal->cu_sem[i] = NULL; + internal->cu_mma[i] = NULL; + internal->cu_array[i] = NULL; + internal->ext_mem[i] = NULL; +#ifdef _WIN32 + if (internal->ext_sem_handle[i]) + CloseHandle(internal->ext_sem_handle[i]); + if (internal->ext_mem_handle[i]) + CloseHandle(internal->ext_mem_handle[i]); + internal->ext_sem_handle[i] = NULL; + internal->ext_mem_handle[i] = NULL; +#endif + } + + av_buffer_unref(&internal->cuda_fc_ref); +} +#endif + static void vulkan_free_internal(VulkanDevicePriv *p, AVVkFrame *f) { av_unused AVVkFrameInternal *internal = f->internal; @@ -2412,31 +2455,7 @@ static void vulkan_free_internal(VulkanDevicePriv *p, AVVkFrame *f) return; #if CONFIG_CUDA - if (internal->cuda_fc_ref) { - AVHWFramesContext *cuda_fc = (AVHWFramesContext *)internal->cuda_fc_ref->data; - int planes = av_pix_fmt_count_planes(cuda_fc->sw_format); - AVHWDeviceContext *cuda_cu = cuda_fc->device_ctx; - AVCUDADeviceContext *cuda_dev = cuda_cu->hwctx; - AVCUDADeviceContextInternal *cu_internal = cuda_dev->internal; - CudaFunctions *cu = cu_internal->cuda_dl; - - for (int i = 0; i < planes; i++) { - if (internal->cu_sem[i]) - CHECK_CU(cu->cuDestroyExternalSemaphore(internal->cu_sem[i])); - if (internal->cu_mma[i]) - CHECK_CU(cu->cuMipmappedArrayDestroy(internal->cu_mma[i])); - if (internal->ext_mem[i]) - CHECK_CU(cu->cuDestroyExternalMemory(internal->ext_mem[i])); -#ifdef _WIN32 - if (internal->ext_sem_handle[i]) - CloseHandle(internal->ext_sem_handle[i]); - if (internal->ext_mem_handle[i]) - CloseHandle(internal->ext_mem_handle[i]); -#endif - } - - av_buffer_unref(&internal->cuda_fc_ref); - } + vulkan_free_cuda_import(internal); #endif if (internal->drm_sync_sem != VK_NULL_HANDLE) @@ -4184,6 +4203,34 @@ static int vulkan_export_to_cuda(AVHWFramesContext *hwfc, dst_f = (AVVkFrame *)frame->data[0]; dst_int = dst_f->internal; + if (dst_int->cuda_fc_ref) { + AVHWFramesContext *old_fc = (AVHWFramesContext *)dst_int->cuda_fc_ref->data; + AVHWDeviceContext *old_cu = old_fc->device_ctx; + AVCUDADeviceContext *old_dev = old_cu->hwctx; + + /* The memory and semaphores were imported into another CUDA context, + * and their handles are not valid in this one. Wait for the work + * queued in that context, release the import and import again. */ + if (old_dev->cuda_ctx != cuda_dev->cuda_ctx) { + CudaFunctions *old_dl = old_dev->internal->cuda_dl; + CUcontext dummy; + + err = FF_CUDA_CHECK_DL(old_cu, old_dl, + old_dl->cuCtxPushCurrent(old_dev->cuda_ctx)); + if (err < 0) + return err; + + err = FF_CUDA_CHECK_DL(old_cu, old_dl, + old_dl->cuStreamSynchronize(old_dev->stream)); + if (err >= 0) + vulkan_free_cuda_import(dst_int); + + FF_CUDA_CHECK_DL(old_cu, old_dl, old_dl->cuCtxPopCurrent(&dummy)); + if (err < 0) + return err; + } + } + if (!dst_int->cuda_fc_ref) { size_t offsets[3] = { 0 }; -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
