PR #24801 opened by Forgejo_Fairy URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24801 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24801.patch
Fixes #24019. When bitstream-buffer creation fails, `nvenc_alloc_surface()` destroys the corresponding input buffer but leaves its handle stored in the surface. The general close path subsequently visits the entire partially initialized surface array, destroying that stale input handle again and passing NULL handles for uncreated buffers to NVENC. Clear the input handle after its early destruction and call the input/output destruction functions only for valid handles. Verification: - Built the NVENC-enabled `ffmpeg` executable. - Successfully encoded the reporter's 4096x4096 H.264 NVENC command on an RTX 5090 with driver 595.84. - Forced the third bitstream-buffer allocation to report `NV_ENC_ERR_OUT_OF_MEMORY`; cleanup completed without an abort. - The configured FATE test set passed. >From ed92317fa5e924bae9ab7afb2c56d22cfddc2141 Mon Sep 17 00:00:00 2001 From: Michael Niedermayer <[email protected]> Date: Tue, 29 Sep 2026 21:14:23 +0000 Subject: [PATCH] avcodec/nvenc: avoid invalid surface destruction after init error A partially initialized surface array contains null handles, and the input handle for the surface whose bitstream allocation failed has already been destroyed. Avoid passing the null handles to NVENC and clear the input handle after its early destruction so it cannot be destroyed twice during close. Fixes issue #24019. Reported-by: hmaarrfk Assisted-by: Fairy --- libavcodec/nvenc.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c index 5ea094e095..6e7331405f 100644 --- a/libavcodec/nvenc.c +++ b/libavcodec/nvenc.c @@ -1937,8 +1937,10 @@ static av_cold int nvenc_alloc_surface(AVCodecContext *avctx, int idx) nv_status = p_nvenc->nvEncCreateBitstreamBuffer(ctx->nvencoder, &allocOut); if (nv_status != NV_ENC_SUCCESS) { int err = nvenc_print_error(avctx, nv_status, "CreateBitstreamBuffer failed"); - if (!IS_HWACCEL(avctx->pix_fmt)) + if (!IS_HWACCEL(avctx->pix_fmt)) { p_nvenc->nvEncDestroyInputBuffer(ctx->nvencoder, ctx->surfaces[idx].input_surface); + ctx->surfaces[idx].input_surface = NULL; + } av_frame_free(&ctx->surfaces[idx].in_ref); return err; } @@ -2072,10 +2074,11 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx) if (ctx->surfaces) { for (i = 0; i < ctx->nb_surfaces; ++i) { - if (!IS_HWACCEL(avctx->pix_fmt)) + if (!IS_HWACCEL(avctx->pix_fmt) && ctx->surfaces[i].input_surface) p_nvenc->nvEncDestroyInputBuffer(ctx->nvencoder, ctx->surfaces[i].input_surface); av_frame_free(&ctx->surfaces[i].in_ref); - p_nvenc->nvEncDestroyBitstreamBuffer(ctx->nvencoder, ctx->surfaces[i].output_surface); + if (ctx->surfaces[i].output_surface) + p_nvenc->nvEncDestroyBitstreamBuffer(ctx->nvencoder, ctx->surfaces[i].output_surface); } } av_freep(&ctx->surfaces); -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
