PR #24484 opened by Zhao Zhili (quink)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24484
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24484.patch
## avcodec/videotoolboxenc: fix use-after-free of strings
av_map_videotoolbox_color_*_from_av() returns borrowed references:
CoreVideo constants, or strings owned by CoreVideo's per-process
codepoint cache. vtenc_reset() released the stored strings, which for
codepoints without a constant name dropped the last reference and
freed the cached instance. CoreVideo kept handing the pointer out, so
the next encoder open or VT session restart in the same process passed
a dangling string to VTSessionSetProperty() and crashed.
```
Crash can be reproduced with:
ffmpeg -f lavfi -i testsrc2=d=2,setparams=color_trc=vlog \
-c:v h264_videotoolbox -y a.mp4 \
-c:v h264_videotoolbox -y b.mp4
```
>From 357956bbb3eb14d17479d0d9e1515d6ece36dedc Mon Sep 17 00:00:00 2001
From: Zhao Zhili <[email protected]>
Date: Mon, 14 Sep 2026 20:50:57 +0800
Subject: [PATCH 1/2] avcodec/videotoolboxenc: fix use-after-free of strings
av_map_videotoolbox_color_*_from_av() returns borrowed references:
CoreVideo constants, or strings owned by CoreVideo's per-process
codepoint cache. vtenc_reset() released the stored strings, which for
codepoints without a constant name dropped the last reference and
freed the cached instance. CoreVideo kept handing the pointer out, so
the next encoder open or VT session restart in the same process passed
a dangling string to VTSessionSetProperty() and crashed.
Crash can be reproduced with:
ffmpeg -f lavfi -i testsrc2=d=2,setparams=color_trc=vlog \
-c:v h264_videotoolbox -y a.mp4 \
-c:v h264_videotoolbox -y b.mp4
---
libavcodec/videotoolboxenc.c | 22 ++++++++--------------
1 file changed, 8 insertions(+), 14 deletions(-)
diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index 536565a252..5fdd7561b0 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -395,20 +395,14 @@ static void vtenc_reset(VTEncContext *vtctx)
vtctx->supported_props = NULL;
}
- if (vtctx->color_primaries) {
- CFRelease(vtctx->color_primaries);
- vtctx->color_primaries = NULL;
- }
-
- if (vtctx->transfer_function) {
- CFRelease(vtctx->transfer_function);
- vtctx->transfer_function = NULL;
- }
-
- if (vtctx->ycbcr_matrix) {
- CFRelease(vtctx->ycbcr_matrix);
- vtctx->ycbcr_matrix = NULL;
- }
+ /* The colorimetry fields hold references borrowed from CoreVideo (Get
+ * semantics). Releasing them would free CoreVideo's cached string for
+ * codepoints without a constant name, and later lookups of the same
+ * codepoint would hand out a dangling pointer.
+ */
+ vtctx->color_primaries = NULL;
+ vtctx->transfer_function = NULL;
+ vtctx->ycbcr_matrix = NULL;
}
static int vtenc_q_pop(VTEncContext *vtctx, bool wait, CMSampleBufferRef *buf,
ExtraSEI *sei)
--
2.52.0
>From c8d7b840dd716af793fc71e91b7a15e2f3cad19b Mon Sep 17 00:00:00 2001
From: Zhao Zhili <[email protected]>
Date: Mon, 14 Sep 2026 23:00:21 +0800
Subject: [PATCH 2/2] avutil/hwcontext_videotoolbox: document mapping string
ownership
The color mapping functions return CFStringRefs borrowed from CoreVideo,
but the header did not say so. Callers that assumed Create semantics and
released the result freed CoreVideo's internal codepoint cache entries,
crashing later users of the same value.
Also return NULL, not 0, in the docs when no equivalent is found.
---
libavutil/hwcontext_videotoolbox.h | 29 +++++++++++++++++++++++++----
1 file changed, 25 insertions(+), 4 deletions(-)
diff --git a/libavutil/hwcontext_videotoolbox.h
b/libavutil/hwcontext_videotoolbox.h
index d35cfbb6c1..b12880b159 100644
--- a/libavutil/hwcontext_videotoolbox.h
+++ b/libavutil/hwcontext_videotoolbox.h
@@ -66,26 +66,47 @@ uint32_t av_map_videotoolbox_format_from_pixfmt2(enum
AVPixelFormat pix_fmt, boo
/**
* Convert an AVChromaLocation to a VideoToolbox/CoreVideo chroma location
string.
- * Returns 0 if no known equivalent was found.
+ *
+ * The returned string is a CoreVideo constant and must not be released.
+ *
+ * Returns NULL if no known equivalent was found.
*/
CFStringRef av_map_videotoolbox_chroma_loc_from_av(enum AVChromaLocation loc);
/**
* Convert an AVColorSpace to a VideoToolbox/CoreVideo color matrix string.
- * Returns 0 if no known equivalent was found.
+ *
+ * The returned string is borrowed from CoreVideo and must not be released.
+ * It is either a constant, or an entry of CoreVideo's internal codepoint
+ * cache; releasing a cached entry makes later lookups of the same value
+ * return a dangling pointer.
+ *
+ * Returns NULL if no known equivalent was found.
*/
CFStringRef av_map_videotoolbox_color_matrix_from_av(enum AVColorSpace space);
/**
* Convert an AVColorPrimaries to a VideoToolbox/CoreVideo color primaries
string.
- * Returns 0 if no known equivalent was found.
+ *
+ * The returned string is borrowed from CoreVideo and must not be released.
+ * It is either a constant, or an entry of CoreVideo's internal codepoint
+ * cache; releasing a cached entry makes later lookups of the same value
+ * return a dangling pointer.
+ *
+ * Returns NULL if no known equivalent was found.
*/
CFStringRef av_map_videotoolbox_color_primaries_from_av(enum AVColorPrimaries
pri);
/**
* Convert an AVColorTransferCharacteristic to a VideoToolbox/CoreVideo color
transfer
* function string.
- * Returns 0 if no known equivalent was found.
+ *
+ * The returned string is borrowed from CoreVideo and must not be released.
+ * It is either a constant, or an entry of CoreVideo's internal codepoint
+ * cache; releasing a cached entry makes later lookups of the same value
+ * return a dangling pointer.
+ *
+ * Returns NULL if no known equivalent was found.
*/
CFStringRef av_map_videotoolbox_color_trc_from_av(enum
AVColorTransferCharacteristic trc);
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
