#11617: heap-use-after-free in libavcodec/libaomenc.c found by AddressSanitizer
---------------------------------+--------------------------------------
Reporter: Ayose | Type: defect
Status: new | Priority: normal
Component: avcodec | Version: git-master
Keywords: | Blocked By:
Blocking: | Reproduced by developer: 0
Analyzed by developer: 0 |
---------------------------------+--------------------------------------
''AddressSanitizer'' found a `heap-use-after-free` in
`libavcodec/libaomenc.c`.
The problem is triggered with a command like the following one, but only
if FFmpeg is compiled with ASAN (`./configure --toolchain=clang-asan`):
{{{
$ ffmpeg -f lavfi -i testsrc -to 1 -c:v libaom-av1 /tmp/0.webm
}}}
If FFmpeg is executed without ASAN, the previous command does not fail.
=== Steps to Reproduce
This test is done against ab37c7e49ff98a8c32c90e6df49dd11145bc64bb, clang
19, and libaom 3.12:
{{{
$ clang --version
Debian clang version 19.1.7 (3)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-19/bin
$ pkg-config --modversion aom
3.12.1
$ git log --format='%h %cs' -1
ab37c7e49f 2025-05-30
}}}
FFmpeg is built with `--toolchain=clang-asan`:
{{{
$ git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
$ cd ffmpeg
$ ./configure \
--toolchain=clang-asan \
--enable-debug \
--enable-libaom
$ make -j16
}}}
Then, a `heap-use-after-free` is detected in
`libavcodec/libaomenc.c:1028:28`:
{{{
$ ./ffmpeg_g -y -v warning -hide_banner -f lavfi -i testsrc -to 1 -c:v
libaom-av1 /tmp/0.webm
[libaom-av1 @ 0x519000002380] Neither bitrate nor constrained quality
specified, using default CRF of 32
=================================================================
==19780==ERROR: AddressSanitizer: heap-use-after-free on address
0x509000007818 at pc 0x564397e7f93a bp 0x7f27e79ad430 sp 0x7f27e79ad428
WRITE of size 8 at 0x509000007818 thread T10 (vf#0:0)
#0 0x564397e7f939 in aom_init
/tmp/ffmpeg/libavcodec/libaomenc.c:1028:28
#1 0x564398f08f62 in avcodec_open2
/tmp/ffmpeg/libavcodec/avcodec.c:336:19
#2 0x564398070117 in enc_open /tmp/ffmpeg/fftools/ffmpeg_enc.c:337:16
#3 0x5643980b7938 in enc_open
/tmp/ffmpeg/fftools/ffmpeg_sched.c:1686:11
#4 0x5643980b7938 in send_to_enc
/tmp/ffmpeg/fftools/ffmpeg_sched.c:1806:19
#5 0x564398086538 in fg_output_frame
/tmp/ffmpeg/fftools/ffmpeg_filter.c:2359:15
#6 0x564398084fba in fg_output_step
/tmp/ffmpeg/fftools/ffmpeg_filter.c:2460:11
#7 0x564398084fba in read_frames
/tmp/ffmpeg/fftools/ffmpeg_filter.c:2521:23
#8 0x56439807916e in filter_thread
/tmp/ffmpeg/fftools/ffmpeg_filter.c:2962:15
#9 0x5643980b85a9 in task_wrapper
/tmp/ffmpeg/fftools/ffmpeg_sched.c:2534:11
#10 0x564398013b3a in asan_thread_start(void*) asan_interceptors.cpp.o
#11 0x7f27f5743b7a (/lib/x86_64-linux-gnu/libc.so.6+0x92b7a)
(BuildId: b46a78e7229ed6fe08549e2bc7ca64155cc5cf1e)
#12 0x7f27f57c15ef in clone (/lib/x86_64-linux-gnu/libc.so.6+0x1105ef)
(BuildId: b46a78e7229ed6fe08549e2bc7ca64155cc5cf1e)
0x509000007818 is located 24 bytes inside of 40-byte region
[0x509000007800,0x509000007828)
freed by thread T10 (vf#0:0) here:
#0 0x564398015f4a in free (/tmp/ffmpeg/ffmpeg_g+0x734f4a) (BuildId:
fc49802141dc7a2a32e6338567bec09a75fe6f79)
#1 0x56439996eb44 in av_packet_side_data_free
/tmp/ffmpeg/libavcodec/packet.c:751:9
previously allocated by thread T10 (vf#0:0) here:
#0 0x564398016ceb in posix_memalign (/tmp/ffmpeg/ffmpeg_g+0x735ceb)
(BuildId: fc49802141dc7a2a32e6338567bec09a75fe6f79)
#1 0x56439b47a9cb in av_malloc /tmp/ffmpeg/libavutil/mem.c:107:9
#2 0x56439b47a9cb in av_mallocz /tmp/ffmpeg/libavutil/mem.c:258:17
#3 0x564399e2452d in av_cpb_properties_alloc
/tmp/ffmpeg/libavcodec/utils.c:956:30
#4 0x564397e7edf4 in aom_init
/tmp/ffmpeg/libavcodec/libaomenc.c:992:17
#5 0x564398f08f62 in avcodec_open2
/tmp/ffmpeg/libavcodec/avcodec.c:336:19
Thread T10 (vf#0:0) created by T0 here:
#0 0x564397ffb6f5 in pthread_create (/tmp/ffmpeg/ffmpeg_g+0x71a6f5)
(BuildId: fc49802141dc7a2a32e6338567bec09a75fe6f79)
#1 0x5643980b2738 in task_start
/tmp/ffmpeg/fftools/ffmpeg_sched.c:414:11
#2 0x5643980e79f3 in transcode /tmp/ffmpeg/fftools/ffmpeg.c:881:11
#3 0x5643980e79f3 in main /tmp/ffmpeg/fftools/ffmpeg.c:1009:11
#4 0x7f27f56daca7 (/lib/x86_64-linux-gnu/libc.so.6+0x29ca7) (BuildId:
b46a78e7229ed6fe08549e2bc7ca64155cc5cf1e)
SUMMARY: AddressSanitizer: heap-use-after-free
/tmp/ffmpeg/libavcodec/libaomenc.c:1028:28 in aom_init
Shadow bytes around the buggy address:
0x509000007580: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
0x509000007600: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
0x509000007680: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
0x509000007700: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x509000007780: 00 00 00 fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x509000007800: fd fd fd[fd]fd fa fa fa fa fa fa fa fa fa fa fa
0x509000007880: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
0x509000007900: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
0x509000007980: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
0x509000007a00: fa fa fa fa fa fa fa fa 00 00 00 00 00 fa fa fa
0x509000007a80: fa fa fa fa fa fa fa fa 00 00 00 fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==19780==ABORTING
}}}
=== Possible Cause
In `libavcodec/libaomenc.c:1028:28`, the code tries to write to memory
pointed by `cpb_props`:
{{{#!c
cpb_props->buffer_size = avctx->rc_buffer_size;
}}}
`cpb_props` is assigned in line `992`, before calling to
`ff_dovi_configure`:
{{{#!c
cpb_props = ff_encode_add_cpb_side_data(avctx);
if (!cpb_props)
return AVERROR(ENOMEM);
ctx->dovi.logctx = avctx;
if ((res = ff_dovi_configure(&ctx->dovi, avctx)) < 0)
return res;
}}}
[https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/ab37c7e49ff98a8c32c90e6df49dd11145bc64bb:/libavcodec/encode.c#l880
`ff_encode_add_cpb_side_data`] returns a new array, and its address is
stored in `avctx->coded_side_data`.
[https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/ab37c7e49ff98a8c32c90e6df49dd11145bc64bb:/libavcodec/dovi_rpuenc.c#l241
`ff_dovi_configure`] calls to
[https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/ab37c7e49ff98a8c32c90e6df49dd11145bc64bb:/libavcodec/codec_par.c#l203
`avcodec_parameters_to_context`], which replaces the memory of
`coded_side_data`:
{{{#!c
int avcodec_parameters_to_context(AVCodecContext *codec,
const AVCodecParameters *par)
{
// ...
av_packet_side_data_free(&codec->coded_side_data,
&codec->nb_coded_side_data);
ret = codec_parameters_copy_side_data(&codec->coded_side_data,
&codec->nb_coded_side_data,
par->coded_side_data,
par->nb_coded_side_data);
// ...
}
}}}
With GDB I can see that the memory stored in the `cpb_props` variable is
freed by `av_free`:
{{{
$ gdb --args ./ffmpeg_g -y -v warning -hide_banner -f lavfi -i testsrc -to
1 -c:v libaom-av1 /tmp/0.webm
[...]
(gdb) break libavcodec/libaomenc.c:993
(gdb) commands
>printf "cpb_props = %p\n", cpb_props
>c
>end
(gdb) break libavcodec/packet.c:751
(gdb) commands
>printf "av_free: %p\n", sd[i].data
>c
>end
(gdb) r
[...]
Thread 18 "vf#0:0" hit Breakpoint 1, aom_init (avctx=0x519000002880,
iface=<optimized out>)
at libavcodec/libaomenc.c:993
993 if (!cpb_props)
cpb_props = 0x509000007800
Thread 18 "vf#0:0" hit Breakpoint 2, av_packet_side_data_free
(psd=psd@entry=0x519000002b88,
pnb_sd=pnb_sd@entry=0x519000002b90) at libavcodec/packet.c:751
751 av_free(sd[i].data);
av_free: 0x509000007800
Thread 18 "vf#0:0" hit Breakpoint 2, av_packet_side_data_free
(psd=psd@entry=0x51100000d9e0,
pnb_sd=pnb_sd@entry=0x51100000d9e8) at libavcodec/packet.c:751
751 av_free(sd[i].data);
av_free: 0x509000007900
=================================================================
==165799==ERROR: AddressSanitizer: heap-use-after-free on address
0x509000007818 at pc 0x555555b3d998 bp 0x7fffdfe67090 sp 0x7fffdfe67088
[...]
}}}
=== Possible Fix
I modified `libavcodec/libaomenc.c` to initialize `cpb_props` ''after''
the call to `ff_dovi_configure`:
{{{#!diff
diff --git a/libavcodec/libaomenc.c b/libavcodec/libaomenc.c
index 9a384fcc3..d2c47b43e 100644
--- a/libavcodec/libaomenc.c
+++ b/libavcodec/libaomenc.c
@@ -989,14 +989,14 @@ static av_cold int aom_init(AVCodecContext *avctx,
if (codec_caps & AOM_CODEC_CAP_HIGHBITDEPTH)
ctx->rawimg.bit_depth = enccfg.g_bit_depth;
- cpb_props = ff_encode_add_cpb_side_data(avctx);
- if (!cpb_props)
- return AVERROR(ENOMEM);
-
ctx->dovi.logctx = avctx;
if ((res = ff_dovi_configure(&ctx->dovi, avctx)) < 0)
return res;
+ cpb_props = ff_encode_add_cpb_side_data(avctx);
+ if (!cpb_props)
+ return AVERROR(ENOMEM);
+
if (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER) {
const AVBitStreamFilter *filter =
av_bsf_get_by_name("extract_extradata");
int ret;
}}}
With the change, the command works with no issues:
{{{
$ ./ffmpeg_g -y -v warning -hide_banner -f lavfi -i testsrc -to 1 -c:v
libaom-av1 /tmp/0.webm
[libaom-av1 @ 0x519000002380] Neither bitrate nor constrained quality
specified, using default CRF of 32
$ ./ffprobe -hide_banner -i /tmp/0.webm
Input #0, matroska,webm, from '/tmp/0.webm':
Metadata:
ENCODER : Lavf62.0.102
Duration: 00:00:01.00, start: 0.000000, bitrate: 33 kb/s
Stream #0:0: Video: av1 (libaom-av1) (High), gbrp(pc,
gbr/bt709/iec61966-2-1, progressive), 320x240 [SAR 1:1 DAR 4:3], 25 fps,
25 tbr, 1k tbn
Metadata:
ENCODER : Lavc62.3.101 libaom-av1
DURATION : 00:00:01.000000000
$
}}}
Though I don't know if this change may break something else.
--
Ticket URL: <https://trac.ffmpeg.org/ticket/11617>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker
_______________________________________________
FFmpeg-trac mailing list
FFmpeg-trac@avcodec.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-trac
To unsubscribe, visit link above, or email
ffmpeg-trac-requ...@ffmpeg.org with subject "unsubscribe".
