PR #25048 opened by urzaru URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25048 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25048.patch
# Summary of changes Three independent fixes to `init_pic_params()` in `libavcodec/vulkan_encode_av1.c`. Each commit explains its own cause and effect. Found on an RTX 4090, where `av1_vulkan` output was undecodable at most bitrates, then drifting, then capped near 37–40 dB PSNR. Details, measurements and a sample are in #25046. 1. **Use `PRIMARY_REF_NONE` for key frames.** Key frames kept `primary_ref_frame = 0`. The encoder's CDFs then no longer match the defaults that decoders use for an error-resilient key frame once `base_q_idx > 20`. 2. **Name the second P reference consistently.** `ref_frame_idx[]` named the older reference GOLDEN, while `referenceNameSlotIndices` named it LAST2, so encoder and decoder predicted from different pictures. 3. **Zero `constantQIndex` when rate control is on**, as `VUID-vkCmdEncodeVideoKHR-constantQIndex-10320` requires and as the H.264 encoder already does. Testing, all on an RTX 4090 with Linux driver 615.78.08 (2560x1440 at 60 fps and 50 Mbit/s is abbreviated 1440p60 at 50M): | | before (9.0.2) | after | | --- | --- | --- | | 1280x720 60 fps CBR 10M, dav1d | 0 of 300 frames decode | decodes, 52.3 dB average (min 50.1) | | same, `-bf 0` | 0 of 300 | 51.7 dB (min 50.1) | | 1280x720 25 fps CBR 2M | 0 of 120 | 42.4 dB | | `-rc_mode cqp -qp 100` | 0 of 120 | 44.9 dB | | `-g 1` | 2 of 60 | 45.8 dB | | 1440p60 at 50M | 0 of 300 | 57.5 dB (min 55.5) | PSNR is measured against the testsrc2 source. Vulkan hardware decoding of these outputs is bit-identical to dav1d. Each commit builds without warnings. `tools/patcheck` and `tools/check_commit_msg.sh` pass, apart from patcheck's note about a changelog entry. I could only test on NVIDIA, so results from AMD, Intel or Mesa (RADV/ANV) would be welcome. AI disclosure: these were found and written with substantial help from an AI assistant (Claude Opus 5.5 in Claude Code). Each commit carries an `Assisted-by:` trailer. I directed the work, and ran the tests on my own hardware. 🤖 Generated with [Claude Code](https://claude.com/claude-code) From c53355b9da46a3d6ab8792f515e88c40cb4d8ed7 Mon Sep 17 00:00:00 2001 From: Uraza Aru <[email protected]> Date: Wed, 14 Oct 2026 02:14:02 +0500 Subject: [PATCH 1/3] avcodec/vulkan_encode_av1: use PRIMARY_REF_NONE for key frames primary_ref_frame is only set in the P and B branches, so key frames keep the initial 0. A shown key frame is error resilient: its header signals no primary reference and decoders start from the default CDFs for the frame's base_q_idx. With primary_ref_frame = 0, NVIDIA's encoder evidently starts from other CDFs; the two agree only while base_q_idx <= 20 (the first coefficient CDF q context). Above that, every key frame and everything after it is undecodable: dav1d reports a symbol decoder overread in the first tile and hardware decoders show garbage. This is why streams at modest bitrates (e.g. 1280x720 at 60 fps and 10 Mbit/s, key frame base_q_idx 39) failed while very high bitrates worked. Set primary_ref_frame to STD_VIDEO_AV1_PRIMARY_REF_NONE for key frames, as NVIDIA's Vulkan Video samples do. Tested on an RTX 4090 (Linux driver 615.78.08): CBR 2-60 Mbit/s and CQP 40-200 now decode, up to 2560x1440 at 120 fps and 3840x2160 at 60 fps. Fixes #25046. Assisted-by: Claude Opus 5.5 Signed-off-by: Uraza Aru <[email protected]> --- libavcodec/vulkan_encode_av1.c | 1 + 1 file changed, 1 insertion(+) diff --git a/libavcodec/vulkan_encode_av1.c b/libavcodec/vulkan_encode_av1.c index 67d81d189b..379b9f2eb9 100644 --- a/libavcodec/vulkan_encode_av1.c +++ b/libavcodec/vulkan_encode_av1.c @@ -355,6 +355,7 @@ static int init_pic_params(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, av_assert0(pic->nb_refs[0] == 0 || pic->nb_refs[1]); ap->av1pic_info.frame_type = STD_VIDEO_AV1_FRAME_TYPE_KEY; ap->av1pic_info.refresh_frame_flags = 0xFF; + ap->av1pic_info.primary_ref_frame = STD_VIDEO_AV1_PRIMARY_REF_NONE; quantization->base_q_idx = enc->q_idx_idr; ap->slot = 0; ap->last_idr_frame = pic->display_order; -- 2.52.0 From 02fcc168532d4a78fc41b24ee4ab63cfde1938be Mon Sep 17 00:00:00 2001 From: Uraza Aru <[email protected]> Date: Wed, 14 Oct 2026 02:14:02 +0500 Subject: [PATCH 2/3] avcodec/vulkan_encode_av1: name the second P reference consistently For P frames with two references, ref_frame_idx[] puts the older reference under GOLDEN and points every other name, LAST2 included, at the newest one. set_name_slot() however gives the older reference the first free allowed name, LAST2, in referenceNameSlotIndices. The encoder then predicts LAST2 blocks from the older picture while decoders, following the frame header, use the newer one, and the picture drifts: on an RTX 4090, 1280x720 CBR 10 Mbit/s without B-frames fell from 44.8 dB PSNR on the key frame to 12.8 dB by frame 299. Index ref_frame_idx[] with the name set_name_slot() returns, so both mappings agree. The B-frame branch already keeps them in step. Fixes #25046. Assisted-by: Claude Opus 5.5 Signed-off-by: Uraza Aru <[email protected]> --- libavcodec/vulkan_encode_av1.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/libavcodec/vulkan_encode_av1.c b/libavcodec/vulkan_encode_av1.c index 379b9f2eb9..9f864732df 100644 --- a/libavcodec/vulkan_encode_av1.c +++ b/libavcodec/vulkan_encode_av1.c @@ -188,6 +188,7 @@ static int init_pic_params(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, uint32_t ref_name_mask = 0x0; int name_slots[STD_VIDEO_AV1_REFS_PER_FRAME]; + int name; StdVideoAV1Segmentation *segmentation = &ap->segmentation; StdVideoAV1LoopFilter *loop_filter = &ap->loop_filter; @@ -397,10 +398,10 @@ static int init_pic_params(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, } ref = pic->refs[0][pic->nb_refs[0] - 2]; ap_ref = ref->codec_priv; - ap->av1pic_info.ref_frame_idx[3] = ap_ref->slot; ap->av1pic_info.ref_order_hint[ap_ref->slot] = ref->display_order - ap_ref->last_idr_frame; -// vpic->ref_frame_ctrl_l0.fields.search_idx1 = AV1_REF_FRAME_GOLDEN; - set_name_slot(ap_ref->av1pic_info.current_frame_id, name_slots, ref_name_mask, 0); + /* The header must name the same reference the encoder uses for it. */ + name = set_name_slot(ap_ref->av1pic_info.current_frame_id, name_slots, ref_name_mask, 0); + ap->av1pic_info.ref_frame_idx[name] = ap_ref->slot; } break; case FF_HW_PICTURE_TYPE_B: -- 2.52.0 From 3652cc4bba6f98ac353f7ebd9d8e5eae35763c85 Mon Sep 17 00:00:00 2001 From: Uraza Aru <[email protected]> Date: Wed, 14 Oct 2026 02:14:02 +0500 Subject: [PATCH 3/3] avcodec/vulkan_encode_av1: zero constantQIndex when rate control is on VUID-vkCmdEncodeVideoKHR-constantQIndex-10320 requires constantQIndex to be 0 unless the rate control mode is DISABLED; the H.264 encoder already does the same for constantQp. The AV1 encoder always passed base_q_idx. On NVIDIA the frame headers then signal delta_q_present = 0 while blocks are evidently quantised differently, so every frame decodes with an error: 2560x1440 at 60 fps and 50 Mbit/s reached only 37.3 dB PSNR, and quality fell as the bitrate rose (165 fps: 40.2, 38.4 and 37.7 dB at 60, 100 and 137 Mbit/s). With this change those become 57.1 dB and 46.8, 53.9 and 59.5 dB, on par with NVIDIA's own encoder sample on the same input. Constant QP is unchanged. Fixes #25046. Assisted-by: Claude Opus 5.5 Signed-off-by: Uraza Aru <[email protected]> --- libavcodec/vulkan_encode_av1.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/libavcodec/vulkan_encode_av1.c b/libavcodec/vulkan_encode_av1.c index 9f864732df..e5d88803a4 100644 --- a/libavcodec/vulkan_encode_av1.c +++ b/libavcodec/vulkan_encode_av1.c @@ -453,6 +453,10 @@ static int init_pic_params(AVCodecContext *avctx, FFHWBaseEncodePicture *pic, .primaryReferenceCdfOnly = 0, .generateObuExtensionHeader = 0, }; + /* With rate control on, the driver picks the quantiser; constantQIndex must be 0 + * (VUID-vkCmdEncodeVideoKHR-constantQIndex-10320). */ + if (ctx->opts.rc_mode != VK_VIDEO_ENCODE_RATE_CONTROL_MODE_DISABLED_BIT_KHR) + ap->vkav1pic_info.constantQIndex = 0; encode_info->pNext = &ap->vkav1pic_info; for (int i = 0; i < FF_ARRAY_ELEMS(ap->vkav1pic_info.referenceNameSlotIndices); i++) -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
