PR #23572 opened by Diego de Souza (ddesouza)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23572
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/23572.patch
Commit 7fd9780a ("avcodec: Remove FF_API_NVDEC_OLD_PIX_FMTS") completed the
switch to the new NVDEC pixel formats, removing the old-format fallbacks. The
NVDEC/CUVID decoders now select the new formats for high-bit-depth content:
- 12-bit 4:2:0 → AV_PIX_FMT_P012
- 10/12-bit 4:2:2 → AV_PIX_FMT_P210 / AV_PIX_FMT_P212
- 10/12-bit 4:4:4 → AV_PIX_FMT_YUV444P10MSB / AV_PIX_FMT_YUV444P12MSB
but two places were never updated to handle them, so decoding these streams
fails on current master:
1. avutil/hwcontext_cuda — P012 and P212 are missing from the CUDA frames
context's supported_formats[], so av_hwframe_ctx_init() rejects them with
AVERROR(ENOSYS) ("Pixel format not supported"). This breaks 12-bit decoding to
CUDA frames (NVDEC and CUVID) and hwdownload of such frames.
2. avcodec/cuviddec — cuvid_handle_video_sequence() never mapped
P012/P210/P212/YUV444P10MSB/YUV444P12MSB to a CUVID surface format, so they hit
the default: arm with "Unsupported output format" (AVERROR(EINVAL)). Because
every packet then errors without producing a frame, this can also surface as an
apparent hang rather than a clean error. The system-memory copy path in
cuvid_output_frame() rejected the same formats with AVERROR_BUG.
**Changes**
avutil/hwcontext_cuda: add P012 and P212 to supported formats
- Add AV_PIX_FMT_P012 and AV_PIX_FMT_P212 next to the existing P010/P016 and
P210/P216 entries. The per-plane device transfer is derived generically from
the pixel descriptor, so no further changes are needed.
avcodec/cuviddec: map new NVDEC 4:2:0/4:2:2/4:4:4 output formats
- Map P012 → cudaVideoSurfaceFormat_P016 (mirroring P010), P210/P212 →
cudaVideoSurfaceFormat_P216 (mirroring P216), and YUV444P10MSB/YUV444P12MSB →
cudaVideoSurfaceFormat_YUV444_16Bit (mirroring YUV444P16). The hardware surface
is physically 16-bit; the pixel format only records the number of valid bits.
- Add the same formats to the system-memory copy path, whose per-plane geometry
is already derived from the pixel descriptor.
From 531a6527a25c5721540f38f89eab8bcb7ff8fe7d Mon Sep 17 00:00:00 2001
From: Diego de Souza <[email protected]>
Date: Wed, 24 Jun 2026 00:30:45 +0200
Subject: [PATCH 1/2] avutil/hwcontext_cuda: add P012 and P212 to supported
formats
NVDEC and CUVID decode 12-bit 4:2:0 content to AV_PIX_FMT_P012 and
12-bit 4:2:2 to AV_PIX_FMT_P212, but these formats were missing from
the CUDA frames context supported format list. As a result
av_hwframe_ctx_init() rejected them ("Pixel format not supported",
AVERROR(ENOSYS)), so decoding 12-bit content to CUDA frames and
hwdownload of such frames both failed.
Add P012 and P212 next to the existing P010/P016 and P210/P216 entries.
The per-plane device transfer is derived generically from the pixel
descriptor, so no other changes are required.
Signed-off-by: Diego de Souza <[email protected]>
---
libavutil/hwcontext_cuda.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/libavutil/hwcontext_cuda.c b/libavutil/hwcontext_cuda.c
index b0b65b2446..c57ea4bb7a 100644
--- a/libavutil/hwcontext_cuda.c
+++ b/libavutil/hwcontext_cuda.c
@@ -47,8 +47,10 @@ static const enum AVPixelFormat supported_formats[] = {
AV_PIX_FMT_YUVA420P,
AV_PIX_FMT_YUV444P,
AV_PIX_FMT_P010,
+ AV_PIX_FMT_P012,
AV_PIX_FMT_P016,
AV_PIX_FMT_P210,
+ AV_PIX_FMT_P212,
AV_PIX_FMT_P216,
AV_PIX_FMT_YUV422P,
AV_PIX_FMT_YUV420P10,
--
2.52.0
From 8108c864f7d9fca1369beebefb688fe8346daedd Mon Sep 17 00:00:00 2001
From: Diego de Souza <[email protected]>
Date: Wed, 24 Jun 2026 00:30:45 +0200
Subject: [PATCH 2/2] avcodec/cuviddec: map new NVDEC 4:2:0/4:2:2/4:4:4 output
formats
The switch to the new NVDEC pixel formats made cuvid_derive_output_format()
and the format probe select AV_PIX_FMT_P012 for 12-bit 4:2:0,
AV_PIX_FMT_P210/P212 for 10/12-bit 4:2:2 and AV_PIX_FMT_YUV444P10MSB/
YUV444P12MSB for 10/12-bit 4:4:4, but cuvid_handle_video_sequence() still
only mapped the older formats to a CUVID surface format. Decoding such
streams therefore failed with "Unsupported output format" (AVERROR(EINVAL));
because every packet then errored without producing a frame, the failure
could also manifest as a hang rather than a clean error exit. The
system-memory copy path in cuvid_output_frame() rejected the same formats
with AVERROR_BUG.
Map P012 to cudaVideoSurfaceFormat_P016 (mirroring P010), P210/P212 to
cudaVideoSurfaceFormat_P216 (mirroring P216) and YUV444P10MSB/YUV444P12MSB
to cudaVideoSurfaceFormat_YUV444_16Bit (mirroring YUV444P16); the hardware
surface is physically 16-bit and the pixel format only records the number
of valid bits. Add the same formats to the system-memory copy path, whose
per-plane geometry is already derived generically from the pixel descriptor.
Signed-off-by: Diego de Souza <[email protected]>
---
libavcodec/cuviddec.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/libavcodec/cuviddec.c b/libavcodec/cuviddec.c
index d423d2611b..292015d18a 100644
--- a/libavcodec/cuviddec.c
+++ b/libavcodec/cuviddec.c
@@ -333,6 +333,7 @@ static int CUDAAPI cuvid_handle_video_sequence(void
*opaque, CUVIDEOFORMAT* form
cuinfo.OutputFormat = cudaVideoSurfaceFormat_NV12;
break;
case AV_PIX_FMT_P010:
+ case AV_PIX_FMT_P012:
case AV_PIX_FMT_P016:
cuinfo.OutputFormat = cudaVideoSurfaceFormat_P016;
break;
@@ -340,6 +341,8 @@ static int CUDAAPI cuvid_handle_video_sequence(void
*opaque, CUVIDEOFORMAT* form
case AV_PIX_FMT_NV16:
cuinfo.OutputFormat = cudaVideoSurfaceFormat_NV16;
break;
+ case AV_PIX_FMT_P210:
+ case AV_PIX_FMT_P212:
case AV_PIX_FMT_P216:
cuinfo.OutputFormat = cudaVideoSurfaceFormat_P216;
break;
@@ -347,6 +350,8 @@ static int CUDAAPI cuvid_handle_video_sequence(void
*opaque, CUVIDEOFORMAT* form
case AV_PIX_FMT_YUV444P:
cuinfo.OutputFormat = cudaVideoSurfaceFormat_YUV444;
break;
+ case AV_PIX_FMT_YUV444P10MSB:
+ case AV_PIX_FMT_YUV444P12MSB:
case AV_PIX_FMT_YUV444P16:
cuinfo.OutputFormat = cudaVideoSurfaceFormat_YUV444_16Bit;
break;
@@ -632,12 +637,17 @@ static int cuvid_output_frame(AVCodecContext *avctx,
AVFrame *frame)
}
} else if (avctx->pix_fmt == AV_PIX_FMT_NV12 ||
avctx->pix_fmt == AV_PIX_FMT_P010 ||
+ avctx->pix_fmt == AV_PIX_FMT_P012 ||
avctx->pix_fmt == AV_PIX_FMT_P016 ||
#ifdef NVDEC_HAVE_422_SUPPORT
avctx->pix_fmt == AV_PIX_FMT_NV16 ||
+ avctx->pix_fmt == AV_PIX_FMT_P210 ||
+ avctx->pix_fmt == AV_PIX_FMT_P212 ||
avctx->pix_fmt == AV_PIX_FMT_P216 ||
#endif
avctx->pix_fmt == AV_PIX_FMT_YUV444P ||
+ avctx->pix_fmt == AV_PIX_FMT_YUV444P10MSB ||
+ avctx->pix_fmt == AV_PIX_FMT_YUV444P12MSB ||
avctx->pix_fmt == AV_PIX_FMT_YUV444P16) {
unsigned int offset = 0;
AVFrame *tmp_frame = av_frame_alloc();
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
