This is an automated email from the git hooks/post-receive script. Git pushed a commit to branch master in repository ffmpeg.
commit 404b97e55176a266600d878d661a8ef6ed7c38df Author: Philip Langdale <[email protected]> AuthorDate: Fri Jul 31 15:59:09 2026 +0800 Commit: Philip Langdale <[email protected]> CommitDate: Fri Aug 7 08:45:59 2026 -0700 avfilter: add vsr_cuda, NVIDIA RTX Video Super Resolution Upscales packed RGB CUDA frames with the RTX VSR network from the NGX SDK snippet nvngx_vsr.dll, replayed directly on libcuda -- no DLL is loaded at run time. vsr_cuda_gen.h describes the captured graph as a function of the input and output size; the filter evaluates it in config_output() and then hands the result to the rtx_cuda core, which does the loading, allocation and replay. What is specific to VSR and therefore lives here: the quality/scale option pair that picks a generated config, the packed-RGB format selectors patched into the argument blocks, and the input-texture/output-surface binding. Quality 0-4 are the upscaling models (0 is a plain bicubic reference, 2x is fixed for low/medium, and high/ultra upscale natively by 2x/3x/4x), 5 and 6 are native-resolution restoration networks forced to 1x, and 7 is a pass-through. Output sizes other than the network's native factor resample that result with the graph's own bicubic compose. Not every model exists in every extraction: the Windows nvngx_vsr.dll carries 0-4 at 2x, and the native Linux library adds the restoration and lossless models and the 3x/4x heads. The option ranges cover the union, and a model the installed data has no network for is refused at config time by name. The cubins and weights are not shipped; the filter is built only when an installed nvidia-video-filters package provides the VSR data. --- configure | 1 + doc/filters.texi | 222 +++++++++++++++++++++++++++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_vsr_cuda.c | 373 ++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 598 insertions(+) diff --git a/configure b/configure index 2a6d0e9725..69e594d5ab 100755 --- a/configure +++ b/configure @@ -4322,6 +4322,7 @@ scale_vt_filter_deps="videotoolbox VTPixelTransferSessionCreate" scale_vulkan_filter_deps="vulkan spirv_compiler swscale" vpp_qsv_filter_deps="libmfx" vpp_qsv_filter_select="qsvvpp" +vsr_cuda_filter_deps="ffnvcodec nvfdata_vsr" xfade_opencl_filter_deps="opencl" xfade_vulkan_filter_deps="vulkan spirv_compiler" yadif_cuda_filter_deps="ffnvcodec" diff --git a/doc/filters.texi b/doc/filters.texi index 985a1615a9..f17469fc96 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -27757,6 +27757,228 @@ Preserve landscape geometry (when @var{width} >= @var{height}). @end table +@anchor{vsr_cuda} +@section vsr_cuda + +Upscale video with NVIDIA RTX Video Super Resolution, running the extracted +VSR network directly on CUDA (no @code{nvngx_vsr.dll} at run time). + +The network operates on packed 32-bit-or-wider @strong{RGB} CUDA frames, so it +is colorspace-agnostic: feed it linear or gamma RGB in whatever primaries you +like, and it processes the pixels as given. + +An output that is exactly the network's native factor takes a fast direct store; +any other size (non-integer, anisotropic, or downscale) resamples the native +result with a bicubic compose. + +The cubins and weights are extracted from the proprietary NVIDIA libraries and +are @emph{not} shipped: the filter is only built when an +@code{nvidia-video-filters} package carrying the VSR data is installed, and +@option{data} defaults to that package's data directory. + +It accepts the following options: + +@table @option +@item quality +Which model to run; see below. Default @code{1}. + +@item scale +Native integer upscale factor for the scale-adaptive @code{high} and +@code{ultra} models: @code{0} (the default), @code{2}, @code{3} or @code{4}. +Ignored by every other model. + +@code{0} derives the factor from the requested output size. Setting it forces +that head and defaults the output to that factor times the input, so +@option{scale} together with @option{w}/@option{h} runs the forced network and +then resamples its result to the rectangle you asked for -- @code{scale=4} with +a 1080p output runs the 4x network and downscales, rather than the 3x the ratio +alone would have picked. + +@item w +@item h +Output width and height, as expressions (as in @ref{scale}); the variables +@var{iw}/@var{in_w} and @var{ih}/@var{in_h} hold the input size. Unset (the +default) means the network's native factor times the input. + +@item format +Output pixel format. Empty (the default) keeps the input format. Must be one +of the supported formats below. + +@item data +Directory holding the extracted cubins and the shared @file{weights.bin}. +@end table + +@subsection Quality levels + +@option{quality} selects the model: + +@table @asis +@item @code{0} bicubic +Not a network at all -- a plain bicubic resample, for a like-for-like reference +to compare the models against. + +@item @code{1} low, @code{2} medium +Fixed 2x super-resolution. + +@item @code{3} high, @code{4} ultra +Scale-adaptive super-resolution: the same model upscales natively by 2x, 3x or +4x, selected with @option{scale}. + +@item @code{5} denoise, @code{6} deblur +Restoration networks rather than upscalers, forced to the input resolution +because upscaling with them produces checkerboard and chroma artefacts. +Restore with one of these first, then upscale in a second @code{vsr_cuda} pass. + +@item @code{7} lossless +A faithful pass-through. +@end table + +Which of these exist depends on where the data came from: the Windows and Linux +builds of VSR do not ship the same set. The Windows @code{nvngx_vsr.dll} +carries @code{0} to @code{4} and 2x only, while the native Linux library adds +the restoration and lossless models and the 3x/4x heads. A model or scale the +installed package has no network for is refused at configuration time with a +message naming it, rather than silently substituted. + +@subsection Supported formats + +Packed RGB formats are accepted for both input and output: @code{rgb0}, +@code{rgba}, @code{bgr0}, @code{bgra} (8-bit) and @code{rgba64le} (16-bit). +Input and output formats are chosen independently (see the @option{format} +option). 16-bit carries the network's full internal precision and avoids +banding. + +The network's kernels branch on a format selector, exposing an R@math{<->}B +swap only on the 8-bit path, so the B-first @code{bgr0}/@code{bgra} are handled +natively. That swap does not exist on the high-bit-depth path, which packs in +array-native order, so 16-bit is R-first only (@code{rgba64le}); there is no +B-first 16-bit variant. + +@subsection Colorspace conversion and HDR (libplacebo) + +@code{vsr_cuda} deliberately does @strong{not} do YUV@math{<->}RGB conversion or +tone mapping itself. Real (YUV) video should be converted with +@code{libplacebo}, whose colorimetry and tone mapping are far better than a +hand-rolled kernel. libplacebo is Vulkan-based, so a Vulkan device is used for +the conversions and the RGB frames are handed to CUDA. + +Both hand-offs are an @code{hwupload} with the other device derived from the +first (@code{hwupload=derive_device=cuda} for Vulkan@math{->}CUDA, +@code{hwupload=derive_device=vulkan} for CUDA@math{->}Vulkan). Each performs a +direct on-GPU transfer: Vulkan exports the image memory and a semaphore, CUDA +imports them and copies. The whole graph therefore stays on the GPU; the only +host copy is the final @code{hwdownload} for a software encoder (route the output +to @code{hevc_nvenc} etc. to avoid even that). + +Create the Vulkan device with @code{disable_multiplane=1} so semi-planar YUV +(@code{nv12}, @code{p010}, @dots{}) is backed by one Vulkan image per plane; +CUDA imports a Vulkan image as a single-plane array, so a multiplanar image +cannot be crossed to CUDA. This matters when a hardware decoder/encoder puts +YUV on the CUDA side of the boundary; RGB (single plane) is unaffected. CUDA is +derived from the Vulkan device, keeping both on the same GPU. + +@example +ffmpeg -init_hw_device vulkan=vk:0,disable_multiplane=1 -filter_hw_device vk \ + -i input.mp4 -vf " + libplacebo=format=rgba, + hwupload=derive_device=cuda, + vsr_cuda=quality=3, + hwupload=derive_device=vulkan, + libplacebo=format=nv12, + hwdownload,format=nv12 +" -c:v libx264 out.mp4 +@end example + +For HDR (e.g. BT.2020 PQ) input, let the first @code{libplacebo} tone-map to +the working space and the second convert back, keeping high-bit-depth precision +through @code{vsr_cuda} with @code{rgba64le}: + +@example +ffmpeg -init_hw_device vulkan=vk:0,disable_multiplane=1 -filter_hw_device vk \ + -i hdr.mp4 -vf " + libplacebo=color_trc=smpte2084:format=rgba64le, + hwupload=derive_device=cuda, + vsr_cuda=quality=3:format=rgba64le, + hwupload=derive_device=vulkan, + libplacebo=color_trc=bt709:tonemapping=bt.2390:format=p010le, + hwdownload,format=p010le +" -c:v libx265 out.mkv +@end example + +@subsection Hardware decode and encode + +The pipeline can be driven entirely by a hardware decoder and encoder, keeping +the whole chain on the GPU. There are two hardware paths for each end, +@code{vulkan} and CUDA (@code{nvdec}/@code{nvenc}), giving four combinations. +Which work depends on where semi-planar YUV has to cross the CUDA@math{<->}Vulkan +boundary, because CUDA imports a Vulkan image as a single-plane array +(see @option{disable_multiplane} above) while @code{hevc_vulkan} requires a +multiplane YUV image. + +@table @asis +@item Vulkan decode @math{->} Vulkan encode +Fully zero-copy. Only packed RGB crosses to CUDA (single plane), so +@code{disable_multiplane} must be @strong{off} and the encoder gets a normal +multiplane @code{nv12}: + +@example +ffmpeg -init_hw_device vulkan=vk:0 -filter_hw_device vk \ + -hwaccel vulkan -hwaccel_output_format vulkan -i input.mp4 -vf " + libplacebo=format=rgba, + hwupload=derive_device=cuda, + vsr_cuda=quality=3, + hwupload=derive_device=vulkan, + libplacebo=format=nv12 +" -c:v hevc_vulkan out.mkv +@end example + +@item nvdec @math{->} Vulkan encode +Works, but the decoded @code{nv12} cannot be crossed to Vulkan zero-copy: that +would need @option{disable_multiplane}, which then breaks @code{hevc_vulkan}. +Bounce the decoded frame through host memory with @code{hwdownload} so +@code{libplacebo} re-uploads it as a multiplane image. Decode and encode are +still on hardware; only the decode hand-off copies to the host: + +@example +ffmpeg -init_hw_device vulkan=vk:0 -init_hw_device cuda=cu@@vk \ + -filter_hw_device vk \ + -hwaccel cuda -hwaccel_device cu -hwaccel_output_format cuda -i input.mp4 -vf " + hwdownload,format=nv12, + libplacebo=format=rgba, + hwupload=derive_device=cuda, + vsr_cuda=quality=3, + hwupload=derive_device=vulkan, + libplacebo=format=nv12 +" -c:v hevc_vulkan out.mkv +@end example + +@item Vulkan or nvdec decode @math{->} nvenc +The @code{nv12} must be crossed from Vulkan to CUDA for @code{nvenc}, so the +Vulkan device needs @code{disable_multiplane=1} and a trailing +@code{hwupload=derive_device=cuda}. With an @code{nvdec} source, prepend +@code{hwupload=derive_device=vulkan} to move its CUDA frame into +@code{libplacebo} (which cannot auto-convert a CUDA frame): + +@example +ffmpeg -init_hw_device vulkan=vk:0,disable_multiplane=1 \ + -init_hw_device cuda=cu@@vk -filter_hw_device vk \ + -hwaccel vulkan -hwaccel_output_format vulkan -i input.mp4 -vf " + libplacebo=format=rgba, + hwupload=derive_device=cuda, + vsr_cuda=quality=3, + hwupload=derive_device=vulkan, + libplacebo=format=nv12, + hwupload=derive_device=cuda +" -c:v hevc_nvenc out.mkv +@end example +@end table + +The trailing @code{libplacebo=format=nv12} is mandatory: the hardware encoders +ingest @strong{YCbCr only}. The Vulkan encoders reject every RGB pixel format +(@code{Pixel format rgba of input frames not supported}), and @code{nvenc} +likewise takes @code{nv12}/@code{p010}, so @code{vsr_cuda}'s RGB output cannot be +fed to an encoder directly. + @section yadif_cuda Deinterlace the input video using the @ref{yadif} algorithm, but implemented diff --git a/libavfilter/Makefile b/libavfilter/Makefile index daa4d552f0..1643686b57 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -575,6 +575,7 @@ OBJS-$(CONFIG_VPP_AMF_FILTER) += vf_vpp_amf.o scale_eval.o vf_amf OBJS-$(CONFIG_FRC_AMF_FILTER) += vf_frc_amf.o vf_amf_common.o OBJS-$(CONFIG_VQE_AMF_FILTER) += vf_vqe_amf.o vf_amf_common.o OBJS-$(CONFIG_VPP_QSV_FILTER) += vf_vpp_qsv.o +OBJS-$(CONFIG_VSR_CUDA_FILTER) += vf_vsr_cuda.o rtx_cuda.o OBJS-$(CONFIG_VSTACK_FILTER) += vf_stack.o framesync.o OBJS-$(CONFIG_W3FDIF_FILTER) += vf_w3fdif.o OBJS-$(CONFIG_WAVEFORM_FILTER) += vf_waveform.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 4af7a3bbbf..3c1e879806 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -539,6 +539,7 @@ extern const FFFilter ff_vf_vif; extern const FFFilter ff_vf_vignette; extern const FFFilter ff_vf_vmafmotion; extern const FFFilter ff_vf_vpp_qsv; +extern const FFFilter ff_vf_vsr_cuda; extern const FFFilter ff_vf_vstack; extern const FFFilter ff_vf_w3fdif; extern const FFFilter ff_vf_waveform; diff --git a/libavfilter/vf_vsr_cuda.c b/libavfilter/vf_vsr_cuda.c new file mode 100644 index 0000000000..be0e3ae308 --- /dev/null +++ b/libavfilter/vf_vsr_cuda.c @@ -0,0 +1,373 @@ +/* + * Copyright (C) 2026 Philip Langdale <[email protected]> + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +/** + * @file + * Super-resolution filter driving NVIDIA RTX Video Super Resolution. The VSR + * network ships as a "pure CUDA" graph inside nvngx_vsr.dll; the cubins were + * extracted per-architecture and the forward pass reverse-engineered by + * intercepting the live CUDA Driver-API launches. vsr_cuda_gen.h encodes, per + * quality (0-4) and per scaling path, how the whole graph (grids, scratch + * allocations, packed arg-buffer scalars incl. division-magic constants and + * float32 resample steps, weight-upload targets, pointer fixups) scales with the + * input W,H and output oW,oH -- derived and validated byte-exact against the DLL + * (rtx-video-re). The filter evaluates that at config time and then just + * replays the graph with libcuda; no DLL is needed at run time. The replay + * machinery itself is rtx_cuda.c. + * + * Most models perform a fixed internal 2x super-resolution: exact isotropic 2x + * output takes the "fast" path (direct postProcess); any other factor (non-integer, + * downscale, anisotropic) takes the "resample" path (bicubic compose to the + * requested rect). The high/ultra models (quality 3/4) are scale-adaptive: their + * SR block selects an internal pixel_shuffle factor ps = min(ceil(max(oW/W,oH/H)),4) + * and writes natively (postProcess) for an isotropic-exact integer 2x/3x/4x, else + * resamples with that ps. Each (quality, path, factor) is a separate config. + * + * The 'scale' option (q3/q4 only) decouples that internal factor from the output + * size: scale=0 (default) derives ps from the requested ratio (byte-exact with the + * snippet), while scale=2/3/4 forces the pixel_shuffle2/3/4 head and defaults the + * output to scale x input. Setting w/h as well runs the chosen native-Nx network + * and resamples its result to that rect -- e.g. scale=4 with a 1080p output runs + * the 4x network then downscales, independent of the 3x the ratio would imply. + * + * Quality 5=denoise and 6=deblur are native-resolution RESTORATION networks, not + * upscalers: they are forced to 1x (output = input size) because any upscale makes + * them emit a checkerboard / chroma hallucination. Restore with them first, then + * upscale with a separate quality=3/4 pass. (7=lossless is a faithful pass-through + * and is left unconstrained.) + * + * I/O binding mirrors the DLL: the input frame is bound as a bindless TEXTURE + * (UNSIGNED_INT8 x4, clamp/linear/normalized -> samples in [0,1]) and the output as + * a bindless SURFACE over a CUDA array. The captured arg blocks carry the tex/surf + * handles at known offsets, which we patch to our own objects. + * + * The cubins and the shared weights blob are external files (the "data" + * option), as for every filter in this family: they are extracted from the + * proprietary DLL and are not shipped with FFmpeg. Input/output are packed RGB + * CUDA frames: 8-bit rgb0/bgr0/rgba/bgra or 16-bit rgba64le. + */ + +#include "libavutil/hwcontext.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +#include "avfilter.h" +#include "filters.h" +#include "rtx_cuda.h" +#include "video.h" + +/* Generated by rtx-video-re from the proprietary NVIDIA library, and + * installed rather than carried here -- located, together with the cubins and + * weights it names, through pkg-config (see configure's nvfdata_* checks). */ +#include <vsr_cuda_gen.h> + +FF_RTX_ASSERT_MODULE_LAYOUT(VsrModule); +FF_RTX_ASSERT_FUNC_LAYOUT(VsrFunc); +FF_RTX_ASSERT_UPLOAD_LAYOUT(VsrGenUpload); +FF_RTX_ASSERT_LAUNCH_LAYOUT(VsrGenLaunch); + +typedef struct VsrCudaContext { + const AVClass *class; + + FFRtxCuda r; + FFRtxImage *in_img, *out_img; + + int W, H, oW, oH; ///< input / output size + int cfg; ///< index into vsr_configs + + const FFRtxPixFmt *inpf, *outpf; ///< input / output packed-RGB descriptors + + /* options */ + int quality; + int scale; ///< native SR factor for q3/q4: 0=auto(from ratio), 2/3/4=forced + char *w_expr; + char *h_expr; + char *data_dir; + char *out_format; +} VsrCudaContext; + +#define OFFSET(x) offsetof(VsrCudaContext, x) +#define FLAGS (AV_OPT_FLAG_VIDEO_PARAM | AV_OPT_FLAG_FILTERING_PARAM) + +static const AVOption vsr_cuda_options[] = { + { "quality", "VSR quality/model (0=bicubic 1=low 2=medium 3=high 4=ultra; 5=denoise 6=deblur are 1x restoration-only; 7=lossless)", OFFSET(quality), AV_OPT_TYPE_INT, {.i64=1}, 0, 7, FLAGS }, + /* Native integer SR factor for the scale-adaptive high/ultra models (quality + * 3/4); ignored for the fixed-2x models. 0 = auto: derive the internal + * pixel_shuffle from the output/input ratio (ps = min(ceil(max(ratio)),4)), + * matching the snippet. 2/3/4 = force that native pixel_shuffle2/3/4 head, + * independent of the requested output size -- so you can run e.g. the ultra 4x + * network and then resample its result to any 'w'x'h'. */ + { "scale", "native integer SR factor for quality 3/4 (0=auto, 2/3/4=forced)", OFFSET(scale), AV_OPT_TYPE_INT, {.i64=0}, 0, 4, FLAGS }, + /* Output size. Unset (default) = the native scale x input: quality 3/4 with a + * forced 'scale' -> that factor, everything else -> 2x. Set either to any + * expression for arbitrary output; a non-native ratio resamples the native + * result to the requested rect. */ + { "w", "output width expression (default: native scale x input)", OFFSET(w_expr), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, + { "h", "output height expression (default: native scale x input)", OFFSET(h_expr), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, + { "data", "directory with extracted VSR cubins + the shared weights.bin", + OFFSET(data_dir), AV_OPT_TYPE_STRING, {.str=VSR_DEFAULT_DATA_DIR}, 0, 0, FLAGS }, + { "format", "output pixel format (empty = same as input); e.g. bgra, rgba64le", + OFFSET(out_format), AV_OPT_TYPE_STRING, {.str=NULL}, 0, 0, FLAGS }, + { NULL } +}; + +AVFILTER_DEFINE_CLASS(vsr_cuda); + +FF_RTX_ASSERT_PRIV_LAYOUT(VsrCudaContext); + +/* A capture only ever yields the capturing GPU's images, so `rtxv extract vsr` + * replaces each one with the snippet's own fatbin -- the captured image is + * byte-for-byte one of the images inside it, so every architecture the DLL + * ships comes with it, plus the PTX the driver can JIT newer ones from -- and + * fills the conv backbone in from the sibling ELFs. A data dir that was built + * with --no-fatbins still holds bare single-arch cubins, hence this hint. + * There is no arch gate to go with it: unlike the driver plugins, nothing here + * is a statically-matched guess that needs an opt-in. Either the vendor ships + * an image for the GPU or the load fails, and this says which. */ +#define VSR_LOAD_HINT \ + "Re-run `rtxv extract vsr <nvngx_vsr.dll>` and `rtxv install`: the " \ + "generator repacks each kernel as the snippet's own multi-arch fatbin." + +/* ------------------------------------------------------------------------- * + * One-time graph setup for the selected config + W,H,oW,oH (context current). + * ------------------------------------------------------------------------- */ +static void fill_sizes(AVFilterContext *ctx, long long *sz) +{ + VsrCudaContext *s = ctx->priv; + vsr_fill_allocs(s->cfg, s->W, s->H, s->oW, s->oH, sz); +} + +static int setup_graph(AVFilterContext *ctx) +{ + VsrCudaContext *s = ctx->priv; + const VsrConfig *c = &vsr_configs[s->cfg]; + VsrGenUpload *up; + int ret, nup; + + if ((ret = ff_rtx_load_modules(ctx, &s->r, s->data_dir, + (const FFRtxModule *)c->modules, c->nmod, VSR_MAX_MID, + (const FFRtxFunc *)c->funcs, c->nfunc, VSR_MAX_FID, + VSR_LOAD_HINT)) < 0) + return ret; + if ((ret = ff_rtx_alloc_arena(ctx, &s->r, c->nalloc, fill_sizes, 0)) < 0) + return ret; + + up = av_calloc(c->nupload, sizeof(*up)); + if (!up) + return AVERROR(ENOMEM); + nup = vsr_fill_uploads(s->cfg, s->W, s->H, s->oW, s->oH, + (const vsr_devptr *)s->r.alloc, up); + ret = ff_rtx_upload_weights(ctx, &s->r, s->data_dir, "weights.bin", + (const FFRtxUpload *)up, nup); + av_freep(&up); + if (ret < 0) + return ret; + + /* Input array + texture: VSR reads the input frame through a texture with + * linear/normalized/clamp sampling (exactly the replay path). The frame is + * copied into this array each frame; the texture handle is patched once. + * The output is the SUST.P target. */ + s->in_img = ff_rtx_image_array(ctx, &s->r, s->W, s->H, s->inpf->cufmt, + FF_RTX_TEX | FF_RTX_CLAMP); + s->out_img = ff_rtx_image_array(ctx, &s->r, s->oW, s->oH, s->outpf->cufmt, + FF_RTX_SURF | FF_RTX_LDST); + if (!s->in_img || !s->out_img) + return AVERROR_EXTERNAL; + + /* Build the graph. vsr_fill_graph() is generated from the same fit as the + * tables above and assigns every field through its named vsr_*_params + * struct, so the argument blocks are constructed rather than patched. Every + * VSR launch stays <= 45568 B dynamic shared, i.e. under the 48 KiB default + * cap, so no MAX_DYNAMIC_SHARED_SIZE_BYTES opt-in is needed. The casts are + * only `unsigned long long *` vs `uint64_t *` on LP64. */ + if ((ret = ff_rtx_alloc_launches(ctx, &s->r, c->nlaunch, sizeof(VsrGenLaunch))) < 0) + return ret; + if (vsr_fill_graph(s->cfg, s->W, s->H, s->oW, s->oH, + (const vsr_devptr *)s->r.alloc, + (vsr_devptr)s->in_img->tex, (vsr_devptr)s->out_img->surf, + s->r.launches) != c->nlaunch) { + av_log(ctx, AV_LOG_ERROR, "generated fill disagrees with the config tables\n"); + return AVERROR_BUG; + } + + /* Format selectors, at the launch and offset the generator recorded. The + * capture leaves both at 0 (raw RGB8), so they are only patched when the + * chosen format is not sel 0. */ + if (s->outpf->sel) { + uint32_t sel = s->outpf->sel; + if (c->sel_launch < 0) { + av_log(ctx, AV_LOG_ERROR, "no store selector for config %s\n", c->tag); + return AVERROR_BUG; + } + memcpy(ff_rtx_launch_at(&s->r, c->sel_launch)->params + c->sel_off, &sel, 4); + } + if (s->inpf->sel) { + uint32_t sel = s->inpf->sel; + if (c->in_sel_launch < 0) { + av_log(ctx, AV_LOG_ERROR, "no input selector for config %s\n", c->tag); + return AVERROR_BUG; + } + memcpy(ff_rtx_launch_at(&s->r, c->in_sel_launch)->params + c->in_sel_off, &sel, 4); + } + + av_log(ctx, AV_LOG_INFO, + "VSR graph ready: %s %dx%d -> %dx%d (%d launches, %d buffers)\n", + c->tag, s->W, s->H, s->oW, s->oH, s->r.nlaunch, s->r.nalloc); + return 0; +} + +/* ------------------------------------------------------------------------- * + * Per-frame: bind the input frame as a texture, replay the graph, copy the + * output surface into the work frame. + * ------------------------------------------------------------------------- */ +static int filter_frame(AVFilterLink *inlink, AVFrame *in) +{ + VsrCudaContext *s = inlink->dst->priv; + /* NGX VSR launches with the full captured arg block, so no FF_RTX_OP_PSIZE. */ + const FFRtxFrameOp op = { + .in_img = s->in_img, .iW = s->W, .iH = s->H, .ibpp = s->inpf->bpp, + .out_img = s->out_img, .oW = s->oW, .oH = s->oH, .obpp = s->outpf->bpp, + .flags = s->outpf->sel == 2 ? FF_RTX_OP_OPAQUE_ALPHA : 0, + }; + + return ff_rtx_filter_frame(inlink, in, &s->r, &op, NULL); +} + +static int config_output(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + AVFilterLink *inlink = ctx->inputs[0]; + VsrCudaContext *s = ctx->priv; + AVHWFramesContext *in_frames_ctx; + FFRtxFormats fmts = { + .in_tbl = ff_rtx_packed_rgb_fmts, .n_in = FF_ARRAY_ELEMS(ff_rtx_packed_rgb_fmts), + .out_tbl = ff_rtx_packed_rgb_fmts, .n_out = FF_ARRAY_ELEMS(ff_rtx_packed_rgb_fmts), + }; + int adaptive, restoration, forced, fast, ret; + + /* This can run again on a link reconfigure or a graph rebuild; drop the + * previous graph first (freed against the context it was built on) so the + * rebuild neither leaks nor inherits stale device pointers. */ + ff_rtx_free_graph(ctx, &s->r); + + fmts.out_format = s->out_format; + if ((ret = ff_rtx_config_formats(ctx, inlink, &fmts, &in_frames_ctx, + &s->inpf, &s->outpf)) < 0) + return ret; + + s->W = inlink->w; + s->H = inlink->h; + + /* Native SR factor. Only high/ultra (q3/q4) are scale-adaptive; the fixed-2x + * models ignore 'scale'. A forced scale (2/3/4) both sets the default output + * (scale x input) and selects the pixel_shuffleN head independent of the output + * size; scale 0 = auto (derive the head from the requested ratio). */ + adaptive = (s->quality == 3 || s->quality == 4); + restoration = (s->quality == 5 || s->quality == 6); /* denoise/deblur: 1x only */ + if (s->scale == 1) { + av_log(ctx, AV_LOG_ERROR, "scale must be 0 (auto) or 2/3/4\n"); + return AVERROR(EINVAL); + } + forced = adaptive ? s->scale : 0; /* forced native factor, 0 = auto */ + if (restoration) { + /* q5=denoise / q6=deblur are native-resolution RESTORATION nets, not + * upscalers: they only produce correct output at 1x (output = input). Any + * upscale yields a period-2 checkerboard (exact-2x path) or gross chroma + * hallucination (~20 dB PSNR even on clean input) -- see rtx-video-re + * docs/output-quality.md. Force 1x and warn if the user asked for scaling; + * upscale with a separate quality=3/4 pass after restoring. */ + if (s->scale || (s->w_expr && *s->w_expr) || (s->h_expr && *s->h_expr)) + av_log(ctx, AV_LOG_WARNING, + "quality %d (%s) is a 1x restoration model; ignoring the requested " + "scale/w/h and forcing output = input size (%dx%d). Upscale with a " + "separate quality=3/4 pass after restoring.\n", + s->quality, s->quality == 5 ? "denoise" : "deblur", s->W, s->H); + s->oW = s->W; + s->oH = s->H; + } else { + if (s->scale && !adaptive) /* q0/q1/q2/q7: fixed internal 2x, scale is moot */ + av_log(ctx, AV_LOG_WARNING, + "scale=%d ignored: quality %d is fixed internal-2x\n", s->scale, s->quality); + if ((ret = ff_rtx_eval_dims(ctx, inlink, s->w_expr, s->h_expr, + forced ? forced : 2, &s->oW, &s->oH)) < 0) + return ret; + } + + /* Scaling-path + native-factor selection. q3/q4 pick an internal pixel_shuffle + * factor and write with the native postProcess kernel for an ISOTROPIC EXACT + * integer 2x/3x/4x, else resample with that same factor. With a forced 'scale' + * we bake that factor and resample its native-Nx result to any other output; + * with scale=0 (auto) we mirror the snippet, ps = min(ceil(max(ratio)),4). Do + * NOT force ps2, or a 3x/4x request would needlessly upscale a 2x image. Fixed + * models pass factor 0 -> vsr_config_index matches their single config. */ + if (adaptive) { + int k; + if (forced) { + k = forced; /* explicit pixel_shuffleN head */ + fast = (s->oW == k * s->W && s->oH == k * s->H); + } else { + int kw = (s->oW + s->W - 1) / s->W; /* ceil(oW/W) */ + int kh = (s->oH + s->H - 1) / s->H; /* ceil(oH/H) */ + k = FFMAX(kw, kh); + if (k < 2) k = 2; else if (k > 4) k = 4; /* native ps ceiling is 4 */ + fast = (s->oW % s->W == 0 && s->oH % s->H == 0 && + s->oW / s->W == s->oH / s->H && + s->oW / s->W >= 2 && s->oW / s->W <= 4); + if (fast) k = s->oW / s->W; /* exact isotropic integer 2x/3x/4x */ + } + s->cfg = vsr_config_index(s->quality, fast ? 0 : 1, k); + } else { + fast = (s->oW == 2 * s->W && s->oH == 2 * s->H); + s->cfg = vsr_config_index(s->quality, fast ? 0 : 1, 0); + } + if (s->cfg < 0) { + av_log(ctx, AV_LOG_ERROR, "no config for quality %d %s path\n", + s->quality, fast ? "fast" : "resample"); + return AVERROR(ENOSYS); + } + + if ((ret = ff_rtx_bind_device(ctx, &s->r, in_frames_ctx)) < 0) + return ret; + if ((ret = ff_rtx_config_hwframes(ctx, outlink, &s->r, s->oW, s->oH, + s->outpf->f)) < 0) + return ret; + return ff_rtx_setup(ctx, &s->r, "VSR", setup_graph); +} + +static const AVFilterPad vsr_cuda_inputs[] = { + { .name = "default", .type = AVMEDIA_TYPE_VIDEO, .filter_frame = filter_frame }, +}; + +static const AVFilterPad vsr_cuda_outputs[] = { + { .name = "default", .type = AVMEDIA_TYPE_VIDEO, .config_props = config_output }, +}; + +const FFFilter ff_vf_vsr_cuda = { + .p.name = "vsr_cuda", + .p.description = NULL_IF_CONFIG_SMALL("NVIDIA RTX Video Super Resolution (CUDA)"), + .p.priv_class = &vsr_cuda_class, + .priv_size = sizeof(VsrCudaContext), + .uninit = ff_rtx_uninit, + FILTER_INPUTS(vsr_cuda_inputs), + FILTER_OUTPUTS(vsr_cuda_outputs), + FILTER_SINGLE_PIXFMT(AV_PIX_FMT_CUDA), + .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, +}; -- To stop receiving notification emails like this one, please contact [email protected]. _______________________________________________ ffmpeg-cvslog mailing list -- [email protected] To unsubscribe send an email to [email protected]
