PR #24862 opened by Leo Bicknell (bicknell)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24862
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24862.patch
# Summary of changes
macOS 26 and 27 added rate control properties to VideoToolbox which the
encoder wrapper cannot reach today. This series exposes them for
h264_videotoolbox and hevc_videotoolbox:
1. variable_bit_rate: kVTCompressionPropertyKey_VariableBitRate, with
the VBV (VBVMaxBitRate, VBVBufferDuration, VBVInitialDelayPercentage)
configured from the generic maxrate, bufsize and rc_init_occupancy
options. macOS 26.
2. preset: applies one of the property dictionaries the encoder reports
through kVTCompressionPropertyKey_SupportedPresetDictionaries
(high_quality, balanced, high_speed, consistent_quality). macOS 26,
consistent_quality macOS 27.
3. constant_quality_factor: kVTCompressionPropertyKey_ConstantQualityFactor.
macOS 27, and only effective together with the consistent_quality
preset.
The motivation is the last one. -q:v maps to
kVTCompressionPropertyKey_Quality, which makes the hardware encoders use
a fixed QP with only a few distinct levels: on an M5 Max, -q:v 51 and 52
give identical output, while 50 and 53 are 20% smaller and 46% larger.
The constant quality factor is continuous and adapts to the content.
24 seconds of 3840x2160 30 fps 10 bit video, hardware HEVC encoder,
M5 Max, macOS 27.0.1, with
-preset consistent_quality -maxrate 100M -bf 2 -g 60 -flags +cgop
for the factor rows and the same -bf, -g and -flags for the -q:v rows:
Setting Bitrate VMAF
constant_quality_factor 0.58 25.4 Mb/s 74.78
constant_quality_factor 0.60 32.8 Mb/s
constant_quality_factor 0.62 43.5 Mb/s 77.47
constant_quality_factor 0.64 57.4 Mb/s
constant_quality_factor 0.66 74.1 Mb/s 79.71
-q:v 50 32.8 Mb/s
-q:v 51 and 52 40.6 Mb/s 77.50
-q:v 53 59.2 Mb/s
Sizes and VMAF match a small native Swift program setting the same
properties directly. The factor is a true float, and relatively small
steps of 0.001 are useful to control the output.
Across five clips from different recordings, a single factor of 0.62
stayed within 0.7 VMAF of libx265 -preset fast -crf 23, while a single
-q:v 51 ranged from equal to 3.4 VMAF below it.
The new rate control modes are slower on this hardware: the clip above
encodes at about 69 fps with variable_bit_rate or the constant quality
factor, against about 128 fps with -q:v or a plain -b:v (wall time,
including decoding the FFV1 source). variable_bit_rate without a preset
is just as slow, so this is not caused by the lookahead of the presets.
All new keys are looked up at runtime like the existing ones, so the
series builds with older SDKs. When the encoder does not support a
property, for example on an older system or with the software encoders,
the option should fail with an error instead of silently falling back.
# Testing
- macOS 27.0.1, Apple M5 Max, hardware H.264 and HEVC encoders.
- Output of the existing modes (-q:v, -b:v, -b:v with -maxrate) and of
prores_videotoolbox is unchanged compared to master.
- Each commit builds on its own, and make fate passes on the series.
- The software encoders (-require_sw 1) reject all three new options
with an error.
- The Forgejo CI has no macOS runner, so it does not build this code.
- Not tested: macOS 15 and 26, Intel Macs, iOS and tvOS. I have no such
systems; reports are welcome.
# Questions for reviewers
I would like guidance on these before polishing further.
1. constant_quality_factor requires the preset: fail, or apply it
implicitly?
The current patch implements the fail option.
With the hardware H.264 and HEVC encoders of macOS 27.0 the factor is
silently ignored unless the consistent_quality preset is applied. The
preset sets a lookahead property which has no public key; setting
VBVMaxBitRate alone is not enough. This is not documented, and the
header says the factor works "with or without bitrate limit
constraints". The software encoders do not support the property at
all, so I know of no encoder which honours the factor on its own, and
the series fails when the factor is given without the preset. The
alternatives are to apply the preset implicitly when the factor is
given, which also brings the preset's 30 Mb/s VBV cap, or to only warn
in case a later release changes this. Which do you prefer?
2. Presets and ffmpeg's defaults.
The current patch seems closest to the existing ffmpeg option behavior.
It also results in less code by omitting the "unset" code. However, it
requires the user to supply additional ffmpeg options to get the macOS
preset default behaviors.
The preset is applied first and every other option afterwards. That
includes defaults the user did not set: with the default -bf 0 the
wrapper sets AllowFrameReordering to false, overriding the presets
which turn it on. The series documents that B-frames still need -bf,
which matches how this encoder behaves today.
It is also worth noting that no preset sets a keyframe interval and
ffmpeg defaults to -g 12. For the user to get the actual Apple preset
behavior they must include -g 0.
Would you rather the -bf and -g defaults were not applied when a preset
is used? That needs code to "unset" defaults for this encoder.
3. -b:v with a preset.
For the presets built on variable bitrate (high_quality, balanced),
-b:v replaces the preset's VariableBitRate target and -maxrate
configures the VBV, while without a preset -b:v means AverageBitRate
and -maxrate means DataRateLimits. Is following the preset's rate
control acceptable, or should this be explicit?
4. Option naming.
variable_bit_rate follows the existing constant_bit_rate. A separate
constant_quality_factor was chosen over reusing global_quality because
the scales differ and the behaviour of -q:v has to stay.
# FATE samples
None. There are no FATE tests for the VideoToolbox encoders.
# Background
The goal was to optimize hardware encoding on a Mac Studio running an M5 Max
and macOS 27. Initial testing showed the `-q:v` parameter on this hardware
did not perform as expected, and was not terribly useful for controlling output
quality.
I used an AI agent to write a Swift program to probe all available options in
the library. Over 100 undocumented properties were discovered. This series
uses none of them directly; it only uses documented public keys. I then had
the Agent research those options to see if any were publicly discussed by Apple
or discussed in community forums. Many of these options do not apply to
general purpose video encoding, for example an entire block of them is to
support FaceTime video conferencing. If anyone is interested in the full
list I am happy to share.
This series adds a continuous, content-adaptive quality setting to the
VideoToolbox H.264 and HEVC encoders (macOS 27's ConstantQualityFactor):
a knob that behaves like x265's CRF, as an alternative to -q:v's handful
of fixed-QP steps. It also exposes Apple's encoder presets, which that setting
depends on, and macOS 26's variable bitrate mode with a VBV.
Using these new options I have performed over 150 test encodes of existing
footage. I had the AI agent run sweeps of the various options and calculate
VMAF scores. I also visually inspected many clips to provide a human eye
comparison to the VMAF scores. The Apple Presets exposed here provide
obvious starting points, but I probed options around them looking at both
the effect on wall clock encoding time and output quality.
Ultimately these options dramatically reduced encode time compared to a
software-only implementation on this hardware and expose the options needed
for fine grained control of output quality.
# Disclosure
This patch was largely researched and developed using AI with careful
human curation and review at each step. AI was also used to develop
extensive tests generating over 150 test encodes to both validate the patch
and to zero in on the useful setting ranges. I have reviewed every change
myself, built and tested the series, and used it locally before submitting.
>From 8977914d3fd162261e2a3ab1a2f4d3855dfb01ea Mon Sep 17 00:00:00 2001
From: Leo Bicknell <[email protected]>
Date: Thu, 1 Oct 2026 18:04:14 -0400
Subject: [PATCH 1/3] avcodec/videotoolboxenc: add variable bitrate mode from
macOS 26
macOS 26 added kVTCompressionPropertyKey_VariableBitRate together with
a video buffering verifier model configured through VBVMaxBitRate,
VBVBufferDuration and VBVInitialDelayPercentage.
Add a variable_bit_rate option, next to constant_bit_rate, which selects
this rate control for H.264 and HEVC. The target is taken from the
bitrate, and the VBV is configured from the generic options: maxrate
sets VBVMaxBitRate, bufsize divided by maxrate gives VBVBufferDuration
in seconds, and rc_init_occupancy as a share of bufsize gives
VBVInitialDelayPercentage. DataRateLimits is not set in this mode, since
VideoToolbox documents it as incompatible with the new properties.
The mode requires a bitrate and cannot be combined with
constant_bit_rate, qscale, global_quality or the low_delay flag.
The keys are looked up at runtime, so this builds with older SDKs.
Requesting the mode from an encoder which does not support it, such as
the software encoders or a system older than macOS 26, is an error
rather than a silent fallback to average bitrate.
Signed-off-by: Leo Bicknell <[email protected]>
---
libavcodec/videotoolboxenc.c | 142 ++++++++++++++++++++++++++++++++++-
1 file changed, 141 insertions(+), 1 deletion(-)
diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index 7dc8c2ac75..d83a0b0813 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -138,6 +138,10 @@ static struct{
CFStringRef kVTCompressionPropertyKey_ReferenceBufferCount;
CFStringRef kVTCompressionPropertyKey_MaxAllowedFrameQP;
CFStringRef kVTCompressionPropertyKey_MinAllowedFrameQP;
+ CFStringRef kVTCompressionPropertyKey_VariableBitRate;
+ CFStringRef kVTCompressionPropertyKey_VBVMaxBitRate;
+ CFStringRef kVTCompressionPropertyKey_VBVBufferDuration;
+ CFStringRef kVTCompressionPropertyKey_VBVInitialDelayPercentage;
getParameterSetAtIndex CMVideoFormatDescriptionGetHEVCParameterSetAtIndex;
} compat_keys;
@@ -218,6 +222,11 @@ static void loadVTEncSymbols(void){
GET_SYM(kVTCompressionPropertyKey_MaxAllowedFrameQP, "MaxAllowedFrameQP");
GET_SYM(kVTCompressionPropertyKey_MinAllowedFrameQP, "MinAllowedFrameQP");
GET_SYM(kVTCompressionPropertyKey_SpatialAdaptiveQPLevel,
"SpatialAdaptiveQPLevel");
+ GET_SYM(kVTCompressionPropertyKey_VariableBitRate, "VariableBitRate");
+ GET_SYM(kVTCompressionPropertyKey_VBVMaxBitRate, "VBVMaxBitRate");
+ GET_SYM(kVTCompressionPropertyKey_VBVBufferDuration, "VBVBufferDuration");
+ GET_SYM(kVTCompressionPropertyKey_VBVInitialDelayPercentage,
+ "VBVInitialDelayPercentage");
}
#define H264_PROFILE_CONSTRAINED_HIGH (AV_PROFILE_H264_HIGH |
AV_PROFILE_H264_CONSTRAINED)
@@ -273,6 +282,7 @@ typedef struct VTEncContext {
int frames_before;
int frames_after;
int constant_bit_rate;
+ int variable_bit_rate;
int allow_sw;
int require_sw;
@@ -1170,6 +1180,92 @@ static int
set_encoder_int_property_or_log(AVCodecContext* avctx,
return 0;
}
+/**
+ * Set a numeric rate control property which only exists on recent OS
+ * releases. Unlike set_encoder_property_or_log(), failure is fatal, since
+ * silently falling back to another rate control mode is not what the user
+ * asked for.
+ */
+static int set_encoder_rc_property(AVCodecContext *avctx,
+ CFStringRef key,
+ const char *print_option_name,
+ const char *min_os_version,
+ CFNumberType type,
+ const void *value)
+{
+ VTEncContext *vtctx = avctx->priv_data;
+ CFNumberRef num = CFNumberCreate(kCFAllocatorDefault, type, value);
+ int status;
+
+ if (!num)
+ return AVERROR(ENOMEM);
+
+ status = VTSessionSetProperty(vtctx->session, key, num);
+ CFRelease(num);
+
+ if (status == kVTPropertyNotSupportedErr) {
+ av_log(avctx, AV_LOG_ERROR,
+ "%s is not supported by this encoder (needs macOS %s or newer,
not available with the software encoder).\n",
+ print_option_name, min_os_version);
+ return AVERROR_EXTERNAL;
+ } else if (status) {
+ av_log(avctx, AV_LOG_ERROR, "Error setting %s: %d\n",
+ print_option_name, status);
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
+/**
+ * Configure the video buffering verifier used by the variable bitrate rate
+ * control from the generic maxrate, bufsize and rc_init_occupancy options.
+ */
+static int vtenc_set_vbv_properties(AVCodecContext *avctx)
+{
+ int64_t max_rate = avctx->rc_max_rate;
+ int status;
+
+ if (max_rate <= 0) {
+ if (avctx->rc_buffer_size > 0)
+ av_log(avctx, AV_LOG_WARNING,
+ "bufsize is ignored without maxrate.\n");
+ return 0;
+ }
+
+ status = set_encoder_rc_property(avctx,
+
compat_keys.kVTCompressionPropertyKey_VBVMaxBitRate,
+ "maxrate (VBVMaxBitRate)", "26",
+ kCFNumberSInt64Type, &max_rate);
+ if (status < 0)
+ return status;
+
+ if (avctx->rc_buffer_size > 0) {
+ Float32 duration = (Float32)avctx->rc_buffer_size / max_rate;
+
+ status = set_encoder_rc_property(avctx,
+
compat_keys.kVTCompressionPropertyKey_VBVBufferDuration,
+ "bufsize (VBVBufferDuration)", "26",
+ kCFNumberFloat32Type, &duration);
+ if (status < 0)
+ return status;
+
+ if (avctx->rc_initial_buffer_occupancy > 0) {
+ Float32 percentage = fminf(100.0f *
avctx->rc_initial_buffer_occupancy /
+ avctx->rc_buffer_size, 100.0f);
+
+ status = set_encoder_rc_property(avctx,
+
compat_keys.kVTCompressionPropertyKey_VBVInitialDelayPercentage,
+ "rc_init_occupancy
(VBVInitialDelayPercentage)", "26",
+ kCFNumberFloat32Type,
&percentage);
+ if (status < 0)
+ return status;
+ }
+ }
+
+ return 0;
+}
+
static int vtenc_create_encoder(AVCodecContext *avctx,
CMVideoCodecType codec_type,
CFStringRef profile_level,
@@ -1270,6 +1366,16 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
CFRelease(bit_rate_num);
return AVERROR_EXTERNAL;
}
+ } else if (vtctx->variable_bit_rate) {
+ status = VTSessionSetProperty(vtctx->session,
+
compat_keys.kVTCompressionPropertyKey_VariableBitRate,
+ bit_rate_num);
+ if (status == kVTPropertyNotSupportedErr) {
+ av_log(avctx, AV_LOG_ERROR, "Variable bit rate is not
supported by this encoder "
+ "(needs macOS 26 or newer, not
available with the software encoder).\n");
+ CFRelease(bit_rate_num);
+ return AVERROR_EXTERNAL;
+ }
} else {
status = VTSessionSetProperty(vtctx->session,
kVTCompressionPropertyKey_AverageBitRate,
@@ -1293,7 +1399,13 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
}
}
- if ((vtctx->codec_id == AV_CODEC_ID_H264 || vtctx->codec_id ==
AV_CODEC_ID_HEVC)
+ if (vtctx->variable_bit_rate) {
+ // kVTCompressionPropertyKey_DataRateLimits is not compatible with
+ // variable bitrate rate control, which has its own VBV model.
+ status = vtenc_set_vbv_properties(avctx);
+ if (status < 0)
+ return status;
+ } else if ((vtctx->codec_id == AV_CODEC_ID_H264 || vtctx->codec_id ==
AV_CODEC_ID_HEVC)
&& max_rate > 0) {
bytes_per_second_value = max_rate >> 3;
bytes_per_second = CFNumberCreate(kCFAllocatorDefault,
@@ -1656,6 +1768,29 @@ static int vtenc_configure_encoder(AVCodecContext *avctx)
vtctx->codec_id = avctx->codec_id;
+ if (vtctx->variable_bit_rate) {
+ if (vtctx->constant_bit_rate) {
+ av_log(avctx, AV_LOG_ERROR, "-variable_bit_rate and
-constant_bit_rate "
+ "are mutually exclusive\n");
+ return AVERROR(EINVAL);
+ }
+ if (avctx->flags & AV_CODEC_FLAG_QSCALE || avctx->global_quality > 0) {
+ av_log(avctx, AV_LOG_ERROR, "-variable_bit_rate cannot be used "
+ "together with a quality target\n");
+ return AVERROR(EINVAL);
+ }
+ if (avctx->flags & AV_CODEC_FLAG_LOW_DELAY) {
+ av_log(avctx, AV_LOG_ERROR, "-variable_bit_rate cannot be used "
+ "in low_delay mode\n");
+ return AVERROR(EINVAL);
+ }
+ if (avctx->bit_rate <= 0) {
+ av_log(avctx, AV_LOG_ERROR, "-variable_bit_rate requires a
bitrate, "
+ "please specify bitrate explicitly\n");
+ return AVERROR(EINVAL);
+ }
+ }
+
if (vtctx->codec_id == AV_CODEC_ID_H264) {
vtctx->get_param_set_func =
CMVideoFormatDescriptionGetH264ParameterSetAtIndex;
@@ -2797,6 +2932,9 @@ static const enum AVPixelFormat prores_pix_fmts[] = {
{ "max_ref_frames", \
"Sets the maximum number of reference frames. This only has an effect
when the value is less than the maximum allowed by the profile/level.", \
OFFSET(max_ref_frames), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, VE
},
+#define RC_OPTIONS \
+ { "variable_bit_rate", "Require variable bit rate, with maxrate, bufsize
and rc_init_occupancy configuring the VBV (macOS 26 or newer)", \
+ OFFSET(variable_bit_rate), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE },
static const AVCodecHWConfigInternal *const vt_encode_hw_configs[] = {
HW_CONFIG_ENCODER_FRAMES(VIDEOTOOLBOX, VIDEOTOOLBOX),
@@ -2834,6 +2972,7 @@ static const AVOption h264_options[] = {
{ "a53cc", "Use A53 Closed Captions (if available)", OFFSET(a53_cc),
AV_OPT_TYPE_BOOL, {.i64 = 1}, 0, 1, VE },
{ "constant_bit_rate", "Require constant bit rate (macOS 13 or newer)",
OFFSET(constant_bit_rate), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE },
+ RC_OPTIONS
{ "max_slice_bytes", "Set the maximum number of bytes in an H.264 slice.",
OFFSET(max_slice_bytes), AV_OPT_TYPE_INT, { .i64 = -1 }, -1, INT_MAX, VE },
COMMON_OPTIONS
{ NULL },
@@ -2882,6 +3021,7 @@ static const AVOption hevc_options[] = {
{ "alpha_quality", "Compression quality for the alpha channel",
OFFSET(alpha_quality), AV_OPT_TYPE_DOUBLE, { .dbl = 0.0 }, 0.0, 1.0, VE },
{ "constant_bit_rate", "Require constant bit rate (macOS 13 or newer)",
OFFSET(constant_bit_rate), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE },
+ RC_OPTIONS
COMMON_OPTIONS
{ NULL },
--
2.52.0
>From 5df187e0fa73ea42a158ad453c74da345c1c266d Mon Sep 17 00:00:00 2001
From: Leo Bicknell <[email protected]>
Date: Thu, 1 Oct 2026 18:11:50 -0400
Subject: [PATCH 2/3] avcodec/videotoolboxenc: expose macOS 26/27 presets
Since macOS 26 an encoder reports named compression presets through
kVTCompressionPropertyKey_SupportedPresetDictionaries. Each preset is a
dictionary of properties tuned by Apple, including properties which have
no public key of their own, such as the number of lookahead frames.
Add a preset option for H.264 and HEVC which applies one of these
dictionaries right after the session is created. All other options are
applied afterwards, so they override the values of the preset. In
particular B-frames still have to be enabled with bf, as for every other
mode of this encoder.
The preset selects the rate control, so it cannot be combined with
constant_bit_rate, variable_bit_rate, qscale or global_quality. The
target bitrate of a preset is only replaced when a bitrate is given, and
the VBV of the presets using variable bitrate or constant quality rate
control is configured through maxrate, bufsize and rc_init_occupancy.
The consistent_quality preset has no bitrate target, so it cannot be
combined with a bitrate. It requires macOS 27.
The VideoConferencing preset is not exposed, since it requires low
latency rate control, which is selected with the low_delay flag and is
not compatible with the properties above. The software encoders report
no presets.
Signed-off-by: Leo Bicknell <[email protected]>
---
libavcodec/videotoolboxenc.c | 136 +++++++++++++++++++++++++++++++++--
1 file changed, 132 insertions(+), 4 deletions(-)
diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index d83a0b0813..23f2a09032 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -142,6 +142,11 @@ static struct{
CFStringRef kVTCompressionPropertyKey_VBVMaxBitRate;
CFStringRef kVTCompressionPropertyKey_VBVBufferDuration;
CFStringRef kVTCompressionPropertyKey_VBVInitialDelayPercentage;
+ CFStringRef kVTCompressionPropertyKey_SupportedPresetDictionaries;
+ CFStringRef kVTCompressionPreset_HighQuality;
+ CFStringRef kVTCompressionPreset_Balanced;
+ CFStringRef kVTCompressionPreset_HighSpeed;
+ CFStringRef kVTCompressionPreset_ConsistentQuality;
getParameterSetAtIndex CMVideoFormatDescriptionGetHEVCParameterSetAtIndex;
} compat_keys;
@@ -227,6 +232,12 @@ static void loadVTEncSymbols(void){
GET_SYM(kVTCompressionPropertyKey_VBVBufferDuration, "VBVBufferDuration");
GET_SYM(kVTCompressionPropertyKey_VBVInitialDelayPercentage,
"VBVInitialDelayPercentage");
+ GET_SYM(kVTCompressionPropertyKey_SupportedPresetDictionaries,
+ "SupportedPresetDictionaries");
+ GET_SYM(kVTCompressionPreset_HighQuality, "HighQuality");
+ GET_SYM(kVTCompressionPreset_Balanced, "Balanced");
+ GET_SYM(kVTCompressionPreset_HighSpeed, "HighSpeed");
+ GET_SYM(kVTCompressionPreset_ConsistentQuality, "ConsistentQuality");
}
#define H264_PROFILE_CONSTRAINED_HIGH (AV_PROFILE_H264_HIGH |
AV_PROFILE_H264_CONSTRAINED)
@@ -237,6 +248,14 @@ typedef enum VTH264Entropy{
VT_CABAC
} VTH264Entropy;
+typedef enum VTPreset {
+ VT_PRESET_NONE,
+ VT_PRESET_HIGH_QUALITY,
+ VT_PRESET_BALANCED,
+ VT_PRESET_HIGH_SPEED,
+ VT_PRESET_CONSISTENT_QUALITY,
+} VTPreset;
+
static const uint8_t start_code[] = { 0, 0, 0, 1 };
typedef struct ExtraSEI {
@@ -283,6 +302,8 @@ typedef struct VTEncContext {
int frames_after;
int constant_bit_rate;
int variable_bit_rate;
+ int preset;
+ bool preset_vbr;
int allow_sw;
int require_sw;
@@ -1266,6 +1287,71 @@ static int vtenc_set_vbv_properties(AVCodecContext
*avctx)
return 0;
}
+/**
+ * Apply one of the compression presets the encoder reports. Options set
+ * afterwards override the values of the preset.
+ */
+static int vtenc_apply_preset(AVCodecContext *avctx)
+{
+ VTEncContext *vtctx = avctx->priv_data;
+ CFDictionaryRef presets = NULL;
+ CFDictionaryRef props;
+ CFStringRef key;
+ int status;
+
+ switch (vtctx->preset) {
+ case VT_PRESET_HIGH_QUALITY:
+ key = compat_keys.kVTCompressionPreset_HighQuality;
+ break;
+ case VT_PRESET_BALANCED:
+ key = compat_keys.kVTCompressionPreset_Balanced;
+ break;
+ case VT_PRESET_HIGH_SPEED:
+ key = compat_keys.kVTCompressionPreset_HighSpeed;
+ break;
+ case VT_PRESET_CONSISTENT_QUALITY:
+ key = compat_keys.kVTCompressionPreset_ConsistentQuality;
+ break;
+ default:
+ return AVERROR_BUG;
+ }
+
+ status = VTSessionCopyProperty(vtctx->session,
+
compat_keys.kVTCompressionPropertyKey_SupportedPresetDictionaries,
+ kCFAllocatorDefault,
+ &presets);
+ if (status || !presets ||
+ CFGetTypeID(presets) != CFDictionaryGetTypeID()) {
+ av_log(avctx, AV_LOG_ERROR,
+ "Presets are not supported by this encoder (needs macOS 26 or
newer, not available with the software encoder).\n");
+ if (presets)
+ CFRelease(presets);
+ return AVERROR_EXTERNAL;
+ }
+
+ props = CFDictionaryGetValue(presets, key);
+ if (!props || CFGetTypeID(props) != CFDictionaryGetTypeID()) {
+ av_log(avctx, AV_LOG_ERROR,
+ "The requested preset is not supported by the encoder%s.\n",
+ vtctx->preset == VT_PRESET_CONSISTENT_QUALITY ?
+ " (requires macOS 27 or newer)" : "");
+ CFRelease(presets);
+ return AVERROR_EXTERNAL;
+ }
+
+ vtctx->preset_vbr = CFDictionaryContainsKey(props,
+
compat_keys.kVTCompressionPropertyKey_VariableBitRate);
+
+ status = VTSessionSetProperties(vtctx->session, props);
+ CFRelease(presets);
+ if (status) {
+ av_log(avctx, AV_LOG_ERROR, "Error applying preset: %d\n", status);
+ return AVERROR_EXTERNAL;
+ }
+
+ return 0;
+}
+
static int vtenc_create_encoder(AVCodecContext *avctx,
CMVideoCodecType codec_type,
CFStringRef profile_level,
@@ -1333,6 +1419,12 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
if (status < 0)
return status;
+ if (vtctx->preset != VT_PRESET_NONE) {
+ status = vtenc_apply_preset(avctx);
+ if (status < 0)
+ return status;
+ }
+
if (avctx->flags & AV_CODEC_FLAG_QSCALE && !vtenc_qscale_enabled()) {
av_log(avctx, AV_LOG_ERROR, "-q:v qscale not available for encoder.
Use -b:v bitrate instead.\n");
return AVERROR_EXTERNAL;
@@ -1351,7 +1443,10 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
kVTCompressionPropertyKey_Quality,
quality_num);
CFRelease(quality_num);
- } else if (avctx->codec_id != AV_CODEC_ID_PRORES) {
+ } else if (avctx->codec_id != AV_CODEC_ID_PRORES &&
+ (vtctx->preset == VT_PRESET_NONE || bit_rate > 0)) {
+ // A preset comes with its own target, which is kept unless a
+ // bitrate is requested explicitly.
bit_rate_num = CFNumberCreate(kCFAllocatorDefault,
kCFNumberSInt32Type,
&bit_rate);
@@ -1366,7 +1461,7 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
CFRelease(bit_rate_num);
return AVERROR_EXTERNAL;
}
- } else if (vtctx->variable_bit_rate) {
+ } else if (vtctx->variable_bit_rate || vtctx->preset_vbr) {
status = VTSessionSetProperty(vtctx->session,
compat_keys.kVTCompressionPropertyKey_VariableBitRate,
bit_rate_num);
@@ -1399,7 +1494,8 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
}
}
- if (vtctx->variable_bit_rate) {
+ if (vtctx->variable_bit_rate || vtctx->preset_vbr ||
+ vtctx->preset == VT_PRESET_CONSISTENT_QUALITY) {
// kVTCompressionPropertyKey_DataRateLimits is not compatible with
// variable bitrate rate control, which has its own VBV model.
status = vtenc_set_vbv_properties(avctx);
@@ -1768,6 +1864,31 @@ static int vtenc_configure_encoder(AVCodecContext *avctx)
vtctx->codec_id = avctx->codec_id;
+ if (vtctx->preset != VT_PRESET_NONE) {
+ // The preset selects the rate control.
+ if (vtctx->constant_bit_rate || vtctx->variable_bit_rate) {
+ av_log(avctx, AV_LOG_ERROR, "-preset cannot be used together with "
+ "-constant_bit_rate or
-variable_bit_rate\n");
+ return AVERROR(EINVAL);
+ }
+ if (avctx->flags & AV_CODEC_FLAG_QSCALE || avctx->global_quality > 0) {
+ av_log(avctx, AV_LOG_ERROR, "-preset cannot be used "
+ "together with qscale or
global_quality\n");
+ return AVERROR(EINVAL);
+ }
+ if (avctx->flags & AV_CODEC_FLAG_LOW_DELAY) {
+ av_log(avctx, AV_LOG_ERROR, "-preset cannot be used "
+ "in low_delay mode\n");
+ return AVERROR(EINVAL);
+ }
+ if (vtctx->preset == VT_PRESET_CONSISTENT_QUALITY &&
+ avctx->bit_rate > 0) {
+ av_log(avctx, AV_LOG_ERROR, "The consistent_quality preset cannot
be used "
+ "together with a bitrate target\n");
+ return AVERROR(EINVAL);
+ }
+ }
+
if (vtctx->variable_bit_rate) {
if (vtctx->constant_bit_rate) {
av_log(avctx, AV_LOG_ERROR, "-variable_bit_rate and
-constant_bit_rate "
@@ -2934,7 +3055,14 @@ static const enum AVPixelFormat prores_pix_fmts[] = {
OFFSET(max_ref_frames), AV_OPT_TYPE_INT, { .i64 = 0 }, 0, INT_MAX, VE
},
#define RC_OPTIONS \
{ "variable_bit_rate", "Require variable bit rate, with maxrate, bufsize
and rc_init_occupancy configuring the VBV (macOS 26 or newer)", \
- OFFSET(variable_bit_rate), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE },
+ OFFSET(variable_bit_rate), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, VE },
\
+ { "preset", "Apply a compression preset of the encoder before all other
options, B-frames still have to be enabled with bf (macOS 26 or newer)", \
+ OFFSET(preset), AV_OPT_TYPE_INT, { .i64 = VT_PRESET_NONE },
VT_PRESET_NONE, VT_PRESET_CONSISTENT_QUALITY, VE, .unit = "preset" }, \
+ { "none", "Do not apply a preset",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_NONE },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
+ { "high_quality", "Higher quality, slower encoding",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_HIGH_QUALITY },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
+ { "balanced", "Balance of quality and speed",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_BALANCED },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
+ { "high_speed", "Faster encoding, lower quality",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_HIGH_SPEED },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
+ { "consistent_quality", "Consistent quality across frames (macOS 27 or
newer)", 0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_CONSISTENT_QUALITY
}, INT_MIN, INT_MAX, VE, .unit = "preset" },
static const AVCodecHWConfigInternal *const vt_encode_hw_configs[] = {
HW_CONFIG_ENCODER_FRAMES(VIDEOTOOLBOX, VIDEOTOOLBOX),
--
2.52.0
>From 3e686f0f8ebc0b47b17ebc870f1e33be39693333 Mon Sep 17 00:00:00 2001
From: Leo Bicknell <[email protected]>
Date: Thu, 1 Oct 2026 18:15:26 -0400
Subject: [PATCH 3/3] avcodec/videotoolboxenc: add constant_quality_factor
option in macOS 27
macOS 27 added kVTCompressionPropertyKey_ConstantQualityFactor, a
rate control targeting consistent visual quality. Unlike
kVTCompressionPropertyKey_Quality, which is what qscale and
global_quality map to and which makes the hardware encoders use a fixed
quantization parameter with only a few distinct levels, the factor is
continuous and adapts to the content. Since the two properties use
different scales and the behaviour of qscale has to stay as it is, this
is a new option.
Add a constant_quality_factor option for H.264 and HEVC, overriding the
factor of the consistent_quality preset. The bitrate is bounded by the
VBV configured through maxrate, bufsize and rc_init_occupancy; without
maxrate the preset's own cap of 30 Mb/s applies.
The option requires the preset. With macOS 27.0 the hardware H.264 and
HEVC encoders silently ignore the factor unless the preset is applied
as well, and the software encoders do not support the property at all,
so accepting the factor on its own would produce output unrelated to
the requested quality.
Encoding 24 seconds of 3840x2160 30 fps 10 bit video with the hardware
HEVC encoder of an M5 Max, using
-preset consistent_quality -maxrate 100M, compared to -q:v:
constant_quality_factor 0.55 17.1 Mb/s
constant_quality_factor 0.58 25.4 Mb/s
constant_quality_factor 0.60 32.8 Mb/s
constant_quality_factor 0.62 43.5 Mb/s
constant_quality_factor 0.64 57.4 Mb/s
constant_quality_factor 0.66 74.1 Mb/s
q:v 50 32.8 Mb/s
q:v 51 and 52 40.6 Mb/s
q:v 53 59.2 Mb/s
Signed-off-by: Leo Bicknell <[email protected]>
---
libavcodec/version.h | 2 +-
libavcodec/videotoolboxenc.c | 31 +++++++++++++++++++++++++++++--
2 files changed, 30 insertions(+), 3 deletions(-)
diff --git a/libavcodec/version.h b/libavcodec/version.h
index 82a86fe9d9..9d0485b068 100644
--- a/libavcodec/version.h
+++ b/libavcodec/version.h
@@ -30,7 +30,7 @@
#include "version_major.h"
#define LIBAVCODEC_VERSION_MINOR 16
-#define LIBAVCODEC_VERSION_MICRO 100
+#define LIBAVCODEC_VERSION_MICRO 101
#define LIBAVCODEC_VERSION_INT AV_VERSION_INT(LIBAVCODEC_VERSION_MAJOR, \
LIBAVCODEC_VERSION_MINOR, \
diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index 23f2a09032..bcdab262fe 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -147,6 +147,7 @@ static struct{
CFStringRef kVTCompressionPreset_Balanced;
CFStringRef kVTCompressionPreset_HighSpeed;
CFStringRef kVTCompressionPreset_ConsistentQuality;
+ CFStringRef kVTCompressionPropertyKey_ConstantQualityFactor;
getParameterSetAtIndex CMVideoFormatDescriptionGetHEVCParameterSetAtIndex;
} compat_keys;
@@ -238,6 +239,8 @@ static void loadVTEncSymbols(void){
GET_SYM(kVTCompressionPreset_Balanced, "Balanced");
GET_SYM(kVTCompressionPreset_HighSpeed, "HighSpeed");
GET_SYM(kVTCompressionPreset_ConsistentQuality, "ConsistentQuality");
+ GET_SYM(kVTCompressionPropertyKey_ConstantQualityFactor,
+ "ConstantQualityFactor");
}
#define H264_PROFILE_CONSTRAINED_HIGH (AV_PROFILE_H264_HIGH |
AV_PROFILE_H264_CONSTRAINED)
@@ -304,6 +307,7 @@ typedef struct VTEncContext {
int variable_bit_rate;
int preset;
bool preset_vbr;
+ float constant_quality_factor;
int allow_sw;
int require_sw;
@@ -1430,7 +1434,16 @@ static int vtenc_create_encoder(AVCodecContext *avctx,
return AVERROR_EXTERNAL;
}
- if (avctx->flags & AV_CODEC_FLAG_QSCALE || avctx->global_quality > 0) {
+ if (vtctx->constant_quality_factor >= 0) {
+ Float32 quality_factor = vtctx->constant_quality_factor;
+
+ status = set_encoder_rc_property(avctx,
+
compat_keys.kVTCompressionPropertyKey_ConstantQualityFactor,
+ "constant_quality_factor", "27",
+ kCFNumberFloat32Type,
&quality_factor);
+ if (status < 0)
+ return status;
+ } else if (avctx->flags & AV_CODEC_FLAG_QSCALE || avctx->global_quality >
0) {
float factor = (avctx->flags & AV_CODEC_FLAG_QSCALE) ?
FF_QP2LAMBDA * 100.0f : 100.0f;
Float32 quality = fminf(avctx->global_quality / factor, 1.0f);
@@ -1889,6 +1902,14 @@ static int vtenc_configure_encoder(AVCodecContext *avctx)
}
}
+ // The encoders ignore the factor unless the preset is applied as well,
+ // which also rules out the incompatible rate control options above.
+ if (vtctx->constant_quality_factor >= 0 &&
+ vtctx->preset != VT_PRESET_CONSISTENT_QUALITY) {
+ av_log(avctx, AV_LOG_ERROR, "-constant_quality_factor requires -preset
consistent_quality\n");
+ return AVERROR(EINVAL);
+ }
+
if (vtctx->variable_bit_rate) {
if (vtctx->constant_bit_rate) {
av_log(avctx, AV_LOG_ERROR, "-variable_bit_rate and
-constant_bit_rate "
@@ -2036,6 +2057,10 @@ static av_cold int vtenc_init(AVCodecContext *avctx)
// It can happen when user set avctx->profile directly.
if (vtctx->profile == AV_PROFILE_UNKNOWN)
vtctx->profile = avctx->profile;
+ // ProRes has no rate control options, so the field is not initialized
+ // by an AVOption default.
+ if (avctx->codec_id == AV_CODEC_ID_PRORES)
+ vtctx->constant_quality_factor = -1;
status = vtenc_configure_encoder(avctx);
if (status) return status;
@@ -3062,7 +3087,9 @@ static const enum AVPixelFormat prores_pix_fmts[] = {
{ "high_quality", "Higher quality, slower encoding",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_HIGH_QUALITY },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
{ "balanced", "Balance of quality and speed",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_BALANCED },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
{ "high_speed", "Faster encoding, lower quality",
0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_HIGH_SPEED },
INT_MIN, INT_MAX, VE, .unit = "preset" }, \
- { "consistent_quality", "Consistent quality across frames (macOS 27 or
newer)", 0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_CONSISTENT_QUALITY
}, INT_MIN, INT_MAX, VE, .unit = "preset" },
+ { "consistent_quality", "Consistent quality across frames (macOS 27 or
newer)", 0, AV_OPT_TYPE_CONST, { .i64 = VT_PRESET_CONSISTENT_QUALITY
}, INT_MIN, INT_MAX, VE, .unit = "preset" }, \
+ { "constant_quality_factor", "Target consistent visual quality, from 0.0
(lowest) to 1.0 (highest), requires the consistent_quality preset, -1 to
disable (macOS 27 or newer)", \
+ OFFSET(constant_quality_factor), AV_OPT_TYPE_FLOAT, { .dbl = -1 }, -1,
1, VE },
static const AVCodecHWConfigInternal *const vt_encode_hw_configs[] = {
HW_CONFIG_ENCODER_FRAMES(VIDEOTOOLBOX, VIDEOTOOLBOX),
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
