PR #25522 opened by nico-lab URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25522 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25522.patch
# Summary of changes Add `mode=all|si|ti` to the SITI filter, keeping both metrics enabled by default. Selecting a single metric skips the unused computation, buffers and output. - Reduce TI from three image passes to one using integer accumulation and centering, and remove the full-frame float motion buffer. - Optimize Sobel calculations, share range conversion between SI and TI, and use a three-row cache in SI-only mode. - Preserve the legacy summary layout and round TI metadata to two decimal places, with exact halfway values rounded to even. - Add SI-only, TI-only, 10-bit and full-range FATE coverage. Against `598857b`, SI matched bit-for-bit across 30,040 normal frames. TI differed by at most 1 ULP, with a maximum observed relative difference of approximately 0.000012%. Two-decimal output matched for those inputs. Constructed rounding-boundary inputs can differ from legacy TI metadata by 0.01; the new output follows mathematical rounding. Before rebasing onto current master, direct benchmarks of the final implementation against `598857b` showed: | Input | SI+TI speedup | CPU time reduction | |---|---:|---:| | 1080p / 4K, 8-bit limited range | 10.0–10.1x | 90.0–90.1% | | 1080p / 4K, 10-bit limited range | 8.5–8.8x | 88.3–88.6% | | 1080p, 8-bit full range | 1.64x | 39.0% | Measurements use six-run medians on Intel N100 / Windows / GCC 15.2 `-O3`, with x86 assembly disabled and one CPU/filter/output thread. CPU time includes input generation, format conversion and null output. The build and all six SITI FATE tests pass after rebasing onto current master. Additional validation covers rounding boundaries, range changes, negative strides, odd dimensions and malformed samples. No external FATE samples are required; the added FATE tests use generated input. From 703efddccbcaaa7eeb243d5528ed67a495023455 Mon Sep 17 00:00:00 2001 From: nico_lab <[email protected]> Date: Sat, 24 Oct 2026 11:44:44 +0900 Subject: [PATCH 1/2] avfilter/vf_siti: add metric selection and optimize SI/TI processing Add mode=all|si|ti (default: all), skipping unselected computation, buffers, metadata and summary output. Preserve inherited metadata. Reduce TI from three image passes to one using 64-bit integer moments and integer centering to avoid cancellation near zero. Remove the float motion buffer, saving 4 * width * height bytes. Move first-frame and range selection outside the pixel loops. Round TI metadata directly from integer moments to two decimals, with ties to even. Use integer Sobel sums while preserving SI's floating-point operations. Share TI's normalized image with SI in all mode; use a three-row cache (6 * width bytes) in SI-only mode. Use integer clipping and constant divisors for limited-to-full conversion. Against 598857b, SI matched bit-for-bit across 30,040 normal frames. TI differed by at most 1 ULP, with maximum observed relative difference about 0.000012%. Normal two-decimal metadata and summaries matched. At constructed rounding boundaries, TI metadata can differ by 0.01 from legacy output; the new values match mathematical rounding. Benchmarks: Intel N100, Windows, GCC 15.2 -O3, x86 assembly disabled, one CPU/filter/output thread, testsrc2 input, six runs per condition. Median CPU time includes input generation, conversion and null output: - SI+TI vs 598857b: 86.4-88.7% less time for 1080p/4K 8/10-bit limited input, and 29.8% less for 1080p 8-bit full-range input. - All mode: 6.1-20.7% less time than the sum of separate SI/TI commands, using the same frame count for all three modes in each condition. Document the options and TI rounding; add SI-only, TI-only and 10-bit FATE coverage. All four SITI FATE tests pass. Final optimizations retain the centered one-pass reference values across 30,040 normal frames, 12 boundary cases and 864 random frames, including range changes, negative strides and odd dimensions. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: nico_lab <[email protected]> --- doc/filters.texi | 42 ++- libavfilter/vf_siti.c | 404 ++++++++++++++++------- tests/fate/filter-video.mak | 9 + tests/ref/fate/filter-refcmp-siti-si-yuv | 10 + tests/ref/fate/filter-refcmp-siti-ti-yuv | 10 + tests/ref/fate/filter-refcmp-siti-yuv10 | 15 + 6 files changed, 368 insertions(+), 122 deletions(-) create mode 100644 tests/ref/fate/filter-refcmp-siti-si-yuv create mode 100644 tests/ref/fate/filter-refcmp-siti-ti-yuv create mode 100644 tests/ref/fate/filter-refcmp-siti-yuv10 diff --git a/doc/filters.texi b/doc/filters.texi index 37f9123df49..e11b7d8b97c 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -23212,11 +23212,33 @@ for multimedia applications. Available PDF at @url{https://www.itu.int/rec/T-REC Note that this is a legacy implementation that corresponds to a superseded recommendation. Refer to ITU-T Rec. P.910 (07/22) for the latest version: @url{https://www.itu.int/rec/T-REC-P.910-202207-I/en} -It accepts the following option: +The @code{lavfi.siti.ti} frame metadata is rounded directly to two decimal +places from the integer pixel differences. Exact halfway values are rounded +to the nearest even hundredth. Summary statistics use floating-point TI +values rather than the rounded frame metadata. + +It accepts the following options: @table @option @item print_summary If set to 1, Summary statistics will be printed to the console. Default 0. + +@item mode +Set which metric to calculate. + +The following values are available: + +@table @samp +@item all +Calculate both Spatial Information (SI) and Temporal Information (TI). +This is the default. + +@item si +Calculate only Spatial Information (SI). + +@item ti +Calculate only Temporal Information (TI). +@end table @end table @subsection Examples @@ -23226,6 +23248,24 @@ To calculate SI/TI metrics and print summary: @example ffmpeg -i input.mp4 -vf siti=print_summary=1 -f null - @end example + +@item +To calculate both SI and TI explicitly: +@example +ffmpeg -i input.mp4 -vf siti=mode=all -f null - +@end example + +@item +To calculate only Temporal Information: +@example +ffmpeg -i input.mp4 -vf siti=mode=ti -f null - +@end example + +@item +To calculate only Spatial Information: +@example +ffmpeg -i input.mp4 -vf siti=mode=si -f null - +@end example @end itemize @anchor{smartblur} diff --git a/libavfilter/vf_siti.c b/libavfilter/vf_siti.c index fde55b49a92..5c0ff131d48 100644 --- a/libavfilter/vf_siti.c +++ b/libavfilter/vf_siti.c @@ -27,6 +27,7 @@ #include <math.h> #include "libavutil/imgutils.h" +#include "libavutil/int128.h" #include "libavutil/internal.h" #include "libavutil/mem.h" #include "libavutil/opt.h" @@ -35,16 +36,10 @@ #include "filters.h" #include "video.h" -static const int X_FILTER[9] = { - 1, 0, -1, - 2, 0, -2, - 1, 0, -1 -}; - -static const int Y_FILTER[9] = { - 1, 2, 1, - 0, 0, 0, - -1, -2, -1 +enum { + MODE_ALL = 0, + MODE_SI = 1, + MODE_TI = 2, }; typedef struct SiTiContext { @@ -60,9 +55,10 @@ typedef struct SiTiContext { float sum_si; float sum_ti; float *gradient_matrix; - float *motion_matrix; + uint16_t *sobel_rows; int full_range; int print_summary; + int mode; } SiTiContext; static const enum AVPixelFormat pix_fmts[] = { @@ -85,20 +81,32 @@ static av_cold void uninit(AVFilterContext *ctx) { SiTiContext *s = ctx->priv; - if (s->print_summary) { - float avg_si = s->sum_si / s->nb_frames; - float avg_ti = s->sum_ti / s->nb_frames; + if (s->print_summary && s->nb_frames) { av_log(ctx, AV_LOG_INFO, - "SITI Summary:\nTotal frames: %"PRId64"\n\n" - "Spatial Information:\nAverage: %f\nMax: %f\nMin: %f\n\n" - "Temporal Information:\nAverage: %f\nMax: %f\nMin: %f\n", - s->nb_frames, avg_si, s->max_si, s->min_si, avg_ti, s->max_ti, s->min_ti - ); + "SITI Summary:\nTotal frames: %"PRId64"\n\n", + s->nb_frames); + + if (s->mode != MODE_TI) { + float avg_si = s->sum_si / s->nb_frames; + av_log(ctx, AV_LOG_INFO, + "Spatial Information:\nAverage: %f\nMax: %f\nMin: %f\n", + avg_si, s->max_si, s->min_si); + } + + if (s->mode == MODE_ALL) + av_log(ctx, AV_LOG_INFO, "\n"); + + if (s->mode != MODE_SI) { + float avg_ti = s->sum_ti / s->nb_frames; + av_log(ctx, AV_LOG_INFO, + "Temporal Information:\nAverage: %f\nMax: %f\nMin: %f\n", + avg_ti, s->max_ti, s->min_ti); + } } av_freep(&s->prev_frame); av_freep(&s->gradient_matrix); - av_freep(&s->motion_matrix); + av_freep(&s->sobel_rows); } static int config_input(AVFilterLink *inlink) @@ -110,7 +118,6 @@ static int config_input(AVFilterLink *inlink) size_t pixel_sz; size_t data_sz; size_t gradient_sz; - size_t motion_sz; const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(inlink->format); av_image_fill_max_pixsteps(max_pixsteps, NULL, desc); @@ -118,7 +125,7 @@ static int config_input(AVFilterLink *inlink) // free previous buffers in case they are allocated already av_freep(&s->prev_frame); av_freep(&s->gradient_matrix); - av_freep(&s->motion_matrix); + av_freep(&s->sobel_rows); s->pixel_depth = max_pixsteps[0]; s->width = inlink->w; @@ -126,16 +133,26 @@ static int config_input(AVFilterLink *inlink) pixel_sz = s->pixel_depth == 1 ? sizeof(uint8_t) : sizeof(uint16_t); data_sz = s->width * pixel_sz * s->height; - s->prev_frame = av_malloc(data_sz); + if (s->mode != MODE_SI) { + s->prev_frame = av_malloc(data_sz); - gradient_sz = (s->width - 2) * sizeof(float) * (s->height - 2); - s->gradient_matrix = av_malloc(gradient_sz); + if (!s->prev_frame) + return AVERROR(ENOMEM); + } - motion_sz = s->width * sizeof(float) * s->height; - s->motion_matrix = av_malloc(motion_sz); + if (s->mode != MODE_TI) { + gradient_sz = (s->width - 2) * sizeof(float) * (s->height - 2); + s->gradient_matrix = av_malloc(gradient_sz); - if (!s->prev_frame || ! s->gradient_matrix || !s->motion_matrix) { - return AVERROR(ENOMEM); + if (!s->gradient_matrix) + return AVERROR(ENOMEM); + + // In all mode, TI already produces the normalized current frame. + if (s->mode == MODE_SI) { + s->sobel_rows = av_malloc_array(s->width, 3 * sizeof(*s->sobel_rows)); + if (!s->sobel_rows) + return AVERROR(ENOMEM); + } } return 0; @@ -162,91 +179,213 @@ static uint16_t convert_full_range(int factor, uint16_t y) shift = 16 * factor; limit_upper = 235 * factor - shift; full_upper = 256 * factor - 1; - limit_y = fminf(fmaxf(y - shift, 0), limit_upper); + limit_y = av_clip(y - shift, 0, limit_upper); return (full_upper * limit_y / limit_upper); } -// Applies sobel convolution -static void convolve_sobel(SiTiContext *s, const uint8_t *src, float *dst, int linesize) -{ - double x_conv_sum; - double y_conv_sum; - float gradient; - int ki; - int kj; - int index; - uint16_t data; - int filter_width = 3; - int filter_size = filter_width * filter_width; - int stride = linesize / s->pixel_depth; - // For 8 bits, limited range goes from 16 to 235, for 10 bits the range is multiplied by 4 - int factor = s->pixel_depth == 1 ? 1 : 4; - - // Dst matrix is smaller than src since we ignore edges that can't be convolved - #define CONVOLVE(bps) \ - { \ - uint##bps##_t *vsrc = (uint##bps##_t*)src; \ - for (int j = 1; j < s->height - 1; j++) { \ - for (int i = 1; i < s->width - 1; i++) { \ - x_conv_sum = 0.0; \ - y_conv_sum = 0.0; \ - for (int k = 0; k < filter_size; k++) { \ - ki = k % filter_width - 1; \ - kj = floor(k / filter_width) - 1; \ - index = (j + kj) * stride + (i + ki); \ - data = s->full_range ? vsrc[index] : convert_full_range(factor, vsrc[index]); \ - x_conv_sum += data * X_FILTER[k]; \ - y_conv_sum += data * Y_FILTER[k]; \ - } \ - gradient = sqrt(x_conv_sum * x_conv_sum + y_conv_sum * y_conv_sum); \ - dst[(j - 1) * (s->width - 2) + (i - 1)] = gradient; \ - } \ - } \ - } - - if (s->pixel_depth == 2) { - CONVOLVE(16); - } else { - CONVOLVE(8); - } +// The integer Sobel sums are exact; retain double sqrt and float gradients. +#define DEFINE_SOBEL_ROW(bps) \ +static inline void sobel_row_##bps(const uint##bps##_t *top, \ + const uint##bps##_t *middle, \ + const uint##bps##_t *bottom, \ + float *dst, int width) \ +{ \ + for (int i = 1; i < width - 1; i++) { \ + int x = top[i - 1] - top[i + 1] + \ + 2 * (middle[i - 1] - middle[i + 1]) + \ + bottom[i - 1] - bottom[i + 1]; \ + int y = top[i - 1] + 2 * top[i] + top[i + 1] - \ + bottom[i - 1] - 2 * bottom[i] - bottom[i + 1]; \ + \ + dst[i - 1] = sqrt((double)x * x + (double)y * y); \ + } \ } -// Calculate pixel difference between current and previous frame, and update previous -static void calculate_motion(SiTiContext *s, const uint8_t *curr, - float *motion_matrix, int linesize) -{ - int stride = linesize / s->pixel_depth; - float motion; - int curr_index; - int prev_index; - uint16_t curr_data; - // For 8 bits, limited range goes from 16 to 235, for 10 bits the range is multiplied by 4 - int factor = s->pixel_depth == 1 ? 1 : 4; +DEFINE_SOBEL_ROW(8) +DEFINE_SOBEL_ROW(16) +#undef DEFINE_SOBEL_ROW - // Previous frame is already converted to full range - #define CALCULATE(bps) \ - { \ - uint##bps##_t *vsrc = (uint##bps##_t*)curr; \ - uint##bps##_t *vdst = (uint##bps##_t*)s->prev_frame; \ - for (int j = 0; j < s->height; j++) { \ - for (int i = 0; i < s->width; i++) { \ - motion = 0; \ - curr_index = j * stride + i; \ - prev_index = j * s->width + i; \ - curr_data = s->full_range ? vsrc[curr_index] : convert_full_range(factor, vsrc[curr_index]); \ - if (s->nb_frames > 1) \ - motion = curr_data - vdst[prev_index]; \ - vdst[prev_index] = curr_data; \ - motion_matrix[j * s->width + i] = motion; \ - } \ - } \ +#define DEFINE_CONVERT_ROW(bps, factor) \ +static inline void convert_row_##bps(const uint##bps##_t *src, \ + uint16_t *dst, int width) \ +{ \ + for (int i = 0; i < width; i++) \ + dst[i] = convert_full_range(factor, src[i]); \ +} + +DEFINE_CONVERT_ROW(8, 1) +DEFINE_CONVERT_ROW(16, 4) +#undef DEFINE_CONVERT_ROW + +// Applies Sobel convolution, converting each limited-range sample once. +static void convolve_sobel(SiTiContext *s, const uint8_t *src, float *dst, + int linesize, int full_range) +{ + const int width = s->width; + const int height = s->height; + const int stride = linesize / s->pixel_depth; + + if (width < 3 || height < 3) + return; + +#define CONVOLVE_ROWS(bps) \ + { \ + const uint##bps##_t *vsrc = (const uint##bps##_t *)src; \ + \ + if (full_range) { \ + for (int j = 1; j < height - 1; j++) \ + sobel_row_##bps(vsrc + (j - 1) * stride, \ + vsrc + j * stride, \ + vsrc + (j + 1) * stride, \ + dst + (j - 1) * (width - 2), width); \ + } else { \ + uint16_t *top = s->sobel_rows; \ + uint16_t *middle = top + width; \ + uint16_t *bottom = middle + width; \ + \ + convert_row_##bps(vsrc, top, width); \ + convert_row_##bps(vsrc + stride, middle, width); \ + for (int j = 1; j < height - 1; j++) { \ + uint16_t *next = top; \ + \ + convert_row_##bps(vsrc + (j + 1) * stride, bottom, width); \ + sobel_row_16(top, middle, bottom, \ + dst + (j - 1) * (width - 2), width); \ + top = middle; \ + middle = bottom; \ + bottom = next; \ + } \ + } \ } if (s->pixel_depth == 2) { - CALCULATE(16); + CONVOLVE_ROWS(16); } else { - CALCULATE(8); + CONVOLVE_ROWS(8); } +#undef CONVOLVE_ROWS +} + +// Round the mathematical TI to hundredths, with ties to even. +static int round_ti_to_hundredths(uint64_t count, int64_t sum, uint64_t sum_sq) +{ + av_int128 n = av_to128i(count); + av_int128 s = av_to128i(sum); + av_int128 q = av_to128i(sum_sq); + av_int128 numerator = av_sub128(av_mul128(n, q), av_mul128(s, s)); + av_int128 scaled = av_mul128(av_to128i(40000), numerator); + av_int128 n_squared = av_mul128(n, n); + int low = 0, high = 1023 * 100; + + if (!av_test128(numerator)) + return 0; + + // Compare TI squared with ((2 * mid + 1) / 200) squared. + // For valid 8/10-bit input and image sizes, all values fit in 98 bits. + while (low < high) { + int mid = low + (high - low) / 2; + int64_t boundary = 2LL * mid + 1; + av_int128 limit = av_mul128(n_squared, av_to128i(boundary * boundary)); + int cmp = av_cmp128(scaled, limit); + + if (cmp > 0 || (!cmp && (mid & 1))) + low = mid + 1; + else + high = mid; + } + return low; +} + +// Calculate TI directly from pixel differences and update previous frame +static float calculate_ti(SiTiContext *s, const uint8_t *curr, int linesize, + int *ti_hundredths) +{ + int stride = linesize / s->pixel_depth; + int64_t integer_sum = 0; + uint64_t integer_sum_sq = 0; + double sum; + double sum_sq; + double center; + double mean; + double variance; + int curr_index; + int prev_index; + int motion; + uint16_t curr_data; + uint64_t count = (uint64_t)s->width * s->height; + + // Previous frame is already converted to full range. + // Squaring as uint32_t handles negative differences modulo 2^32. + // |motion| <= UINT16_MAX, so its exact square fits in uint32_t; + // accumulate in 64 bits and convert to double once before centering. +#define CALCULATE(bps, factor, full, first) \ + { \ + uint##bps##_t *vsrc = (uint##bps##_t *)curr; \ + uint##bps##_t *vdst = (uint##bps##_t *)s->prev_frame; \ + for (int j = 0; j < s->height; j++) { \ + for (int i = 0; i < s->width; i++) { \ + motion = 0; \ + curr_index = j * stride + i; \ + prev_index = j * s->width + i; \ + curr_data = full ? vsrc[curr_index] \ + : convert_full_range(factor, vsrc[curr_index]); \ + if (!first) \ + motion = curr_data - vdst[prev_index]; \ + vdst[prev_index] = curr_data; \ + integer_sum += motion; \ + integer_sum_sq += (uint32_t)motion * (uint32_t)motion; \ + } \ + } \ + } + + if (s->nb_frames > 1) { + if (s->pixel_depth == 2) { + if (s->full_range) { + CALCULATE(16, 4, 1, 0); + } else { + CALCULATE(16, 4, 0, 0); + } + } else { + if (s->full_range) { + CALCULATE(8, 1, 1, 0); + } else { + CALCULATE(8, 1, 0, 0); + } + } + } else { + if (s->pixel_depth == 2) { + if (s->full_range) { + CALCULATE(16, 4, 1, 1); + } else { + CALCULATE(16, 4, 0, 1); + } + } else { + if (s->full_range) { + CALCULATE(8, 1, 1, 1); + } else { + CALCULATE(8, 1, 0, 1); + } + } + *ti_hundredths = 0; + return 0.0f; + } +#undef CALCULATE + + sum = integer_sum; + sum_sq = integer_sum_sq; + + // Recenter the integer moments before division to avoid cancellation + // when motion is nearly constant. These operations are exact in double + // for valid image sizes and 8/10-bit input. + center = floor(sum / count + 0.5); + sum_sq = sum_sq - 2 * center * sum + center * center * count; + sum -= center * count; + *ti_hundredths = round_ti_to_hundredths(count, (int64_t)sum, (uint64_t)sum_sq); + mean = sum / count; + variance = sum_sq / count - mean * mean; + + // Guard against a tiny negative value caused by floating-point rounding. + return sqrt(fmax(variance, 0.0)); } static float std_deviation(float *img_metrics, int width, int height) @@ -278,33 +417,52 @@ static void set_meta(AVDictionary **metadata, const char *key, float d) av_dict_set(metadata, key, value, 0); } +static void set_meta_ti(AVDictionary **metadata, int hundredths) +{ + char value[128]; + + snprintf(value, sizeof(value), "%d.%02d", hundredths / 100, hundredths % 100); + av_dict_set(metadata, "lavfi.siti.ti", value, 0); +} + static int filter_frame(AVFilterLink *inlink, AVFrame *frame) { AVFilterContext *ctx = inlink->dst; SiTiContext *s = ctx->priv; - float si; - float ti; + const int have_ti = s->mode != MODE_SI; + float ti = 0; + int ti_hundredths = 0; s->full_range = is_full_range(frame); s->nb_frames++; - // Calculate si and ti - convolve_sobel(s, frame->data[0], s->gradient_matrix, frame->linesize[0]); - calculate_motion(s, frame->data[0], s->motion_matrix, frame->linesize[0]); - si = std_deviation(s->gradient_matrix, s->width - 2, s->height - 2); - ti = std_deviation(s->motion_matrix, s->width, s->height); + // Updating TI leaves a complete normalized current frame in prev_frame. + if (have_ti) + ti = calculate_ti(s, frame->data[0], frame->linesize[0], &ti_hundredths); - // Calculate statistics - s->max_si = fmaxf(si, s->max_si); - s->max_ti = fmaxf(ti, s->max_ti); - s->sum_si += si; - s->sum_ti += ti; - s->min_si = s->nb_frames == 1 ? si : fminf(si, s->min_si); - s->min_ti = s->nb_frames == 1 ? ti : fminf(ti, s->min_ti); + if (s->mode != MODE_TI) { + const uint8_t *si_src = have_ti ? s->prev_frame : frame->data[0]; + const int si_linesize = have_ti ? s->width * s->pixel_depth : frame->linesize[0]; + float si; - // Set si ti information in frame metadata - set_meta(&frame->metadata, "lavfi.siti.si", si); - set_meta(&frame->metadata, "lavfi.siti.ti", ti); + convolve_sobel(s, si_src, s->gradient_matrix, si_linesize, + have_ti || s->full_range); + si = std_deviation(s->gradient_matrix, s->width - 2, s->height - 2); + + s->max_si = fmaxf(si, s->max_si); + s->sum_si += si; + s->min_si = s->nb_frames == 1 ? si : fminf(si, s->min_si); + + set_meta(&frame->metadata, "lavfi.siti.si", si); + } + + if (have_ti) { + s->max_ti = fmaxf(ti, s->max_ti); + s->sum_ti += ti; + s->min_ti = s->nb_frames == 1 ? ti : fminf(ti, s->min_ti); + + set_meta_ti(&frame->metadata, ti_hundredths); + } return ff_filter_frame(inlink->dst->outputs[0], frame); } @@ -313,7 +471,11 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) #define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM static const AVOption siti_options[] = { - { "print_summary", "Print summary showing average values", OFFSET(print_summary), AV_OPT_TYPE_BOOL, { .i64=0 }, 0, 1, FLAGS }, + { "print_summary", "Print summary showing average values", OFFSET(print_summary), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "mode", "Set metric to calculate", OFFSET(mode), AV_OPT_TYPE_INT, { .i64 = MODE_ALL }, MODE_ALL, MODE_TI, FLAGS, .unit = "mode" }, + { "all", "Calculate spatial and temporal information", 0, AV_OPT_TYPE_CONST, { .i64 = MODE_ALL }, 0, 0, FLAGS, .unit = "mode" }, + { "si", "Calculate spatial information", 0, AV_OPT_TYPE_CONST, { .i64 = MODE_SI }, 0, 0, FLAGS, .unit = "mode" }, + { "ti", "Calculate temporal information", 0, AV_OPT_TYPE_CONST, { .i64 = MODE_TI }, 0, 0, FLAGS, .unit = "mode" }, { NULL } }; diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 95254ff3fcf..4891a8827f7 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -960,6 +960,15 @@ fate-filter-refcmp-blurdetect-yuv: CMD = cmp_metadata blurdetect yuv420p 0.015 FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-yuv fate-filter-refcmp-siti-yuv: CMD = cmp_metadata siti yuv420p 0.015 +FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-si-yuv +fate-filter-refcmp-siti-si-yuv: CMD = cmp_metadata siti=mode=si yuv420p 0.015 + +FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-ti-yuv +fate-filter-refcmp-siti-ti-yuv: CMD = cmp_metadata siti=mode=ti yuv420p 0.015 + +FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-yuv10 +fate-filter-refcmp-siti-yuv10: CMD = cmp_metadata siti yuv420p10le 0.015 + FATE_FILTER-$(call ALLYES, TESTSRC2_FILTER METADATA_FILTER WRAPPED_AVFRAME_ENCODER \ NULL_MUXER PIPE_PROTOCOL) += $(FATE_FILTER_CMP_METADATA-yes) diff --git a/tests/ref/fate/filter-refcmp-siti-si-yuv b/tests/ref/fate/filter-refcmp-siti-si-yuv new file mode 100644 index 00000000000..e5f4d70a02d --- /dev/null +++ b/tests/ref/fate/filter-refcmp-siti-si-yuv @@ -0,0 +1,10 @@ +frame:0 pts:0 pts_time:0 +lavfi.siti.si=106.68 +frame:1 pts:1 pts_time:1 +lavfi.siti.si=109.44 +frame:2 pts:2 pts_time:2 +lavfi.siti.si=109.22 +frame:3 pts:3 pts_time:3 +lavfi.siti.si=113.20 +frame:4 pts:4 pts_time:4 +lavfi.siti.si=110.85 diff --git a/tests/ref/fate/filter-refcmp-siti-ti-yuv b/tests/ref/fate/filter-refcmp-siti-ti-yuv new file mode 100644 index 00000000000..216e582dc06 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-siti-ti-yuv @@ -0,0 +1,10 @@ +frame:0 pts:0 pts_time:0 +lavfi.siti.ti=0.00 +frame:1 pts:1 pts_time:1 +lavfi.siti.ti=28.07 +frame:2 pts:2 pts_time:2 +lavfi.siti.ti=28.43 +frame:3 pts:3 pts_time:3 +lavfi.siti.ti=33.42 +frame:4 pts:4 pts_time:4 +lavfi.siti.ti=30.54 diff --git a/tests/ref/fate/filter-refcmp-siti-yuv10 b/tests/ref/fate/filter-refcmp-siti-yuv10 new file mode 100644 index 00000000000..ecad1e3797b --- /dev/null +++ b/tests/ref/fate/filter-refcmp-siti-yuv10 @@ -0,0 +1,15 @@ +frame:0 pts:0 pts_time:0 +lavfi.siti.si=429.62 +lavfi.siti.ti=0.00 +frame:1 pts:1 pts_time:1 +lavfi.siti.si=440.55 +lavfi.siti.ti=112.80 +frame:2 pts:2 pts_time:2 +lavfi.siti.si=439.70 +lavfi.siti.ti=114.00 +frame:3 pts:3 pts_time:3 +lavfi.siti.si=455.97 +lavfi.siti.ti=134.51 +frame:4 pts:4 pts_time:4 +lavfi.siti.si=446.51 +lavfi.siti.ti=123.05 -- 2.54.0 From 023838f14be8ff44b8b611e5aaf5597c6e1723c0 Mon Sep 17 00:00:00 2001 From: nico_lab <[email protected]> Date: Tue, 27 Oct 2026 18:04:46 +0900 Subject: [PATCH 2/2] avfilter/vf_siti: fix summary formatting and streamline TI Build the summary with AVBPrint and log it once to preserve the legacy prefix and blank-line layout. Use PRIu64 for the frame count. Cache TI dimensions locally to enable 8-bit loop vectorization, and handle the first frame with a copy or row conversion. Round metadata from the original integer moments to avoid a potentially undefined double-to-unsigned cast on malformed input; allow the full uint16_t sample range. Retain exact rounding and valid-input results. Assert the limited-range Sobel buffer invariant, add full-range all/SI FATE coverage, and simplify the documentation. Against ae1c68a, six-run median CPU time on Intel N100, Windows, GCC 15.2 -O3, no x86 assembly, one CPU/filter/output thread: - 8-bit SI+TI: 23.2-23.7% less time for 1080p/4K limited range; 10.1% less for 1080p full range. - 1080p TI-only: 61.8% less for 8-bit limited, 37.0% less for full. - 10-bit TI-only: within 0.1%; SI+TI took 0.6-1.4% more time. Timing includes testsrc2 generation, format conversion and null output. Build and all six SITI FATE tests pass. Internal SI/TI values and two-decimal metadata match the prior branch across 30,040 normal frames, 12 boundary cases and 3,456 random/stride frames. Native and portable integer rounding match a high-precision reference in 31,786 cases. Legacy summary layout and first-frame copies were also verified. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: nico_lab <[email protected]> --- doc/filters.texi | 14 +-- libavfilter/vf_siti.c | 122 ++++++++++++---------- tests/fate/filter-video.mak | 6 ++ tests/ref/fate/filter-refcmp-siti-si-yuvj | 10 ++ tests/ref/fate/filter-refcmp-siti-yuvj | 15 +++ 5 files changed, 100 insertions(+), 67 deletions(-) create mode 100644 tests/ref/fate/filter-refcmp-siti-si-yuvj create mode 100644 tests/ref/fate/filter-refcmp-siti-yuvj diff --git a/doc/filters.texi b/doc/filters.texi index e11b7d8b97c..cf37837139b 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -23212,10 +23212,10 @@ for multimedia applications. Available PDF at @url{https://www.itu.int/rec/T-REC Note that this is a legacy implementation that corresponds to a superseded recommendation. Refer to ITU-T Rec. P.910 (07/22) for the latest version: @url{https://www.itu.int/rec/T-REC-P.910-202207-I/en} -The @code{lavfi.siti.ti} frame metadata is rounded directly to two decimal -places from the integer pixel differences. Exact halfway values are rounded -to the nearest even hundredth. Summary statistics use floating-point TI -values rather than the rounded frame metadata. +The @code{lavfi.siti.ti} frame metadata is rounded to two decimal places. +Exact halfway values are rounded to the nearest even hundredth. +Summary statistics use floating-point TI values rather than the rounded frame +metadata. It accepts the following options: @@ -23249,12 +23249,6 @@ To calculate SI/TI metrics and print summary: ffmpeg -i input.mp4 -vf siti=print_summary=1 -f null - @end example -@item -To calculate both SI and TI explicitly: -@example -ffmpeg -i input.mp4 -vf siti=mode=all -f null - -@end example - @item To calculate only Temporal Information: @example diff --git a/libavfilter/vf_siti.c b/libavfilter/vf_siti.c index 5c0ff131d48..69db71d44fb 100644 --- a/libavfilter/vf_siti.c +++ b/libavfilter/vf_siti.c @@ -26,6 +26,8 @@ #include <math.h> +#include "libavutil/avassert.h" +#include "libavutil/bprint.h" #include "libavutil/imgutils.h" #include "libavutil/int128.h" #include "libavutil/internal.h" @@ -82,26 +84,30 @@ static av_cold void uninit(AVFilterContext *ctx) SiTiContext *s = ctx->priv; if (s->print_summary && s->nb_frames) { - av_log(ctx, AV_LOG_INFO, - "SITI Summary:\nTotal frames: %"PRId64"\n\n", - s->nb_frames); + AVBPrint buf; + + av_bprint_init(&buf, 0, AV_BPRINT_SIZE_AUTOMATIC); + av_bprintf(&buf, "SITI Summary:\nTotal frames: %"PRIu64"\n\n", + s->nb_frames); if (s->mode != MODE_TI) { float avg_si = s->sum_si / s->nb_frames; - av_log(ctx, AV_LOG_INFO, - "Spatial Information:\nAverage: %f\nMax: %f\nMin: %f\n", - avg_si, s->max_si, s->min_si); + av_bprintf(&buf, + "Spatial Information:\nAverage: %f\nMax: %f\nMin: %f\n", + avg_si, s->max_si, s->min_si); } if (s->mode == MODE_ALL) - av_log(ctx, AV_LOG_INFO, "\n"); + av_bprintf(&buf, "\n"); if (s->mode != MODE_SI) { float avg_ti = s->sum_ti / s->nb_frames; - av_log(ctx, AV_LOG_INFO, - "Temporal Information:\nAverage: %f\nMax: %f\nMin: %f\n", - avg_ti, s->max_ti, s->min_ti); + av_bprintf(&buf, + "Temporal Information:\nAverage: %f\nMax: %f\nMin: %f\n", + avg_ti, s->max_ti, s->min_ti); } + + av_log(ctx, AV_LOG_INFO, "%s", buf.str); } av_freep(&s->prev_frame); @@ -205,16 +211,17 @@ DEFINE_SOBEL_ROW(8) DEFINE_SOBEL_ROW(16) #undef DEFINE_SOBEL_ROW -#define DEFINE_CONVERT_ROW(bps, factor) \ -static inline void convert_row_##bps(const uint##bps##_t *src, \ - uint16_t *dst, int width) \ +#define DEFINE_CONVERT_ROW(bps, dst_bps, factor) \ +static inline void convert_row_##bps##_##dst_bps(const uint##bps##_t *src, \ + uint##dst_bps##_t *dst, int width) \ { \ for (int i = 0; i < width; i++) \ dst[i] = convert_full_range(factor, src[i]); \ } -DEFINE_CONVERT_ROW(8, 1) -DEFINE_CONVERT_ROW(16, 4) +DEFINE_CONVERT_ROW(8, 8, 1) +DEFINE_CONVERT_ROW(8, 16, 1) +DEFINE_CONVERT_ROW(16, 16, 4) #undef DEFINE_CONVERT_ROW // Applies Sobel convolution, converting each limited-range sample once. @@ -228,6 +235,8 @@ static void convolve_sobel(SiTiContext *s, const uint8_t *src, float *dst, if (width < 3 || height < 3) return; + av_assert1(full_range || s->sobel_rows); + #define CONVOLVE_ROWS(bps) \ { \ const uint##bps##_t *vsrc = (const uint##bps##_t *)src; \ @@ -243,12 +252,12 @@ static void convolve_sobel(SiTiContext *s, const uint8_t *src, float *dst, uint16_t *middle = top + width; \ uint16_t *bottom = middle + width; \ \ - convert_row_##bps(vsrc, top, width); \ - convert_row_##bps(vsrc + stride, middle, width); \ + convert_row_##bps##_16(vsrc, top, width); \ + convert_row_##bps##_16(vsrc + stride, middle, width); \ for (int j = 1; j < height - 1; j++) { \ uint16_t *next = top; \ \ - convert_row_##bps(vsrc + (j + 1) * stride, bottom, width); \ + convert_row_##bps##_16(vsrc + (j + 1) * stride, bottom, width); \ sobel_row_16(top, middle, bottom, \ dst + (j - 1) * (width - 2), width); \ top = middle; \ @@ -275,13 +284,13 @@ static int round_ti_to_hundredths(uint64_t count, int64_t sum, uint64_t sum_sq) av_int128 numerator = av_sub128(av_mul128(n, q), av_mul128(s, s)); av_int128 scaled = av_mul128(av_to128i(40000), numerator); av_int128 n_squared = av_mul128(n, n); - int low = 0, high = 1023 * 100; + int low = 0, high = UINT16_MAX * 100; if (!av_test128(numerator)) return 0; // Compare TI squared with ((2 * mid + 1) / 200) squared. - // For valid 8/10-bit input and image sizes, all values fit in 98 bits. + // All values fit in 110 bits, even for out-of-range 16-bit samples. while (low < high) { int mid = low + (high - low) / 2; int64_t boundary = 2LL * mid + 1; @@ -300,7 +309,9 @@ static int round_ti_to_hundredths(uint64_t count, int64_t sum, uint64_t sum_sq) static float calculate_ti(SiTiContext *s, const uint8_t *curr, int linesize, int *ti_hundredths) { - int stride = linesize / s->pixel_depth; + const int width = s->width; + const int height = s->height; + const int stride = linesize / s->pixel_depth; int64_t integer_sum = 0; uint64_t integer_sum_sq = 0; double sum; @@ -312,25 +323,40 @@ static float calculate_ti(SiTiContext *s, const uint8_t *curr, int linesize, int prev_index; int motion; uint16_t curr_data; - uint64_t count = (uint64_t)s->width * s->height; + uint64_t count = (uint64_t)width * height; + + if (s->nb_frames == 1) { + if (s->full_range) { + const int row_bytes = width * s->pixel_depth; + av_image_copy_plane(s->prev_frame, row_bytes, curr, linesize, + row_bytes, height); + } else if (s->pixel_depth == 2) { + for (int j = 0; j < height; j++) + convert_row_16_16((const uint16_t *)curr + j * stride, + (uint16_t *)s->prev_frame + j * width, width); + } else { + for (int j = 0; j < height; j++) + convert_row_8_8(curr + j * stride, s->prev_frame + j * width, width); + } + *ti_hundredths = 0; + return 0.0f; + } // Previous frame is already converted to full range. // Squaring as uint32_t handles negative differences modulo 2^32. // |motion| <= UINT16_MAX, so its exact square fits in uint32_t; // accumulate in 64 bits and convert to double once before centering. -#define CALCULATE(bps, factor, full, first) \ +#define CALCULATE(bps, factor, full) \ { \ - uint##bps##_t *vsrc = (uint##bps##_t *)curr; \ + const uint##bps##_t *vsrc = (const uint##bps##_t *)curr; \ uint##bps##_t *vdst = (uint##bps##_t *)s->prev_frame; \ - for (int j = 0; j < s->height; j++) { \ - for (int i = 0; i < s->width; i++) { \ - motion = 0; \ + for (int j = 0; j < height; j++) { \ + for (int i = 0; i < width; i++) { \ curr_index = j * stride + i; \ - prev_index = j * s->width + i; \ + prev_index = j * width + i; \ curr_data = full ? vsrc[curr_index] \ : convert_full_range(factor, vsrc[curr_index]); \ - if (!first) \ - motion = curr_data - vdst[prev_index]; \ + motion = curr_data - vdst[prev_index]; \ vdst[prev_index] = curr_data; \ integer_sum += motion; \ integer_sum_sq += (uint32_t)motion * (uint32_t)motion; \ @@ -338,39 +364,22 @@ static float calculate_ti(SiTiContext *s, const uint8_t *curr, int linesize, } \ } - if (s->nb_frames > 1) { - if (s->pixel_depth == 2) { - if (s->full_range) { - CALCULATE(16, 4, 1, 0); - } else { - CALCULATE(16, 4, 0, 0); - } + if (s->pixel_depth == 2) { + if (s->full_range) { + CALCULATE(16, 4, 1); } else { - if (s->full_range) { - CALCULATE(8, 1, 1, 0); - } else { - CALCULATE(8, 1, 0, 0); - } + CALCULATE(16, 4, 0); } } else { - if (s->pixel_depth == 2) { - if (s->full_range) { - CALCULATE(16, 4, 1, 1); - } else { - CALCULATE(16, 4, 0, 1); - } + if (s->full_range) { + CALCULATE(8, 1, 1); } else { - if (s->full_range) { - CALCULATE(8, 1, 1, 1); - } else { - CALCULATE(8, 1, 0, 1); - } + CALCULATE(8, 1, 0); } - *ti_hundredths = 0; - return 0.0f; } #undef CALCULATE + *ti_hundredths = round_ti_to_hundredths(count, integer_sum, integer_sum_sq); sum = integer_sum; sum_sq = integer_sum_sq; @@ -380,7 +389,6 @@ static float calculate_ti(SiTiContext *s, const uint8_t *curr, int linesize, center = floor(sum / count + 0.5); sum_sq = sum_sq - 2 * center * sum + center * center * count; sum -= center * count; - *ti_hundredths = round_ti_to_hundredths(count, (int64_t)sum, (uint64_t)sum_sq); mean = sum / count; variance = sum_sq / count - mean * mean; @@ -471,7 +479,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *frame) #define FLAGS AV_OPT_FLAG_VIDEO_PARAM|AV_OPT_FLAG_FILTERING_PARAM static const AVOption siti_options[] = { - { "print_summary", "Print summary showing average values", OFFSET(print_summary), AV_OPT_TYPE_BOOL, { .i64 = 0 }, 0, 1, FLAGS }, + { "print_summary", "Print summary showing average values", OFFSET(print_summary), AV_OPT_TYPE_BOOL, { .i64=0 }, 0, 1, FLAGS }, { "mode", "Set metric to calculate", OFFSET(mode), AV_OPT_TYPE_INT, { .i64 = MODE_ALL }, MODE_ALL, MODE_TI, FLAGS, .unit = "mode" }, { "all", "Calculate spatial and temporal information", 0, AV_OPT_TYPE_CONST, { .i64 = MODE_ALL }, 0, 0, FLAGS, .unit = "mode" }, { "si", "Calculate spatial information", 0, AV_OPT_TYPE_CONST, { .i64 = MODE_SI }, 0, 0, FLAGS, .unit = "mode" }, diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 4891a8827f7..0a460e76922 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -969,6 +969,12 @@ fate-filter-refcmp-siti-ti-yuv: CMD = cmp_metadata siti=mode=ti yuv420p 0.015 FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-yuv10 fate-filter-refcmp-siti-yuv10: CMD = cmp_metadata siti yuv420p10le 0.015 +FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-yuvj +fate-filter-refcmp-siti-yuvj: CMD = cmp_metadata siti yuvj420p 0.015 + +FATE_FILTER_CMP_METADATA-$(CONFIG_SITI_FILTER) += fate-filter-refcmp-siti-si-yuvj +fate-filter-refcmp-siti-si-yuvj: CMD = cmp_metadata siti=mode=si yuvj420p 0.015 + FATE_FILTER-$(call ALLYES, TESTSRC2_FILTER METADATA_FILTER WRAPPED_AVFRAME_ENCODER \ NULL_MUXER PIPE_PROTOCOL) += $(FATE_FILTER_CMP_METADATA-yes) diff --git a/tests/ref/fate/filter-refcmp-siti-si-yuvj b/tests/ref/fate/filter-refcmp-siti-si-yuvj new file mode 100644 index 00000000000..ad62a9a1ad1 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-siti-si-yuvj @@ -0,0 +1,10 @@ +frame:0 pts:0 pts_time:0 +lavfi.siti.si=106.84 +frame:1 pts:1 pts_time:1 +lavfi.siti.si=109.59 +frame:2 pts:2 pts_time:2 +lavfi.siti.si=109.37 +frame:3 pts:3 pts_time:3 +lavfi.siti.si=113.38 +frame:4 pts:4 pts_time:4 +lavfi.siti.si=111.04 diff --git a/tests/ref/fate/filter-refcmp-siti-yuvj b/tests/ref/fate/filter-refcmp-siti-yuvj new file mode 100644 index 00000000000..3ad687465f9 --- /dev/null +++ b/tests/ref/fate/filter-refcmp-siti-yuvj @@ -0,0 +1,15 @@ +frame:0 pts:0 pts_time:0 +lavfi.siti.si=106.84 +lavfi.siti.ti=0.00 +frame:1 pts:1 pts_time:1 +lavfi.siti.si=109.59 +lavfi.siti.ti=28.07 +frame:2 pts:2 pts_time:2 +lavfi.siti.si=109.37 +lavfi.siti.ti=28.38 +frame:3 pts:3 pts_time:3 +lavfi.siti.si=113.38 +lavfi.siti.ti=33.45 +frame:4 pts:4 pts_time:4 +lavfi.siti.si=111.04 +lavfi.siti.ti=30.59 -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
