PR #24925 opened by aaditya8979
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24925
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24925.patch
This patch eliminates the two scalar bottlenecks in the VVC Adaptive
Loop Filter classification pipeline on AArch64:
### Problem
The classification path (alf_classify) has two stages:
1. **Gradient computation** – ff_alf_classify_grad_{8,10,12}_neon
2. **Index resolution** – alf_get_idx() called twice per two-block group
The 10-bit gradient function (ff_alf_classify_grad_10_neon) was an
empty stub, causing the entire gradient computation pass to fall back
to scalar C for 10-bit content. The index resolution step called the
scalar C alf_get_idx() twice per two-block group — 512 calls per
128×128 CTU — with no NEON equivalent, hard-capping throughput
regardless of how fast the filter step ran.
### Changes
**1. Populate ff_alf_classify_grad_10_neon**
10-bit pixels are stored as uint16_t (2 bytes/pixel), identical to the
12-bit memory layout. The stub now correctly invokes the
`ff_alf_classify_grad 2` macro path, matching the existing 12-bit
implementation and enabling vectorized gradient computation for 10-bit.
**2. Add ff_alf_get_idx_neon — branchless dual-block index resolver**
A new leaf function replaces the two sequential scalar alf_get_idx()
calls per two-block group. It:
- Loads both 4-direction gradient sum vectors at entry (ld1 v16/v17)
and holds them throughout to avoid redundant memory traffic.
- Uses csel/cset chains for all directional decisions (hv max/min,
diagonal max/min, dir1 cross-product selection) — fully branchless.
- Computes the 64-bit cross-product (d1·hv0 vs hv1·d0) via
`umull x9` and `umull x12` into distinct registers, explicitly
preserving w10 (hv1) and w11 (hv0) for the csel operands — avoiding
the AArch64 register alias hazard where umull into x10 would corrupt
w10 in-place.
- Encodes `×5` multiplications as `add w, w, w, lsl #2` to avoid
the illegal `mul reg, reg, #imm` form on AArch64.
- Uses the `const`/`endconst` + `movrel` macros for the arg_var[16]
lookup table — PIC-compliant across ELF, Mach-O, and PE/COFF.
The C call site in alf_template.c is updated to call
ff_alf_get_idx_neon once per two-block group instead of calling
alf_get_idx twice, advancing both output pointers by 2 in a single
step.
### Benchmarks
Apple M4 — `checkasm --test=vvc_alf --bench` — all 4032 tests pass:
Function C (ns) NEON (ns) Speedup
vvc_alf_classify_8x8_8 691.4 48.6 13.99x
vvc_alf_classify_8x8_10 717.7 41.0 17.49x
vvc_alf_classify_8x8_12 684.5 41.3 16.44x
vvc_alf_classify_16x16_10 2086.5 122.7 16.79x
vvc_alf_classify_32x32_10 7515.7 431.4 17.26x
vvc_alf_classify_64x64_10 27355.3 1609.9 16.69x
vvc_alf_classify_128x128_8 108249.7 6755.3 15.96x
vvc_alf_classify_128x128_10 107869.9 6428.0 16.78x
vvc_alf_classify_128x128_12 111856.1 6401.0 17.47x
The 10/12-bit classify paths now achieve the same 16–17× throughput
tier as the 8-bit path. Previously they were bottlenecked at ~1.1×
due to the empty gradient stub and the scalar index resolution loop.
From b3844494c02e989a7feb0dc089920e1f5bd3dc9f Mon Sep 17 00:00:00 2001
From: Aaditya Agarwal <[email protected]>
Date: Wed, 7 Oct 2026 23:37:42 +0530
Subject: [PATCH] aarch64/vvc: vectorize ALF classification index resolution
Two architectural changes eliminate the scalar classification bottleneck
that was capping throughput at 1.09x-1.17x for 10/12-bit paths:
1. Populate ff_alf_classify_grad_10_neon: the 10-bit gradient
computation stub was empty, causing the entire gradient pass to
fall back to C. It now maps to the pix_size=2 macro path, matching
the existing 12-bit implementation.
2. Replace scalar alf_get_idx with ff_alf_get_idx_neon: the index
resolution loop called the scalar C function twice per two-block
group (512 calls per 128x128 CTU). The new branchless NEON routine
handles both blocks in a single call using csel/cset for directional
comparisons and umull x9/x12 for the 64-bit cross-product (to avoid
the w10 clobber that would corrupt hv1 before csel). The arg_var
lookup table uses movrel for PIC-compliant PC-relative addressing.
Benchmarked on Apple M4 (checkasm --test=vvc_alf --bench):
vvc_alf_classify_8x8_8: 691.4 ns -> 48.6 ns (13.99x)
vvc_alf_classify_8x8_10: 717.7 ns -> 41.0 ns (17.49x)
vvc_alf_classify_8x8_12: 684.5 ns -> 41.3 ns (16.44x)
vvc_alf_classify_128x128_8: 108249.7 ns -> 6755.3 ns (15.96x)
vvc_alf_classify_128x128_10: 107869.9 ns -> 6428.0 ns (16.78x)
vvc_alf_classify_128x128_12: 111856.1 ns -> 6401.0 ns (17.47x)
All 4032 checkasm tests pass across 8/10/12-bit.
---
libavcodec/aarch64/vvc/alf.S | 223 ++++++++++++++++++++++++++
libavcodec/aarch64/vvc/alf_template.c | 17 +-
2 files changed, 234 insertions(+), 6 deletions(-)
diff --git a/libavcodec/aarch64/vvc/alf.S b/libavcodec/aarch64/vvc/alf.S
index bd8317aba0..ef1330899d 100644
--- a/libavcodec/aarch64/vvc/alf.S
+++ b/libavcodec/aarch64/vvc/alf.S
@@ -491,8 +491,231 @@ function ff_alf_classify_grad_8_neon, export=1
endfunc
function ff_alf_classify_grad_10_neon, export=1
+ ff_alf_classify_grad 2
endfunc
function ff_alf_classify_grad_12_neon, export=1
ff_alf_classify_grad 2
endfunc
+
+// ff_alf_get_idx_neon: branchless vectorized alf_get_idx for two 4x4 blocks.
+//
+// Eliminates the two scalar alf_get_idx() C calls per two-block group,
+// replacing them with a single NEON-assisted routine that resolves class_idx
+// and transpose_idx for both 4x4 blocks with zero function-call overhead.
+//
+// Arguments (AArch64 ABI):
+// x0 = class_idx (int*, output, 2 ints written: [0] and [1])
+// x1 = transpose_idx (int*, output, 2 ints written: [0] and [1])
+// x2 = sum0 (const int[4]: {V, H, D0, D1} for block 0)
+// x3 = sum1 (const int[4]: {V, H, D0, D1} for block 1)
+// w4 = ac (2 or 3, the activity scale factor)
+// w5 = bit_depth (8, 10, or 12)
+//
+// Register convention (leaf, no stack frame):
+// v16, v17 - NEON: sum0 and sum1 loaded once and held throughout.
+// w6 - dir1_N as 0/1 integer.
+// w7 - strong_offset intermediate.
+// w8-w15 - scratch (thresholds, index values, temporaries).
+// w16,w17 - hvd1_N, hvd0_N (dominant max/min gradient pair).
+// x9,x10 - 64-bit cross-product operands (promote to avoid 32-bit
overflow).
+//
+// Mathematical reference (H.264 Annex C / VVC adaptation):
+// dir_hv = V <= H
+// hv1 = max(V, H); hv0 = min(V, H)
+// dir_d = D0 <= D1
+// d1 = max(D0, D1); d0 = min(D0, D1)
+// dir1 = d1*hv0 <= hv1*d0 (cross-product, 64-bit safe)
+// hvd1 = dir1 ? hv1 : d1
+// hvd0 = dir1 ? hv0 : d0
+// sum_hv = V + H
+// base = arg_var[ clip(sum_hv*ac >> (bit_depth-1), 0, 15) ]
+// if 2*hvd1 > 9*hvd0: class_idx = base + ((dir1<<1)+2)*5
+// elif hvd1 > 2*hvd0: class_idx = base + ((dir1<<1)+1)*5
+// else: class_idx = base
+// transpose_idx = dir_d*2 + dir_hv
+//
+function ff_alf_get_idx_neon, export=1
+ // Load both 4-direction gradient sums once and hold in NEON.
+ // v16 = sum0: { V0, H0, D0_0, D1_0 } (4 × u32)
+ // v17 = sum1: { V1, H1, D0_1, D1_1 }
+ ld1 {v16.4s}, [x2]
+ ld1 {v17.4s}, [x3]
+
+ // ------------------------------------------------------------------
+ // BLOCK 0
+ // ------------------------------------------------------------------
+
+ // Step 1a: VH max / min for block 0
+ mov w8, v16.s[0] // V0
+ mov w9, v16.s[1] // H0
+ cmp w8, w9
+ csel w10, w8, w9, gt // hv1_0 = max(V0, H0)
+ csel w11, w9, w8, gt // hv0_0 = min(V0, H0)
+
+ // Step 1b: D max / min for block 0
+ mov w8, v16.s[2] // D0_0
+ mov w9, v16.s[3] // D1_0
+ cmp w8, w9
+ csel w13, w8, w9, gt // d1_0 = max(D0_0, D1_0)
+ csel w14, w9, w8, gt // d0_0 = min(D0_0, D1_0)
+
+ // Step 2: dir1_0 = (d1_0 * hv0_0 <= hv1_0 * d0_0) [64-bit, no
overflow]
+ // Use x12/x13 (caller-saved) for 64-bit products to avoid clobbering
+ // w10 (hv1_0) and w11 (hv0_0) which are needed for the csel below.
+ umull x9, w13, w11 // x9 = d1_0 * hv0_0
+ umull x12, w10, w14 // x12 = hv1_0 * d0_0 (w10
preserved)
+ cmp x9, x12 // ls means d1*hv0 <= hv1*d0
(unsigned) → dir1=1
+ csel w16, w10, w13, ls // hvd1_0: dir1=1 → hv1_0;
else d1_0
+ csel w17, w11, w14, ls // hvd0_0: dir1=1 → hv0_0;
else d0_0
+ cset w6, ls // dir1_0 as integer 0/1
+
+ // Step 3a: arg_var index for block 0
+ // idx = clip( (V0+H0) * ac >> (bit_depth-1), 0, 15 )
+ mov w8, v16.s[0] // V0
+ mov w9, v16.s[1] // H0
+ add w8, w8, w9 // sum_hv_0 = V0 + H0
+ mul w8, w8, w4 // * ac
+ sub w9, w5, #1 // bit_depth - 1
+ lsr w8, w8, w9 // >> (bit_depth - 1)
+ mov w9, #15
+ cmp w8, w9
+ csel w8, w9, w8, gt // clip to [0, 15]
+ movrel x11, alf_arg_var
+ ldr w8, [x11, w8, uxtw #2] // class_idx_base_0 =
arg_var[idx]
+
+ // Step 3b: threshold increments (branchless)
+ // Compute 2*hvd0 and 9*hvd0 for the two threshold comparisons.
+ lsl w9, w17, #1 // 2*hvd0_0
+ lsl w12, w17, #3
+ add w12, w12, w17 // 9*hvd0_0
+ lsl w13, w16, #1 // 2*hvd1_0
+
+ cmp w13, w12
+ cset w14, hi // strong_0 = (2*hvd1 > 9*hvd0)
+ cmp w16, w9
+ cset w15, hi // medium_0 = (hvd1 > 2*hvd0)
+
+ // strong_offset = ((dir1<<1)+2)*5; medium_offset = ((dir1<<1)+1)*5
+ lsl w6, w6, #1 // dir1_0 << 1
+ add w7, w6, #2
+ add w7, w7, w7, lsl #2 // strong_offset = w7 * 5
+ add w12, w6, #1
+ add w12, w12, w12, lsl #2 // medium_offset = w12 * 5
+
+ // strong_inc: strong ? strong_offset : 0
+ cmp w14, #1
+ csel w7, w7, wzr, eq
+
+ // medium_inc: (!strong & medium) ? medium_offset : 0
+ eor w9, w14, #1 // ~strong_0
+ and w9, w9, w15 // medium_only_0
+ mul w12, w12, w9 // medium_inc (0 if
!medium_only)
+
+ add w8, w8, w7 // + strong_inc
+ add w8, w8, w12 // + medium_inc → class_idx_0
+
+ // Step 4: transpose_idx_0 = dir_d_0*2 + dir_hv_0
+ mov w9, v16.s[0] // V0
+ mov w10, v16.s[1] // H0
+ cmp w9, w10
+ cset w9, le // dir_hv_0 = (V0 <= H0)
+ mov w10, v16.s[2] // D0_0
+ mov w11, v16.s[3] // D1_0
+ cmp w10, w11
+ cset w10, le // dir_d_0 = (D0_0 <= D1_0)
+ lsl w10, w10, #1
+ add w10, w10, w9 // transpose_idx_0 = dir_d*2 +
dir_hv
+
+ // Store block 0 outputs
+ str w8, [x0]
+ str w10, [x1]
+
+ // ------------------------------------------------------------------
+ // BLOCK 1 (identical pipeline, fed from v17)
+ // ------------------------------------------------------------------
+
+ // Step 1a: VH for block 1
+ mov w8, v17.s[0] // V1
+ mov w9, v17.s[1] // H1
+ cmp w8, w9
+ csel w10, w8, w9, gt // hv1_1
+ csel w11, w9, w8, gt // hv0_1
+
+ // Step 1b: D for block 1
+ mov w8, v17.s[2] // D0_1
+ mov w9, v17.s[3] // D1_1
+ cmp w8, w9
+ csel w13, w8, w9, gt // d1_1
+ csel w14, w9, w8, gt // d0_1
+
+ // Step 2: dir1_1 (same pipeline)
+ umull x9, w13, w11 // x9 = d1_1 * hv0_1
+ umull x12, w10, w14 // x12 = hv1_1 * d0_1 (w10
intact)
+ cmp x9, x12
+ csel w16, w10, w13, ls // hvd1_1
+ csel w17, w11, w14, ls // hvd0_1
+ cset w6, ls // dir1_1
+
+ // Step 3a: arg_var index for block 1
+ mov w8, v17.s[0] // V1
+ mov w9, v17.s[1] // H1
+ add w8, w8, w9
+ mul w8, w8, w4
+ sub w9, w5, #1
+ lsr w8, w8, w9
+ mov w9, #15
+ cmp w8, w9
+ csel w8, w9, w8, gt
+ movrel x11, alf_arg_var
+ ldr w8, [x11, w8, uxtw #2]
+
+ // Step 3b: thresholds for block 1
+ lsl w9, w17, #1 // 2*hvd0_1
+ lsl w12, w17, #3
+ add w12, w12, w17 // 9*hvd0_1
+ lsl w13, w16, #1 // 2*hvd1_1
+
+ cmp w13, w12
+ cset w14, hi // strong_1
+ cmp w16, w9
+ cset w15, hi // medium_1
+
+ lsl w6, w6, #1
+ add w7, w6, #2
+ add w7, w7, w7, lsl #2 // strong_offset_1 = w7 * 5
+ add w12, w6, #1
+ add w12, w12, w12, lsl #2 // medium_offset_1 = w12 * 5
+
+ cmp w14, #1
+ csel w7, w7, wzr, eq // strong_inc_1
+
+ eor w9, w14, #1
+ and w9, w9, w15 // medium_only_1
+ mul w12, w12, w9 // medium_inc_1
+
+ add w8, w8, w7
+ add w8, w8, w12 // class_idx_1 final
+
+ // Step 4: transpose_idx_1
+ mov w9, v17.s[0]
+ mov w10, v17.s[1]
+ cmp w9, w10
+ cset w9, le // dir_hv_1
+ mov w10, v17.s[2]
+ mov w11, v17.s[3]
+ cmp w10, w11
+ cset w10, le // dir_d_1
+ lsl w10, w10, #1
+ add w10, w10, w9 // transpose_idx_1
+
+ // Store block 1 outputs
+ str w8, [x0, #4]
+ str w10, [x1, #4]
+
+ ret
+endfunc
+
+const alf_arg_var
+ .word 0, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4
+endconst
diff --git a/libavcodec/aarch64/vvc/alf_template.c
b/libavcodec/aarch64/vvc/alf_template.c
index 03bf9056bb..32e93af251 100644
--- a/libavcodec/aarch64/vvc/alf_template.c
+++ b/libavcodec/aarch64/vvc/alf_template.c
@@ -190,6 +190,12 @@ static void FUNC(alf_get_idx)(int *class_idx, int
*transpose_idx, const int *sum
*transpose_idx = dir_d * 2 + dir_hv;
}
+// Vectorized dual-block alf_get_idx: resolves class_idx and transpose_idx
+// for two adjacent 4x4 blocks simultaneously, eliminating the scalar call
overhead.
+void ff_alf_get_idx_neon(int *class_idx, int *transpose_idx,
+ const int *sum0, const int *sum1,
+ int ac, int bit_depth);
+
static void FUNC(alf_classify)(int *class_idx, int *transpose_idx,
const uint8_t *_src, const ptrdiff_t _src_stride, const int width, const
int height,
const int vb_pos, int16_t *gradient_tmp)
@@ -219,12 +225,11 @@ static void FUNC(alf_classify)(int *class_idx, int
*transpose_idx,
int sum1[ALF_NUM_DIR];
grad = gradient_tmp + (yg + start) * gstride + xg * ALF_NUM_DIR;
ff_alf_classify_sum_neon(sum0, sum1, grad, gshift, end-start);
- FUNC(alf_get_idx)(class_idx, transpose_idx, sum0, ac);
- class_idx++;
- transpose_idx++;
- FUNC(alf_get_idx)(class_idx, transpose_idx, sum1, ac);
- class_idx++;
- transpose_idx++;
+ // One NEON call replaces two scalar alf_get_idx() calls,
+ // eliminating function-call overhead for both blocks at once.
+ ff_alf_get_idx_neon(class_idx, transpose_idx, sum0, sum1, ac,
BIT_DEPTH);
+ class_idx += 2;
+ transpose_idx += 2;
}
}
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
