Kyrylo Tkachov <[email protected]> writes: > Hi Tamar, > >> On 16 Sep 2026, at 15:45, Tamar Christina <[email protected]> wrote: >> >> Hi Kyrill, >> >>> -----Original Message----- >>> From: [email protected] <[email protected]> >>> Sent: 15 September 2026 12:12 >>> To: [email protected] >>> Cc: Tamar Christina <[email protected]>; Kyrylo Tkachov >>> <[email protected]> >>> Subject: [PATCH] aarch64: Use exact trip counts to select SVE or Advanced >>> SIMD >>> >>> From: Kyrylo Tkachov <[email protected]> >>> >>> aarch64_vector_costs::better_main_loop_than_p normally compares two >>> cycles-per-element rates. It multiplies each cost per vector iteration >>> by the other candidate's vectorization factor to avoid division. >>> >>> This comparison can misrank a fixed-count loop that fits in one vector >>> iteration of both candidates. Both candidates execute once, but the >>> cross multiplication gives the wider vector credit for lanes that the >>> loop does not use. >>> >>> For example: >>> >>> void >>> wadd8 (int16_t *restrict d, const uint8_t *restrict a, >>> const uint8_t *restrict b) >>> { >>> for (int i = 0; i < 8; ++i) >>> d[i] = (int16_t) (a[i] + b[i]); >>> } >>> >>> At -O3 -mcpu=olympus the previous comparison selects masked SVE code: >>> >>> ldr d31, [x1] >>> ptrue p15.b, vl8 >>> uunpklo z28.h, z31.b >>> punpklo p6.h, p15.b >>> punpkhi p7.h, p15.b >>> uunpkhi z31.h, z31.b >>> ldr d30, [x2] >>> uunpklo z29.h, z30.b >>> uunpkhi z30.h, z30.b >>> add z28.h, z29.h, z28.h >>> add z31.h, z30.h, z31.h >>> st1h z31.h, p7, [x0, #1, mul vl] >>> st1h z28.h, p6, [x0] >>> >>> When the exact trip count is no greater than either candidate's >>> vectorization factor, compare the two costs per vector iteration >>> directly. Do this only for loops without early exits. Keep the >>> existing comparison for unknown counts and for loops that need more >>> than one iteration of either candidate. >>> >>> This selects Advanced SIMD for the example: >>> >>> ldr d30, [x1] >>> ldr d31, [x2] >>> uaddl v30.8h, v31.8b, v30.8b >>> str q30, [x0] >>> >>> The tests also cover a fixed count that needs an Advanced SIMD >>> remainder, a symbolic bounded count, and an early-exit loop. These >>> cases keep their previous SVE choice. >>> >> >> It was confusing to me why this was happening, and more importantly why only >> with the Olympus cost model. So I had spent a few hours digging into this. >> >> The difference is that the cost model says the CPU is wide enough that the >> unrolled instructions can all issue in the same cycle. So in terms of cycles >> per iteration it is correct in saying that V4QI unrolled twice has the >> same cost as >> V8QI not unrolled. >> >> What was confusing to me is that I know we already take this into >> account during >> costing, where we cap the VF to MIN (known_iters, vf) such that the >> code takes >> into account whether unrolling has any benefits and how much the vector >> bandwidth is used. >> >> This naturally gets it to prefer in cases of a tie the code with the >> higher VF (and so >> lower cycles per iterations). >> >> But it looks like we did this when comparing scalar vs vector but >> forgot it when >> comparing the two vector loops which is why the weighted cycles per >> iterations >> show up differently in the dump file between initial costing and cross >> mode costing. >> >> Does the attach patch instead fix the workload you wanted this for? >> (It fixes the >> testcases at least). >> > > Yes, it does.
LGTM too FWIW. Thanks for doing this. Richard
