Ping. https://gcc.gnu.org/pipermail/gcc-patches/2026-June/719927.html
Thanks, Kyrill > On 24 Aug 2026, at 19:01, Kyrylo Tkachov <[email protected]> wrote: > > Ping. > Thanks, > Kyrill > >> On 10 Jun 2026, at 13:37, Kyrylo Tkachov <[email protected]> wrote: >> >> From: Kyrylo Tkachov <[email protected]> >> >> eliminated_by_inlining_prob only credited a write to the return value as >> free-after-inlining when the RHS was a register or invariant (a scalar >> "return x;"). A function that returns an aggregate by value >> ("V t = a; ...; return t;") therefore had the copy of the result into the >> return slot charged in full to the inliner's growth estimate, even though >> return-slot optimization / SRA eliminates that copy once the call is inlined >> (the inlined callee writes the caller's destination in place). >> >> For a small value-type wrapper -- a SIMD/array lane type, a std::array-like >> POD, std::complex, a short Tensor/Matrix -- whose overloaded operators are >> written "V op(const V &a, const V &b) { V t = a; t op= b; return t; }", the >> mispriced return-slot copy pushes the wrapper over --param >> early-inlining-insns, so the early inliner leaves it out of line. Such >> operators are called pervasively, and keeping them out of line also forces >> their operands and result through memory in the caller. >> >> Credit aggregate copies whose destination is the return value (a RESULT_DECL, >> or a store through the invisible-reference return pointer) as eliminated by >> inlining, mirroring the existing treatment of reads of by-reference >> parameters. Statements with volatile operands are excluded: a volatile >> access is never elided, so such a copy is not free after inlining. >> >> This is a size-model refinement only; no IR is changed and no semantics are >> affected. >> >> This gives about 8.5% on an aarch64 machine on the 766.femflow_r >> benchmark from SPEC2026 where these routines are critical to performance >> and their vectorised form needs to be inlined for good performance. >> >> Bootstrapped and tested on aarch64-none-linux-gnu. >> >> Signed-off-by: Kyrylo Tkachov <[email protected]> >> >> gcc/ChangeLog: >> >> * ipa-fnsummary.cc (eliminated_by_inlining_prob): Credit aggregate >> copies into the return slot (a RESULT_DECL or invisible-reference >> return) as eliminated by inlining. >> >> gcc/testsuite/ChangeLog: >> >> * g++.dg/ipa/inline-aggregate-retval.C: New test. >> * gcc.dg/ipa/inline-aggregate-retval.c: New test. >> --- >> gcc/ipa-fnsummary.cc | 27 ++++++++++++++++ >> .../g++.dg/ipa/inline-aggregate-retval.C | 21 +++++++++++++ >> .../gcc.dg/ipa/inline-aggregate-retval.c | 31 +++++++++++++++++++ >> 3 files changed, 79 insertions(+) >> create mode 100644 gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C >> create mode 100644 gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c >> >> diff --git a/gcc/ipa-fnsummary.cc b/gcc/ipa-fnsummary.cc >> index a6ccdb1852a..c05ffc313ed 100644 >> --- a/gcc/ipa-fnsummary.cc >> +++ b/gcc/ipa-fnsummary.cc >> @@ -1476,6 +1476,33 @@ eliminated_by_inlining_prob (ipa_func_body_info *fbi, >> gimple *stmt) >> if (lhs_free >> && (is_gimple_reg (rhs) || is_gimple_min_invariant (rhs))) >> rhs_free = true; >> + /* A by-value aggregate return ("T t = ...; return t;") stores the >> + result into the return slot. Under the return-slot optimization the >> + inlined callee writes the caller's destination in place and SRA then >> + scalarises the small aggregate, so the store is eliminated by inlining >> + -- the return-value analogue of the by-reference parameter reads >> + credited above. The is_gimple_reg/is_gimple_min_invariant test only >> + credits scalar returns; also credit an aggregate return whose source is >> + an ordinary memory value. This is the value-type wrapper idiom >> + (a SIMD/array lane type, std::array, std::complex, a short Tensor): >> + "T operator OP (const T &a, const T &b) { T t = a; t OP= b; return t; >> }". >> + This is the whole-aggregate-copy case. */ >> + if (!rhs_free >> + && !gimple_has_volatile_ops (stmt) >> + && AGGREGATE_TYPE_P (TREE_TYPE (lhs)) >> + && gimple_assign_rhs_class (stmt) == GIMPLE_SINGLE_RHS >> + && (TREE_CODE (inner_lhs) == RESULT_DECL >> + || (TREE_CODE (inner_lhs) == MEM_REF >> + && TREE_CODE (TREE_OPERAND (inner_lhs, 0)) == SSA_NAME >> + && SSA_NAME_VAR (TREE_OPERAND (inner_lhs, 0)) >> + && TREE_CODE (SSA_NAME_VAR (TREE_OPERAND (inner_lhs, 0))) >> + == RESULT_DECL)) >> + && (TREE_CODE (inner_rhs) == MEM_REF >> + || TREE_CODE (inner_rhs) == COMPONENT_REF >> + || VAR_P (inner_rhs) >> + || TREE_CODE (inner_rhs) == PARM_DECL >> + || TREE_CODE (inner_rhs) == RESULT_DECL)) >> + return 2; >> if (lhs_free && rhs_free) >> return 1; >> } >> diff --git a/gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C >> b/gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C >> new file mode 100644 >> index 00000000000..a60bad00b66 >> --- /dev/null >> +++ b/gcc/testsuite/g++.dg/ipa/inline-aggregate-retval.C >> @@ -0,0 +1,21 @@ >> +// The value-type operator-overload idiom: a by-value 'operator*' returns >> its >> +// result aggregate, and the copy of that result into the return slot is >> +// eliminated by inlining (return-slot optimization / SRA). It used to be >> +// charged to the early-inliner growth, pushing the operator over >> +// --param early-inlining-insns. -fno-inline-functions isolates the early >> +// inliner so the operator is inlined only when the copy is credited. >> +// { dg-do compile } >> +// { dg-options "-O3 --param early-inlining-insns=14 -fno-inline-functions >> -fno-partial-inlining -fdump-tree-einline-optimized" } >> + >> +struct V { >> + double d[8]; >> + V &operator*= (const V &o) { for (int i = 0; i < 8; i++) d[i] *= o.d[i]; >> return *this; } >> +}; >> + >> +static V operator* (const V &a, const V &b) { V t = a; return t *= b; } >> + >> +V ga, gb, gc, r; >> + >> +void use () { r = ga * gb * gc; } >> + >> +// { dg-final { scan-tree-dump "Inlining.*operator.*into.*use" "einline" } } >> diff --git a/gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c >> b/gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c >> new file mode 100644 >> index 00000000000..e21cff3438f >> --- /dev/null >> +++ b/gcc/testsuite/gcc.dg/ipa/inline-aggregate-retval.c >> @@ -0,0 +1,31 @@ >> +/* A small by-value wrapper that returns its aggregate result by value. The >> + copy of the result into the return slot is eliminated by inlining >> + (return-slot optimization / SRA combines it with the caller's >> destination), >> + but it used to be charged in full to the early-inliner growth estimate, >> + pushing the wrapper over --param early-inlining-insns and leaving it out >> of >> + line. -fno-inline-functions isolates the early inliner, so 'mul' can >> only >> + be inlined when the return-slot copy is correctly credited as >> eliminable. */ >> +/* { dg-do compile } */ >> +/* { dg-options "-O3 --param early-inlining-insns=14 -fno-inline-functions >> -fno-partial-inlining -fdump-tree-einline-optimized" } */ >> + >> +struct V { double d[8]; }; >> + >> +static struct V >> +mul (const struct V *a, const struct V *b) >> +{ >> + struct V t = *a; >> + for (int i = 0; i < 8; i++) >> + t.d[i] *= b->d[i]; >> + return t; >> +} >> + >> +struct V ga, gb, gc, r; >> + >> +void >> +use (void) >> +{ >> + struct V x = mul (&ga, &gb); >> + r = mul (&x, &gc); >> +} >> + >> +/* { dg-final { scan-tree-dump "Inlining mul.* into use" "einline" } } */ >> -- >> 2.50.1 (Apple Git-155) >> >
