On Tue, 6 Oct 2026, [email protected] wrote: > From: Kyrylo Tkachov <[email protected]> > > genmatch writes one function for each outer operation, and output > partitioning places those functions whole. One operation matcher can > therefore dominate an output part: on trunk gimple_simplify_COND_EXPR is a > single function of over 58000 lines, and the largest GIMPLE part is 2.42 > times the size of the smallest. Compile time grows faster than function > size, so that part is also the slowest to compile and sets the critical path > of a parallel build. > > Split a large per-operation matcher between its top-level alternatives. The > entry function keeps the dispatch, and calls the function that holds the > selected group of alternatives in place of the group's body. > > Before, one function holds the dispatch and every alternative it reaches: > > gimple_simplify_COND_EXPR (gimple_match_op *res_op, ...) > { > switch (TREE_CODE (_p0)) > { > case SSA_NAME: > if (gimple *_d1 = get_def (valueize, _p0)) > { > if (gassign *_a1 = dyn_cast <gassign *> (_d1)) > switch (gimple_assign_rhs_code (_a1)) > { > case LT_EXPR: > { > tree _q20 = gimple_assign_rhs1 (_a1); > ... > } > ... > case NE_EXPR: > { ... } > > After, the entry holds the dispatch and the runs of alternatives that are too > small to move: > > gimple_simplify_COND_EXPR (gimple_match_op *res_op, ...) > { > switch (TREE_CODE (_p0)) > { > case SSA_NAME: > if (gimple *_d1 = get_def (valueize, _p0)) > { > if (gassign *_a1 = dyn_cast <gassign *> (_d1)) > switch (gimple_assign_rhs_code (_a1)) > { > case LT_EXPR: > case GE_EXPR: > case GT_EXPR: > case LE_EXPR: > case EQ_EXPR: > { > if (genmatch_gimple_simplify_COND_EXPR_chunk_1 (res_op, > seq, valueize, code, type, _p0, _p1, _p2)) > return true; > break; > } > case NE_EXPR: > { > if (genmatch_gimple_simplify_COND_EXPR_chunk_2 (...)) > return true; > break; > } > ... > } > switch (TREE_CODE (_p1)) > { > case INTEGER_CST: > ... <- a small run, kept here > > and the groups that move are packed into new functions, each placed in the > shortest output part other than the one that holds the entry function: > > genmatch_gimple_simplify_COND_EXPR_chunk_1 (gimple_match_op *res_op, ...) > { > switch (TREE_CODE (_p0)) > { > case SSA_NAME: > if (gimple *_d1 = get_def (valueize, _p0)) > { > if (gassign *_a1 = dyn_cast <gassign *> (_d1)) > switch (gimple_assign_rhs_code (_a1)) > { > case LT_EXPR: > { > tree _q20 = gimple_assign_rhs1 (_a1); > ... > > The case labels of consecutive groups that land in the same function fall > through to one call, which is what the run of labels above is. > > The groups are the ones collect_operand_groups already forms, which merges > any two alternatives whose cases overlap. Distinct groups therefore have > disjoint cases, and selecting one is what trying them in turn does. Since the > previous patch, a pattern that fails no longer leaves the matcher, so a > pattern behaves the same whichever function holds it, and the order in which > patterns are tried does not change. The predicates and the remaining > decision-tree nodes stay in the entry function. So do two kinds of group > that the entry emits a form of whatever function holds them: the GENERIC > SSA_NAME group of a run with GIMPLE alternatives, which goes inside the > GIMPLE SSA_NAME case, and a group with a COND_EXPR that also matches a PHI. > > An order-preserving barrier in the decision tree ends one run of alternatives > and starts the next, and a function never holds groups of two runs. Within a > run, groups are packed into functions of about one tenth of the average > number of decision-tree nodes per output part. A run of at most a quarter of > that stays in the entry function, because splitting it out costs a function, > a declaration and a call but moves little between the parts. With a single > output file there is nothing to balance, so nothing is split. > > On aarch64 with the default ten parts, 91 functions are split out of the > GIMPLE matchers and 93 out of the GENERIC ones. The largest-to-smallest > GIMPLE part size ratio goes from 2.42 to 1.05, and the largest per-operation > matcher function is 81% smaller. Compiling the twenty generated sources in > parallel with the host compiler at -O2, the slowest file takes about a third > of the time it did, and the total CPU time falls by about a third because a > large function costs more than its size to compile. A bootstrap at -j352 on > a 352-thread aarch64 machine takes 12.1% less time. > > The .text of cc1 and cc1plus grows by 0.35% and 0.13%. Over seven large > preprocessed translation units, retired instructions change by +0.004%. > Improving bootstrap time is the main motivation for this patch. > I'm hoping that a <1% cc1 growth is worth the >10% bootstrap time > improvement. The cc1 size has already been reduced in this cycle with > g:4756a4c0cfaeb18e7fff9108cd9c5d1e1e116690 and the other patches in this > series try to improve it further. > > Bootstrapped and tested on aarch64-none-linux-gnu. > Ok for trunk?
OK. Thanks, Richard. > Thanks, > Kyrill > > gcc/ChangeLog: > > * genmatch.cc (MATCHER_FN_FILE_FRACTION) > (MATCHER_FN_MIN_RUN_FRACTION): New constants. > (ssa_name_group_p, phi_cond_group_p): New functions. > (class fn_splitter): New class. > (fn_splitter::split_out_p, fn_splitter::same_chunk_p) > (fn_splitter::full, fn_splitter::print_params) > (fn_splitter::print_args, fn_splitter::choose_chunk_file) > (fn_splitter::open_chunk, fn_splitter::close_chunk) > (fn_splitter::emit_dispatch, collect_alt_units) > (fn_splitter::emit_alternatives): New functions. > (dt_node::gen_kids): Add an optional fn_splitter argument and use it > at valid function-level boundaries. > (dt_node::gen_kids_1): Add an optional fn_splitter argument and call > the function holding a group in place of its body. Check that the > GENERIC SSA_NAME group and groups with a PHI form stay in the entry > function. Use ssa_name_group_p. > (decision_tree::gen): Set the matcher function budget and use > fn_splitter for large per-operation matchers. > > Signed-off-by: Kyrylo Tkachov <[email protected]> > --- > gcc/genmatch.cc | 522 +++++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 494 insertions(+), 28 deletions(-) > > diff --git a/gcc/genmatch.cc b/gcc/genmatch.cc > index 1a89b7c13c2..9bb7cbbb427 100644 > --- a/gcc/genmatch.cc > +++ b/gcc/genmatch.cc > @@ -2516,6 +2516,7 @@ lower (vec<simplify *>& simplifiers, bool gimple) > simplifies and has those as its leafs. */ > > class dt_simplify; > +class fn_splitter; > > /* A hash-map collecting semantically equivalent leafs in the decision > tree for splitting out to separate functions. */ > @@ -2571,11 +2572,12 @@ public: > > virtual void gen (FILE *, int, bool, int) {} > > - void gen_kids (FILE *, int, bool, int, bool); > + void gen_kids (FILE *, int, bool, int, bool, fn_splitter * = NULL); > void gen_kids_1 (FILE *, int, bool, int, > const vec<dt_operand *> &, const vec<dt_operand *> &, > const vec<dt_operand *> &, const vec<dt_operand *> &, > - const vec<dt_operand *> &, const vec<dt_node *> &, bool); > + const vec<dt_operand *> &, const vec<dt_node *> &, bool, > + fn_splitter * = NULL); > > void analyze (sinfo_map_t &); > }; > @@ -4260,11 +4262,395 @@ emit_fn_group (FILE *f, int indent, bool gimple, int > depth, > fprintf_indent (f, indent, " break;\n"); > } > > -/* Generate matching code for the children of the decision tree node. */ > +/* A split matcher packs its alternatives into functions of about > + 1 / MATCHER_FN_FILE_FRACTION of the decision-tree nodes that one output > + file receives on average. Smaller functions spread more evenly across the > + files, but each one costs a declaration and a call. */ > + > +static const unsigned int MATCHER_FN_FILE_FRACTION = 10; > + > +/* A run of alternatives no larger than 1 / MATCHER_FN_MIN_RUN_FRACTION of > + that size stays in the entry function. */ > + > +static const unsigned int MATCHER_FN_MIN_RUN_FRACTION = 4; > + > +/* Return true if GROUP is the GENERIC group whose only case is SSA_NAME and > + GIMPLE_MATCH_P says that the matcher also has GIMPLE alternatives. Such a > + matcher emits the group inside its own SSA_NAME case, not as a case of the > + GENERIC dispatch. */ > + > +static bool > +ssa_name_group_p (const dt_operand_group *group, bool gimple_match_p) > +{ > + return (gimple_match_p > + && group->cases.length () == 1 > + && *group->cases[0] == SSA_NAME); > +} > + > +/* Return true if GROUP of GIMPLE alternatives holds a COND_EXPR that also > + matches a PHI. The function that dispatches on the definition emits the > + PHI form of such an alternative itself. */ > + > +static bool > +phi_cond_group_p (const dt_operand_group *group) > +{ > + for (auto op : group->operands) > + { > + expr *e = as_a <expr *> (op->op); > + if (*e->operation == COND_EXPR && e->match_phi) > + return true; > + } > + return false; > +} > + > +/* Split a generated matcher across functions. The entry function keeps the > + dispatch that selects between the top-level alternatives, and calls the > + function that holds the selected group in place of the group's body. */ > + > +class fn_splitter > +{ > +public: > + /* The categories of alternatives that can move to another function, in the > + order gen_kids_1 emits them. ALT_COUNT is their number. */ > + enum alt_cat > + { > + ALT_GIMPLE_EXPRS, > + ALT_FNS, > + ALT_GENERIC_EXPRS, > + ALT_GENERIC_FNS, > + ALT_COUNT > + }; > + > + /* One complete group from category CAT at INDEX. NODES estimates the > + amount of generated code for the group. */ > + struct alt_unit > + { > + alt_cat cat; > + unsigned index; > + unsigned nodes; > + }; > + > + fn_splitter (vec<FILE *> &files_, FILE *f_, const char *fname_, > + bool gimple_, unsigned nops_, unsigned budget_) > + : f (f_), files (files_), fname (fname_), gimple (gimple_), > + nops (nops_), budget (budget_), nchunks (0), dispatching (false) > + { > + gcc_checking_assert (budget); > + } > + > + void emit_alternatives (dt_node *, int, int, > + const vec<dt_operand *> &, > + const vec<dt_operand *> &, > + const vec<dt_operand *> &, > + const vec<dt_operand *> &, > + const vec<dt_operand *> &, > + const vec<dt_node *> &); > + bool emit_dispatch (FILE *, int, alt_cat, unsigned); > + > + /* Return true if, while the entry function is being emitted, group INDEX > + of category CAT is held by another function. */ > + bool split_out_p (alt_cat cat, unsigned index) const > + { > + return dispatching && index < owner[cat].length () && owner[cat][index]; > + } > + > + /* Return true if groups A and B of category CAT go to the same function, > + so that the labels of A can fall through to the call emitted for B. */ > + bool same_chunk_p (alt_cat cat, unsigned a, unsigned b) const > + { > + return (split_out_p (cat, a) > + && split_out_p (cat, b) > + && owner[cat][a] == owner[cat][b]); > + } > + > +private: > + /* Return true if adding MORE nodes to a function that holds HAVE nodes > + would take it over the budget. An empty function accepts any one > + group. */ > + bool full (unsigned have, unsigned more) const > + { > + return have && have + more > budget; > + } > + > + void print_params (FILE *); > + void print_args (FILE *); > + FILE *open_chunk (unsigned); > + void close_chunk (FILE *); > + FILE *choose_chunk_file (); > + > + /* The output file for the entry function. */ > + FILE *f; > + /* All output files used for the generated matcher. */ > + vec<FILE *> &files; > + /* The base name for new functions. */ > + const char *fname; > + /* True for GIMPLE matching and false for GENERIC matching. */ > + bool gimple; > + /* The number of operands passed to each function. */ > + unsigned nops; > + /* The target maximum number of nodes accumulated in one function. */ > + unsigned budget; > + /* The number of new functions. */ > + unsigned nchunks; > + /* True while the entry function is being emitted, when the body of a > + group is replaced by a call to the function that holds it. */ > + bool dispatching; > + /* For each category, the function holding each group, or zero for the > + entry function. */ > + auto_vec<unsigned> owner[ALT_COUNT]; > +}; > + > +/* Print the parameter list of a new function to TO. Also print its > + declaration to the generated header. */ > + > +void > +fn_splitter::print_params (FILE *to) > +{ > + if (gimple) > + fp_decl (to, " (gimple_match_op *ARG_UNUSED (res_op)," > + " gimple_seq *ARG_UNUSED (seq),\n" > + "\t\t tree (*valueize)(tree) ATTRIBUTE_UNUSED,\n" > + "\t\t code_helper ARG_UNUSED (code)," > + " tree ARG_UNUSED (type)"); > + else > + fp_decl (to, " (location_t ARG_UNUSED (loc)," > + " enum tree_code ARG_UNUSED (code),\n" > + "\t\t const tree ARG_UNUSED (type)"); > + for (unsigned i = 0; i < nops; ++i) > + fp_decl (to, ", tree ARG_UNUSED (_p%d)", i); > + fp_decl (to, ")"); > +} > + > +/* Print to TO the arguments that forward the current matcher state to a new > + function. */ > + > +void > +fn_splitter::print_args (FILE *to) > +{ > + fputs (gimple > + ? " (res_op, seq, valueize, code, type" > + : " (loc, code, type", to); > + for (unsigned i = 0; i < nops; ++i) > + fprintf (to, ", _p%d", i); > + fputs (")", to); > +} > + > +/* Return the shortest output file that is not the one the entry function is > + being written to, which must not receive another function while it is > + open. */ > + > +FILE * > +fn_splitter::choose_chunk_file () > +{ > + FILE *best = NULL; > + long best_len = 0; > + for (FILE *cand : files) > + { > + if (cand == f) > + continue; > + long len = ftell (cand); > + if (!best || len < best_len) > + { > + best = cand; > + best_len = len; > + } > + } > + /* Splitting is only enabled with more than one output file. */ > + gcc_assert (best); > + return best; > +} > + > +/* Open the function holding chunk N and return its output file. */ > + > +FILE * > +fn_splitter::open_chunk (unsigned n) > +{ > + /* Multiple output files require a header, which also carries the > + declarations for functions emitted in a different file. */ > + gcc_assert (header_file); > + > + FILE *cf = choose_chunk_file (); > + fp_decl (cf, "\n%s\n%s_chunk_%u", gimple ? "bool" : "tree", fname, n); > + print_params (cf); > + fp_decl_done (cf, ""); > + fputs ("{\n", cf); > + fprintf_indent (cf, 2, "const bool debug_dump = " > + "dump_file && (dump_flags & TDF_FOLDING);\n"); > + return cf; > +} > + > +/* Close the function being written to CF. */ > > void > -dt_node::gen_kids (FILE *f, int indent, bool gimple, int depth, bool > in_gcond) > +fn_splitter::close_chunk (FILE *cf) > +{ > + fprintf (cf, gimple ? " return false;\n}\n" : " return NULL_TREE;\n}\n"); > +} > + > +/* Called by gen_kids_1 in place of emitting the body of group INDEX of > + category CAT to TO at INDENT. If another function holds the group, emit a > + call to it and return true. Otherwise return false, and gen_kids_1 emits > + the body. */ > + > +bool > +fn_splitter::emit_dispatch (FILE *to, int indent, alt_cat cat, unsigned > index) > { > + if (!split_out_p (cat, index)) > + return false; > + > + fprintf_indent (to, indent, " {\n"); > + if (gimple) > + { > + fprintf_indent (to, indent, " if (%s_chunk_%u", fname, > + owner[cat][index]); > + print_args (to); > + fprintf (to, ")\n"); > + fprintf_indent (to, indent, " return true;\n"); > + } > + else > + { > + fprintf_indent (to, indent, " if (tree _chunk_res = %s_chunk_%u", > + fname, owner[cat][index]); > + print_args (to); > + fprintf (to, ")\n"); > + fprintf_indent (to, indent, " return _chunk_res;\n"); > + } > + fprintf_indent (to, indent, " break;\n"); > + fprintf_indent (to, indent, " }\n"); > + return true; > +} > + > +/* Append to UNITS the operand GROUPS of category CAT that can move to > + another function. GIMPLE_MATCH_P says whether the matcher has GIMPLE > + alternatives. */ > + > +static void > +collect_alt_units (vec<fn_splitter::alt_unit> &units, > + fn_splitter::alt_cat cat, > + const vec<dt_operand_group *> &groups, bool gimple_match_p) > +{ > + for (unsigned i = 0; i < groups.length (); ++i) > + { > + /* gen_kids_1 emits these groups, or a form of them, in the entry > + function whichever function holds them. */ > + if ((cat == fn_splitter::ALT_GENERIC_EXPRS > + && ssa_name_group_p (groups[i], gimple_match_p)) > + || (cat == fn_splitter::ALT_GIMPLE_EXPRS > + && phi_cond_group_p (groups[i]))) > + continue; > + /* Only alternatives with a level-one parent inspect a parameter of the > + matcher entry function. */ > + unsigned nodes = 0; > + for (auto op : groups[i]->operands) > + { > + gcc_assert (op->parent && op->parent->level == 1); > + nodes += op->total_size; > + } > + units.safe_push (fn_splitter::alt_unit { cat, i, nodes }); > + } > +} > + > +/* Emit the alternatives of NODE at INDENT and DEPTH. GIMPLE_EXPRS, > + GENERIC_EXPRS, FNS, GENERIC_FNS, PREDS and OTHERS are the vectors that > + gen_kids_1 accepts. Unless the whole run is small, the groups of the > first > + four move to new functions, packed in order up to the budget. The entry > + function keeps the dispatch, PREDS and OTHERS. Groups reached from one > + dispatch have disjoint cases, so selecting one is what trying them in turn > + does. */ > + > +void > +fn_splitter::emit_alternatives (dt_node *node, int indent, int depth, > + const vec<dt_operand *> &gimple_exprs, > + const vec<dt_operand *> &generic_exprs, > + const vec<dt_operand *> &fns, > + const vec<dt_operand *> &generic_fns, > + const vec<dt_operand *> &preds, > + const vec<dt_node *> &others) > +{ > + auto_delete_vec<dt_operand_group> group_storage[ALT_COUNT]; > + collect_operand_groups (gimple_exprs, group_storage[ALT_GIMPLE_EXPRS]); > + collect_operand_groups (fns, group_storage[ALT_FNS]); > + collect_operand_groups (generic_exprs, group_storage[ALT_GENERIC_EXPRS]); > + collect_operand_groups (generic_fns, group_storage[ALT_GENERIC_FNS]); > + > + const bool gimple_match_p = gimple_exprs.length () || fns.length (); > + auto_vec<alt_unit> units; > + for (unsigned cat = ALT_GIMPLE_EXPRS; cat < ALT_COUNT; ++cat) > + collect_alt_units (units, (alt_cat) cat, group_storage[cat], > + gimple_match_p); > + > + /* Decide which function holds each group. Zero keeps it in the entry, > + which is where a group stays when nothing has to be split out. */ > + for (unsigned cat = ALT_GIMPLE_EXPRS; cat < ALT_COUNT; ++cat) > + { > + owner[cat].truncate (0); > + owner[cat].safe_grow_cleared (group_storage[cat].length ()); > + } > + > + /* An order-preserving barrier ends a run of alternatives, so a matcher > + reaches here once per run, and a function never holds groups of two > + runs. A small run moves too little weight between the output files to > + pay for a function, a declaration and a call, so leave it in the entry > + function. */ > + unsigned total = 0; > + for (auto &u : units) > + total += u.nodes; > + if (total * MATCHER_FN_MIN_RUN_FRACTION <= budget) > + units.truncate (0); > + > + const unsigned first_chunk = nchunks + 1; > + unsigned have = 0; > + bool split_p = false; > + for (auto &u : units) > + { > + if (!split_p || full (have, u.nodes)) > + { > + ++nchunks; > + have = 0; > + split_p = true; > + } > + owner[u.cat][u.index] = nchunks; > + have += u.nodes; > + } > + > + /* The entry function: the dispatch, and whatever cannot be reached > + through it. */ > + dispatching = true; > + node->gen_kids_1 (f, indent, gimple, depth, gimple_exprs, generic_exprs, > + fns, generic_fns, preds, others, false, this); > + dispatching = false; > + > + /* Each new function, with the groups it holds. */ > + for (unsigned n = first_chunk; n <= nchunks; ++n) > + { > + auto_vec<dt_operand *> sub[ALT_COUNT]; > + for (unsigned cat = ALT_GIMPLE_EXPRS; cat < ALT_COUNT; ++cat) > + for (unsigned i = 0; i < owner[cat].length (); ++i) > + if (owner[cat][i] == n) > + for (auto op : group_storage[cat][i]->operands) > + sub[cat].safe_push (op); > + > + FILE *cf = open_chunk (n); > + auto_vec<dt_operand *> no_preds; > + auto_vec<dt_node *> no_others; > + node->gen_kids_1 (cf, 2, gimple, depth, sub[ALT_GIMPLE_EXPRS], > + sub[ALT_GENERIC_EXPRS], sub[ALT_FNS], > + sub[ALT_GENERIC_FNS], no_preds, no_others, false); > + close_chunk (cf); > + } > +} > + > +/* Generate matching code for the children of the decision tree node. If SP > + is not null, it splits the matcher between functions. */ > + > +void > +dt_node::gen_kids (FILE *f, int indent, bool gimple, int depth, bool > in_gcond, > + fn_splitter *sp) > +{ > + /* A gcond matcher starts inside an open scope and is not a valid function > + boundary. */ > + gcc_assert (!sp || !in_gcond); > + > auto_vec<dt_operand *> gimple_exprs; > auto_vec<dt_operand *> generic_exprs; > auto_vec<dt_operand *> fns; > @@ -4333,10 +4719,23 @@ dt_node::gen_kids (FILE *f, int indent, bool gimple, > int depth, bool in_gcond) > Like DT_TRUE, DT_MATCH serves as a barrier as it can cause > dependent matches to get out-of-order. Generate code now > for what we have collected sofar. */ > - gen_kids_1 (f, indent, gimple, depth, gimple_exprs, generic_exprs, > - fns, generic_fns, preds, others, in_gcond); > + if (sp) > + sp->emit_alternatives (this, indent, depth, gimple_exprs, > + generic_exprs, fns, generic_fns, preds, > + others); > + else > + gen_kids_1 (f, indent, gimple, depth, gimple_exprs, > + generic_exprs, fns, generic_fns, preds, others, > + in_gcond); > /* And output the true operand itself. */ > - kids[i]->gen (f, indent, gimple, depth); > + if (sp && kids[i]->type == dt_node::DT_TRUE) > + { > + /* A DT_TRUE node emits no code and opens no scope. Its children > + remain valid function-level split points. */ > + kids[i]->gen_kids (f, indent, gimple, depth, false, sp); > + } > + else > + kids[i]->gen (f, indent, gimple, depth); > gimple_exprs.truncate (0); > generic_exprs.truncate (0); > fns.truncate (0); > @@ -4349,11 +4748,17 @@ dt_node::gen_kids (FILE *f, int indent, bool gimple, > int depth, bool in_gcond) > } > > /* Generate code for the remains. */ > - gen_kids_1 (f, indent, gimple, depth, gimple_exprs, generic_exprs, > - fns, generic_fns, preds, others, in_gcond); > + if (sp) > + sp->emit_alternatives (this, indent, depth, gimple_exprs, generic_exprs, > + fns, generic_fns, preds, others); > + else > + gen_kids_1 (f, indent, gimple, depth, gimple_exprs, generic_exprs, > + fns, generic_fns, preds, others, in_gcond); > } > > -/* Generate matching code for the children of the decision tree node. */ > +/* Generate matching code for the children of the decision tree node. If SP > + is not null, call the function that holds a group in place of emitting its > + body. */ > > void > dt_node::gen_kids_1 (FILE *f, int indent, bool gimple, int depth, > @@ -4363,7 +4768,7 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool gimple, > int depth, > const vec<dt_operand *> &generic_fns, > const vec<dt_operand *> &preds, > const vec<dt_node *> &others, > - bool in_gcond) > + bool in_gcond, fn_splitter *sp) > { > char buf[128]; > char *kid_opname = buf; > @@ -4431,10 +4836,23 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool > gimple, int depth, > char code[64]; > snprintf (code, sizeof (code), > "gimple_assign_rhs_code (_a%d)", depth); > - for (auto group : gimple_expr_groups) > + for (unsigned gi = 0; gi < gimple_expr_groups.length (); ++gi) > { > - emit_group_labels (f, indent, group); > - emit_expr_group (f, indent, true, depth, code, group); > + /* The PHI form is emitted below, so a function holding the > + group would carry a second copy of it. */ > + gcc_checking_assert (!sp > + || !phi_cond_group_p (gimple_expr_groups[gi]) > + || !sp->split_out_p > + (fn_splitter::ALT_GIMPLE_EXPRS, gi)); > + emit_group_labels (f, indent, gimple_expr_groups[gi]); > + if (sp > + && sp->same_chunk_p (fn_splitter::ALT_GIMPLE_EXPRS, gi, > + gi + 1)) > + continue; > + if (!sp || !sp->emit_dispatch (f, indent, > + fn_splitter::ALT_GIMPLE_EXPRS, gi)) > + emit_expr_group (f, indent, true, depth, code, > + gimple_expr_groups[gi]); > } > fprintf_indent (f, indent, "default:;\n"); > fprintf_indent (f, indent, "}\n"); > @@ -4476,10 +4894,15 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool > gimple, int depth, > char code[64]; > snprintf (code, sizeof (code), > "gimple_call_combined_fn (_c%d)", depth); > - for (auto group : fn_groups) > + for (unsigned fi = 0; fi < fn_groups.length (); ++fi) > { > - emit_group_labels (f, indent, group); > - emit_fn_group (f, indent, true, depth, NULL, code, group); > + emit_group_labels (f, indent, fn_groups[fi]); > + if (sp && sp->same_chunk_p (fn_splitter::ALT_FNS, fi, fi + 1)) > + continue; > + if (!sp || !sp->emit_dispatch (f, indent, fn_splitter::ALT_FNS, > + fi)) > + emit_fn_group (f, indent, true, depth, NULL, code, > + fn_groups[fi]); > } > fprintf_indent (f, indent, "default:;\n"); > fprintf_indent (f, indent, "}\n"); > @@ -4491,11 +4914,16 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool > gimple, int depth, > fprintf_indent (f, indent, " }\n"); > /* See if there is SSA_NAME among generic_exprs and if yes, emit it > here rather than in the next loop. */ > - for (auto group : generic_expr_groups) > - for (auto id : group->cases) > + for (unsigned gei = 0; gei < generic_expr_groups.length (); ++gei) > + for (auto id : generic_expr_groups[gei]->cases) > if (*id == SSA_NAME) > { > + dt_operand_group *group = generic_expr_groups[gei]; > gcc_checking_assert (group->cases.length () == 1); > + /* A function holding the group would run it a second time. */ > + gcc_checking_assert (!sp > + || !sp->split_out_p > + (fn_splitter::ALT_GENERIC_EXPRS, gei)); > for (auto op : group->operands) > { > fprintf_indent (f, indent + 4, "{\n"); > @@ -4512,14 +4940,27 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool > gimple, int depth, > char generic_code[160]; > snprintf (generic_code, sizeof (generic_code), "TREE_CODE (%s)", > kid_opname); > - for (auto group : generic_expr_groups) > + /* The SSA_NAME group was emitted above, so it is not here to fall > + through to, and a run of labels must not be merged across it. */ > + auto emitted_above_p = [&] (unsigned i) > + { > + return ssa_name_group_p (generic_expr_groups[i], gimple_match_p); > + }; > + for (unsigned gei = 0; gei < generic_expr_groups.length (); ++gei) > { > - if (gimple_match_p > - && group->cases.length () == 1 > - && *group->cases[0] == SSA_NAME) > + dt_operand_group *group = generic_expr_groups[gei]; > + if (emitted_above_p (gei)) > continue; > emit_group_labels (f, indent, group); > - emit_expr_group (f, indent, gimple, depth, generic_code, group); > + if (sp > + && gei + 1 < generic_expr_groups.length () > + && !emitted_above_p (gei + 1) > + && sp->same_chunk_p (fn_splitter::ALT_GENERIC_EXPRS, gei, > + gei + 1)) > + continue; > + if (!sp || !sp->emit_dispatch (f, indent, > + fn_splitter::ALT_GENERIC_EXPRS, gei)) > + emit_expr_group (f, indent, gimple, depth, generic_code, group); > } > } > > @@ -4537,10 +4978,16 @@ dt_node::gen_kids_1 (FILE *f, int indent, bool > gimple, int depth, > char code[160]; > snprintf (code, sizeof (code), "get_call_combined_fn (%s)", > kid_opname); > - for (auto group : generic_fn_groups) > + for (unsigned gfi = 0; gfi < generic_fn_groups.length (); ++gfi) > { > - emit_group_labels (f, indent, group); > - emit_fn_group (f, indent, false, depth, kid_opname, code, group); > + emit_group_labels (f, indent, generic_fn_groups[gfi]); > + if (sp && sp->same_chunk_p (fn_splitter::ALT_GENERIC_FNS, gfi, > + gfi + 1)) > + continue; > + if (!sp || !sp->emit_dispatch (f, indent, > + fn_splitter::ALT_GENERIC_FNS, gfi)) > + emit_fn_group (f, indent, false, depth, kid_opname, code, > + generic_fn_groups[gfi]); > } > fprintf_indent (f, indent, "default:;\n"); > > @@ -5325,6 +5772,15 @@ decision_tree::gen (vec <FILE *> &files, bool gimple) > } > fprintf (stderr, "removed %u duplicate tails\n", rcnt); > > + /* A file receives about ROOT->TOTAL_SIZE / NFILES decision-tree nodes. > + Keep matcher functions small enough to spread them across the files. > + A function split out of a matcher goes to a file other than the one > + that holds the matcher, so a single file splits nothing. */ > + unsigned nfiles = files.length (); > + unsigned budget = (nfiles > 1 > + ? root->total_size / nfiles / MATCHER_FN_FILE_FRACTION > + : 0); > + > for (unsigned n = 1; n <= 7; ++n) > { > bool has_kids_p = false; > @@ -5367,7 +5823,17 @@ decision_tree::gen (vec <FILE *> &files, bool gimple) > fprintf (f, "{\n"); > fprintf_indent (f, 2, "const bool debug_dump = " > "dump_file && (dump_flags & TDF_FOLDING);\n"); > - dop->gen_kids (f, 2, gimple, 0, false); > + if (budget && dop->total_size > budget) > + { > + char *fname = xasprintf ("genmatch_%s_simplify_%s", > + gimple ? "gimple" : "generic", > + e->operation->id); > + fn_splitter sp (files, f, fname, gimple, n, budget); > + dop->gen_kids (f, 2, gimple, 0, false, &sp); > + free (fname); > + } > + else > + dop->gen_kids (f, 2, gimple, 0, false); > if (gimple) > fprintf (f, " return false;\n"); > else > -- Richard Biener <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri; (HRB 36809, AG Nuernberg)
