On Tue, Jul 7, 2026 at 11:55 AM Andrea Pinski <[email protected]> wrote: > > On Tue, Jul 7, 2026 at 11:38 AM Daniel Henrique Barboza > <[email protected]> wrote: > > > > Hi Andrea, > > > > On 7/4/2026 1:30 AM, Andrea Pinski wrote: > > > On Mon, May 18, 2026 at 5:16 AM Daniel Barboza > > > <[email protected]> wrote: > > >> > > >> Consider the following code that checks if a given bit is set, setting > > >> it in case it isn't: > > >> > > >> bit_val = 1 << num; > > >> if ((ptr[x] & bit_val) == 0) > > >> { > > >> ptr[x] |= bit_val; > > >> } > > >> return ptr[x]; > > >> > > >> The generated gimple is something similar to: > > >> > > >> ;; basic block 2 > > >> bitshift_6 = 1 << bit_5(D); > > >> # VUSE <.MEM_7(D)> > > >> _1 = arrD.4593[n_8(D)]; > > >> _2 = _1 & bitshift_6; > > >> if (_2 == 0) goto <bb 3>; else goto <bb 4>; > > >> > > >> ;; basic block 3 > > >> _3 = _1 | bitshift_6; > > >> # .MEM_9 = VDEF <.MEM_7(D)> > > >> arrD.4593[n_8(D)] = _3; > > >> ;; succ: 4 [always] (FALLTHRU,EXECUTABLE) > > >> > > >> ;; basic block 4, > > >> ;; prev block 3 > > >> # .MEM_4 = PHI <.MEM_7(D)(2), .MEM_9(3)> > > >> # VUSE <.MEM_4> > > >> _10 = arrD.4593[n_8(D)]; > > >> # .MEM_11 = VDEF <.MEM_4> > > >> arrD.4593 ={v} {CLOBBER(eos)}; > > >> # VUSE <.MEM_11> > > >> return _10; > > >> > > >> If we have the right conditions (e.g. we don't have store data races to > > >> worry about, we're not dealing with read-only memory) we can move the > > >> bitset operation to the cond_bb (block 2 in the example), removing the > > >> potential branch mispredict, as long as we're able to identify this "bit > > >> N is either already set or will end up being set" scenario: > > >> > > >> bitshift_6 = 1 << bit_5(D); > > >> # VUSE <.MEM_7(D)> > > >> _1 = arrD.4593[n_8(D)]; > > >> _2 = _1 & bitshift_6; > > >> _3 = _1 | bitshift_6; > > >> # .MEM_9 = VDEF <.MEM_7(D)> > > >> arrD.4593[n_8(D)] = _3; > > >> if (_2 == 0) goto <bb 3>; else goto <bb 4>; > > >> > > >> ;; basic block 3 > > >> ;; succ: 4 [always] (FALLTHRU,EXECUTABLE) > > >> (...) > > >> > > >> If the bitcheck result isn't being used as a PHI result there's a good > > >> chance that this optimization will get rid of both the bitcheck and the > > >> gcond. The 'optimized' dump for the example above looks like this: > > >> > > >> ;; basic block 2 > > >> ;; prev block 0 > > >> bitshift_6 = 1 << bit_5(D); > > >> # VUSE <.MEM_7(D)> > > >> _1 = arrD.4593[n_8(D)]; > > >> _3 = _1 | bitshift_6; > > >> # .MEM_11 = VDEF <.MEM_7(D)> > > >> arrD.4593 ={v} {CLOBBER(eos)}; > > >> # VUSE <.MEM_11> > > >> return _3; > > >> > > >> This optimization was motivated by GCC's bitmap_set_bit() before > > >> PR119482. We're also covering the bitclear equivalent of this opt > > >> (check if a bit is set, if positive clear it). The bitset > > >> transformation only works for single bits. The bitclear variation > > >> can handle single or multiple bit masks. > > >> > > >> Bootstrapped and regression tested in x86, aarch64 and RISC-V. > > > > > > Looking into this slightly. And with my patches at: > > > https://gcc.gnu.org/pipermail/gcc-patches/2026-July/722737.html > > > https://gcc.gnu.org/pipermail/gcc-patches/2026-July/722738.html > > > Which improves the generic cselim parts to reuse a previous load (or > > > store) that alone allows to use it in phiopt (patch 2). > > > > > > Then only thing left for the testcase bitset1 of pr124667.c is a > > > match pattern like: > > > ``` > > > (simplify > > > (cond (eq (bit_and:c @0 (lshift@1 integer_onep @2)) integer_zerop) > > > (bit_ior:c@3 @0 @1) @0) > > > @3) > > > ``` > > > And that will optimize it in phiopt1. (hopefully I did that correctly). > > > > > > The others are handled by: > > > ``` > > > /* If we have a "if a bit is not set, set it" case, > > > just set the bit all the time (PR 64567). Note that > > > this does not work if we're checking for more than one > > > bit, e.g. (a & 5 ? a | 5 : a) will fail for a = 1 (we > > > would return 5 instead of 1). */ > > > (simplify > > > (cond (eq (bit_and @0 INTEGER_CST@1) integer_zerop) > > > (bit_ior@2 @0 INTEGER_CST@1) @0) > > > (if (wi::popcount (wi::to_wide (@1)) == 1) > > > @2)) > > > ``` > > > So this new pattern should just be right below this one. (Oh I just > > > noticed you added that one even :) ) > > > . > > > For bitclear1, it is a bit more complex since match phiopt currently > > > does not support more than one statement in the middle bbs. We could > > > improve that to support 2 or more statements (been needing it for much > > > more complex ones even; e.g. recombining of ctz). > > > > How much work is involved in extending match.pd to match 2 stmts in > > middle_bbs? I'm taking a look at genmatch.cc and friends, trying to > > understand how the 'match_phi' flag works and so on. > > It is not about extending match.pd/genmatch but rather extending > match_simplify_replacement and it's stmt_to_move/stmt_to_move_alt. > Right now since there is no cost model figuring out if moving these > statements to be non-conditional, extending it to 2 or more statements > need to have some kind of cost associated with it. > For an example only allowing logical/casts expressions might be > enough. No FP (that is trapping in the general case anyways). Maybe > even no extra addition/subtraction. > Extending to allow 3 statements where the first and last statements > are casts might be a good way to start. > That way cond_removal_in_builtin_zero_pattern can be removed and moved > over to just match.pd. > > And then extending it to allow say ~ as either before or after the > other statement will allow for the pattern you want to create there. > > And then if you are feeling extra then we can extend it to support > what is needed for PR 126035. > > Or maybe the way we can cost it for that case is if the new sequence > contains any reference to one statement that is in the bbs, see how > many dependent instructions there are and figure out the cost for > moving that sequence. I had went for the easy way out of moving the > statements from being conditional to unconditional for easy > implementation (and delete them afterwards if they become > non-referenced).
So in the end I went to implement this idea as I see a need for PR 126035 as the current set of patches for PR 126035 by Kyrylo are just way too complex for my taste (the loops with a hard coded limit for an example). Much more complex than even my prototype which it is based on. And I finally decided that we really need to stop doing this manual matching after all. Also while working on it, I saw other places it would fire earlier; mostly dealing with comparisons combining. I should have a set of patches by Friday. I am testing and will be submitting the first patch by tonight; the first one is a cleanup of what match returns as it can add unused statements in some cases; this is also useful for another patch which Kyrylo and I are working on dealing with early phi acceptance (which is needed for a different patch I am working on too). Thanks, Andrea > > > Thanks, > Andrea > > > > > > Not sure if I'm capable of pulling this one off but it seems to be a > > time better spent, given that this would also help other matches in > > match.pd, than trying to hammer in a bitclear pattern in tree-ssa-phiopt. > > > > Thanks, > > Daniel > > > > > > > > So you will need to manual match and create a manual one in phiopt: > > > ``` > > > bitshift_8 = 1 << bit_7(D); > > > _2 = _1 & bitshift_8; > > > if (_2 != 0) > > > goto <bb 3>; [INV] > > > else > > > goto <bb 4>; [INV] > > > > > > <bb 3> : > > > _4 = ~bitshift_8; > > > _5 = _1 & _4; > > > > > > <bb 4> : > > > # cstore_3 = PHI <_1(2), _5(3)> > > > ``` > > > > > > One note there is another thing which should be added. > > > Right now tree_could_trap_p for `a[n]` does not check the range of n > > > when n is non-constant and just assumes it traps. > > > We could/should improve the cselim code to have that check the range of n > > > here. > > > > > > Thanks, > > > Andrea > > > > > > > > >> > > >> PR tree-optimization/124667 > > >> > > >> gcc/ChangeLog: > > >> > > >> * tree-ssa-phiopt.cc (stmt_is_memory_load_assignment): helper to > > >> check if a gimple stmt is a memory load. > > >> (stmt_is_memory_store_assignment): helper to check if a gimple > > >> stmt is a memory store. > > >> (cond_removal_mispredict_validate_memregs): helper to check if a > > >> memory load and a memory store are using the same memory > > >> address. > > >> (cond_removal_mispredict_valid_bitmask): helper to validate if > > >> the bit/bitmask is valid for the current optimization. > > >> (cond_removal_mispredict_check_cond): helper to validate the > > >> gcond and cond_stmt format. > > >> (cond_removal_mispredict_memop): new cselim optimization that, > > >> after doing checks and validations, move a bitset/bitclear > > >> operation to the end of cond_bb, making it unconditional. > > >> (pass_cselim::execute): call cond_removal_mispredict_memop. > > >> > > >> gcc/testsuite/ChangeLog: > > >> > > >> * gcc.dg/tree-ssa/pr124667.c: New test. > > >> --- > > >> > > >> Changes from v2: > > >> - added "is_gimple_assign()" checks before using the result of > > >> SSA_NAME_DEF_STMT; > > >> - changed stmt_is_memory_store_assignment() to a positive check; > > >> - simplified stmt_is_load_assignment() to a simple > > >> "gimple_assign_load_p()" check; > > >> - v2 link: > > >> https://gcc.gnu.org/pipermail/gcc-patches/2026-April/713411.html > > >> > > >> gcc/testsuite/gcc.dg/tree-ssa/pr124667.c | 77 ++++ > > >> gcc/tree-ssa-phiopt.cc | 425 ++++++++++++++++++++++- > > >> 2 files changed, 497 insertions(+), 5 deletions(-) > > >> create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/pr124667.c > > >> > > >> diff --git a/gcc/testsuite/gcc.dg/tree-ssa/pr124667.c > > >> b/gcc/testsuite/gcc.dg/tree-ssa/pr124667.c > > >> new file mode 100644 > > >> index 00000000000..1074169ac20 > > >> --- /dev/null > > >> +++ b/gcc/testsuite/gcc.dg/tree-ssa/pr124667.c > > >> @@ -0,0 +1,77 @@ > > >> +/* { dg-additional-options -O2 } */ > > >> +/* { dg-additional-options -fdump-tree-cselim } */ > > >> + > > >> +int bitset1 (int n, int bit) > > >> +{ > > >> + int arr[16]; > > >> + > > >> + int bitshift = 1 << bit; > > >> + > > >> + if ((arr[n] & bitshift) == 0) > > >> + arr[n] |= bitshift; > > >> + > > >> + return arr[n]; > > >> +} > > >> + > > >> +int bitset2 (int n) > > >> +{ > > >> + int arr[16]; > > >> + > > >> + int bit = 0x4; > > >> + > > >> + if ((arr[n] & bit) == 0) > > >> + arr[n] |= bit; > > >> + > > >> + return arr[n]; > > >> +} > > >> + > > >> +int bitset3 (int n) > > >> +{ > > >> + int arr[16]; > > >> + > > >> + int bits = 0xF; > > >> + > > >> + if ((arr[n] & bits) == 0) > > >> + arr[n] |= bits; > > >> + > > >> + return arr[n]; > > >> +} > > >> + > > >> +int bitclear1 (int n, int bit) > > >> +{ > > >> + int arr[16]; > > >> + > > >> + int bitshift = 1 << bit; > > >> + > > >> + if ((arr[n] & bitshift) != 0) > > >> + arr[n] &= ~bitshift; > > >> + > > >> + return arr[n]; > > >> +} > > >> + > > >> +int bitclear2 (int n) > > >> +{ > > >> + int arr[16]; > > >> + > > >> + int bit = 0x4; > > >> + > > >> + if ((arr[n] & bit) != 0) > > >> + arr[n] &= ~bit; > > >> + > > >> + return arr[n]; > > >> +} > > >> + > > >> +int bitclear3 (int n) > > >> +{ > > >> + int arr[16]; > > >> + > > >> + int bits = 0xF; > > >> + > > >> + if ((arr[n] & bits) != 0) > > >> + arr[n] &= ~bits; > > >> + > > >> + return arr[n]; > > >> +} > > >> + > > >> +/* bitset3 won't be optimized all willl kept its branch. */ > > >> +/* { dg-final { scan-tree-dump-times "goto" 2 cselim } } */ > > >> diff --git a/gcc/tree-ssa-phiopt.cc b/gcc/tree-ssa-phiopt.cc > > >> index d9e1edb9b14..0a6e561d029 100644 > > >> --- a/gcc/tree-ssa-phiopt.cc > > >> +++ b/gcc/tree-ssa-phiopt.cc > > >> @@ -3123,6 +3123,418 @@ cond_store_replacement (basic_block middle_bb, > > >> basic_block join_bb, > > >> return true; > > >> } > > >> > > >> +/* Return TRUE if STMT is a memory load, FALSE otherwise. */ > > >> + > > >> +static bool > > >> +stmt_is_memory_load_assignment (gimple *stmt) > > >> +{ > > >> + return stmt && gimple_assign_load_p (stmt); > > >> +} > > >> + > > >> +/* Return TRUE if STMT is a memory store, FALSE otherwise. */ > > >> + > > >> +static bool > > >> +stmt_is_memory_store_assignment (gimple *stmt) > > >> +{ > > >> + return stmt > > >> + && gimple_assign_single_p (stmt) > > >> + && gimple_store_p (stmt) > > >> + && gimple_references_memory_p (stmt); > > >> +} > > >> + > > >> +/* cond_removal_mispredict_memop helper that checks if a > > >> + given memreg operand of a bitop_stmt is a memory load, > > >> + and it loads the same mem addr that is later stored > > >> + in store_stmt: > > >> + > > >> + # VUSE <.MEM_11> > > >> + _1 = ptr_10->bits[word_num_12]; (load_stmt) > > >> + (...) > > >> + _3 = _1 OP bitmask; (bitop_stmt) > > >> + # .MEM_14 = VDEF <.MEM_11> > > >> + ptr_10->bits[word_num_12] = _3; (store_stmt) > > >> + > > >> + For the case above "_1" matches the criteria. > > >> + > > >> + We're also validating whether store_stmt supports the > > >> + transformation by testing its LHS for read-only memory > > >> + and store data races. */ > > >> + > > >> +static bool > > >> +cond_removal_mispredict_validate_memregs (gimple *store_stmt, > > >> + tree memreg, > > >> + hash_set<tree> *nontrap) > > >> +{ > > >> + gimple *load_stmt = SSA_NAME_DEF_STMT (memreg); > > >> + > > >> + if (!load_stmt || !is_gimple_assign (load_stmt)) > > >> + return false; > > >> + > > >> + if (!operand_equal_p (gimple_assign_rhs1 (load_stmt), > > >> + gimple_assign_lhs (store_stmt))) > > >> + return false; > > >> + > > >> + tree lhs = gimple_assign_lhs (store_stmt); > > >> + if (!nontrap->contains (lhs) && tree_could_trap_p (lhs)) > > >> + return false; > > >> + > > >> + if (ref_can_have_store_data_races (lhs)) > > >> + return false; > > >> + > > >> + tree base = get_base_address (lhs); > > >> + if (DECL_P (base) && TREE_READONLY (base)) > > >> + return false; > > >> + > > >> + return true; > > >> +} > > >> + > > >> +/* Check if a given node represents a valid bitmask for > > >> + the cond_removal_mispredict_memop transformation: > > >> + single bit mask for unconditional bit set, multiple > > >> + bits mask for unconditional bit clear. */ > > >> + > > >> +static bool > > >> +cond_removal_mispredict_valid_bitmask (tree bitmask, bool > > >> only_single_bit) > > >> +{ > > >> + if (TREE_CODE (bitmask) == INTEGER_CST) > > >> + { > > >> + if (!only_single_bit) > > >> + return true; > > >> + return wi::popcount (wi::to_wide (bitmask)) == 1; > > >> + } > > >> + > > >> + /* There are several ops that can generate any bitmask, but in this > > >> + case we want to detect "SSA_NAME = 1 << X" that represents a > > >> + single bit mask. */ > > >> + if (TREE_CODE (bitmask) == SSA_NAME) > > >> + { > > >> + gimple *def_stmt = SSA_NAME_DEF_STMT (bitmask); > > >> + return def_stmt > > >> + && is_gimple_assign (def_stmt) > > >> + && gimple_assign_rhs_code (def_stmt) == LSHIFT_EXPR > > >> + && integer_onep (gimple_assign_rhs1 (def_stmt)); > > >> + } > > >> + return false; > > >> +} > > >> + > > >> +/* cond_removal_mispredict helper that checks if the 'cond' > > >> + stamement is on the expected format for the possible > > >> + transformation we can have: > > >> + - a "bitcheck EQ 0" comparison that follows a bitset > > >> + - a "bitcheck NE 0" comparison that follows a bitclear > > >> + > > >> + This also includes checking if the bitmasks involved are > > >> + compatible with each other. E.g. if we're checking for > > >> + bit N and then clearing a bit other than N, we can't do > > >> + the transformation. */ > > >> + > > >> +static bool > > >> +cond_removal_mispredict_check_cond (gcond *cond, tree_code bitop_code, > > >> + tree memreg, tree bitop_bitmask, > > >> + bool has_not_stmt) > > >> +{ > > >> + /* First check if the conditional has the following format: > > >> + > > >> + # VUSE <.MEM_11> > > >> + _1 = ptr_10->bits[word_num_12]; > > >> + _2 = _1 & bitmask; > > >> + if (_2 ==/!= 0) > > >> + goto <bb 4>; [50.00%] > > >> + else > > >> + goto <bb 5>; [50.00%] > > >> + > > >> + I.e. there is a check for an absent bitmask (_2 == 0) that follows > > >> + a bit set or a check for an existing bitmask (_2 != 0) that follows > > >> + a bit clear. */ > > >> + if (TREE_CODE (gimple_cond_lhs (cond)) != SSA_NAME) > > >> + return false; > > >> + > > >> + if (!integer_zerop (gimple_cond_rhs (cond))) > > >> + return false; > > >> + > > >> + gimple *cond_stmt = SSA_NAME_DEF_STMT (gimple_cond_lhs (cond)); > > >> + if (!cond_stmt || !is_gimple_assign (cond_stmt)) > > >> + return false; > > >> + > > >> + tree_code cond_code = gimple_cond_code (cond); > > >> + > > >> + if (cond_code != EQ_EXPR && cond_code != NE_EXPR) > > >> + return false; > > >> + > > >> + if (gimple_cond_code (cond) == EQ_EXPR && bitop_code != BIT_IOR_EXPR) > > >> + return false; > > >> + else if (gimple_cond_code (cond) == NE_EXPR && bitop_code != > > >> BIT_AND_EXPR) > > >> + return false; > > >> + > > >> + tree cond_rhs1 = gimple_assign_rhs1 (cond_stmt); > > >> + tree cond_rhs2 = gimple_assign_rhs2 (cond_stmt); > > >> + tree cond_bitmask = NULL_TREE; > > >> + > > >> + /* cond_stmt must use the same memreg as bitop_stmt. */ > > >> + if (cond_rhs1 == memreg) > > >> + cond_bitmask = cond_rhs2; > > >> + else if (cond_rhs2 == memreg) > > >> + cond_bitmask = cond_rhs1; > > >> + else > > >> + return false; > > >> + > > >> + /* If "bitop_stmt == bit_ior" 'bitmask' must also match. > > >> + > > >> + (cond_bb) > > >> + _1 = ptr_10->bits[word_num_12]; > > >> + _2 = _1 & bit_val_9; <====== cond_stmt > > >> + > > >> + (middle_bb) > > >> + _3 = _1 | bit_val_9; <====== bit_ior > > >> + # .MEM_14 = VDEF <.MEM_11> > > >> + ptr_10->bits[word_num_12] = _3; > > >> + > > >> + Same thing for bit_and with a 'not': > > >> + > > >> + (cond_bb) > > >> + _1 = ptr_10->bits[word_num_12]; > > >> + _2 = _1 & bit_val_9; <==== cond_stmt > > >> + > > >> + (middle_bb) > > >> + _3 = ~bit_val_9; <==== not_stmt > > >> + _4 = _1 & _3; <==== bit_and > > >> + # .MEM_14 = VDEF <.MEM_11> > > >> + ptr_10->bits[word_num_12] = _4; */ > > >> + if (bitop_code == BIT_IOR_EXPR || has_not_stmt) > > >> + return cond_bitmask == bitop_bitmask; > > >> + > > >> + /* Finally, for "bitop_stmt == bit_and" with an INTEGER_CST > > >> + bitop_bitmask, check if we're clearing exactly what we're > > >> + checking in cond_bitmask: > > >> + > > >> + (cond_bb) > > >> + # VUSE <.MEM_11> > > >> + _1 = ptr_10->bits[word_num_12]; > > >> + _2 = _1 & 15; <==== cond_stmt > > >> + if (_2 ==/!= 0) > > >> + > > >> + (middle_bb) > > >> + _4 = _1 & 18446744073709551600; <==== ~15 > > >> + # .MEM_11 = VDEF <.MEM_8> > > >> + ptr_10->bits[word_num_12] = _4; */ > > >> + if (TREE_CODE (bitop_bitmask) == INTEGER_CST > > >> + && TREE_CODE (cond_bitmask) == INTEGER_CST > > >> + && wi::to_wide (bitop_bitmask) == ~wi::to_wide (cond_bitmask)) > > >> + return true; > > >> + > > >> + return false; > > >> +} > > >> + > > >> + > > >> +/* This transformation aims to optimize cases where conditional > > >> + bit clear and bit set operations can be made unconditional > > >> + if the end result in memory is the same. A conditional > > >> + bitset that can be optimized would be: > > >> + > > >> + ;; bb 2 > > >> + bitshift_6 = 1 << bit_5; > > >> + # VUSE <.MEM_7> > > >> + _1 = arrD.4593[n_8]; (load_stmt) > > >> + _2 = _1 & bitshift_6; (cond_stmt) > > >> + if (_2 == 0) goto <bb 3>; else goto <bb 4>; > > >> + > > >> + ;; bb 3 > > >> + _3 = _1 | bitshift_6; (bitop_stmt) > > >> + # .MEM_9 = VDEF <.MEM_7> > > >> + arrD.4593[n_8] = _3; (store_stmt) > > >> + ;; succ: 4 (FALLTHRU,EXECUTABLE) > > >> + > > >> + As far as the memory pointed by MEM_7 goes the end result at the > > >> + start of bb4 is "bitshift_6 is set", either because it was already > > >> + set before or because it is set it in bb3. > > >> + > > >> + In this case, depending on constraints like store data races and > > >> + read only memory, we want to move the stms from bb3 to the end of > > >> + bb2, i.e. always do the bitset: > > >> + > > >> + ;; bb 2 > > >> + bitshift_6 = 1 << bit_5; > > >> + # VUSE <.MEM_7> > > >> + _1 = arrD.4593[n_8]; (load_stmt) > > >> + _2 = _1 & bitshift_6; (cond_stmt) > > >> + _3 = _1 | bitshift_6; (bitop_stmt) > > >> + # .MEM_9 = VDEF <.MEM_7> > > >> + arrD.4593[n_8] = _3; (store_stmt) > > >> + if (_2 == 0) goto <bb 3>; else goto <bb 4>; > > >> + > > >> + This will not just remove the gcond but it can also get rid of > > >> + 'cond_stmt' in case it's not used anywhere else. */ > > >> + > > >> +static bool > > >> +cond_removal_mispredict_memop (basic_block cond_bb, > > >> + basic_block middle_bb, > > >> + basic_block join_bb, > > >> + hash_set<tree> *nontrap) > > >> +{ > > >> + /* 'middle_bb' must have no PHI nodes, it must come via a > > >> + TRUE_VALUE edge, and it must have a store preceeding > > >> + a bitop: > > >> + > > >> + _3 = _1 BITOP bitmask; > > >> + # .MEM_14 = VDEF <.MEM_11> > > >> + ptr_10->bits[word_num_12] = _3; */ > > >> + if (!gimple_seq_empty_p (phi_nodes (middle_bb))) > > >> + return false; > > >> + > > >> + edge e_cond_middle = single_pred_edge (middle_bb); > > >> + if (!(e_cond_middle->flags & EDGE_TRUE_VALUE)) > > >> + return false; > > >> + > > >> + gimple_stmt_iterator gsi = gsi_last_nondebug_bb (middle_bb); > > >> + gimple *store_stmt = gsi_stmt (gsi); > > >> + if (!store_stmt) > > >> + return false; > > >> + > > >> + if (!stmt_is_memory_store_assignment (store_stmt) > > >> + || gimple_has_volatile_ops (store_stmt)) > > >> + return false; > > >> + > > >> + gsi = gsi_start_nondebug_after_labels_bb (middle_bb); > > >> + gimple *bitop_stmt = gsi_stmt (gsi); > > >> + > > >> + if (!is_gimple_assign (bitop_stmt)) > > >> + return false; > > >> + > > >> + tree_code bitop_code = gimple_assign_rhs_code (bitop_stmt); > > >> + gimple *not_stmt = NULL; > > >> + > > >> + if (bitop_code != BIT_IOR_EXPR && bitop_code != BIT_AND_EXPR) > > >> + { > > >> + /* For a bit clear case we can also expect a pattern like this: > > >> + if (_2 != 0) > > >> + goto <bb 4>; [50.00%] > > >> + else > > >> + goto <bb 5>; [50.00%] > > >> + > > >> + ;; basic block 4, > > >> + _3 = ~bit_val_9; > > >> + _4 = _1 & _3; > > >> + # .MEM_14 = VDEF <.MEM_11> > > >> + ptr_10->bitsD.4594[word_num_12] = _4; */ > > >> + if (bitop_code != BIT_NOT_EXPR) > > >> + return false; > > >> + > > >> + not_stmt = bitop_stmt; > > >> + gsi_next (&gsi); > > >> + bitop_stmt = gsi_stmt (gsi); > > >> + > > >> + if (!is_gimple_assign (bitop_stmt)) > > >> + return false; > > >> + > > >> + bitop_code = gimple_assign_rhs_code (bitop_stmt); > > >> + if (bitop_code != BIT_AND_EXPR) > > >> + return false; > > >> + } > > >> + > > >> + /* Verify that after bitop_stmt we only have store_stmt. */ > > >> + gsi_next (&gsi); > > >> + if (gsi_stmt (gsi) != store_stmt) > > >> + return false; > > >> + > > >> + /* Check if the register being stored by 'store_stmt' > > >> + is the result of the previous bitop_stmt. */ > > >> + tree store_rhs1 = gimple_assign_rhs1 (store_stmt); > > >> + if (TREE_CODE (store_rhs1) != SSA_NAME > > >> + || SSA_NAME_DEF_STMT (store_rhs1) != bitop_stmt) > > >> + return false; > > >> + > > >> + /* One of the BITOP operands must be a memory load. Assume for > > >> + now that the other operand will be a valid bitmask. */ > > >> + tree memreg = NULL_TREE, bitmask = NULL_TREE; > > >> + > > >> + if (TREE_CODE (gimple_assign_rhs1 (bitop_stmt)) == SSA_NAME > > >> + && stmt_is_memory_load_assignment ( > > >> + SSA_NAME_DEF_STMT (gimple_assign_rhs1 (bitop_stmt)))) > > >> + { > > >> + memreg = gimple_assign_rhs1 (bitop_stmt); > > >> + bitmask = gimple_assign_rhs2 (bitop_stmt); > > >> + } > > >> + else if (TREE_CODE (gimple_assign_rhs2 (bitop_stmt)) == SSA_NAME > > >> + && stmt_is_memory_load_assignment ( > > >> + SSA_NAME_DEF_STMT (gimple_assign_rhs2 (bitop_stmt)))) > > >> + { > > >> + memreg = gimple_assign_rhs2 (bitop_stmt); > > >> + bitmask = gimple_assign_rhs1 (bitop_stmt); > > >> + } > > >> + > > >> + if (!memreg) > > >> + return false; > > >> + > > >> + /* For the conditional bitclear case with a not_stmt, > > >> + 'bitmask' would be pointing to the LHS of not_stmt, > > >> + and the actual bitmask we want to verify is its RHS1. */ > > >> + if (not_stmt) > > >> + { > > >> + if (gimple_assign_lhs (not_stmt) == bitmask) > > >> + bitmask = gimple_assign_rhs1 (not_stmt); > > >> + else > > >> + return false; > > >> + } > > >> + > > >> + /* Validate 'bitmask' before proceeding. Only single bit masks > > >> + are supported for the bit_ior pattern. */ > > >> + if (!cond_removal_mispredict_valid_bitmask (bitmask, > > >> + bitop_code == > > >> BIT_IOR_EXPR)) > > >> + return false; > > >> + > > >> + /* Validate store_stmt LHS and memreg. */ > > >> + if (!cond_removal_mispredict_validate_memregs (store_stmt, memreg, > > >> nontrap)) > > >> + return false; > > >> + > > >> + gcond *cond = safe_dyn_cast <gcond *> (*gsi_last_bb (cond_bb)); > > >> + if (!cond) > > >> + return false; > > >> + > > >> + if (!cond_removal_mispredict_check_cond (cond, bitop_code, > > >> + memreg, bitmask, > > >> + not_stmt != NULL)) > > >> + return false; > > >> + > > >> + /* At this point we're certain we can always execute > > >> + the store. We could make more analysis to determine > > >> + if the gcond result is being used as a PHI result, > > >> + or we can just move things to cond_bb, right before > > >> + the gcond, and trust that cfg_cleanup will do > > >> + the right thing. */ > > >> + gimple_stmt_iterator gsi_from; > > >> + gsi = gsi_for_stmt (cond); > > >> + > > >> + if (not_stmt) > > >> + { > > >> + gsi_from = gsi_for_stmt (not_stmt); > > >> + gsi_move_before (&gsi_from, &gsi); > > >> + update_stmt (not_stmt); > > >> + } > > >> + > > >> + gsi_from = gsi_for_stmt (bitop_stmt); > > >> + gsi_move_before (&gsi_from, &gsi); > > >> + update_stmt (bitop_stmt); > > >> + > > >> + gsi_from = gsi_for_stmt (store_stmt); > > >> + gsi_move_before (&gsi_from, &gsi); > > >> + update_stmt (store_stmt); > > >> + > > >> + gphi *vphi = get_virtual_phi (join_bb); > > >> + edge e_cond_join = find_edge (cond_bb, join_bb); > > >> + SET_PHI_ARG_DEF (vphi, e_cond_join->dest_idx, gimple_vdef > > >> (store_stmt)); > > >> + update_stmt (vphi); > > >> + > > >> + if (dump_file && (dump_flags & TDF_DETAILS)) > > >> + { > > >> + fprintf (dump_file, "\n Conditional store turned unconditional."); > > >> + print_gimple_stmt (dump_file, store_stmt, 0, > > >> TDF_VOPS|TDF_MEMSYMS); > > >> + } > > >> + statistics_counter_event (cfun, > > >> + "conditional store turned unconditional", 1); > > >> + > > >> + return true; > > >> +} > > >> + > > >> /* Do the main work of conditional store replacement. */ > > >> > > >> static bool > > >> @@ -4264,11 +4676,14 @@ pass_cselim::execute (function *) > > >> return; > > >> > > >> /* bb1 is the middle block, bb2 the join block, bb the split > > >> block, > > >> - e1 the fallthrough edge from bb1 to bb2. We can't do the > > >> - optimization if the join block has more than two predecessors. > > >> */ > > >> - if (EDGE_COUNT (bb2->preds) > 2) > > >> - return; > > >> - if (cond_store_replacement (bb1, bb2, e1, e2, nontrap)) > > >> + e1 the fallthrough edge from bb1 to bb2. */ > > >> + > > >> + /* We can't do cond_store_replacement if the join block has more > > >> + than two predecessors. */ > > >> + if (EDGE_COUNT (bb2->preds) <= 2 > > >> + && cond_store_replacement (bb1, bb2, e1, e2, nontrap)) > > >> + cfgchanged = true; > > >> + else if (cond_removal_mispredict_memop (bb, bb1, bb2, nontrap)) > > >> cfgchanged = true; > > >> }; > > >> > > >> -- > > >> 2.43.0 > > >> > >
