Hi,
this patch bring about more sophisticated building of aggregate jump
functions under the optimistic assumption the aggregate will not
escape. If it is then discovered to escape, the appropriate jump
functions are invalidated at IPA time before anything else can see
them. Invalidating is done by storing NULL_TREE value, some other
functions had to be made aware of the possibility.
The implementation is not surprising, the patch keeps information
about aggregate contents at the beginning of each BB and deltas from
there to each call and BB end. Merging information from different BB
predecessors and applying deltas is very simplistic and replaced in
the subsequent patch.
The only thing which could be perhaps unexpected is that whether an
aggregate jump function requires non-escaped-ness is stored on item by
item bases in order not to regress. This way, all stores to one
particular aggregate performed immediately before the call will result
in a valid jump function regardless of whether the aggregate somehow
escapes - exactly the previous behavior.
Bootstrapped and tested on x86_64-linux, also passes LTO-bootstrap and
I have successfully built Firefox with LTO with it. I'm sure there
will be comments but eventually I'd like to commit this to trunk.
Thanks,
Martin
2014-02-18 Martin Jambor <mjam...@suse.cz>
* ipa-prop.h (ipa_agg_jf_item): New fields size and only_unescaped.
* ipa-prop.c (ipa_known_agg_contents_list): Moved up, new fields
counter and only_unescaped.
(AGG_CONTENTS_TOP): New macro.
(dump_agg_contents_list): New function.
(debug_agg_contents_list): Likewise.
(ipa_bb_info): New fields begin_agg_cnt, agg_deltas and queued.
(ipa_escape): New field decl_p.
(func_body_info): New fields call_agg_deltas_map agg_contents_pool and
worklist.
(ipa_print_node_jump_functions_for_edge): Expect NULL aggregate
values, also dump only_unescaped flag.
(build_agg_jump_func_from_list): New parameter honor_unescaped_flag,
also fill in new fields only_unescaped and size. Assert that the
actual size does not exceed precision of size field.
(determine_locally_known_aggregate_parts): Fill in new field
only_unescaped, pass true honor_unescaped_flag.
(apply_agg_contents_deltas): New function.
(ipa_compute_jump_functions_for_edge): Build aggregate jump functions
from global information.
(ipa_analyze_stmt_uses): Removed, functionality integrated to
ipa_analyze_bb_statements.
(present_tracked_refs_p): New function.
(escape_all_tracked_references): Likewise.
(copy_aggg_deltas_to_cg_edge): Likewise.
(update_agg_deltas_for_stmt): Likewise.
(ipa_analyze_bb_statements): Create BB and call statement aggregate
deltas.
(free_ipa_bb_info): Also free aggregate contents vectors.
(analysis_dom_walker): Put into anonymous namespace. Do not compute
jump functions.
(jfunc_builder_dom_walker): New class.
(create_escape_structures): Initialize call_agg_deltas_map,
agg_contents_pool and worklist fields.
(free_escape_structures): Deallocate the same three data structures.
(merge_agg_contents): New function.
(propagate_agg_cnts_through_bb): Likewise.
(propagate_agg_contents_accross_bbs): Likewise.
(ipa_verify_escaped_flags_match): Likewise.
(ipa_analyze_node): Propagate aggregate contents accross the function
and only then build jump functions.
(kill_invalid_escaped_agg_jfs): New function.
(ipa_spread_escapes): Call kill_invalid_escaped_agg_jfs on all call
graph edges.
(ipa_find_agg_cst_for_param): Ignore NULL values.
(ipa_write_jump_function): Stream new fields size and only_unescaped.
(ipa_read_jump_function): Likewise.
(ipcp_transform_function): Initialize new func_body_info fields.
* ipa-cp.c (propagate_aggs_accross_jump_function): Use size directly
from the aggregate jump function, check it fits the constant. Ignore
NULL values.
testsuite/
* gcc.dg/ipa/ipcp-agg-11.c: New test.
* gcc.dg/ipa/ipcp-agg-12.c: Likewise.
* gcc.dg/ipa/ipcp-agg-13.c: Likewise.
* gcc.dg/ipa/ipcp-agg-14.c: Likewise.
* g++.dg/ipa/devirt-24.C: Bump scan-times to two times.
Index: src/gcc/ipa-prop.c
===================================================================
--- src.orig/gcc/ipa-prop.c
+++ src/gcc/ipa-prop.c
@@ -83,6 +83,82 @@ struct param_aa_status
bool parm_modified, ref_modified, pt_modified;
};
+/* Simple linked list describing known contents of an aggregate. */
+
+struct ipa_known_agg_contents_list
+{
+ /* Offset and size of the described part of the aggregate. */
+ HOST_WIDE_INT offset, size;
+ /* Known constant value or NULL if the contents is known to be unknown. */
+ tree constant;
+ /* Pointer to the next structure in the list. */
+ struct ipa_known_agg_contents_list *next;
+
+ /* In chains describing BBs, this field is a BB local counter of
+ possibly-clobbering statements. */
+ unsigned counter : 31;
+
+ /* In chains describing call-deltas, this is set if the contents is only
+ valid if the memory referenced is not escaped. */
+ unsigned only_unescaped : 1;
+};
+
+/* Special value for an aggregate description during the phase propagating them
+ over BBs. */
+
+#define AGG_CONTENTS_TOP ((struct ipa_known_agg_contents_list *) -1)
+
+/* Dump list P into file F in human readable form begining each new line with
+ INDENT spaces. */
+
+static void
+dump_agg_contents_list (FILE *f, struct ipa_known_agg_contents_list *p,
+ int indent)
+{
+ if (p == AGG_CONTENTS_TOP)
+ {
+ for (int i = 0; i < indent; ++i)
+ fputc (' ', f);
+ fprintf (f, "AGG_CONTENTS_TOP\n");
+ return;
+ }
+
+ if (!p)
+ {
+ for (int i = 0; i < indent; ++i)
+ fputc (' ', f);
+ fprintf (f, "EMPTY\n");
+ return;
+ }
+
+ while (p)
+ {
+ for (int i = 0; i < indent; ++i)
+ fputc (' ', f);
+ fprintf (f, "offset: " HOST_WIDE_INT_PRINT_DEC
+ " size: " HOST_WIDE_INT_PRINT_DEC " ", p->offset, p->size);
+ if (p->constant)
+ {
+ fprintf (f, "cst: ");
+ print_generic_expr (f, p->constant, 0);
+ fprintf (f, " (counter: %u, only_unesc: %u)\n",
+ p->counter, p->only_unescaped);
+ }
+ else
+ fprintf (f, "VARIABLE\n");
+ p = p->next;
+ }
+}
+
+/* Dump P to stderr in human readable format. */
+
+DEBUG_FUNCTION void
+debug_agg_contents_list (struct ipa_known_agg_contents_list *p)
+{
+ dump_agg_contents_list (stderr, p, 0);
+}
+
+
/* Information related to a given BB that used only when looking at function
body. */
@@ -92,6 +168,15 @@ struct ipa_bb_info
vec<cgraph_edge_p> cg_edges;
/* Alias analysis statuses of each formal parameter at this bb. */
vec<param_aa_status> param_aa_statuses;
+
+ /* Aggregate contents of tracked references at the beginning of each BB. */
+ vec<ipa_known_agg_contents_list *> begin_agg_cnt;
+
+ /* Changes in aggregate contents of tracked references. */
+ vec<ipa_known_agg_contents_list *> agg_deltas;
+
+ /* Set when the associated BB is already present in the work list. */
+ bool queued;
};
/* Structure used for intra-procedural escape analysis (and associated
@@ -119,6 +204,10 @@ struct ipa_escape
one. Zero means this structure will remain unused. */
int result_index;
+ /* Set when this structure represents a declaration rather than memory
+ pointed at by an SSA_NAME. */
+ bool decl_p;
+
/* True if we have already dealt with this SSA name. Valid even if target is
non-NULL. */
bool analyzed;
@@ -168,6 +257,15 @@ struct func_body_info
/* Mapping from VAR_DECLS to escape information. */
pointer_map <ipa_escape *> *decl_escapes;
+ /* Mapping from call statements to aggregate content deltas. */
+ pointer_map <vec <ipa_known_agg_contents_list *> > *call_agg_deltas_map;
+
+ /* Allocation pool for instances of ipa_known_agg_contents_list. */
+ alloc_pool agg_contents_pool;
+
+ /* Propagation work list of basic blocks. */
+ vec <basic_block> worklist;
+
/* Number of parameters. */
int param_count;
@@ -423,6 +521,9 @@ ipa_print_node_jump_functions_for_edge (
jump_func->agg.by_ref ? "reference" : "value");
FOR_EACH_VEC_SAFE_ELT (jump_func->agg.items, j, item)
{
+ if (!item->value)
+ continue;
+
fprintf (f, " offset: " HOST_WIDE_INT_PRINT_DEC ", ",
item->offset);
if (TYPE_P (item->value))
@@ -433,6 +534,8 @@ ipa_print_node_jump_functions_for_edge (
fprintf (f, "cst: ");
print_generic_expr (f, item->value, 0);
}
+ if (item->only_unescaped)
+ fprintf (f, " only_unescaped");
fprintf (f, "\n");
}
}
@@ -1666,19 +1769,6 @@ get_ssa_def_if_simple_copy (tree rhs)
return rhs;
}
-/* Simple linked list, describing known contents of an aggregate beforere
- call. */
-
-struct ipa_known_agg_contents_list
-{
- /* Offset and size of the described part of the aggregate. */
- HOST_WIDE_INT offset, size;
- /* Known constant value or NULL if the contents is known to be unknown. */
- tree constant;
- /* Pointer to the next structure in the list. */
- struct ipa_known_agg_contents_list *next;
-};
-
/* Find the proper place in linked list of ipa_known_agg_contents_list
structures where to put a new one with the given LHS_OFFSET and LHS_SIZE,
unless there is a partial overlap, in which case return NULL, or such
@@ -1729,7 +1819,11 @@ build_agg_jump_func_from_list (struct ip
struct ipa_agg_jf_item item;
item.offset = list->offset - arg_offset;
gcc_assert ((item.offset % BITS_PER_UNIT) == 0);
+ gcc_checking_assert (list->size
+ == (HOST_WIDE_INT) (unsigned) list->size);
+ item.size = (unsigned) list->size;
item.value = unshare_expr_without_location (list->constant);
+ item.only_unescaped = list->only_unescaped;
jfunc->agg.items->quick_push (item);
}
list = list->next;
@@ -1871,6 +1965,7 @@ determine_locally_known_aggregate_parts
n = XALLOCA (struct ipa_known_agg_contents_list);
n->size = lhs_size;
n->offset = lhs_offset;
+ n->only_unescaped = false;
if (is_gimple_ip_invariant (rhs))
{
n->constant = rhs;
@@ -1898,6 +1993,28 @@ determine_locally_known_aggregate_parts
}
}
+/* Apply basic block DELTAS to INITial aggregate contents description. */
+
+static struct ipa_known_agg_contents_list *
+apply_agg_contents_deltas (struct ipa_known_agg_contents_list *init,
+ struct ipa_known_agg_contents_list *deltas)
+{
+ /* TODO: This over-conservative but should work for Fortran descriptors.
+ Will be replaced in a subsequent patches with real merging. */
+
+ gcc_assert (init != AGG_CONTENTS_TOP);
+ if (deltas)
+ return deltas;
+ else
+ {
+#ifdef ENABLE_CHECKING
+ for (struct ipa_known_agg_contents_list *p = init; p; p = p->next)
+ gcc_assert (p->only_unescaped);
+#endif
+ return init;
+ }
+}
+
static tree
ipa_get_callee_param_type (struct cgraph_edge *e, int i)
{
@@ -2103,19 +2220,30 @@ ipa_compute_jump_functions_for_edge (str
if (!param_type)
param_type = TREE_TYPE (arg);
- HOST_WIDE_INT dummy_offset;
- struct ipa_escape *esc = get_escape_for_value (fbi, arg, &dummy_offset);
+ HOST_WIDE_INT arg_offset;
+ struct ipa_escape *esc = get_escape_for_value (fbi, arg, &arg_offset);
int ref_index;
- if (esc && valid_escape_result_index (esc, &ref_index))
+ if (esc && !esc->escaped && valid_escape_result_index (esc, &ref_index))
{
+ bool build_agg_jfs;
if (jfunc->type == IPA_JF_UNKNOWN)
- ipa_set_jf_unknown_ref_index (jfunc, ref_index);
+ {
+ ipa_set_jf_unknown_ref_index (jfunc, ref_index);
+ build_agg_jfs = true;
+ }
else if (jfunc->type == IPA_JF_KNOWN_TYPE)
- ipa_set_jf_known_type_ref_index (jfunc, ref_index);
+ {
+ ipa_set_jf_known_type_ref_index (jfunc, ref_index);
+ build_agg_jfs = true;
+ }
else if (jfunc->type == IPA_JF_CONST)
- ipa_set_jf_constant_ref_index (jfunc, ref_index);
+ {
+ ipa_set_jf_constant_ref_index (jfunc, ref_index);
+ build_agg_jfs = true;
+ }
else
{
+ build_agg_jfs = false;
gcc_checking_assert
(jfunc->type != IPA_JF_PASS_THROUGH
|| ipa_get_jf_pass_through_formal_id (jfunc) == ref_index);
@@ -2123,14 +2251,39 @@ ipa_compute_jump_functions_for_edge (str
(jfunc->type != IPA_JF_ANCESTOR
|| ipa_get_jf_ancestor_formal_id (jfunc) == ref_index);
}
- }
- /* TODO: We should allow aggregate jump functions even for these types of
- jump functions but we need to be able to combine them first. */
- if (jfunc->type != IPA_JF_PASS_THROUGH
- && jfunc->type != IPA_JF_ANCESTOR
- && (AGGREGATE_TYPE_P (TREE_TYPE (arg))
- || POINTER_TYPE_P (param_type)))
+ /* TODO: We should allow aggregate jump functions even for other
+ types of jump functions but we need to be able to combine them
+ first. */
+ if (build_agg_jfs)
+ {
+ vec <ipa_known_agg_contents_list *> *dvec;
+ dvec = fbi->call_agg_deltas_map->contains (cs->call_stmt);
+ if (dvec)
+ {
+ struct ipa_bb_info *bi;
+ bi = ipa_get_bb_info (fbi, gimple_bb (cs->call_stmt));
+ struct ipa_known_agg_contents_list *begin, *final, *p;
+ begin = bi->begin_agg_cnt[ref_index];
+ final = apply_agg_contents_deltas (begin, (*dvec)[n]);
+
+ int const_count = 0;
+ for (p = final; p; p = p->next)
+ if (p->constant)
+ const_count++;
+ if (const_count)
+ {
+ jfunc->agg.by_ref = true;
+ build_agg_jump_func_from_list (final, const_count,
+ arg_offset, jfunc);
+ }
+ }
+ }
+ }
+ else if (jfunc->type != IPA_JF_PASS_THROUGH
+ && jfunc->type != IPA_JF_ANCESTOR
+ && (AGGREGATE_TYPE_P (TREE_TYPE (arg))
+ || POINTER_TYPE_P (param_type)))
determine_locally_known_aggregate_parts (call, arg, param_type, jfunc);
}
}
@@ -2520,18 +2673,6 @@ ipa_analyze_call_uses (struct func_body_
ipa_analyze_virtual_call_uses (fbi, call, target);
}
-
-/* Analyze the call statement STMT with respect to formal parameters (described
- in INFO) of caller given by FBI->NODE. Currently it only checks whether
- formal parameters are called. */
-
-static void
-ipa_analyze_stmt_uses (struct func_body_info *fbi, gimple stmt)
-{
- if (is_gimple_call (stmt))
- ipa_analyze_call_uses (fbi, stmt);
-}
-
/* Callback of walk_stmt_load_store_addr_ops. If OP is a parameter
declaration, mark it as used in the info structure passed in DATA. */
@@ -2552,6 +2693,186 @@ visit_ref_mark_it_used (gimple, tree op,
return false;
}
+/* Return true if there are any tracked references. */
+
+static bool
+present_tracked_refs_p (struct func_body_info *fbi)
+{
+ return ipa_get_tracked_refs_count (fbi->info) != 0;
+}
+
+/* Mark all tracked references as escaped. */
+
+static void
+escape_all_tracked_references (struct func_body_info *fbi)
+{
+ for (int i = 0; i < ipa_get_tracked_refs_count (fbi->info); ++i)
+ ipa_set_ref_escaped (fbi->info, i, 1);
+ unsigned int ui;
+ struct ipa_escape *esc;
+ FOR_EACH_VEC_ELT (fbi->escapes, ui, esc)
+ esc->escaped = true;
+}
+
+/* Copy aggregate deltas gathered at this point to the
+ FBI->call_agg_deltas_map. FBI describes the current function and BB the
+ current basic block. CUR_COUNTER is the current value of a BB-local
+ possibly clobbering statements counter. */
+
+static void
+copy_aggg_deltas_to_cg_edge (struct func_body_info *fbi,
+ struct ipa_bb_info *bi,
+ unsigned cur_counter,
+ gimple call)
+{
+ vec<ipa_known_agg_contents_list *> deltas = vNULL;
+ int arg_num = gimple_call_num_args (call);
+
+ for (int i = 0; i < arg_num; ++i)
+ {
+ ipa_known_agg_contents_list *list = NULL;
+ HOST_WIDE_INT dummy_offset;
+ int ri;
+ tree arg = gimple_call_arg (call, i);
+ struct ipa_escape *esc = get_escape_for_value (fbi, arg, &dummy_offset);
+ if (esc && !esc->escaped
+ && valid_escape_result_index (esc, &ri)
+ && !bi->agg_deltas.is_empty ())
+ {
+ ipa_known_agg_contents_list *o, **p = &list;
+ for (o = bi->agg_deltas[ri]; o; o = o->next)
+ {
+ struct ipa_known_agg_contents_list *n;
+ n = (struct ipa_known_agg_contents_list *)
+ pool_alloc (fbi->agg_contents_pool);
+ n->offset = o->offset;
+ n->size = o->size;
+ n->constant = o->constant;
+ n->only_unescaped = o->counter != cur_counter;
+ n->next = NULL;
+ *p = n;
+ p = &n->next;
+ }
+ }
+ deltas.safe_push (list);
+ }
+ *fbi->call_agg_deltas_map->insert (call) = deltas;
+}
+
+/* Update current aggregate offset deltas. Return true if the statement might
+ have modified also an escaped memory reference. */
+
+static bool
+update_agg_deltas_for_stmt (struct func_body_info *fbi,
+ struct ipa_bb_info *bi,
+ int *cnt_desc_counts,
+ unsigned counter, gimple stmt)
+{
+ tree lhs, rhs;
+ HOST_WIDE_INT offset, size, max_size;
+ bool res;
+
+ if (!gimple_vdef (stmt) || !present_tracked_refs_p (fbi))
+ return false;
+
+ if (gimple_assign_single_p (stmt))
+ {
+ lhs = gimple_assign_lhs (stmt);
+ rhs = gimple_assign_rhs1 (stmt);
+ res = false;
+ }
+ else if (is_gimple_call (stmt))
+ {
+ lhs = gimple_call_lhs (stmt);
+ if (!lhs)
+ return true;
+ rhs = NULL_TREE;
+ res = true;
+ }
+ else
+ {
+ if (dump_file)
+ {
+ fprintf (dump_file, "Marking all tracked references in %s/%i "
+ "because of statement: \n", fbi->node->name (),
+ fbi->node->order);
+ print_gimple_stmt (dump_file, stmt, 2, TDF_SLIM);
+ fprintf (dump_file, "\n");
+ }
+ escape_all_tracked_references (fbi);
+ return true;
+ }
+
+ struct ipa_escape *esc = get_escape_for_ref (fbi, lhs, &offset, &size,
+ &max_size);
+
+ int ri;
+ if (!esc || esc->escaped || !valid_escape_result_index (esc, &ri))
+ return true;
+
+ gcc_checking_assert (ri < ipa_get_tracked_refs_count (fbi->info));
+ if (max_size < 0 || size < 0
+ || TREE_CODE (lhs) == BIT_FIELD_REF
+ || contains_bitfld_component_ref_p (lhs))
+ {
+ /* TODO: Eventually this should only be a clobber on paths through the
+ BB. */
+ ipa_set_ref_escaped (fbi->info, ri, 1);
+ esc->escaped = true;
+ return true;
+ }
+
+ ipa_set_ref_clobbered (fbi->info, ri, 1);
+ if (bi->agg_deltas.is_empty ())
+ bi->agg_deltas.safe_grow_cleared (ipa_get_tracked_refs_count (fbi->info));
+
+ struct ipa_known_agg_contents_list *n, **p, **list = &bi->agg_deltas[ri];
+ bool already_there = false;
+ p = get_place_in_agg_contents_list (list, offset, max_size, &already_there);
+ if (!p)
+ {
+ /* TODO: Eventually this should only be a clobber on paths through the
+ BB. */
+ ipa_set_ref_escaped (fbi->info, ri, 1);
+ esc->escaped = true;
+ return true;
+ }
+
+ if (already_there)
+ n = *p;
+ else
+ {
+ cnt_desc_counts[ri]++;
+ if (cnt_desc_counts[ri] > PARAM_VALUE (PARAM_IPA_MAX_AGG_ITEMS))
+ {
+ /* TODO: Eventually this should only be a clobber on paths
+ through the BB. */
+ ipa_set_ref_escaped (fbi->info, ri, 1);
+ esc->escaped = true;
+ return true;
+ }
+
+ n = (struct ipa_known_agg_contents_list *)
+ pool_alloc (fbi->agg_contents_pool);
+ n->size = max_size;
+ n->offset = offset;
+ n->next = *p;
+ *p = n;
+ }
+
+ if (rhs
+ && size == max_size
+ && is_gimple_reg_type (TREE_TYPE (rhs))
+ && is_gimple_ip_invariant (rhs))
+ n->constant = rhs;
+ else
+ n->constant = NULL_TREE;
+
+ n->counter = counter;
+ n->only_unescaped = true;
+ return res && esc->decl_p;
+}
+
/* Scan the statements in BB and inspect the uses of formal parameters, escape
analysis and so on. FBI holds various data about the function being
analyzed. */
@@ -2559,6 +2880,24 @@ visit_ref_mark_it_used (gimple, tree op,
static void
ipa_analyze_bb_statements (struct func_body_info *fbi, basic_block bb)
{
+ struct ipa_bb_info *bi = ipa_get_bb_info (fbi, bb);
+ unsigned clobbering_counter = 0;
+ int *cnt_desc_counts;
+
+ if (present_tracked_refs_p (fbi))
+ {
+ cnt_desc_counts = XALLOCAVEC (int,
+ ipa_get_tracked_refs_count (fbi->info));
+ memset (cnt_desc_counts, 0,
+ sizeof (int) * ipa_get_tracked_refs_count (fbi->info));
+
+ bi->begin_agg_cnt.safe_grow (ipa_get_tracked_refs_count (fbi->info));
+ for (int i = 0; i < ipa_get_tracked_refs_count (fbi->info); ++i)
+ bi->begin_agg_cnt[i] = AGG_CONTENTS_TOP;
+ }
+ else
+ cnt_desc_counts = NULL;
+
gimple_stmt_iterator gsi;
for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
{
@@ -2567,11 +2906,21 @@ ipa_analyze_bb_statements (struct func_b
if (is_gimple_debug (stmt))
continue;
- ipa_analyze_stmt_uses (fbi, stmt);
+ if (is_gimple_call (stmt))
+ {
+ ipa_analyze_call_uses (fbi, stmt);
+ if (present_tracked_refs_p (fbi))
+ copy_aggg_deltas_to_cg_edge (fbi, bi, clobbering_counter, stmt);
+ }
+
+ if (update_agg_deltas_for_stmt (fbi, bi, cnt_desc_counts,
+ clobbering_counter, stmt))
+ clobbering_counter++;
walk_stmt_load_store_addr_ops (stmt, fbi->info,
visit_ref_mark_it_used,
visit_ref_mark_it_used,
visit_ref_mark_it_used);
+
}
for (gsi = gsi_start_phis (bb); !gsi_end_p (gsi); gsi_next (&gsi))
walk_stmt_load_store_addr_ops (gsi_stmt (gsi), fbi->info,
@@ -2632,6 +2981,8 @@ free_ipa_bb_info (struct ipa_bb_info *bi
{
bi->cg_edges.release ();
bi->param_aa_statuses.release ();
+ bi->begin_agg_cnt.release ();
+ bi->agg_deltas.release ();
}
/* Dominator walker driving the analysis. */
@@ -2652,6 +3003,25 @@ void
analysis_dom_walker::before_dom_children (basic_block bb)
{
ipa_analyze_bb_statements (m_fbi, bb);
+}
+
+/* Dominator walker driving jump_function creation. */
+
+class jfunc_builder_dom_walker : public dom_walker
+{
+public:
+ jfunc_builder_dom_walker (struct func_body_info *fbi)
+ : dom_walker (CDI_DOMINATORS), m_fbi (fbi) {}
+
+ virtual void before_dom_children (basic_block);
+
+private:
+ struct func_body_info *m_fbi;
+};
+
+void
+jfunc_builder_dom_walker::before_dom_children (basic_block bb)
+{
ipa_compute_jump_functions_for_bb (m_fbi, bb);
}
@@ -2685,6 +3055,7 @@ static void
analyze_ssa_escape (struct func_body_info *fbi, tree ssa,
struct ipa_escape *esc)
{
+ gcc_assert (fbi->info->ref_descs.is_empty ());
esc->analyzed = true;
if (!POINTER_TYPE_P (TREE_TYPE (ssa)))
{
@@ -2826,7 +3197,7 @@ analyze_all_ssa_escapes (struct func_bod
continue;
struct ipa_escape *esc = &fbi->escapes[SSA_NAME_VERSION (ssa)];
if (esc->analyzed)
- return;
+ continue;
analyze_ssa_escape (fbi, ssa, esc);
}
}
@@ -2863,6 +3234,15 @@ create_escape_structures (struct func_bo
FOR_EACH_LOCAL_DECL (fbi->func, i, var)
if (TREE_CODE (var) == VAR_DECL && TREE_ADDRESSABLE (var))
*fbi->decl_escapes->insert (var) = &fbi->escapes[var_idx++];
+
+ for (unsigned j = SSANAMES (fbi->func)->length (); j < var_idx; ++j)
+ fbi->escapes[j].decl_p = true;
+
+ fbi->call_agg_deltas_map
+ = new pointer_map <vec <ipa_known_agg_contents_list *> >;
+ fbi->agg_contents_pool = create_alloc_pool ("IPA-PROP agg contents pool",
+ sizeof (struct ipa_known_agg_contents_list),32);
+ fbi->worklist = vNULL;
}
/* Free escape analysis structures in the FBI. */
@@ -2872,6 +3252,10 @@ free_escape_structures (struct func_body
{
fbi->escapes.release ();
delete fbi->decl_escapes;
+ delete fbi->call_agg_deltas_map;
+ free_alloc_pool (fbi->agg_contents_pool);
+ gcc_checking_assert (fbi->worklist.is_empty ());
+ fbi->worklist.release ();
}
/* Go over call argument of CS and if any warrants a result_index for an escape
@@ -2929,6 +3313,113 @@ gather_picked_escapes (struct func_body_
}
}
+/* Merge aggregate contents FINAL with those in *TARGET. Return true if those
+ in *TARGET have changed. */
+
+static bool
+merge_agg_contents (struct ipa_known_agg_contents_list *final,
+ struct ipa_known_agg_contents_list **target)
+{
+ /* TODO: This over-conservative but should work for Fortran descriptors.
+ Will be replaced in a subsequent patches by real merging. */
+ if (*target == AGG_CONTENTS_TOP)
+ {
+ *target = final;
+ return true;
+ }
+ else if (*target != final)
+ {
+ if (*target)
+ {
+ *target = NULL;
+ return true;
+ }
+ else
+ return false;
+ }
+ return false;
+}
+
+/* Apply all computed aggregate deltas for the given BB and merge results into
+ corresponding aggregate contents description at the beginning of successors.
+ Enqueue all successors with changed information. */
+
+static void
+propagate_agg_cnts_through_bb (struct func_body_info *fbi,
+ basic_block bb, ipa_bb_info *bi)
+{
+ for (int i = 0; i < ipa_get_tracked_refs_count (fbi->info); ++i)
+ {
+ struct ipa_known_agg_contents_list *deltas, *final;
+ if (bi->agg_deltas.is_empty ())
+ deltas = NULL;
+ else
+ deltas = bi->agg_deltas[i];
+
+ final = apply_agg_contents_deltas (bi->begin_agg_cnt[i], deltas);
+
+ edge e;
+ edge_iterator ei;
+ FOR_EACH_EDGE (e, ei, bb->succs)
+ {
+ basic_block succ = e->dest;
+
+ if (succ->index == EXIT_BLOCK)
+ continue;
+
+ ipa_bb_info *succ_info = ipa_get_bb_info (fbi, succ);
+
+ gcc_checking_assert (succ_info->begin_agg_cnt.length ()
+ >= (unsigned) i);
+ if (merge_agg_contents (final, &succ_info->begin_agg_cnt[i])
+ && !succ_info->queued)
+ {
+ succ_info->queued = true;
+ fbi->worklist.safe_push (succ);
+ }
+ }
+ }
+}
+
+/* Global propagation of aggregate contents accross all BBs. */
+
+static void
+propagate_agg_contents_accross_bbs (struct func_body_info *fbi)
+{
+ if (!present_tracked_refs_p (fbi))
+ return;
+
+ basic_block first = single_succ (ENTRY_BLOCK_PTR_FOR_FN (fbi->func));
+ struct ipa_bb_info *first_bi = ipa_get_bb_info (fbi, first);
+ for (int i = 0; i < ipa_get_tracked_refs_count (fbi->info); ++i)
+ first_bi->begin_agg_cnt[i] = NULL;
+ propagate_agg_cnts_through_bb (fbi, first, first_bi);
+
+ while (!fbi->worklist.is_empty ())
+ {
+ basic_block bb = fbi->worklist.pop ();
+ struct ipa_bb_info *bi = ipa_get_bb_info (fbi, bb);
+ bi->queued = false;
+ propagate_agg_cnts_through_bb (fbi, bb, bi);
+ }
+}
+
+/* Verify that the correspondng escaped flags in ipa_escape structures and in
+ reference descriptors match. */
+
+DEBUG_FUNCTION void
+ipa_verify_escaped_flags_match (struct func_body_info *fbi)
+{
+ int i;
+ struct ipa_escape *esc;
+ FOR_EACH_VEC_ELT (fbi->escapes, i, esc)
+ {
+ int ridx;
+ if (valid_escape_result_index (esc, &ridx))
+ gcc_assert (esc->escaped == ipa_is_ref_escaped (fbi->info, ridx));
+ }
+}
+
/* Initialize the array describing properties of of formal parameters
of NODE, analyze their uses and compute jump functions associated
with actual arguments of calls from within NODE. */
@@ -3002,6 +3493,12 @@ ipa_analyze_node (struct cgraph_node *no
gather_picked_escapes (&fbi, ri);
analysis_dom_walker (&fbi).walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
+ propagate_agg_contents_accross_bbs (&fbi);
+ jfunc_builder_dom_walker (&fbi).walk (ENTRY_BLOCK_PTR_FOR_FN (cfun));
+
+#ifdef ENABLE_CHECKING
+ ipa_verify_escaped_flags_match (&fbi);
+#endif
free_escape_structures (&fbi);
int i;
@@ -3211,6 +3708,38 @@ spread_escapes_down (struct escape_sprea
}
}
+/* Invalidate all aggregate jump function items in all jump functions
associated
+ with CS that are marked as only valid if the reference is unescaped if the
+ reference is actually escaped or found to be modified by a callee. */
+
+static void
+kill_invalid_escaped_agg_jfs (struct ipa_node_params *info,
+ struct cgraph_edge *cs)
+{
+ struct ipa_edge_args *args = IPA_EDGE_REF (cs);
+ int count = ipa_get_cs_argument_count (args);
+
+ for (int i = 0; i < count; i++)
+ {
+ struct ipa_jump_func *jf = ipa_get_ith_jump_func (args, i);
+ if (!jf->agg.items)
+ continue;
+
+ int origin = escape_origin_from_jfunc (jf);
+ if (origin >= 0
+ && !ipa_is_ref_escaped (info, origin)
+ && !ipa_is_ref_callee_clobbered (info, origin))
+ continue;
+
+ int j;
+ struct ipa_agg_jf_item *ai;
+ FOR_EACH_VEC_ELT (*jf->agg.items, j, ai)
+ if (ai->only_unescaped)
+ ai->value = NULL_TREE;
+ }
+}
+
+
/* Spread escape flags through jump functions accross the call graph. */
void
@@ -3275,6 +3804,13 @@ ipa_spread_escapes ()
if (!ipa_is_ref_clobbered (info, i))
cgraph_set_param_noclobber (node, i);
}
+
+ for (struct cgraph_edge *cs = node->callees; cs; cs = cs->next_callee)
+ kill_invalid_escaped_agg_jfs (info, cs);
+ for (struct cgraph_edge *cs = node->indirect_calls;
+ cs;
+ cs = cs->next_callee)
+ kill_invalid_escaped_agg_jfs (info, cs);
}
}
@@ -3626,6 +4162,8 @@ ipa_find_agg_cst_for_param (struct ipa_a
FOR_EACH_VEC_SAFE_ELT (agg->items, i, item)
if (item->offset == offset)
{
+ if (!item->value)
+ return NULL;
/* Currently we do not have clobber values, return NULL for them once
we do. */
gcc_checking_assert (is_gimple_ip_invariant (item->value));
@@ -5374,6 +5912,11 @@ ipa_write_jump_function (struct output_b
{
streamer_write_uhwi (ob, item->offset);
stream_write_tree (ob, item->value, true);
+ streamer_write_uhwi (ob, item->size);
+
+ bp = bitpack_create (ob->main_stream);
+ bp_pack_value (&bp, item->only_unescaped, 1);
+ streamer_write_bitpack (&bp);
}
}
@@ -5478,6 +6021,11 @@ ipa_read_jump_function (struct lto_input
struct ipa_agg_jf_item item;
item.offset = streamer_read_uhwi (ib);
item.value = stream_read_tree (ib, data_in);
+ item.size = streamer_read_uhwi (ib);
+
+ struct bitpack_d bp = streamer_read_bitpack (ib);
+ item.only_unescaped = bp_unpack_value (&bp, 1);
+
jump_func->agg.items->quick_push (item);
}
}
@@ -6136,6 +6684,9 @@ ipcp_transform_function (struct cgraph_n
fbi.aa_walked = 0;
fbi.escapes = vNULL;
fbi.decl_escapes = NULL;
+ fbi.call_agg_deltas_map = NULL;
+ fbi.agg_contents_pool = NULL;
+ fbi.worklist = vNULL;
descriptors.safe_grow_cleared (param_count);
ipa_populate_param_decls (node, descriptors);
Index: src/gcc/ipa-cp.c
===================================================================
--- src.orig/gcc/ipa-cp.c
+++ src/gcc/ipa-cp.c
@@ -1390,17 +1390,23 @@ propagate_aggs_accross_jump_function (st
FOR_EACH_VEC_ELT (*jfunc->agg.items, i, item)
{
- HOST_WIDE_INT val_size;
-
if (item->offset < 0)
continue;
- gcc_checking_assert (is_gimple_ip_invariant (item->value));
- val_size = tree_to_uhwi (TYPE_SIZE (TREE_TYPE (item->value)));
- if (merge_agg_lats_step (dest_plats, item->offset, val_size,
+ if (merge_agg_lats_step (dest_plats, item->offset,
+ (HOST_WIDE_INT) item->size,
&aglat, pre_existing, &ret))
{
- ret |= add_value_to_lattice (*aglat, item->value, cs, NULL, 0, 0);
+ if (item->value)
+ {
+ gcc_checking_assert (is_gimple_ip_invariant (item->value));
+ gcc_checking_assert (item->size == tree_to_uhwi
+ (TYPE_SIZE (TREE_TYPE (item->value))));
+ ret |= add_value_to_lattice (*aglat, item->value, cs, NULL,
+ 0, 0);
+ }
+ else
+ ret |= set_lattice_contains_variable (*aglat);
aglat = &(*aglat)->next;
}
else if (dest_plats->aggs_bottom)
Index: src/gcc/ipa-prop.h
===================================================================
--- src.orig/gcc/ipa-prop.h
+++ src/gcc/ipa-prop.h
@@ -178,8 +178,15 @@ struct GTY(()) ipa_agg_jf_item
/* The offset at which the known value is located within the aggregate. */
HOST_WIDE_INT offset;
- /* The known constant or type if this is a clobber. */
+ /* The known constant or NULL if the entry has been invalidated after
+ creation or is a clobber. */
tree value;
+
+ /* Size of the constant. */
+ unsigned size;
+
+ /* Set if the value is only valid if the referring pointer is unescaped. */
+ unsigned only_unescaped : 1;
};
Index: src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-11.c
===================================================================
--- /dev/null
+++ src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-11.c
@@ -0,0 +1,25 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -fno-ipa-sra -fdump-ipa-cp-details" } */
+/* { dg-add-options bind_pic_locally } */
+
+volatile int g;
+
+static void __attribute__ ((noinline))
+bar (int *i)
+{
+ g = *i;
+}
+
+int
+main (int argc, char **argv)
+{
+ int i = 8;
+
+ bar (&i);
+ bar (&i);
+
+ return 0;
+}
+
+/* { dg-final { scan-ipa-dump "Creating a specialized node of bar.*for all
known contexts" "cp" } } */
+/* { dg-final { cleanup-ipa-dump "cp" } } */
Index: src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-12.c
===================================================================
--- /dev/null
+++ src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-12.c
@@ -0,0 +1,33 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -fno-ipa-sra -fdump-ipa-cp-details" } */
+/* { dg-add-options bind_pic_locally } */
+
+volatile int g;
+static int *e;
+
+static void __attribute__ ((noinline))
+foo (int *i)
+{
+ e = i;
+}
+
+static void __attribute__ ((noinline))
+bar (int *i)
+{
+ g = *i;
+ foo (i);
+}
+
+int
+main (int argc, char **argv)
+{
+ int i = 8;
+
+ bar (&i);
+ bar (&i);
+
+ return 0;
+}
+
+/* { dg-final { scan-ipa-dump-not "Creating a specialized node" "cp" } } */
+/* { dg-final { cleanup-ipa-dump "cp" } } */
Index: src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-13.c
===================================================================
--- /dev/null
+++ src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-13.c
@@ -0,0 +1,31 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -fno-ipa-sra -fdump-ipa-cp-details" } */
+/* { dg-add-options bind_pic_locally } */
+
+volatile int g;
+
+static void __attribute__ ((noinline))
+foo (int *i)
+{
+ *i = 16;
+}
+
+static void __attribute__ ((noinline))
+bar (int *i)
+{
+ g = *i;
+}
+
+int
+main (int argc, char **argv)
+{
+ int i = 8;
+
+ bar (&i);
+ foo (&i);
+ bar (&i);
+
+ return 0;
+}
+/* { dg-final { scan-ipa-dump-not "Creating a specialized node" "cp" } } */
+/* { dg-final { cleanup-ipa-dump "cp" } } */
Index: src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-14.c
===================================================================
--- /dev/null
+++ src/gcc/testsuite/gcc.dg/ipa/ipcp-agg-14.c
@@ -0,0 +1,34 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -fno-ipa-sra -fdump-ipa-cp-details" } */
+/* { dg-add-options bind_pic_locally } */
+
+volatile int g;
+
+static void __attribute__ ((noinline))
+bar (int *i)
+{
+ g = *i;
+}
+
+static int __attribute__ ((noinline, noclone))
+something_unpredictable (void)
+{
+ return 1;
+}
+
+
+int
+main (int argc, char **argv)
+{
+ int i = 8;
+
+ if (something_unpredictable ())
+ g = 6;
+
+ bar (&i);
+
+ return 0;
+}
+
+/* { dg-final { scan-ipa-dump "Creating a specialized node of bar.*for all
known contexts" "cp" } } */
+/* { dg-final { cleanup-ipa-dump "cp" } } */
Index: src/gcc/testsuite/g++.dg/ipa/devirt-24.C
===================================================================
--- src.orig/gcc/testsuite/g++.dg/ipa/devirt-24.C
+++ src/gcc/testsuite/g++.dg/ipa/devirt-24.C
@@ -38,5 +38,5 @@ C *b = new (C);
}
/* { dg-final { scan-ipa-dump-times "Discovered a virtual call to a known
target" 1 "inline" } } */
/* { dg-final { cleanup-ipa-dump "inline" } } */
-/* { dg-final { scan-ipa-dump-times "Aggregate passed by reference" 1 "cp" }
} */
+/* { dg-final { scan-ipa-dump-times "Aggregate passed by reference" 2 "cp" }
} */
/* { dg-final { cleanup-ipa-dump "cp" } } */
