This reduces GC walk recursion depth in two ways. First by re-ordering tree_type_common members to move 'name' last and 'canonical' before 'next_variant'. That makes us first recurse downward (type, pointer_to/reference_to), then on the same level (canonical, next_variant, main_variant) and finally upward (context, name->decl_context). For TS_TYPE_NON_COMMON we still walk down afterwards via values, on the same level via minval/maxval and upwards via binfo, so that the patch helps is maybe too much handwaving? (but it helps a reduced testcase without doing the 2nd part)
Second by choosing sth different for chain_next for types than TREE_CHAIN (which is TYPE_STUB_DECL, no chain at all). That makes the unreduced testcase work and apart from the issue below should be obvious enough (though there usually shouldn't be so many type variants - still if for every type we save two or three recursions that still helps). Martin verified this fixes PR60553. I've changed chain_next only for the LTO frontend as while (ggc_test_and_set_mark (xlimit)) xlimit = (CODE_CONTAINS_STRUCT (TREE_CODE (&(*xlimit).generic), TS_TYPE_COMMON) ? ((union lang_tree_node *) (*xlimit).generic.type_common.next_variant) : CODE_CONTAINS_STRUCT (TREE_CODE (&(*xlimit).generic), TS_COMMON) ? ((union lang_tree_node *) (*xlimit).generic.common.chain) : NULL); likely doesn't create great code ... (note duplicate tree checks with checking here for other frontends, fixed LTO with the patch below). LTO bootstrap running on x86_64-unknown-linux-gnu. Ok for trunk? Thanks, Richard. 2014-03-19 Richard Biener <rguent...@suse.de> PR middle-end/60553 * tree-core.h (tree_type_common): Re-order pointer members to reduce recursion depth during GC walks. lto/ * lto-tree.h (lang_tree_node): For types use TYPE_NEXT_VARIANT instead of TREE_CHAIN as chain_next. Index: gcc/tree-core.h =================================================================== --- gcc/tree-core.h (revision 208642) +++ gcc/tree-core.h (working copy) @@ -1265,11 +1265,11 @@ struct GTY(()) tree_type_common { const char * GTY ((tag ("TYPE_SYMTAB_IS_POINTER"))) pointer; struct die_struct * GTY ((tag ("TYPE_SYMTAB_IS_DIE"))) die; } GTY ((desc ("debug_hooks->tree_type_symtab_field"))) symtab; - tree name; + tree canonical; tree next_variant; tree main_variant; tree context; - tree canonical; + tree name; }; struct GTY(()) tree_type_with_lang_specific { Index: gcc/lto/lto-tree.h =================================================================== --- gcc/lto/lto-tree.h (revision 208642) +++ gcc/lto/lto-tree.h (working copy) @@ -48,7 +48,7 @@ enum lto_tree_node_structure_enum { }; union GTY((desc ("lto_tree_node_structure (&%h)"), - chain_next ("CODE_CONTAINS_STRUCT (TREE_CODE (&%h.generic), TS_COMMON) ? ((union lang_tree_node *) TREE_CHAIN (&%h.generic)) : NULL"))) + chain_next ("CODE_CONTAINS_STRUCT (TREE_CODE (&%h.generic), TS_TYPE_COMMON) ? ((union lang_tree_node *) %h.generic.type_common.next_variant) : CODE_CONTAINS_STRUCT (TREE_CODE (&%h.generic), TS_COMMON) ? ((union lang_tree_node *) %h.generic.common.chain) : NULL"))) lang_tree_node { union tree_node GTY ((tag ("TS_LTO_GENERIC"),