Honza figured that variably_modified_type_p uses TREE_VISITED to not run into an Ada abdomination. That causes havoc during WPA streaming which happens in multiple forked processes and thus causes a lot of COW faulting and resident memory usage. It also stands in the way of using threads here.
LTO bootstrapped and tested on x86_64-unknown-linux-gnu. Honza - does this look OK? Thanks, Richard. 2019-10-15 Richard Biener <rguent...@suse.de> * lto-streamer-out.c (lto_variably_modified_type_p): New. (tree_is_indexable): Use it. * tree-streamer-out.c (pack_ts_type_common_value_fields): Stream variably_modified_type_p as TYPE_LANG_FLAG_0. * tree-streamer-in.c (unpack_ts_type_common_value_fields): Likewise. Index: gcc/lto-streamer-out.c =================================================================== --- gcc/lto-streamer-out.c (revision 276985) +++ gcc/lto-streamer-out.c (working copy) @@ -120,6 +120,17 @@ output_type_ref (struct output_block *ob lto_output_type_ref_index (ob->decl_state, ob->main_stream, node); } +/* Wrapper around variably_modified_type_p avoiding type modification + during WPA streaming. */ + +static bool +lto_variably_modified_type_p (tree type) +{ + return (in_lto_p + ? TYPE_LANG_FLAG_0 (TYPE_MAIN_VARIANT (type)) + : variably_modified_type_p (type, NULL_TREE)); +} + /* Return true if tree node T is written to various tables. For these nodes, we sometimes want to write their phyiscal representation @@ -134,7 +145,7 @@ tree_is_indexable (tree t) definition. */ if ((TREE_CODE (t) == PARM_DECL || TREE_CODE (t) == RESULT_DECL) && DECL_CONTEXT (t)) - return variably_modified_type_p (TREE_TYPE (DECL_CONTEXT (t)), NULL_TREE); + return lto_variably_modified_type_p (TREE_TYPE (DECL_CONTEXT (t))); /* IMPORTED_DECL is put into BLOCK and thus it never can be shared. We should no longer need to stream it. */ else if (TREE_CODE (t) == IMPORTED_DECL) @@ -154,10 +165,10 @@ tree_is_indexable (tree t) them we have to localize their members as well. ??? In theory that includes non-FIELD_DECLs as well. */ else if (TYPE_P (t) - && variably_modified_type_p (t, NULL_TREE)) + && lto_variably_modified_type_p (t)) return false; else if (TREE_CODE (t) == FIELD_DECL - && variably_modified_type_p (DECL_CONTEXT (t), NULL_TREE)) + && lto_variably_modified_type_p (DECL_CONTEXT (t))) return false; else return (TYPE_P (t) || DECL_P (t) || TREE_CODE (t) == SSA_NAME); Index: gcc/tree-streamer-in.c =================================================================== --- gcc/tree-streamer-in.c (revision 276985) +++ gcc/tree-streamer-in.c (working copy) @@ -378,6 +378,7 @@ unpack_ts_type_common_value_fields (stru TYPE_RESTRICT (expr) = (unsigned) bp_unpack_value (bp, 1); TYPE_USER_ALIGN (expr) = (unsigned) bp_unpack_value (bp, 1); TYPE_READONLY (expr) = (unsigned) bp_unpack_value (bp, 1); + TYPE_LANG_FLAG_0 (expr) = (unsigned) bp_unpack_value (bp, 1); if (RECORD_OR_UNION_TYPE_P (expr)) { TYPE_TRANSPARENT_AGGR (expr) = (unsigned) bp_unpack_value (bp, 1); Index: gcc/tree-streamer-out.c =================================================================== --- gcc/tree-streamer-out.c (revision 276985) +++ gcc/tree-streamer-out.c (working copy) @@ -326,6 +326,12 @@ pack_ts_type_common_value_fields (struct bp_pack_value (bp, TYPE_RESTRICT (expr), 1); bp_pack_value (bp, TYPE_USER_ALIGN (expr), 1); bp_pack_value (bp, TYPE_READONLY (expr), 1); + unsigned vla_p; + if (in_lto_p) + vla_p = TYPE_LANG_FLAG_0 (TYPE_MAIN_VARIANT (expr)); + else + vla_p = variably_modified_type_p (expr, NULL_TREE); + bp_pack_value (bp, vla_p, 1); /* We used to stream TYPE_ALIAS_SET == 0 information to let frontends mark types that are opaque for TBAA. This however did not work as intended, because TYPE_ALIAS_SET == 0 was regularly lost in type merging. */