https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65536
--- Comment #56 from GCC Commits <cvs-commit at gcc dot gnu.org> --- The master branch has been updated by Lewis Hyatt <[email protected]>: https://gcc.gnu.org/g:5ca252314bf097bfec299ea8564edc4885c1bd6f commit r17-2492-g5ca252314bf097bfec299ea8564edc4885c1bd6f Author: Lewis Hyatt <[email protected]> Date: Sat Jul 11 14:21:12 2026 -0400 lto: Overhaul approach to location streaming [PR65536] PR65536 is about location-related issues that arise when the LTO front end reads one or several large object files and runs out of location_t space to encode all the locations. The main reason for the potential problem is that the libcpp linemap is designed to be used for incrementally reading source files (and their included header files) in the natural order. When it is used in another way, such as by LTO, which adds locations in the order in which it happened to read different entities, then the assumptions that justify its design are no longer applicable, and it is not hard to run out of location_t values (e.g., because of a large number of file changes, or lines being added out of order.) The PR remains open because it is not theoretically resolved, but it has been resolved for all practical purposes by the following two improvements: 1) The lto_location_cache class now tries hard to optimize the number of maps that it creates, especially by sorting the locations before adding them. This approach only goes so far, because it can only work at the LTO section level, so each function is handled independently of the others, but it helps a lot with reducing the number of maps required for a single function. 2) We moved to 64-bit location_t, which means everything is a lot more forgiving of wasting location_t space. Point 2) made this not much of an issue in practice, although sufficiently large files (especially with very long lines) could still trigger a problem in theory. While locations are now by and large working fine in LTO, there was an interesting discussion on PR65536, starting around: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65536#c8 about how the location streaming might be optimized by streaming the linemap structure itself, rather than a (file name, line number, column number) triplet for each location. In the end, this approach was not taken because solution 1) above was adequate and less disruptive. But the arguments presented in favor of streaming the linemap directly are still interesting. That discussion did not touch on another topic, namely the need to make `#pragma GCC diagnostic' usable in LTO. Right now, it does not work, because the diagnostic pragmas are not streamed out. But even if they were streamed out, the existing approach to locations in the LTO front end is not compatible with enforcing the pragmas. Diagnostic pragmas require a global ordering on location_t values, so that the machinery in option-classifier.cc can determine which pragmas are in force at each location. This requires remembering the order in which each source line was encountered, and that is exactly what is encoded in the linemap. This cannot be reconstructed just from source file names and line numbers; consider, for example, that the same file might be used in different translation units or multiple times in the same translation unit with different diagnostic pragmas in place each time. These considerations tip the scales in favor of adding linemap streaming as Manuel proposed on the PR. This patch prepares to support `#pragma GCC diagnostic' in the LTO front end by changing the approach to location streaming along these lines. The new approach has these general characteristics: o There is a new LTO section (LTO_section_linemap) that contains the information needed to reconstruct the linemap. There is an entry for each line_map_ordinary object that was used by at least one streamed-out location. When the LTO front end reads one of these map entries, it adds a new map with the corresponding properties to its own linemap using the new line_map_add_raw_map interface in libcpp. o When a location needs to be output, we stream out two integers: one to identify which linemap contains the location, and one containing the offset from the start of that map to the location. o When the language front ends stream out their data, they produce a single linemap section (labeled as linemap.0) that applies to the whole translation unit. When WPA prepares partitioned files for LTRANS, it may copy function bodies into the LTRANS files without reading them, so it needs also to copy the linemap sections they refer to. Since the same linemap section will often be needed by more than one partition, this is done by putting all of the linemap sections into one additional object file, which is provided as input to each LTRANS process via the new option -fltrans-linemap-file. o The naming of the LTRANS linemap sections is stable so as not to inhibit incremental LTO. o Once the reader has processed the linemap sections, there is no further overhead associated with inputting a location, unless it is an adhoc location for the purpose of associating a discriminator or a tree with the location. For the adhoc location case, the existing lto_location_cache setup is still useful to avoid creating unneeded adhoc locations, so this has been left in place as before. The restriction that there be only one lto_location_cache at a time is no longer strictly necessary; but it is still useful so that the IPA passes can access the currently active one simply, so I have not changed this for now. With this new setup, PR65536 can be closed. The subsequent patches in this series will enable support for diagnostic pragmas. gcc/lto/ChangeLog: PR lto/65536 * lang.opt: Add -fltrans-linemap-file. * lto-common.cc (lto_read_in_decl_state): Read the linemap ID from function sections. (linemap_section_id): New function. (create_subid_section_table): Note how many linemap sections were found in each input section. (loc_map_decl_data): New GC root. (lto_file_read): Make ORDER into a static variable so it counts continuously across all files and sub-files provided on the command line. (lto_file_finalize): Handle reading the new LTRANS linemap file. (read_cgraph_and_symbols): Likewise. Also stop freeing ALL_FILE_DECL_DATA at the end; the ordered list of files is now useful later on for lto_copy_linemaps(). * lto.cc (stream_out_linemaps): New function. (lto_wpa_write_files): Stream out the linemaps for LTRANS to use. gcc/testsuite/ChangeLog: PR lto/65536 * gcc.misc-tests/outputs.exp: Adjust LTO -save-temps tests to expect the new linemap file. gcc/ChangeLog: PR lto/65536 * doc/lto.texi: Document the new LTO_section_linemap and the new option -fltrans-linemap-file. * lto-opts.cc (lto_write_options): Handle the new option. * opts.cc (gen_command_line_string): Likewise. * lto-section-in.cc (lto_section_name[]): Add new name for LTO_section_linemap. * lto-streamer-in.cc (get_location_from_idx): New function. (lto_location_cache::cmp_loc): Remove. (bp_unpack_delta): New function. (create_loc_map): New function. (get_loc_map): New function. (lto_location_cache::override_loc_map): New function. (lto_location_cache::apply_location_cache): Pervasive changes to implement new location streaming format. (lto_location_cache::input_location_and_block): Likewise. (lto_location_cache::input_location): Rename argument LOC -> DEST for clarity. (lto_read_body_or_constructor): Handle LINEMAP_ID in the decl state. (lto_data_in_create): Add NEED_LOCATION_CACHE argument. Rename local variable DATA_IN to D to avoid clash with the type name. * lto-streamer-out.cc (clear_line_info): Update for change to class output_block. (compute_map_hash): New function. (class location_output): New class. (location_output::record_location): New function. (bp_pack_delta): New function. (location_output::produce_linemap_section): New function. (lto_output_location_1): Pervasive changes to implement new location streaming format. (copy_function_or_variable): Set the LINEMAP_ID in the decl state. (copy_linemap_section): New function. (copy_linemap_sections): New function. (lto_copy_linemaps): New function. (lto_register_linemap_for_output): New function. (lto_output_decl_state_refs): Output the LINEMAP_ID for function decls. (lto_out_decl_state_written_size): Adapt for new LINEMAP_ID output. (produce_asm_for_decls): Output the linemap section when needed. * lto-streamer.cc (lto_get_section_name): Handle linemap sections, which need an order suffix. * lto-streamer.h (enum lto_section_type): Fix typo in the comment. Add LTO_section_linemap. (struct lto_loc_map): New struct. (class lto_location_cache): Pervasive changes to implement new location streaming format. (struct lto_in_decl_state): Add LINEMAP_ID member. (struct lto_out_decl_state): Likewise. (struct lto_file_decl_data): Add LOC_MAP_DECL_DATA, LOC_MAPS, and NUM_LINEMAP_SECTIONS members. (lto_linemap_output_id): New function. (struct output_block): Adjust members for new streaming format. (data_in::data_in): New function. (lto_data_in_create): Adjust prototype for new NEED_LOCATION_CACHE argument. (lto_register_linemap_for_output): Declare. (lto_copy_linemaps): Declare. * lto-wrapper.cc (run_gcc): Pass new argument -fltrans-linemap-file. * timevar.def (TV_IPA_LTO_LINEMAP_IN): New timevar. (TV_IPA_LTO_LINEMAP_OUT): New timevar. (TV_IPA_LTO_LINEMAP_COPY): New timevar.
