Hi David,
Here is a new version of the linemap patch (see my earlier emails for an updated
version of the test code). I spent several days poring over the line map code,
and I think I understand it a little better now. I also discovered
-fdump-internal-locations, and it was a big help. If I use this switch with the
code from the test I posted with an initial offset of 0x60000001, I get the
following (I have removed some unnecessary fields for brevity):
ORDINARY MAP: 21
source_location interval: 1610612805 <= loc < 1610612809
file: location-overflow-test-pr83173.c
starting at line: 3
reason: 2 (LC_RENAME)
location-overflow-test-pr83173.c: 3|loc:1610612805| { dg-do preprocess }
location-overflow-test-pr83173.c: 4|loc:1610612806|*/
location-overflow-test-pr83173.c: 5|loc:1610612807|
location-overflow-test-pr83173.c: 6|loc:1610612808|#include
"location-overflow-test-pr83173.h"
ORDINARY MAP: 22
source_location interval: 1610612809 <= loc < 1610612810
file: location-overflow-test-pr83173.h
starting at line: 1
reason: 0 (LC_ENTER)
location-overflow-test-pr83173.h: 1|loc:1610612809|#include
"location-overflow-test-pr83173-1.h"
ORDINARY MAP: 23
source_location interval: 1610612810 <= loc < 1610612812
file: location-overflow-test-pr83173-1.h
starting at line: 1
reason: 0 (LC_ENTER)
location-overflow-test-pr83173-1.h: 1|loc:1610612810|#pragma once
location-overflow-test-pr83173-1.h: 2|loc:1610612811|#define
LOCATION_OVERFLOW_TEST_PR83173_1_H
ORDINARY MAP: 24
source_location interval: 1610612812 <= loc < 1610612812
file: location-overflow-test-pr83173.h
starting at line: 2
reason: 1 (LC_LEAVE)
ORDINARY MAP: 25
source_location interval: 1610612812 <= loc < 1610612814
file: location-overflow-test-pr83173-2.h
starting at line: 1
reason: 0 (LC_ENTER)
location-overflow-test-pr83173-2.h: 1|loc:1610612812|#pragma once
location-overflow-test-pr83173-2.h: 2|loc:1610612813|#define
LOCATION_OVERFLOW_TEST_PR83173_2_H
ORDINARY MAP: 26
source_location interval: 1610612814 <= loc < 1610612815
file: location-overflow-test-pr83173.h
starting at line: 2
reason: 1 (LC_LEAVE)
location-overflow-test-pr83173.h: 2|loc:1610612814|#include
"location-overflow-test-pr83173-2.h"
ORDINARY MAP: 27
source_location interval: 1610612815 <= loc < 1610612823
file: location-overflow-test-pr83173.c
starting at line: 7
reason: 1 (LC_LEAVE)
ORDINARY MAP: 28
source_location interval: 1610612823 <= loc < 1610612825
file: location-overflow-test-pr83173.c
starting at line: 15
reason: 2 (LC_RENAME)
Notice that map 24 and 25 have the same start_location. This is because
line_table->highest_location is being decremented when it should not be. This
new patch uses a more robust method to check whether the source line that
highest_location refers to is greater than loc (which is the source_location of
the #include). Instead of comparing the source_location values directly, I use
linemap_get_expansion_line to map the source location to a line number, and then
compare the line numbers. This handles the case that loc is an adhoc location.
I noticed that after updating the filenames in the test to be
'location-overflow-test-pr83173-1.h', the source locations passed into
_cpp_stack_include were adhoc locations when I wasn't using the location offset
plugin. This is because these filenames are now 34 characters long, which is
longer than the 5 range bits that an ordinary source_location allows.
I spent a lot of time comparing the -fdump-internal-locations output when this
patch is in use to the unpatched version. The only differences occur when a
#include is the last line in the file. For the case where the source location >
LINE_MAP_MAX_SOURCE_LOCATION, the ordinary map 25 (see above) now starts at
1610612813 instead of 1610612812. For the case where the source location <
LINE_MAP_MAX_LOCATION_WITH_COLS, the start location of that map has increased by
32 (which is effectively one column). For the case where the source location is
between LINE_MAP_MAX_LOCATION_WITH_COLS and LINE_MAP_MAX_SOURCE_LOCATION (so
that there are no range bits), the output was unchanged. I can post these
location map dumps if you would like to see them.
There are no regressions in the gcc test suite from this patch.
The other addition in this version if the patch is a small improvement to the
-fdump-internal-locations output to include the reason and included_from fields
from the line_map_ordinary data structure. I aos fixed an issue with the output
alignment. I found this useful when evaluating these changes.
Thanks,
Mike
From e3f23f66ad18598fc21a9223b7f29572a14d1588 Mon Sep 17 00:00:00 2001
From: Mike Gulick <mgul...@mathworks.com>
Date: Fri, 1 Dec 2017 09:43:22 -0500
Subject: [PATCH v2] PR preprocessor/83173: Additional check before
decrementing highest_location
2017-12-01 Mike Gulick <mgul...@mathworks.com>
PR preprocessor/83173
* libcpp/files.c (_cpp_stack_include): Check if
line_table->highest_location is past current line before
decrementing.
* gcc/input.c (dump_location_info): Dump reason and
included_from fields from line_map_ordinary struct. Fix
indentation when location > 5 digits.
* libcpp/location-example.txt: Update example
-fdump-internal-locations output.
---
gcc/input.c | 43 +++++++-
libcpp/files.c | 32 ++++--
libcpp/location-example.txt | 261 +++++++++++++++++++++++++-------------------
3 files changed, 214 insertions(+), 122 deletions(-)
diff --git a/gcc/input.c b/gcc/input.c
index 081e7856916..c7619703625 100644
--- a/gcc/input.c
+++ b/gcc/input.c
@@ -1077,6 +1077,17 @@ dump_labelled_location_range (FILE *stream,
fprintf (stream, "\n");
}
+#define NUM_DIGITS(x) ((x) >= 1000000000 ? 10 : \
+ (x) >= 100000000 ? 9 : \
+ (x) >= 10000000 ? 8 : \
+ (x) >= 1000000 ? 7 : \
+ (x) >= 100000 ? 6 : \
+ (x) >= 10000 ? 5 : \
+ (x) >= 1000 ? 4 : \
+ (x) >= 100 ? 3 : \
+ (x) >= 10 ? 2 : \
+ 1)
+
/* Write a visualization of the locations in the line_table to STREAM. */
void
@@ -1106,6 +1117,29 @@ dump_location_info (FILE *stream)
map->m_column_and_range_bits - map->m_range_bits);
fprintf (stream, " range bits: %i\n",
map->m_range_bits);
+ const char * reason;
+ switch (map->reason) {
+ case LC_ENTER:
+ reason = "LC_ENTER";
+ break;
+ case LC_LEAVE:
+ reason = "LC_LEAVE";
+ break;
+ case LC_RENAME:
+ reason = "LC_RENAME";
+ break;
+ case LC_RENAME_VERBATIM:
+ reason = "LC_RENAME_VERBATIM";
+ break;
+ case LC_ENTER_MACRO:
+ reason = "LC_RENAME_MACRO";
+ break;
+ default:
+ reason = "Unknown";
+ }
+ fprintf (stream, " reason: %d (%s)\n", map->reason, reason);
+ fprintf (stream, " included from idx: %d\n",
+ ORDINARY_MAP_INCLUDER_FILE_INDEX (map));
/* Render the span of source lines that this "map" covers. */
for (source_location loc = MAP_START_LOCATION (map);
@@ -1141,7 +1175,14 @@ dump_location_info (FILE *stream)
if (max_col > line_size)
max_col = line_size + 1;
- int indent = 14 + strlen (exploc.file);
+ int len_lnum = NUM_DIGITS (exploc.line);
+ if (len_lnum < 3)
+ len_lnum = 3;
+ int len_loc = NUM_DIGITS (loc);
+ if (len_loc < 5)
+ len_loc = 5;
+
+ int indent = 6 + strlen (exploc.file) + len_lnum + len_loc;
/* Thousands. */
if (end_location > 999)
diff --git a/libcpp/files.c b/libcpp/files.c
index e8d21b28e62..543730b9267 100644
--- a/libcpp/files.c
+++ b/libcpp/files.c
@@ -1012,6 +1012,7 @@ _cpp_stack_include (cpp_reader *pfile, const char *fname, int angle_brackets,
struct cpp_dir *dir;
_cpp_file *file;
bool stacked;
+ bool decremented = false;
/* For -include command-line flags we have type == IT_CMDLINE.
When the first -include file is processed we have the case, where
@@ -1035,20 +1036,33 @@ _cpp_stack_include (cpp_reader *pfile, const char *fname, int angle_brackets,
return false;
/* Compensate for the increment in linemap_add that occurs if
- _cpp_stack_file actually stacks the file. In the case of a
- normal #include, we're currently at the start of the line
- *following* the #include. A separate source_location for this
- location makes no sense (until we do the LC_LEAVE), and
- complicates LAST_SOURCE_LINE_LOCATION. This does not apply if we
- found a PCH file (in which case linemap_add is not called) or we
- were included from the command-line. */
+ _cpp_stack_file actually stacks the file. In the case of a normal
+ #include, we're currently at the start of the line *following* the
+ #include. A separate source_location for this location makes no
+ sense (until we do the LC_LEAVE), and complicates
+ LAST_SOURCE_LINE_LOCATION. This does not apply if we found a PCH
+ file (in which case linemap_add is not called) or we were included
+ from the command-line. In the case that the #include is the last
+ line in the file, highest_location still points to the current
+ line, not the start of the next line, so we do not decrement in
+ this case. See plugin/location-overflow-test-pr83173.h for an
+ example. */
if (file->pchname == NULL && file->err_no == 0
&& type != IT_CMDLINE && type != IT_DEFAULT)
- pfile->line_table->highest_location--;
+ {
+ int highest_line = linemap_get_expansion_line (pfile->line_table,
+ pfile->line_table->highest_location);
+ int source_line = linemap_get_expansion_line (pfile->line_table, loc);
+ if (highest_line > source_line)
+ {
+ pfile->line_table->highest_location--;
+ decremented = true;
+ }
+ }
stacked = _cpp_stack_file (pfile, file, type == IT_IMPORT, loc);
- if (!stacked)
+ if (decremented && !stacked)
/* _cpp_stack_file didn't stack the file, so let's rollback the
compensation dance we performed above. */
pfile->line_table->highest_location++;
diff --git a/libcpp/location-example.txt b/libcpp/location-example.txt
index 14b5c2e284a..c6b8f7be881 100644
--- a/libcpp/location-example.txt
+++ b/libcpp/location-example.txt
@@ -33,8 +33,11 @@ ORDINARY MAP: 0
source_location interval: 32 <= loc < 64
file: test.c
starting at line: 1
- column bits: 12
+ column and range bits: 12
+ column bits: 7
range bits: 5
+ reason: 0 (LC_ENTER)
+ included from idx: -1
test.c: 1|loc: 32|#include "test.h"
|69269258258148147
|46802468024680246
@@ -43,186 +46,220 @@ ORDINARY MAP: 1
source_location interval: 64 <= loc < 96
file: <built-in>
starting at line: 0
+ column and range bits: 0
column bits: 0
range bits: 0
+ reason: 2 (LC_RENAME)
+ included from idx: -1
ORDINARY MAP: 2
source_location interval: 96 <= loc < 128
file: <command-line>
starting at line: 0
+ column and range bits: 0
column bits: 0
range bits: 0
+ reason: 2 (LC_RENAME)
+ included from idx: -1
ORDINARY MAP: 3
- source_location interval: 128 <= loc < 160128
+ source_location interval: 128 <= loc < 250240
file: /usr/include/stdc-predef.h
starting at line: 1
- column bits: 12
+ column and range bits: 12
+ column bits: 7
range bits: 5
+ reason: 0 (LC_ENTER)
+ included from idx: 2
(contents of /usr/include/stdc-predef.h snipped for brevity)
ORDINARY MAP: 4
- source_location interval: 160128 <= loc < 160160
+ source_location interval: 250240 <= loc < 250272
file: <command-line>
starting at line: 32
- column bits: 12
+ column and range bits: 12
+ column bits: 7
range bits: 5
+ reason: 1 (LC_LEAVE)
+ included from idx: -1
ORDINARY MAP: 5
- source_location interval: 160160 <= loc < 164256
+ source_location interval: 250272 <= loc < 254368
file: test.c
starting at line: 1
- column bits: 12
+ column and range bits: 12
+ column bits: 7
range bits: 5
-test.c: 1|loc:160160|#include "test.h"
- |00000000000000000
- |12223334445556667
- |92582581481470470
- |24680246802468024
+ reason: 2 (LC_RENAME)
+ included from idx: -1
+test.c: 1|loc:250272|#include "test.h"
+ |00000000000000000
+ |33344445556667778
+ |03603692692582581
+ |46802468024680246
ORDINARY MAP: 6
- source_location interval: 164256 <= loc < 173280
+ source_location interval: 254368 <= loc < 263392
file: test.h
starting at line: 1
- column bits: 12
+ column and range bits: 12
+ column bits: 7
range bits: 5
-test.h: 1|loc:164256|extern int foo ();
- |444444444444444444
- |233344455566677788
- |825814814704703603
- |802468024680246802
-test.h: 2|loc:168352|
- |
- |
- |
- |
-test.h: 3|loc:172448|#define PLUS(A, B) A + B
- |222222222222222223333333
- |455566677788889990001112
- |814704703603692692582581
- |024680246802468024680246
+ reason: 0 (LC_ENTER)
+ included from idx: 5
+test.h: 1|loc:254368|extern int foo ();
+ |444444444444444444
+ |444455566677788899
+ |036926925825814814
+ |024680246802468024
+test.h: 2|loc:258464|
+ |
+ |
+ |
+ |
+test.h: 3|loc:262560|#define PLUS(A, B) A + B
+ |222222222222233333333333
+ |566677788899900011122223
+ |925825814814704703603692
+ |246802468024680246802468
ORDINARY MAP: 7
- source_location interval: 173280 <= loc < 202016
+ source_location interval: 263392 <= loc < 292128
file: test.c
starting at line: 2
- column bits: 12
+ column and range bits: 12
+ column bits: 7
range bits: 5
-test.c: 2|loc:173280|
- |
- |
- |
- |
-test.c: 3|loc:177376|int
- |777
- |444
- |047
- |802
-test.c: 4|loc:181472|main (int argc, char **argv)
- |1111111111111111222222222222
- |5556666777888999000111222333
- |0360369269258258148147047036
- |4680246802468024680246802468
-test.c: 5|loc:185568|{
- |5
- |6
- |0
- |0
-test.c: 6|loc:189664| int a = PLUS (1,2);
- |999999999900000000000
- |677788899900011122233
- |926925825814814704703
- |680246802468024680246
-test.c: 7|loc:193760| int b = PLUS (3,4);
- |333333344444444444444
- |788899900011122233344
- |925825814814704703603
- |246802468024680246802
-test.c: 8|loc:197856| return 0;
- |77778888888
- |89990001112
- |82581481470
- |80246802468
-test.c: 9|loc:201952|}
- |1
- |9
- |8
- |4
+ reason: 1 (LC_LEAVE)
+ included from idx: -1
+test.c: 2|loc:263392|
+ |
+ |
+ |
+ |
+test.c: 3|loc:267488|int
+ |777
+ |555
+ |258
+ |024
+test.c: 4|loc:271584|main (int argc, char **argv)
+ |1111111111112222222222222222
+ |6667778889990000111222333444
+ |1481470470360369269258258148
+ |6802468024680246802468024680
+test.c: 5|loc:275680|{
+ |5
+ |7
+ |1
+ |2
+test.c: 6|loc:279776| int a = PLUS (1,2);
+ |999999000000000000000
+ |888999000011122233344
+ |047036036926925825814
+ |802468024680246802468
+test.c: 7|loc:283872| int b = PLUS (3,4);
+ |333444444444444444444
+ |999000011122233344455
+ |036036926925825814814
+ |468024680246802468024
+test.c: 8|loc:287968| return 0;
+ |88888888888
+ |00001112223
+ |03692692582
+ |02468024680
+test.c: 9|loc:292064|}
+ |2
+ |0
+ |9
+ |6
UNALLOCATED LOCATIONS
- source_location interval: 202016 <= loc < 2147483633
+ source_location interval: 292128 <= loc < 2147483631
-MACRO 1: PLUS (7 tokens)
- source_location interval: 2147483633 <= loc < 2147483640
-test.c:7:11: note: expansion point is location 194115
+MACRO 3: PLUS (7 tokens)
+ source_location interval: 2147483631 <= loc < 2147483638
+test.c:7:11: note: expansion point is location 284227
int b = PLUS (3,4);
^~~~
-
- map->start_location: 2147483633
+ map->start_location: 2147483631
macro_locations:
- 0: 194304, 173088
-test.c:7:17: note: token 0 has x-location == 194304
+ 0: 284416, 263200
+test.c:7:17: note: token 0 has x-location == 284416
int b = PLUS (3,4);
^
-
-test.c:7:17: note: token 0 has y-location == 173088
- 1: 173152, 173152
-In file included from test.c:1:0:
-test.h:3:22: note: token 1 has x-location == y-location == 173152
+test.c:7:17: note: token 0 has y-location == 263200
+ 1: 263264, 263264
+In file included from test.c:1:
+test.h:3:22: note: token 1 has x-location == y-location == 263264
#define PLUS(A, B) A + B
^
-
- 2: 194368, 173216
-test.c:7:19: note: token 2 has x-location == 194368
+ 2: 284480, 263328
+test.c:7:19: note: token 2 has x-location == 284480
int b = PLUS (3,4);
^
-
-test.c:7:19: note: token 2 has y-location == 173216
+test.c:7:19: note: token 2 has y-location == 263328
3: 0, 2947526575
cc1: note: token 3 has x-location == 0
cc1: note: token 3 has y-location == 2947526575
4: 2947526575, 2947526575
-x-location == y-location == 2947526575 encodes token # 800042942
+x-location == y-location == 2947526575 encodes token # 800042944
5: 2947526575, 2947526575
-x-location == y-location == 2947526575 encodes token # 800042942
+x-location == y-location == 2947526575 encodes token # 800042944
6: 2947526575, 2947526575
-x-location == y-location == 2947526575 encodes token # 800042942
+x-location == y-location == 2947526575 encodes token # 800042944
-MACRO 0: PLUS (7 tokens)
- source_location interval: 2147483640 <= loc < 2147483647
-test.c:6:11: note: expansion point is location 190019
+MACRO 2: PLUS (7 tokens)
+ source_location interval: 2147483638 <= loc < 2147483645
+test.c:6:11: note: expansion point is location 280131
int a = PLUS (1,2);
^~~~
-
- map->start_location: 2147483640
+ map->start_location: 2147483638
macro_locations:
- 0: 190208, 173088
-test.c:6:17: note: token 0 has x-location == 190208
+ 0: 280320, 263200
+test.c:6:17: note: token 0 has x-location == 280320
int a = PLUS (1,2);
^
-
-test.c:6:17: note: token 0 has y-location == 173088
- 1: 173152, 173152
-In file included from test.c:1:0:
-test.h:3:22: note: token 1 has x-location == y-location == 173152
+test.c:6:17: note: token 0 has y-location == 263200
+ 1: 263264, 263264
+In file included from test.c:1:
+test.h:3:22: note: token 1 has x-location == y-location == 263264
#define PLUS(A, B) A + B
^
-
- 2: 190272, 173216
-test.c:6:19: note: token 2 has x-location == 190272
+ 2: 280384, 263328
+test.c:6:19: note: token 2 has x-location == 280384
int a = PLUS (1,2);
^
-
-test.c:6:19: note: token 2 has y-location == 173216
+test.c:6:19: note: token 2 has y-location == 263328
3: 0, 2947526575
cc1: note: token 3 has x-location == 0
cc1: note: token 3 has y-location == 2947526575
4: 2947526575, 2947526575
-x-location == y-location == 2947526575 encodes token # 800042935
+x-location == y-location == 2947526575 encodes token # 800042937
5: 2947526575, 2947526575
-x-location == y-location == 2947526575 encodes token # 800042935
+x-location == y-location == 2947526575 encodes token # 800042937
6: 2947526575, 2947526575
-x-location == y-location == 2947526575 encodes token # 800042935
+x-location == y-location == 2947526575 encodes token # 800042937
+
+MACRO 1: __GCC_IEC_559_COMPLEX (1 tokens)
+ source_location interval: 2147483645 <= loc < 2147483646
+In file included from <command-line>:31:
+/usr/include/stdc-predef.h:45:6: note: expansion point is location 180564
+ # if __GCC_IEC_559_COMPLEX > 0
+ ^~~~~~~~~~~~~~~~~~~~~
+ map->start_location: 2147483645
+ macro_locations:
+ 0: 1, 1
+<built-in>: note: token 0 has x-location == y-location == 1
+
+MACRO 0: __GCC_IEC_559 (1 tokens)
+ source_location interval: 2147483646 <= loc < 2147483647
+/usr/include/stdc-predef.h:37:6: note: expansion point is location 147788
+ # if __GCC_IEC_559 > 0
+ ^~~~~~~~~~~~~
+ map->start_location: 2147483646
+ macro_locations:
+ 0: 1, 1
+<built-in>: note: token 0 has x-location == y-location == 1
MAX_SOURCE_LOCATION
source_location interval: 2147483647 <= loc < 2147483648
--
2.11.0
