[Bug tree-optimization/128589] New: [14/15/16/17 Regression] wrong code at -O2: strlen-like loop with a signed char counter is replaced by __builtin_strlen plus a narrow signed add, and VRP folds the compare

[lzy20000419 at outlook dot com via Gcc-bugs]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=128589

            Bug ID: 128589
           Summary: [14/15/16/17 Regression] wrong code at -O2:
                    strlen-like loop with a signed char counter is
                    replaced by __builtin_strlen plus a narrow signed add,
                    and VRP folds the compare
           Product: gcc
           Version: 17.0
            Status: UNCONFIRMED
          Keywords: wrong-code
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: lzy20000419 at outlook dot com
  Target Milestone: ---

Fails with trunk (gcc 17.0.0 20261109, a74179d08523) and with 12.1.0 through
16.2.0; 9.5.0 through 11.5.0 work.
Target: x86_64-pc-linux-gnu (target-independent).

    #include <stdlib.h>

    __attribute__((noipa)) int
    sc_cmp (const char *s)
    {
      signed char n = 100;
      for (; *s; s++)
        n++;
      return n < -28;
    }

    __attribute__((noipa)) int
    sh_cmp (const char *s)
    {
      short n = 32000;
      for (; *s; s++)
        n++;
      return n < 0;
    }

    char buf[1200];

    int
    main (void)
    {
      for (int i = 0; i < 50; i++)
        buf[i] = 'a';
      if (sc_cmp (buf) != 1)
        abort ();
      for (int i = 50; i < 1000; i++)
        buf[i] = 'a';
      if (sh_cmp (buf) != 1)
        abort ();
      return 0;
    }

    gcc -O2 t.c && ./a.out

Observed: abort. Expected: exit 0. n++ on a signed char is an int addition
followed by an implementation-defined conversion, so n wraps to -106 and n <
-28 is true. The short case is the same.
-O3 and -Os fail too. -O1, -O2 -fno-tree-loop-distribute-patterns and -O2
-fwrapv work.

Godbolt (trunk): https://godbolt.org/z/r1sja7b4b

Right after ldist the IR is

      _18 = __builtin_strlen (_13); _12 = (signed char) _18; _6 = _12 + 101;
_19 = _6 < -28;

generate_strlen_builtin_1 (tree-loop-distribution.cc) converts the strlen
result to the reduction type and then adds the start value in that narrow
signed type. transform_reduction_loop accepts the reduction because
TYPE_OVERFLOW_UNDEFINED is true for the type. The narrow PLUS adds signed
overflow that the source did not have, and later passes fold _6 < -28 to 0.

Possible fix (not tested): in generate_strlen_builtin_1, do the addition in an
unsigned or wider type before converting to the reduction type.

Disclaimer: This test case was found with LLM-assisted targeted testing, and
the report was written with LLM assistance.