[Bug tree-optimization/105650] New: [13 Regression] Possibly wrong code on fontforge -fvect-cost-model=unlimited

[slyfox at gcc dot gnu.org via Gcc-bugs]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105650

            Bug ID: 105650
           Summary: [13 Regression] Possibly wrong code on fontforge
                    -fvect-cost-model=unlimited
           Product: gcc
           Version: 13.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: slyfox at gcc dot gnu.org
  Target Milestone: ---

Initially encountered wrong code on fontforge-2022-03-08 where it failed to
generate djvu fonts and crashed. I spent some time extracting example and I
hope I got it without without breaking it too much:

// $ cat a.c
// Extracted from fontforge-2022-03-08:
//   $ gcc-13/bin/gcc -O2 a.c -o a && ./a
//   ok
//   $ gcc-13/bin/gcc -O2 -fvect-cost-model=unlimited a.c -o a && ./a
//   SIGSEGV
#include <stdlib.h>

typedef struct SplineChar2 {
    int * p1;
    int * p2;
} SplineChar2;

volatile SplineChar2 gp;

static void dumpgposkernclass(void) __attribute__((noipa));
static void dumpgposkernclass(void) {
    volatile SplineChar2 * a[6] = {&gp, &gp, 0, 0, 0, &gp, };
    volatile int l = 6;

    volatile SplineChar2 **gs = a;
    int numGlyphs = l;

    int i, cnt;
    volatile int v;
    volatile SplineChar2 **glyphs__;

    // count non-null
    for ( i=cnt=0; i<numGlyphs; ++i )
        if ( gs[i]!=NULL ) ++cnt;

    size_t bl = (cnt+1)*sizeof(volatile SplineChar2 *);
    v = cnt+1;
    glyphs__ = malloc(bl);

    // store non-nulls
    for ( i=cnt=0; i<numGlyphs; ++i )
      if ( gs[i]!=NULL )
        glyphs__[cnt++] = gs[i];

    // null-terminate
    glyphs__[cnt++] = NULL;

    // corruption happens here
    if (v != cnt) __builtin_trap();
}

int main(void) {
    dumpgposkernclass();
}

Triggering:

$ gcc-13/bin/gcc -O2 a.c -o a && ./a
$ gcc-13/bin/gcc -O2 -fvect-cost-model=unlimited a.c -o a && ./a
Illegal instruction (core dumped)

It's a weekly gcc-13 from this week:

$ gcc-13/bin/gcc -v |& unnix
Using built-in specs.
COLLECT_GCC=/<<NIX>>/gcc-13.0.0/bin/gcc
COLLECT_LTO_WRAPPER=/<<NIX>>/gcc-13.0.0/libexec/gcc/x86_64-unknown-linux-gnu/13.0.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with:
Thread model: posix
Supported LTO compression algorithms: zlib
gcc version 13.0.0 20220515 (experimental) (GCC)