[Bug c++/85795] New: bad_alloc thrown from allocation function when bad_array_length is expected

[msebor at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85795

            Bug ID: 85795
           Summary: bad_alloc thrown from allocation function when
                    bad_array_length is expected
           Product: gcc
           Version: 8.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: msebor at gcc dot gnu.org
  Target Milestone: ---

(This report was prompted by bug 85783.)

In [expr.new], p7, the C++ standard requires implementations to avoid calling
an allocation function when the non-constant size expression in a new
expression is in excess of some implementation-defined limit.  I can't find
this limit documented anywhere but it stands to reason it's no more than
SIZE_MAX.  The program below shows that G++ violates this requirement and calls
the allocation function regardless.  The code that causes this was added in
r190546 as a fix for bug 19351.  (Jason privately confirmed that this is a
likely bug.)

$ cat u.C && g++ -O2 -Wall u.C && ./a.out 
#include <new>

typedef __SIZE_TYPE__ size_t;

void* operator new[](size_t n)
{
  __builtin_printf ("%s (%zx)\n", __func__, n);

  if (void* p = __builtin_malloc (n))
    return p;

  throw std::bad_alloc ();
}

struct A
{
  int i;

  A ();
  ~A ();
};

__attribute__ ((noipa)) A::A () { }
__attribute__ ((noipa)) A::~A () { }

size_t n = __SIZE_MAX__ / 2 - 1;

int main ()
{
  __builtin_printf ("calling new A[%zx]: expect bad_array_new_length\n", n);

  try {
    void *p = new A[n];
    __builtin_printf ("%p: failure\n", p);
  }
  catch (const std::bad_array_new_length&) {
    __builtin_printf ("caught bad_array_new_length: success\n");
    return 0;
  }
  catch (const std::bad_alloc&) {
    __builtin_printf ("caught bad_alloc: failure\n");
  }
  catch (...) {
    __builtin_printf ("caught unknown exception: failure\n");
  }

  __builtin_abort ();
}
calling new A[7ffffffffffffffe]: expect bad_array_new_length
operator new [] (ffffffffffffffff)
caught bad_alloc: failure
Aborted (core dumped)