Re: [PATCH] Fix qsort ordering violation in tree-vrp.c

2017-07-17 Thread Richard Biener 
On Sat, 15 Jul 2017, Yuri Gribov wrote:

> Hi,
> 
> This is a follow-up on https://gcc.gnu.org/ml/gcc/2017-07/msg00078.html
> 
> compare_assert_loc in tree-vrp.c could return unpredictable results
> due to implicit conversion of unsigned subtraction to int here:
>   return ha - hb;
> 
> This could return inconsistent results for objects with the following hashes:
>   a: 0xdc8e4f72U
>   b: 0x912ab538U
>   c: 0x5ae66e3bU
> Then
>   a > b because (int)(0xdc8e4f72U - 0x912ab538U) > 0
>   b > c because (int)(0x912ab538U - 0x5ae66e3bU) > 0
> but
>   a < c because (int)(0xdc8e4f72U - 0x5ae66e3bU) == (int)0x81a7e137U < 0
> 
> Bug was found with https://github.com/yugr/sortcheck
> 
> Bootstrapped and regtested in x64, ok for trunk?

Ok.

Thanks,
Richard.

[PATCH] Fix qsort ordering violation in tree-vrp.c

2017-07-15 Thread Yuri Gribov 
Hi,

This is a follow-up on https://gcc.gnu.org/ml/gcc/2017-07/msg00078.html

compare_assert_loc in tree-vrp.c could return unpredictable results
due to implicit conversion of unsigned subtraction to int here:
  return ha - hb;

This could return inconsistent results for objects with the following hashes:
  a: 0xdc8e4f72U
  b: 0x912ab538U
  c: 0x5ae66e3bU
Then
  a > b because (int)(0xdc8e4f72U - 0x912ab538U) > 0
  b > c because (int)(0x912ab538U - 0x5ae66e3bU) > 0
but
  a < c because (int)(0xdc8e4f72U - 0x5ae66e3bU) == (int)0x81a7e137U < 0

Bug was found with https://github.com/yugr/sortcheck

Bootstrapped and regtested in x64, ok for trunk?

-Y


fix-order-1.patch
Description: Binary data