Fix selectivity estimation integer overflow in contrib/intarray This fixes a poorly written integer comparison function which was performing subtraction in an attempt to return a negative value when a < b and a positive value when a > b, and 0 when the values were equal. Unfortunately that didn't always work correctly due to two's complement having the INT_MIN 1 further from zero than INT_MAX. This could result in an overflow and cause the comparison function to return an incorrect result, which would result in the binary search failing to find the value being searched for.
This could cause poor selectivity estimates when the statistics stored the value of INT_MAX (2147483647) and the value being searched for was large enough to result in the binary search doing a comparison with that INT_MAX value. Author: Chao Li <[email protected]> Reviewed-by: David Rowley <[email protected]> Discussion: https://postgr.es/m/caeowx2ng1ot5lokbvu-dh---dftuzwjrh8wv2chbu29fnnd...@mail.gmail.com Backpatch-through: 14 Branch ------ REL_16_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/54f82c4aae768037fe14d909b2b535cfbe89f900 Modified Files -------------- contrib/intarray/_int_selfuncs.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
