btree_gist: fix NaN handling in float4/float8 opclasses. The float4 and float8 btree_gist opclasses compared keys with raw C operators (==, <, >). IEEE 754 makes every comparison involving NaN false, so GiST disagreed with the regular float comparison operators and with the btree opclass, which uses float[4|8]_cmp_internal() (so that all NaNs are equal and NaN sorts after every non-NaN value).
In addition, the penalty and distance functions were not careful about NaNs, and the penalty functions could also misbehave for IEEE infinities. Wrong answers from the penalty functions would probably do no more than make the index non-optimal, but the distance mistakes were visible from SQL. To fix, make the comparison functions rely on the same NaN-aware comparison functions the core code uses, and rewrite the penalty and distance functions to follow the rules that NaNs are equal but maximally far away from non-NaNs. The penalty_num() code was formerly shared between integral and float cases, but I chose to make two copies so that the integral cases are not saddled with the extra logic for NaNs and infinities/overflows. I also rewrote it as static inline functions instead of an unreadable and uncommented macro. The float penalty functions were previously unreached by the regression tests, so add new test cases to exercise them. There's no on-disk format change, but users who have NaN entries in a btree_gist index would be well advised to reindex it. Bug: #19501 Bug: #19524 Reported-by: Man Zeng <[email protected]> Reported-by: Yuelin Wang <[email protected]> Author: Bill Kim <[email protected]> Co-authored-by: Tom Lane <[email protected]> Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/camqxxcgbtd2lxfx0tpgvoizxp-xxrchv2zdy4by_tznjmsx...@mail.gmail.com Backpatch-through: 14 Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/7d3448961da3f8cb5c78b9d58c5e03b6bff53364 Modified Files -------------- contrib/btree_gist/btree_float4.c | 59 +++++++++----- contrib/btree_gist/btree_float8.c | 51 +++++++++---- contrib/btree_gist/btree_utils_num.h | 131 ++++++++++++++++++++++++++++---- contrib/btree_gist/data/float4.data | 3 + contrib/btree_gist/data/float8.data | 3 + contrib/btree_gist/expected/float4.out | 51 +++++++++++-- contrib/btree_gist/expected/float8.out | 51 +++++++++++-- contrib/btree_gist/expected/numeric.out | 48 ++++++------ contrib/btree_gist/sql/float4.sql | 17 +++++ contrib/btree_gist/sql/float8.sql | 17 +++++ 10 files changed, 345 insertions(+), 86 deletions(-)
