zanmato1984 commented on code in PR #51325:
URL: https://github.com/apache/arrow/pull/51325#discussion_r4582583455
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -2582,6 +2615,49 @@ TEST(TestPrimitiveAdHoc, FloatingUlpDistanceEquality) {
CheckFloatApproxEqualsWithUlpDistance<DoubleType>();
}
+TEST(TestPrimitiveAdHoc, FloatingPhysicalRepresentationEquality) {
+ CheckFloatEqualsWithPhysicalRepresentation<HalfFloatType>();
+ CheckFloatEqualsWithPhysicalRepresentation<FloatType>();
+ CheckFloatEqualsWithPhysicalRepresentation<DoubleType>();
+}
+
+TEST(TestRunEncodedBuilder, MergeConsecutiveNaNs) {
+ auto run_end_encoded_type = run_end_encoded(int32(), float32());
+ std::shared_ptr<RunEndEncodedArray> result;
+ auto run_end_builder = std::make_shared<Int32Builder>();
+ auto value_builder = std::make_shared<FloatBuilder>();
+ RunEndEncodedBuilder builder(default_memory_pool(), run_end_builder,
value_builder,
+ run_end_encoded_type);
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 1), 15));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 1), 10));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 2), 20));
+ ASSERT_OK(builder.AppendScalar(
+ **MakeScalar(float32(), std::numeric_limits<float>::quiet_NaN()), 25));
+ ASSERT_OK(builder.AppendScalar(
+ **MakeScalar(float32(), std::numeric_limits<float>::signaling_NaN()),
5));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 3), 15));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), std::nanf("")), 20));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 4), 25));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), +0.0), 10));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), 5.0), 15));
+ ASSERT_OK(builder.AppendScalar(**MakeScalar(float32(), -0.0), 25));
Review Comment:
Could this test append the same NaN in two separate `AppendScalar` calls?
The consecutive NaNs here are quiet versus signaling, which should remain
separate. The other NaN is isolated, and `+0` and `-0` are separated by `5`.
As a narrow negative control, reverting only the builder's `Equals(scalar,
options_)` call to `Equals(scalar)`, while keeping the rest of the current code
and tests unchanged, still leaves this test passing. An independent
repeated-NaN probe fails under that same reversion.
Two calls to `AppendScalar(nan, 3)` with identical NaN bits should produce
run ends `[6]`, not `[3, 6]`. Could we cover that case, two quiet NaNs with
different payloads, and adjacent signed zeros, asserting both the run
boundaries and exact decoded bits? This would protect the intended merge
without allowing payload or sign loss.
##########
cpp/src/arrow/compare.h:
##########
@@ -108,6 +108,19 @@ class EqualOptions {
return res;
}
+ /// Whether to compare floating-point values by their physical
representation.
+ bool physical_repr_equal() const { return physical_repr_equal_; }
+
+ /// Return a copy of this EqualOptions object with the "physical_repr_equal"
+ /// property set to the specified value.
+ /// If both "physical_repr_equal" and "nans_equal" are enabled, two values
+ /// are considered equal if either condition is satisfied.
+ EqualOptions physical_repr_equal(bool v) const {
Review Comment:
Could we also propagate this new option to sparse floating-point equality?
`FloatSparseTensorDataEquals` currently only checks `nans_equal()`, so it
ignores `physical_repr_equal()`.
For independently allocated float32 and float64 tensors containing identical
NaN bits,
`EqualOptions::Defaults().signed_zeros_equal(false).physical_repr_equal(true)`
gives `true` for dense tensor equality but `false` after conversion to COO
sparse tensors. Enabling `nans_equal(true)` is not equivalent, since that also
accepts different NaN payloads.
Since both equality APIs accept `EqualOptions` and this new flag has no
documented sparse exclusion, could the sparse implementation reuse the common
floating-point equality visitor, with regression coverage for identical and
different NaN payloads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]