sunchao opened a new pull request, #5404: URL: https://github.com/apache/datafusion-comet/pull/5404
## Which issue does this PR close? Closes #5402. Follow-up to #5271 and #5393. ## Why are the changes needed? Repairing the negative-zero literals did not make every affected assertion sign-sensitive. The mixed-zero `array_intersect` inputs still produce the same result if an implementation loses the sign distinction, and the special-value percentile query never returns zero. The `arrays_zip` literal change also narrowed one field from double to float. ## What changes were proposed in this PR? - Preserve the existing double `arrays_zip` test shape and add Parquet-backed float/double arrays containing both zero signs and null padding. - Add opposite-sign singleton intersections in both directions and a same-sign negative-zero control for float and double. - Add exact percentile medians for separate negative-zero and positive-zero groups. Keeping the signs in separate groups avoids assertions that depend on the ordering of equal mixed-sign zeros. All added assertions use plain `query`, retaining native operator coverage and exact result comparison. The intersection tests use the running Spark version as the oracle, including Spark 4.2's floating-point normalization. No runtime implementation changes are included. ## How was this PR tested? - Built the unchanged native code with `cargo build --locked --offline` and JDK 17. - Ran the focused Comet SQL tests on Spark 3.4.3 and 4.1.3: 8 tests passed on each version. - Replayed the exact affected SQL records on Spark 3.4.3, 3.5.9, 4.0.2, 4.1.3, and 4.2.0, with Parquet dictionary encoding disabled and enabled: 50 checked queries passed, including raw sign-bit and schema assertions. - Spotless and `git diff --check` passed. Each selector was run separately so ScalaTest selected nonzero tests: ```sh ./mvnw test -Pspark-4.1 -Dtest=none -Dscalastyle.skip=true -Dsuites="org.apache.comet.CometSqlFileTestSuite arrays_zip" ./mvnw test -Pspark-4.1 -Dtest=none -Dscalastyle.skip=true -Dsuites="org.apache.comet.CometSqlFileTestSuite array_intersect" ./mvnw test -Pspark-4.1 -Dtest=none -Dscalastyle.skip=true -Dsuites="org.apache.comet.CometSqlFileTestSuite percentile" ``` Repeated with `-Pspark-3.4`, using a clean reactor build when switching profiles. Local runs used dependency-cache and native-library-directory overrides. The full test suite was not run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
