sunchao commented on code in PR #5854: URL: https://github.com/apache/datafusion-comet/pull/5854#discussion_r4315759678
########## spark/src/test/resources/sql-tests/expressions/map/map_builders_strict_fp.sql: ########## @@ -0,0 +1,43 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- With `spark.comet.exec.strictFloatingPoint` on, the native map constructors decline a +-- floating-point key type, which they compare by its raw bits where Spark 4.0+ normalizes it (see +-- the map_funcs expression audit). Each constructor goes a different way: `CometMapFromArrays` has +-- no codegen dispatcher, so the projection falls back to Spark, while `CometMapFromEntries` mixes +-- in `CodegenDispatchFallback`, so it stays in the Comet pipeline running Spark's own generated +-- code. A floating-point value type is not affected. + +-- Config: spark.comet.exec.strictFloatingPoint=true + +statement +CREATE TABLE test_map_builders_strict_fp(k double, v int) USING parquet + +statement +INSERT INTO test_map_builders_strict_fp VALUES (1.0D, 1), (double('-0.0'), 2), (2.5D, 3) + +query expect_fallback(Map construction on a floating-point key) Review Comment: [P2] [P2] Expect codegen dispatch in the new map routing fixtures. With the default `spark.comet.exec.scalaUDF.codegen.enabled=true`, `CometMapFromArrays` mixes in `CodegenDispatchFallback`, so this strict-floating-point query executes Spark’s generated code inside `CometProject`. It does not produce the operator fallback required by `expect_fallback`, and the test fails despite returning the correct answer. The same mismatch affects `map_builders_collation.sql`, `map_builders_nested_fp.sql`, `map_builders_nested_fp_last_win.sql`, and `map_from_arrays_nondeterministic_child.sql`. These five failures independently keep required expression CI red. Please use `expect_dispatch(map_from_arrays)` for these cases and align their routing explanations. Evidence: Exact-head expression CI job https://github.com/apache/datafusion-comet/actions/runs/39112998076/job/117639227900 fails this query with `Expected fallback reason 'Map construction on a floating-point key' but no fallback reasons were found`, showing `CometProject` and codegen-dispatched execution. The four sibling fixtures fail with the corresponding missing-fallback assertion. Verified that all five fixtures, `maps.scala`, `QueryPlanSerde.scala`, and the test harness match HEAD in the tested merge revision `62dc23651a2a23cf14ad402cbc3008767c03af47`. With the native library built, the focused reproduction is `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite map_builders_strict_fp"`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
