comphead commented on issue #6570: URL: https://github.com/apache/datafusion-comet/issues/6570#issuecomment-6003034089
Reproduced on `main` (`83285bbe1b`, Spark 4.1). Each query was run with Comet off and on: | Plan shape | Dispatched expression | Rows that differ from Spark | | --- | --- | --- | | `UNION ALL`, second branch | `map(1, spark_partition_id())` | 8 of 16 | | `coalesce(1)` | `map(1, spark_partition_id())` | 4 of 8 | | `UNION ALL` over Parquet | `round(rand(42L) * 100, 2)` | 400 of 800 | | `coalesce(1)` over Parquet | `regexp_replace(uuid(), '-', '')` | 300 of 400 | | `UNION ALL` over Parquet | `udf(monotonically_increasing_id())` | 400 of 800 | | Cross join (`CartesianProductExec`) | `round(rand(42), 6)` on the left | 24 of 32 | The same expressions in a plan without a union, coalesce or cross join match Spark, and `spark.comet.exec.scalaUDF.codegen.enabled=false` restores Spark's answers. Additions to the description: - `CartesianProductExec` over native children is affected too. Each left row ends up with two different `rand(42)` values, where Spark gives one. - Common queries hit this. `round` on a double and `regexp_replace` dispatch their whole subtree by default, so `round(rand(42) * 100, 2)` and `regexp_replace(uuid(), '-', '')` qualify. - A plain `rand(42)` or `monotonically_increasing_id()` in the same row stays correct, so the two columns disagree within one row. - No duplicate uuids or ids came up in any run. - Passing the right index alone does not fix `coalesce`. The kernel cache is per task, so the second input partition reuses the first one's kernel and its random state. `priority:critical` looks right to me. The fix is in #6700. It passes the partition index and plan id from the native planner, and caches nondeterministic kernels per native plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
