KrishnaSudarshan7 opened a new pull request, #3991: URL: https://github.com/apache/datafusion-comet/pull/3991
## Which issue does this PR close? Closes #2706. ## Rationale for this change Comet already translated MapFromEntries to the upstream datafusion-spark map_from_entries UDF, but three Spark-compatibility gaps remained: Comet already delegated MapFromEntries to the upstream datafusion-spark UDF, but Spark-parity was incomplete: duplicate-key policy was effectively hardcoded to LAST_WIN, so Comet fell back to Spark for the default EXCEPTION case. Null struct entries and null keys also diverged from Spark, which raises NULL_MAP_KEY / DUPLICATED_MAP_KEY. The companion datafusion PR makes the policy config-driven and adds Spark error-class prefixes; this PR wires that through Comet so both policies run natively and errors match Spark. ## What changes are included in this PR? 1. Flip map_from_entries to supported in spark_expressions_support.md. 2. Remove the LAST_WIN fallback branch in maps.scala, only BinaryType keys/values remain unsupported. 3. Forward spark.sql.mapKeyDedupPolicy across JNI via the ConfigMap proto in CometExecIterator.scala. 4. Translate it onto datafusion.execution.map_key_dedup_policy in jni_api.rs. ## How are these changes tested? Four new tests in CometMapExpressionSuite.scala 1. LAST_WIN runs natively and matches Spark (checkSparkAnswerAndOperator under withSQLConf). 2. Duplicate keys under default EXCEPTION policy throw with DUPLICATED_MAP_KEY. 3. Null struct entry throws with NULL_MAP_KEY. 4. Null key throws with NULL_MAP_KEY. Existing Parquet and BinaryType-fallback tests continue to cover the happy path and unsupported cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
