KrishnaSudarshan7 opened a new pull request, #21720: URL: https://github.com/apache/datafusion/pull/21720
## Which issue does this PR close? Closes # (none — prerequisite for https://github.com/apache/datafusion-comet/issues/2706; follow-up to #17779 and #19274). ## Rationale for this change The existing Spark map_from_entries UDF hardcoded LAST_WIN duplicate-key behavior and silently propagated null struct entries / null keys. Spark's default is EXCEPTION via spark.sql.mapKeyDedupPolicy, and Spark 4 raises SparkRuntimeException with error classes DUPLICATED_MAP_KEY / NULL_MAP_KEY. Without a configurable policy and matching error messages, downstream engines (e.g. datafusion-comet) must fall back to Spark for the common case. ## What changes are included in this PR? 1. Add execution.map_key_dedup_policy to ExecutionConfig in config.rs, mirroring the enable_ansi_mode pattern. 2. Introduce MapKeyDedupPolicy enum and thread it through the dedup helper in utils.rs; raise [DUPLICATED_MAP_KEY] ... under EXCEPTION. 3. Read the policy from ScalarFunctionArgs::config_options in map_from_entries.rs; validate null struct entries and null keys with [NULL_MAP_KEY] ...; preserve outer list-row nulls. 4. Pass EXCEPTION explicitly in map_from_arrays.rs. ## Are these changes tested? Yes — four new Rust unit tests in map_from_entries.rs cover: happy path, duplicate under EXCEPTION, duplicate under LAST_WIN, null struct entry, and null key. Existing sqllogictest coverage for map_from_entries continues to pass. ## Are there any user-facing changes? Yes — one new config key datafusion.execution.map_key_dedup_policy (default "EXCEPTION", accepts "LAST_WIN"). Error messages from map_from_entries now carry Spark-style bracketed error-class prefixes. No breaking API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
