rizzabhdwaj opened a new pull request, #27371:
URL: https://github.com/apache/datafusion/pull/27371
## Which issue does this PR close?
issue - #27342
## Rationale for this change
Spark-specific configuration was stored in DataFusion core even though it is
only consumed by Spark-compatible functions. This caused Spark options to
appear in every DataFusion session and required new Spark options to wait for a
DataFusion release.
## What changes are included in this PR?
- Removed `SparkOptions` and `MapKeyDedupPolicy` from DataFusion core.
- Added them to the Spark crate as a `ConfigExtension` using the `spark`
namespace.
- Registered the Spark configuration extension when Spark features are
enabled.
- Updated Spark map functions to read `SparkOptions` from configuration
extensions.
- Changed the map deduplication configuration key from
`datafusion.spark.map_key_dedup_policy` to `spark.map_key_dedup_policy`.
- Preserved the existing `EXCEPTION` and `LAST_WIN` duplicate-key behavior.
- Updated SQL logic tests, configuration documentation, and upgrade
documentation.
- Left `enable_ansi_mode` unchanged because its ownership between DataFusion
core and Spark remains unresolved.
## What is the testing strategy for this PR?
- Verified the Spark crate with:
```bash
cargo check -p datafusion-spark
- Ran formatting and diff checks:
cargo fmt --all -- --check
git diff --check
- Updated Spark map SQL logic tests to cover the new
spark.map_key_dedup_policy key and existing duplicate-key behavior.
Are there any user-facing changes?
Yes. The map deduplication option is no longer a built-in DataFusion option.
The old key:
datafusion.spark.map_key_dedup_policy
is replaced by:
spark.map_key_dedup_policy
The new key is available when Spark features are enabled. The default
behavior remains EXCEPTION, and LAST_WIN remains available for preserving the
previous permissive behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
