Subodh-17 opened a new pull request, #27260: URL: https://github.com/apache/datafusion/pull/27260
## Summary Addresses #27186 by adding selectable Spark 4-compatible behavior to the `datafusion-spark` `encode` expression. - Adds `datafusion.spark.legacy_java_charsets` and `datafusion.spark.legacy_coding_error_action`, both defaulting to `false`. - Enforces Spark 4's seven canonical charset names by default while retaining legacy aliases when `legacy_java_charsets` is enabled. - Raises `MALFORMED_CHARACTER_CODING` (SQLSTATE `22000`) for unmappable characters by default. - Restores legacy replacement behavior when `legacy_coding_error_action` is enabled. - Adds unit tests and SQLLogicTests for charset handling, configuration combinations, error behavior, and settings exposure. - Documents both configuration options. ## Validation - `cargo check -p datafusion-spark -p datafusion-common` - `cargo clippy -p datafusion-spark -p datafusion-common -- -D warnings` - `cargo fmt --check` - Focused encode unit tests - Full `datafusion-spark` library tests (333 passed) - Focused `encode.slt` SQLLogicTest - Focused `information_schema.slt` SQLLogicTest - `git diff HEAD^ HEAD --check` ## Compatibility note The existing empty-input UTF-16 behavior is unchanged: DataFusion emits the UTF-16 BOM, whereas Spark returns an empty byte array. This pre-existing divergence is documented and retained. Closes #27186. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
