Subodh-17 opened a new pull request, #116:
URL: https://github.com/apache/datafusion-spark/pull/116

   ## Summary
   
   Addresses #46 by migrating the Spark 4 `encode` charset compatibility fix to 
the dedicated `datafusion-spark` repository, following maintainer instructions 
on apache/datafusion#27260 and #87.
   
   In Spark 4.0, `encode` tightened charset validation and error handling in 
two key ways:
   1. **Canonical Charset Whitelist:** Rejects Java charset aliases (`UTF8`, 
`LATIN1`, `ASCII`, `UTF-32BE`, `UTF-32LE`, etc.) with 
`[INVALID_PARAMETER_VALUE.CHARSET]`, accepting only the 7 canonical Spark 
charsets: `US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16`, `UTF-16BE`, `UTF-16LE`, 
and `UTF-32`.
   2. **Unmappable Character Handling:** Raises `[MALFORMED_CHARACTER_CODING] 
... SQLSTATE: 22000` instead of silently replacing unmappable characters with 
`?`.
   
   This PR implements Spark 4.0 strict semantics by default while providing 
programmatic builder methods to restore pre-Spark 4.0 legacy behavior:
   - `SparkEncode::with_legacy_java_charsets(true)` restores legacy Java 
charset alias resolution (Spark 3.5 behavior).
   - `SparkEncode::with_legacy_coding_error_action(true)` restores legacy 
replacement of unmappable characters with `?` (Spark 3.5 behavior).
   - `SparkEncode::with_options(legacy_java_charsets, 
legacy_coding_error_action)` configures both switches.
   
   ## Architectural & Configuration Compatibility
   
   Per #92 and #93, `datafusion-spark` is moving toward local configuration 
extensions and version-derived flags. Following the precedent established in PR 
#70 (`SparkArrayInsert::with_legacy_negative_index`), `SparkEncode` defines 
explicit builder methods on the struct. This allows downstream consumers (such 
as Comet) to configure legacy behavior directly, and provides the exact hooks 
needed once `SparkOptions` (#92) is merged into the crate.
   
   ## Changes
   
   - **`crates/spark/src/function/string/encode.rs`**:
     - Enforce canonical 7-charset whitelist in strict mode; return 
`[INVALID_PARAMETER_VALUE.CHARSET]` for non-canonical names.
     - Case-insensitive matching for canonical names.
     - Raise `[MALFORMED_CHARACTER_CODING]` (SQLSTATE 22000) for unmappable 
characters in `US-ASCII` and `ISO-8859-1`.
     - Provide `with_legacy_java_charsets`, `with_legacy_coding_error_action`, 
and `with_options` builder methods.
     - Validate scalar charsets upfront so invalid charsets fail immediately 
even if the input is empty.
     - Update `#[user_doc]` documentation and examples.
     - Add 10 comprehensive unit tests covering canonical sets, case 
insensitivity, alias rejection, legacy mode acceptance, unmappable characters, 
independent flags, nulls, and array inputs.
   - **`crates/sqllogictest/test_files/string/encode.slt`**:
     - Update test cases to expect Spark 4 strict errors 
(`INVALID_PARAMETER_VALUE.CHARSET` for `utf-32be`, `utf-32le`, `EBCDIC`; 
`MALFORMED_CHARACTER_CODING` for `é` in `us-ascii` and `Ā` in `iso-8859-1`).
   
   ## Validation
   
   - `cargo fmt --all -- --check` (clean)
   - `cargo clippy --all-targets --all-features -- -D warnings` (clean, 0 
warnings)
   - `cargo test -p datafusion-spark -- encode` (10/10 focused unit tests 
passed)
   - `cargo test -p datafusion-spark` (341/341 crate tests passed)
   - `cargo test -p datafusion-spark-sqllogictest -- string/encode` (passed)
   - `cargo test -p datafusion-spark-sqllogictest` (245/245 test files passed, 
100%)
   - `git diff --check` (clean)
   
   Closes #46.
   
   ---
   
   *AI assistance disclosure: This implementation was prepared and verified 
with the assistance of Claude Code / Antigravity.*
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to