Subodh-17 opened a new pull request, #116:
URL: https://github.com/apache/datafusion-spark/pull/116
## Summary
Addresses #46 by migrating the Spark 4 `encode` charset compatibility fix to
the dedicated `datafusion-spark` repository, following maintainer instructions
on apache/datafusion#27260 and #87.
In Spark 4.0, `encode` tightened charset validation and error handling in
two key ways:
1. **Canonical Charset Whitelist:** Rejects Java charset aliases (`UTF8`,
`LATIN1`, `ASCII`, `UTF-32BE`, `UTF-32LE`, etc.) with
`[INVALID_PARAMETER_VALUE.CHARSET]`, accepting only the 7 canonical Spark
charsets: `US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16`, `UTF-16BE`, `UTF-16LE`,
and `UTF-32`.
2. **Unmappable Character Handling:** Raises `[MALFORMED_CHARACTER_CODING]
... SQLSTATE: 22000` instead of silently replacing unmappable characters with
`?`.
This PR implements Spark 4.0 strict semantics by default while providing
programmatic builder methods to restore pre-Spark 4.0 legacy behavior:
- `SparkEncode::with_legacy_java_charsets(true)` restores legacy Java
charset alias resolution (Spark 3.5 behavior).
- `SparkEncode::with_legacy_coding_error_action(true)` restores legacy
replacement of unmappable characters with `?` (Spark 3.5 behavior).
- `SparkEncode::with_options(legacy_java_charsets,
legacy_coding_error_action)` configures both switches.
## Architectural & Configuration Compatibility
Per #92 and #93, `datafusion-spark` is moving toward local configuration
extensions and version-derived flags. Following the precedent established in PR
#70 (`SparkArrayInsert::with_legacy_negative_index`), `SparkEncode` defines
explicit builder methods on the struct. This allows downstream consumers (such
as Comet) to configure legacy behavior directly, and provides the exact hooks
needed once `SparkOptions` (#92) is merged into the crate.
## Changes
- **`crates/spark/src/function/string/encode.rs`**:
- Enforce canonical 7-charset whitelist in strict mode; return
`[INVALID_PARAMETER_VALUE.CHARSET]` for non-canonical names.
- Case-insensitive matching for canonical names.
- Raise `[MALFORMED_CHARACTER_CODING]` (SQLSTATE 22000) for unmappable
characters in `US-ASCII` and `ISO-8859-1`.
- Provide `with_legacy_java_charsets`, `with_legacy_coding_error_action`,
and `with_options` builder methods.
- Validate scalar charsets upfront so invalid charsets fail immediately
even if the input is empty.
- Update `#[user_doc]` documentation and examples.
- Add 10 comprehensive unit tests covering canonical sets, case
insensitivity, alias rejection, legacy mode acceptance, unmappable characters,
independent flags, nulls, and array inputs.
- **`crates/sqllogictest/test_files/string/encode.slt`**:
- Update test cases to expect Spark 4 strict errors
(`INVALID_PARAMETER_VALUE.CHARSET` for `utf-32be`, `utf-32le`, `EBCDIC`;
`MALFORMED_CHARACTER_CODING` for `é` in `us-ascii` and `Ā` in `iso-8859-1`).
## Validation
- `cargo fmt --all -- --check` (clean)
- `cargo clippy --all-targets --all-features -- -D warnings` (clean, 0
warnings)
- `cargo test -p datafusion-spark -- encode` (10/10 focused unit tests
passed)
- `cargo test -p datafusion-spark` (341/341 crate tests passed)
- `cargo test -p datafusion-spark-sqllogictest -- string/encode` (passed)
- `cargo test -p datafusion-spark-sqllogictest` (245/245 test files passed,
100%)
- `git diff --check` (clean)
Closes #46.
---
*AI assistance disclosure: This implementation was prepared and verified
with the assistance of Claude Code / Antigravity.*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]