rdtr opened a new pull request, #13642:
URL: https://github.com/apache/gluten/pull/13642
## What changes are proposed in this pull request?
On Spark 3.4 and 3.5, `Encode` calls `String.getBytes(charset)`. Characters
the charset cannot represent become `?`, and any JDK charset name or alias is
accepted. Since the Velox pin moved to dft-2026_11_05 (#13540), Gluten offloads
`Encode` to Velox `encode` (facebookincubator/velox#18534), which throws
`encode: input contains a character that cannot be encoded using 'US-ASCII'`
unless the query config `spark.legacy_coding_error_action` is set, and accepts
only seven charset names. `encode('日本語', 'US-ASCII')` and `encode('café',
'utf8')` return values in Spark and fail in Gluten.
This sets `spark.legacy_coding_error_action=true` for the Velox backend on
Spark 3.x. `GlutenConfig.getNativeSessionConf` adds it as
`spark.gluten.sql.columnar.backend.velox.spark.legacy_coding_error_action`,
which `overwriteVeloxConf` passes to the Velox query config, so no C++ change
is needed. It is added before the user's backend configs are copied, so an
explicit setting still wins.
`VeloxSparkPlanExecApi.extraExpressionConverter` now falls back when the
charset is not a string literal naming one of `US-ASCII`, `ISO-8859-1`,
`UTF-8`, `UTF-16BE`, `UTF-16LE`, `UTF-16` or `UTF-32` (ASCII case-insensitive,
like Velox), or a NULL literal. A non-literal charset could hold any alias, so
it falls back too. The fallback reason is the new
`EncodeRestrictions.ONLY_STANDARD_CHARSET_LITERAL`.
Spark 4.x is unchanged. There `Encode` is replaced by a `StaticInvoke` that
Gluten does not offload, so the config is not set.
Fixes #13641
## How was this patch tested?
New test `encode` in `ScalarFunctionsValidateSuite`, run on Spark 3.5 and
earlier. Input is a Parquet table with ASCII, Latin-1, CJK, a supplementary
character, an empty string and NULL.
- For each of the seven charsets (in mixed case), the query is offloaded
(`ProjectExecTransformer`) and the result equals vanilla Spark.
- `encode(s, 'utf8')`, `encode(s, 'latin1')` and `encode(s, cs)` with a
column charset fall back, and the fallback reason contains the new restriction
message.
Ran `ScalarFunctionsValidateSuite` on Spark 3.5: 91 succeeded. Without the
config, the test fails with `encode: input contains a character that cannot be
encoded using 'US-ASCII'`. Without the charset check, it fails with `encode:
unsupported charset 'utf8'`. `test-compile` also passes on Spark 4.1 with Scala
2.13.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]