rdtr opened a new pull request, #13572: URL: https://github.com/apache/gluten/pull/13572
## What changes are proposed in this pull request? Spark 4.0 added `try_url_decode` (SPARK-48865), which returns NULL on an invalid `%` escape. Spark replaces it with `StaticInvoke(UrlCodec, "decode", Seq(child, Literal(false)))`, and `UrlCodec.decode` returns NULL when `failOnError` is false. Gluten mapped every `UrlCodec.decode` call to Velox `url_decode` with only the first argument. Velox's `url_decode` always throws, so `try_url_decode` on a column with `http%3A%2F%2spark.apache.org` failed with `Illegal hex characters in escape (%) pattern: 2s` instead of returning NULL. `ExpressionConverter.replaceStaticInvokeWithExpressionTransformer` now throws `GlutenNotSupportException` for `UrlCodec.decode` when the second argument is `Literal(false)`, so the project falls back to vanilla Spark. `url_decode` (`failOnError = true`) and `url_encode` are offloaded as before. On Spark 3.5 the second argument of `UrlCodec.decode` is the charset literal `"UTF-8"`, so 3.5 is unchanged. Wrapping `url_decode` in Velox's `try` would turn errors from the child expression into NULL as well, which Spark does not do. The fallback can be removed once Velox has a sparksql `try_url_decode` (tracked in facebookincubator/velox#20057). Fixes #13571 ## How was this patch tested? Added `try_url_decode falls back` to `ScalarFunctionsValidateSuite` (Spark 4.0+). It reads valid, invalid (`%2s`), truncated (`abc%2`) and NULL inputs from Parquet, checks that the result equals vanilla Spark and that the project runs as `ProjectExec`. Ran the `url_*` tests of the suite on Spark 4.1 with `SPARK_ANSI_SQL_MODE=false`. Without the change the new test fails with the `VeloxUserError` above. Compiled on Spark 3.5 (Scala 2.12) as well. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
