ReemaAlzaid commented on code in PR #13642:
URL: https://github.com/apache/gluten/pull/13642#discussion_r4578866834


##########
backends-velox/src/main/scala/org/apache/gluten/expression/ExpressionRestrictions.scala:
##########
@@ -112,7 +123,8 @@ object ExpressionRestrictions {
       ToJsonRestrictions,
       Unbase64Restrictions,
       Base64Restrictions,
-      FormatNumberRestrictions
+      FormatNumberRestrictions,
+      EncodeRestrictions

Review Comment:
   I think this breaks the function support doc. The script runs on Spark 4.x 
where `encode` isn't offloaded at all, but anything in `listAllRestrictions()` 
gets marked PS, and PS wins over unsupported. Maybe keep the object but don't 
list it here? (Only read the script, didn't run it.)



##########
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:
##########
@@ -613,6 +614,14 @@ object GlutenConfig extends ConfigRegistry {
           nativeConfMap
             .put(SQLConf.LEGACY_TIME_PARSER_POLICY.key, 
v.toUpperCase(Locale.ROOT)))
 
+    // Spark 3.x Encode uses String.getBytes, which replaces characters the 
charset cannot
+    // represent with '?'. Velox encode throws on them unless this query 
config is set. Spark 4.x
+    // Encode is a StaticInvoke that Gluten does not offload. Set before 
copying the backend
+    // configs below, so an explicit setting by the user still wins.
+    if (backendName == "velox" && !SparkVersionUtil.gteSpark40) {
+      
nativeConfMap.put(s"${prefixOf(backendName)}.spark.legacy_coding_error_action", 
"true")
+    }

Review Comment:
   Not a blocker, but this is Velox-specific code in the common module. Could 
it be set in `getQueryContextConf` like `legacy_date_formatter`, so we don't 
depend on a hand-typed key string? Also, do we really want users to override 
this to false on 3.x?



##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala:
##########
@@ -1811,4 +1815,19 @@ object VeloxSparkPlanExecApi {
   // A new VeloxSparkPlanExecApi is created per call to 
Backend.sparkPlanExecApi(), so state that
   // must be shared across calls lives here.
   private val warnedBroadcastBuildOncePerExecutorIgnored = new 
AtomicBoolean(false)
+
+  // The charset names Velox encode accepts, compared ignoring ASCII case.
+  private val veloxEncodeCharsets =
+    Set("US-ASCII", "ISO-8859-1", "UTF-8", "UTF-16BE", "UTF-16LE", "UTF-16", 
"UTF-32")
+

Review Comment:
   nit: the charset names are repeated here, in the restriction message, and in 
Velox. Maybe build the message from this list.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to