pedrumj2 commented on code in PR #13613:
URL: https://github.com/apache/gluten/pull/13613#discussion_r4570110994


##########
gluten-substrait/src/main/scala/org/apache/gluten/config/GlutenConfig.scala:
##########
@@ -613,6 +612,21 @@ object GlutenConfig extends ConfigRegistry {
           nativeConfMap
             .put(SQLConf.LEGACY_TIME_PARSER_POLICY.key, 
v.toUpperCase(Locale.ROOT)))
 
+    // Spark accepts this value in any case, and native code compares it in 
upper case.
+    nativeConfMap.put(
+      SQLConf.MAP_KEY_DEDUP_POLICY.key,
+      conf
+        .getOrElse(
+          SQLConf.MAP_KEY_DEDUP_POLICY.key,
+          SQLConf.MAP_KEY_DEDUP_POLICY.defaultValueString)
+        .toUpperCase(Locale.ROOT)
+    )
+
+    // Spark accepts the codec name in any case, and Velox matches it in lower 
case.
+    conf

Review Comment:
   Thank for the review @rui-mo 
   
   > Would it make sense to make name matching case-insensitive in Velox or 
elsewhere, so we can avoid having to use toUpperCase in some places and 
toLowerCase in others?
   
   Thats a good point, there are a few different options but please let me know 
if you feel this is not the right repository for this change:
   
   ### Option1 Handle in the Engines
   - Every engine would need to handle the case translation on their end 
(Velox, Bolt, ...)
   - The `spark.io.compression.codec` can be handled in the engines since we 
pass down the value as is but the `spark.sql.mapKeyDedupPolicy` might be a bit 
more involved since [we convert it into a 
"true/false"](https://github.com/apache/gluten/blob/3dae1409f9c8656dbb79d26f31ad5b4b8e0f3dbc/cpp/velox/compute/WholeStageResultIterator.cc#L692)
 value in gluten 
   
   
   ### Option2 Handle in Gluten
   This was the path in this PR
   
   ### Option3: Spark
   
   Leave it to Spark to ensure all spark configs are handle in a consistent 
way. Today it seems some configs are changed to uppercase while others are not. 
For example 
[mapKeyDedupPolicy](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3785-L3794)
   ```
     val MAP_KEY_DEDUP_POLICY = buildConf("spark.sql.mapKeyDedupPolicy")
       .doc("The policy to deduplicate map keys in builtin function: CreateMap, 
MapFromArrays, " +
         "MapFromEntries, StringToMap, MapConcat and TransformKeys. When 
EXCEPTION, the query " +
         "fails if duplicated map keys are detected. When LAST_WIN, the map key 
that is inserted " +
         "at last takes precedence.")
       .version("3.0.0")
       .stringConf
       transform(_.toUpperCase(Locale.ROOT))
       .checkValues(MapKeyDedupPolicy.values.map(_.toString))
       .createWithDefault(MapKeyDedupPolicy.EXCEPTION.toString)
   ```
   
   But this route would only work for Spark versions going forward and we'd 
still need some way to handle the previous versions. 
   
   Please let me know what you think, thanks again!
   
   
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to