pedrumj2 opened a new pull request, #13613:
URL: https://github.com/apache/gluten/pull/13613

   ## What changes are proposed in this pull request?
   
   Spark reads 
[spark.sql.mapKeyDedupPolicy](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3785-L3794)
 and 
[spark.io.compression.codec](https://github.com/apache/spark/blob/v3.5.5/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L84)
 in any case, but Gluten [compared the 
policy](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/cpp/velox/compute/WholeStageResultIterator.cc#L692-L696)
 and [the codec 
name](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/common/compression/Compression.cpp#L90-L106)
 case-sensitively. This PR makes Gluten read both settings in any case.
   
   | Setting | Spark | Gluten before | Gluten after |
   |---|---|---|---|
   | `spark.sql.mapKeyDedupPolicy=exception` | recognized | not recognized | 
recognized |
   | `spark.sql.mapKeyDedupPolicy=EXCEPTION` | recognized | recognized | 
recognized |
   | `spark.io.compression.codec=zstd` | recognized | recognized | recognized |
   | `spark.io.compression.codec=ZSTD` | recognized | not recognized | 
recognized |
   
   ## How was this patch tested?
   
   ### Reproduction
   
   1. Clone the repo and check out the base commit:
   
   ```bash
   git clone https://github.com/apache/gluten.git
   cd gluten
   git checkout 31fb4dad58e7339ac4f4b820931d648160dc4746
   ```
   
   2. Build the native libraries and the bundle jar:
   
   ```bash
   docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z 
apache/gluten:centos-9-jdk8 bash -c '
     source /opt/rh/gcc-toolset-12/enable
     ./dev/builddeps-veloxbe.sh --run_setup_script=OFF
     export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
     ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests
   '
   ```
   
   3. Fetch Spark 3.5.5:
   
   ```bash
   curl -sSL 
https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | 
tar -xz
   ```
   
   4. Write the input table and the queries:
   
   ```bash
   cat > write_input.sql <<'EOF'
   CREATE TABLE t USING parquet AS SELECT 'a:1,b:2,a:3' AS s;
   EOF
   cat > query.sql <<'EOF'
   SET spark.sql.mapKeyDedupPolicy=exception;
   EXPLAIN SELECT str_to_map(s, ',', ':') FROM t;
   SELECT str_to_map(s, ',', ':') FROM t;
   EOF
   cat > agg.sql <<'EOF'
   SET spark.sql.adaptive.enabled=false;
   EXPLAIN SELECT s, count(*) FROM t GROUP BY s;
   SELECT s, count(*) FROM t GROUP BY s;
   EOF
   ```
   
   5. Write the table with plain Spark, then run `query.sql` with Gluten:
   
   ```bash
   docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z 
apache/gluten:centos-9-jdk8 bash -c '
     export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
     rm -rf metastore_db spark-warehouse
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f 
write_input.sql 2>/dev/null
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
       --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
       --conf spark.plugins=org.apache.gluten.GlutenPlugin \
       --conf spark.memory.offHeap.enabled=true \
       --conf spark.memory.offHeap.size=4g \
       --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
       --conf 
spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
       --conf 
spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
       -f query.sql 2>err.log
     grep -m 1 "Duplicate" err.log
   '
   ```
   
   6. Run `agg.sql` with Gluten and `spark.io.compression.codec=ZSTD`:
   
   ```bash
   docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z 
apache/gluten:centos-9-jdk8 bash -c '
     export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
     BUNDLE=$(ls 
package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar)
     spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \
       --jars "$BUNDLE" --driver-class-path "$BUNDLE" \
       --conf spark.plugins=org.apache.gluten.GlutenPlugin \
       --conf spark.memory.offHeap.enabled=true \
       --conf spark.memory.offHeap.size=4g \
       --conf 
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
       --conf 
spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
       --conf 
spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \
       --conf spark.io.compression.codec=ZSTD \
       -f agg.sql 2>err.log
     grep -m 1 "compression kind" err.log
   '
   ```
   
   On the base commit, step 5 runs natively and keeps the last value of the 
duplicate key, and the grep finds nothing:
   
   ```
   spark.sql.mapKeyDedupPolicy  exception
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, 
:)#22]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, 
Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], 
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> 
NativeFilters: []
   
   
   {"a":"3","b":"2"}
   ```
   
   Step 6 runs natively and fails on the codec name, and the grep finds the 
reason:
   
   ```
   spark.sql.adaptive.enabled   false
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], 
isStreamingAgg=false)
      +- ^(2) InputIteratorTransformer[s#22, count#26L]
         +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, 
[s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash]
            +- VeloxResizeBatches
               +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS 
hash_partition_key#28, s#22, count#26L]
                  +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], 
functions=[partial_count(1)], isStreamingAgg=false)
                     +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, 
Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], 
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> 
NativeFilters: []
   
   
   Reason: Not support compression kind ZSTD
   ```
   
   On this pull request's branch, with the bundle jar rebuilt, step 5 fails 
natively on the duplicate key, and the grep finds the reason:
   
   ```
   spark.sql.mapKeyDedupPolicy  exception
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, 
:)#22]
      +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, 
Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], 
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> 
NativeFilters: []
   
   
   Reason: Duplicate keys are not allowed: 'a'.
   ```
   
   Step 6 runs natively and returns the count, and the grep finds nothing:
   
   ```
   spark.sql.adaptive.enabled   false
   == Physical Plan ==
   VeloxColumnarToRow
   +- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], 
isStreamingAgg=false)
      +- ^(2) InputIteratorTransformer[s#22, count#26L]
         +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, 
[s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash]
            +- VeloxResizeBatches
               +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS 
hash_partition_key#28, s#22, count#26L]
                  +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], 
functions=[partial_count(1)], isStreamingAgg=false)
                     +- ^(1) FileFileSourceScanExecTransformer parquet 
spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, 
Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], 
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> 
NativeFilters: []
   
   
   a:1,b:2,a:3  1
   ```
   
   Spark without Gluten, which is the Gluten `spark-sql` command in each step 
without the `--jars` and `--driver-class-path` options and without every 
`--conf` option except `spark.io.compression.codec`, fails step 5 and returns 
the same count in step 6:
   
   ```
   spark.sql.mapKeyDedupPolicy  exception
   == Physical Plan ==
   *(1) Project [str_to_map(s#15, ,, :) AS str_to_map(s, ,, :)#16]
   +- *(1) ColumnarToRow
      +- FileScan parquet spark_catalog.default.t[s#15] Batched: true, 
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<s:string>
   
   
   org.apache.spark.SparkRuntimeException: [DUPLICATED_MAP_KEY] Duplicate map 
key a was found, please check the input data. If you want to remove the 
duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so 
that the key inserted at last takes precedence.
   ```
   
   ```
   spark.sql.adaptive.enabled   false
   == Physical Plan ==
   *(2) HashAggregate(keys=[s#16], functions=[count(1)])
   +- Exchange hashpartitioning(s#16, 200), ENSURE_REQUIREMENTS, [plan_id=29]
      +- *(1) HashAggregate(keys=[s#16], functions=[partial_count(1)])
         +- *(1) ColumnarToRow
            +- FileScan parquet spark_catalog.default.t[s#16] Batched: true, 
DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 
paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], 
PushedFilters: [], ReadSchema: struct<s:string>
   
   
   a:1,b:2,a:3  1
   ```
   
   ### Automated tests
   
   A new test in 
[ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1943-L1998),
 `str_to_map with MAP_KEY_DEDUP_POLICY set in lower case`, sets `exception` and 
checks that native 
[str_to_map](https://spark.apache.org/docs/3.5.6/api/sql/index.html#str_to_map) 
fails on a duplicate key. A new test in 
[MiscOperatorSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala#L2667-L2692),
 `native aggregation with spark.io.compression.codec set in upper case`, sets 
`ZSTD` and checks that a native aggregation returns the same rows as Spark.
   
   With this pull request's tests on the base commit's code, both tests fail:
   
   ```bash
   ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
"-Dsuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite @str_to_map 
with MAP_KEY_DEDUP_POLICY set in lower case"
   ```
   
   ```
   - str_to_map with MAP_KEY_DEDUP_POLICY set in lower case *** FAILED ***
     Expected exception org.apache.spark.SparkException to be thrown, but no 
exception was thrown (ScalarFunctionsValidateSuite.scala:2013)
   …
   Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
   *** 1 TEST FAILED ***
   ```
   
   ```bash
   ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
"-Dsuites=org.apache.gluten.execution.MiscOperatorSuite @native aggregation 
with spark.io.compression.codec set in upper case"
   ```
   
   ```
   - native aggregation with spark.io.compression.codec set in upper case *** 
FAILED ***
   …
   Error Source: USER
   Error Code: UNSUPPORTED
   Reason: Not support compression kind ZSTD
   Retriable: False
   Function: stringToCompressionKind
   …
   Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0
   *** 1 TEST FAILED ***
   ```
   
   With this change:
   
   ```bash
   ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox 
-Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true 
-Dtest=none -DfailIfNoTests=false 
-DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite,org.apache.gluten.execution.MiscOperatorSuite
   ```
   
   ```
   ScalarFunctionsValidateSuite:
   …
   - str_to_map with MAP_KEY_DEDUP_POLICY set in lower case
   …
   MiscOperatorSuite:
   …
   - native aggregation with spark.io.compression.codec set in upper case
   …
   Tests: succeeded 191, failed 0, canceled 6, ignored 1, pending 0
   All tests passed.
   ```
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Co-authored with Claude Code (claude-opus-5-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to