pedrumj2 opened a new pull request, #13613: URL: https://github.com/apache/gluten/pull/13613
## What changes are proposed in this pull request? Spark reads [spark.sql.mapKeyDedupPolicy](https://github.com/apache/spark/blob/v3.5.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3785-L3794) and [spark.io.compression.codec](https://github.com/apache/spark/blob/v3.5.5/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L84) in any case, but Gluten [compared the policy](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/cpp/velox/compute/WholeStageResultIterator.cc#L692-L696) and [the codec name](https://github.com/IBM/velox/blob/6deac80271569277bf54b6d04547620c17438348/velox/common/compression/Compression.cpp#L90-L106) case-sensitively. This PR makes Gluten read both settings in any case. | Setting | Spark | Gluten before | Gluten after | |---|---|---|---| | `spark.sql.mapKeyDedupPolicy=exception` | recognized | not recognized | recognized | | `spark.sql.mapKeyDedupPolicy=EXCEPTION` | recognized | recognized | recognized | | `spark.io.compression.codec=zstd` | recognized | recognized | recognized | | `spark.io.compression.codec=ZSTD` | recognized | not recognized | recognized | ## How was this patch tested? ### Reproduction 1. Clone the repo and check out the base commit: ```bash git clone https://github.com/apache/gluten.git cd gluten git checkout 31fb4dad58e7339ac4f4b820931d648160dc4746 ``` 2. Build the native libraries and the bundle jar: ```bash docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' source /opt/rh/gcc-toolset-12/enable ./dev/builddeps-veloxbe.sh --run_setup_script=OFF export JAVA_HOME=/usr/lib/jvm/java-17-openjdk ./build/mvn clean install -Pbackends-velox -Pspark-3.5 -DskipTests ' ``` 3. Fetch Spark 3.5.5: ```bash curl -sSL https://archive.apache.org/dist/spark/spark-3.5.5/spark-3.5.5-bin-hadoop3.tgz | tar -xz ``` 4. Write the input table and the queries: ```bash cat > write_input.sql <<'EOF' CREATE TABLE t USING parquet AS SELECT 'a:1,b:2,a:3' AS s; EOF cat > query.sql <<'EOF' SET spark.sql.mapKeyDedupPolicy=exception; EXPLAIN SELECT str_to_map(s, ',', ':') FROM t; SELECT str_to_map(s, ',', ':') FROM t; EOF cat > agg.sql <<'EOF' SET spark.sql.adaptive.enabled=false; EXPLAIN SELECT s, count(*) FROM t GROUP BY s; SELECT s, count(*) FROM t GROUP BY s; EOF ``` 5. Write the table with plain Spark, then run `query.sql` with Gluten: ```bash docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' export JAVA_HOME=/usr/lib/jvm/java-17-openjdk rm -rf metastore_db spark-warehouse spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" -f write_input.sql 2>/dev/null BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ -f query.sql 2>err.log grep -m 1 "Duplicate" err.log ' ``` 6. Run `agg.sql` with Gluten and `spark.io.compression.codec=ZSTD`: ```bash docker run --rm -w /work/gluten -v "$PWD":/work/gluten:z apache/gluten:centos-9-jdk8 bash -c ' export JAVA_HOME=/usr/lib/jvm/java-17-openjdk BUNDLE=$(ls package/target/gluten-velox-bundle-spark3.5_2.12-linux_amd64-*.jar) spark-3.5.5-bin-hadoop3/bin/spark-sql --master "local[2]" \ --jars "$BUNDLE" --driver-class-path "$BUNDLE" \ --conf spark.plugins=org.apache.gluten.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=4g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --conf spark.driver.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ --conf spark.executor.extraJavaOptions=-Dio.netty.tryReflectionSetAccessible=true \ --conf spark.io.compression.codec=ZSTD \ -f agg.sql 2>err.log grep -m 1 "compression kind" err.log ' ``` On the base commit, step 5 runs natively and keeps the last value of the duplicate key, and the grep finds nothing: ``` spark.sql.mapKeyDedupPolicy exception == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, :)#22] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] {"a":"3","b":"2"} ``` Step 6 runs natively and fails on the codec name, and the grep finds the reason: ``` spark.sql.adaptive.enabled false == Physical Plan == VeloxColumnarToRow +- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], isStreamingAgg=false) +- ^(2) InputIteratorTransformer[s#22, count#26L] +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, [s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash] +- VeloxResizeBatches +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS hash_partition_key#28, s#22, count#26L] +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], functions=[partial_count(1)], isStreamingAgg=false) +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] Reason: Not support compression kind ZSTD ``` On this pull request's branch, with the bundle jar rebuilt, step 5 fails natively on the duplicate key, and the grep finds the reason: ``` spark.sql.mapKeyDedupPolicy exception == Physical Plan == VeloxColumnarToRow +- ^(1) ProjectExecTransformer [str_to_map(s#21, ,, :) AS str_to_map(s, ,, :)#22] +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#21] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] Reason: Duplicate keys are not allowed: 'a'. ``` Step 6 runs natively and returns the count, and the grep finds nothing: ``` spark.sql.adaptive.enabled false == Physical Plan == VeloxColumnarToRow +- ^(2) HashAggregateTransformer(keys=[s#22], functions=[count(1)], isStreamingAgg=false) +- ^(2) InputIteratorTransformer[s#22, count#26L] +- ColumnarExchange hashpartitioning(s#22, 200), ENSURE_REQUIREMENTS, [s#22, count#26L], [plan_id=740], [shuffle_writer_type=hash] +- VeloxResizeBatches +- ^(1) ProjectExecTransformer [hash(s#22, 42) AS hash_partition_key#28, s#22, count#26L] +- ^(1) FlushableHashAggregateTransformer(keys=[s#22], functions=[partial_count(1)], isStreamingAgg=false) +- ^(1) FileFileSourceScanExecTransformer parquet spark_catalog.default.t[s#22] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> NativeFilters: [] a:1,b:2,a:3 1 ``` Spark without Gluten, which is the Gluten `spark-sql` command in each step without the `--jars` and `--driver-class-path` options and without every `--conf` option except `spark.io.compression.codec`, fails step 5 and returns the same count in step 6: ``` spark.sql.mapKeyDedupPolicy exception == Physical Plan == *(1) Project [str_to_map(s#15, ,, :) AS str_to_map(s, ,, :)#16] +- *(1) ColumnarToRow +- FileScan parquet spark_catalog.default.t[s#15] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> org.apache.spark.SparkRuntimeException: [DUPLICATED_MAP_KEY] Duplicate map key a was found, please check the input data. If you want to remove the duplicated keys, you can set "spark.sql.mapKeyDedupPolicy" to "LAST_WIN" so that the key inserted at last takes precedence. ``` ``` spark.sql.adaptive.enabled false == Physical Plan == *(2) HashAggregate(keys=[s#16], functions=[count(1)]) +- Exchange hashpartitioning(s#16, 200), ENSURE_REQUIREMENTS, [plan_id=29] +- *(1) HashAggregate(keys=[s#16], functions=[partial_count(1)]) +- *(1) ColumnarToRow +- FileScan parquet spark_catalog.default.t[s#16] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/work/gluten/spark-warehouse/t], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<s:string> a:1,b:2,a:3 1 ``` ### Automated tests A new test in [ScalarFunctionsValidateSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala#L1943-L1998), `str_to_map with MAP_KEY_DEDUP_POLICY set in lower case`, sets `exception` and checks that native [str_to_map](https://spark.apache.org/docs/3.5.6/api/sql/index.html#str_to_map) fails on a duplicate key. A new test in [MiscOperatorSuite](https://github.com/apache/gluten/blob/31fb4dad58e7339ac4f4b820931d648160dc4746/backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala#L2667-L2692), `native aggregation with spark.io.compression.codec set in upper case`, sets `ZSTD` and checks that a native aggregation returns the same rows as Spark. With this pull request's tests on the base commit's code, both tests fail: ```bash ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite @str_to_map with MAP_KEY_DEDUP_POLICY set in lower case" ``` ``` - str_to_map with MAP_KEY_DEDUP_POLICY set in lower case *** FAILED *** Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown (ScalarFunctionsValidateSuite.scala:2013) … Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` ```bash ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false "-Dsuites=org.apache.gluten.execution.MiscOperatorSuite @native aggregation with spark.io.compression.codec set in upper case" ``` ``` - native aggregation with spark.io.compression.codec set in upper case *** FAILED *** … Error Source: USER Error Code: UNSUPPORTED Reason: Not support compression kind ZSTD Retriable: False Function: stringToCompressionKind … Tests: succeeded 0, failed 1, canceled 0, ignored 0, pending 0 *** 1 TEST FAILED *** ``` With this change: ```bash ./build/mvn -B test -Pbackends-velox -Pspark-3.5 -pl backends-velox -Dspotless.check.skip=true -Dscalastyle.skip=true -Dcheckstyle.skip=true -Dtest=none -DfailIfNoTests=false -DwildcardSuites=org.apache.gluten.functions.ScalarFunctionsValidateSuite,org.apache.gluten.execution.MiscOperatorSuite ``` ``` ScalarFunctionsValidateSuite: … - str_to_map with MAP_KEY_DEDUP_POLICY set in lower case … MiscOperatorSuite: … - native aggregation with spark.io.compression.codec set in upper case … Tests: succeeded 191, failed 0, canceled 6, ignored 1, pending 0 All tests passed. ``` ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Co-authored with Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
