This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 9099c0212b17 [SPARK-51496][SQL] CaseInsensitiveStringMap comparison
should ignore case
9099c0212b17 is described below
commit 9099c0212b1793e46c68dabef053587b8e26ac16
Author: huaxingao <[email protected]>
AuthorDate: Fri Mar 21 07:03:45 2025 +0800
[SPARK-51496][SQL] CaseInsensitiveStringMap comparison should ignore case
### What changes were proposed in this pull request?
Since both `commandOptions` and `dsOptions` are `CaseInsensitiveStringMap`
objects, I think we should convert the keys and values to lowercase before
comparing them
### Why are the changes needed?
In iceberg/spark4.0 integration, I got a few assertion errors:
```
assertion failed
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:264)
at
org.apache.spark.sql.execution.datasources.v2.V2Writes$.org$apache$spark$sql$execution$datasources$v2$V2Writes$$mergeOptions(V2Writes.scala:128)
```
The assertion error occurs when comparing commandOptions and dsOptions; the
cases of the keys don't match.
```
assert(commandOptions == dsOptions)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
I have verified that the iceberg tests can pass with this fix.
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #50275 from huaxingao/mergeOptions.
Authored-by: huaxingao <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit bfe63a3d3e537fdd5b3f0df9da432e7f404a62d7)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala
index 9d059416766a..6f30ade1c7d3 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala
@@ -45,7 +45,7 @@ object V2Writes extends Rule[LogicalPlan] with
PredicateHelper {
override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
case a @ AppendData(r: DataSourceV2Relation, query, options, _, None, _) =>
- val writeOptions = mergeOptions(options, r.options.asScala.toMap)
+ val writeOptions = mergeOptions(options,
r.options.asCaseSensitiveMap.asScala.toMap)
val writeBuilder = newWriteBuilder(r.table, writeOptions, query.schema)
val write = writeBuilder.build()
val newQuery = DistributionAndOrderingUtils.prepareQuery(write, query,
r.funCatalog)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
