[
https://issues.apache.org/jira/browse/SPARK-40388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602307#comment-17602307
]
zzzzming95 commented on SPARK-40388:
------------------------------------
Hi [~praetp] , can you provide a way to reproduce this bug?
> SQL configuration spark.sql.mapKeyDedupPolicy not always applied
> ----------------------------------------------------------------
>
> Key: SPARK-40388
> URL: https://issues.apache.org/jira/browse/SPARK-40388
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.2
> Reporter: Paul Praet
> Priority: Major
>
> I have set spark.sql.mapKeyDedupPolicy to LAST_WIN.
> However, I had still one failure where I got
> {quote}Caused by: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 7 in stage 1201.0 failed 4 times, most recent failure: Lost
> task 7.3 in stage 1201.0 (TID 1011313)
> (ip-10-1-34-47.eu-west-1.compute.internal executor 228):
> java.lang.RuntimeException: Duplicate map key domain was found, please check
> the input data. If you want to remove the duplicated keys, you can set
> spark.sql.mapKeyDedupPolicy to LAST_WIN so that the key inserted at last
> takes precedence.
> {quote}
> We are confident we set the right configuration in SparkConf (we can find it
> on the Spark UI -> Environment).
> It is our impression this configuration is not propagated reliably to the
> executors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
