[ https://issues.apache.org/jira/browse/SPARK-40388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17602307#comment-17602307 ]
zzzzming95 commented on SPARK-40388: ------------------------------------ Hi [~praetp] , can you provide a way to reproduce this bug? > SQL configuration spark.sql.mapKeyDedupPolicy not always applied > ---------------------------------------------------------------- > > Key: SPARK-40388 > URL: https://issues.apache.org/jira/browse/SPARK-40388 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.1.2 > Reporter: Paul Praet > Priority: Major > > I have set spark.sql.mapKeyDedupPolicy to LAST_WIN. > However, I had still one failure where I got > {quote}Caused by: org.apache.spark.SparkException: Job aborted due to stage > failure: Task 7 in stage 1201.0 failed 4 times, most recent failure: Lost > task 7.3 in stage 1201.0 (TID 1011313) > (ip-10-1-34-47.eu-west-1.compute.internal executor 228): > java.lang.RuntimeException: Duplicate map key domain was found, please check > the input data. If you want to remove the duplicated keys, you can set > spark.sql.mapKeyDedupPolicy to LAST_WIN so that the key inserted at last > takes precedence. > {quote} > We are confident we set the right configuration in SparkConf (we can find it > on the Spark UI -> Environment). > It is our impression this configuration is not propagated reliably to the > executors. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org