Paul Praet created SPARK-40388:
----------------------------------
Summary: SQL configuration spark.sql.mapKeyDedupPolicy not always
applied
Key: SPARK-40388
URL: https://issues.apache.org/jira/browse/SPARK-40388
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.1.2
Reporter: Paul Praet
I have set spark.sql.mapKeyDedupPolicy to LAST_WIN.
However, I had still one failure where I got
{quote}Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task 7 in stage 1201.0 failed 4 times, most recent failure: Lost task
7.3 in stage 1201.0 (TID 1011313) (ip-10-1-34-47.eu-west-1.compute.internal
executor 228): java.lang.RuntimeException: Duplicate map key domain was found,
please check the input data. If you want to remove the duplicated keys, you can
set spark.sql.mapKeyDedupPolicy to LAST_WIN so that the key inserted at last
takes precedence.
{quote}
We are confident we set the right configuration in SparkConf (we can find it on
the Spark UI -> Environment).
It is our impression this configuration is not propagated reliably to the
executors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
