steveloughran commented on a change in pull request #30141:
URL: https://github.com/apache/spark/pull/30141#discussion_r511955017
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
##########
@@ -164,6 +164,10 @@ object FileFormatWriter extends Logging {
SQLExecution.checkSQLExecutionId(sparkSession)
+ // propagate the decription UUID into the jobs, so that committers
+ // get an ID guaranteed to be unique.
+ job.getConfiguration.set("spark.sql.sources.writeJobUUID",
description.uuid)
Review comment:
It picked it up if set, so yes, it was being used. We've hit a problem
where if >1 job kicks off in the same second for that user, the generated app
ID is the same for both, so the staging committers end up using the same dir in
HDFS. The committers already use the writeJobUUID property if set: restoring
the original config option will mean that the shipping artifacts will work
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
