zsxwing commented on a change in pull request #30073:
URL: https://github.com/apache/spark/pull/30073#discussion_r506790079
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2052,6 +2052,20 @@ object SQLConf {
.stringConf
.createWithDefault("")
+ val FASTFAIL_ON_FILEFORMAT_OUTPUT =
+ buildConf("spark.sql.execution.fastFailOnFileFormatOutput")
+ .internal()
+ .doc("Whether to fast fail task execution when writing output to
FileFormat datasource. " +
+ "If this is enabled, in `FileFormatWriter` we will catch
`FileAlreadyExistsException` " +
+ "and fast fail output task without further task retry. Only enabling
this if you know " +
+ "the `FileAlreadyExistsException` of the output task is unrecoverable,
i.e., further " +
+ "task attempts won't be able to success. If the
`FileAlreadyExistsException` might be " +
+ "recoverable, you should keep this as disabled and let Spark to retry
output tasks. " +
+ "This is disabled by default.")
+ .version("3.1.0")
Review comment:
I think it's better to merge to branch-3.0 as well as currently user
don't have a flag for this in 3.0.x.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
