[
https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492810#comment-17492810
]
Steve Loughran commented on SPARK-38115:
----------------------------------------
* stop using the classic FileOutputCommitter for your work, unless you like
waiting a long time for your jobs to complete. along with a risk of corrupt
data in the presence of worker failures.
* the choice of where temporary paths go is a function of the committer, not
the spark codebase. the s3a staging committer uses the local fs. for example
* the magic committer does work under _temporary, but it doesn't write the
final data there. it's "magic", after all. l
> No spark conf to control the path of _temporary when writing to target
> filesystem
> ---------------------------------------------------------------------------------
>
> Key: SPARK-38115
> URL: https://issues.apache.org/jira/browse/SPARK-38115
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.8, 3.2.1
> Reporter: kk
> Priority: Minor
> Labels: spark, spark-conf, spark-sql, spark-submit
>
> No default spark conf or param to control the '_temporary' path when writing
> to filesystem.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
