[
https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488445#comment-17488445
]
kk commented on SPARK-38115:
----------------------------
Thanks [~hyukjin.kwon] for responding.
Basically I am trying to write data to s3 from spark dataframe. And this will
use FileOutputCommitter by spark.
[https://stackoverflow.com/questions/46665299/spark-avoid-creating-temporary-directory-in-s3]
Now my requirement is to either change the '{*}_temporary{*}' path to write to
different s3 bucket and copy to original s3 by setting any spark conf or
parameter part of write step.
or
stop creating *_temporary* when writing to s3.
As we have version enabled bucket the _temporary is being stored in the version
even though it is not physically present.
Below is the write step:
df.coalesce(1).write.format('parquet').mode('overwrite').save('{*}s3a{*}://outpath')
> No spark conf to control the path of _temporary when writing to target
> filesystem
> ---------------------------------------------------------------------------------
>
> Key: SPARK-38115
> URL: https://issues.apache.org/jira/browse/SPARK-38115
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.8, 3.2.1
> Reporter: kk
> Priority: Minor
> Labels: spark, spark-conf, spark-sql, spark-submit
>
> No default spark conf or param to control the '_temporary' path when writing
> to filesystem.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
