[
https://issues.apache.org/jira/browse/SPARK-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557248#comment-15557248
]
holdenk commented on SPARK-13585:
---------------------------------
What is the use case for overwriting the old pyFile? The current scala API
doesn't have an overwrite flag we can just plumb through directly but we could
maybe think about alternatives or if it would be worth adding on the scala side
too.
> addPyFile behavior change between 1.6 and before
> ------------------------------------------------
>
> Key: SPARK-13585
> URL: https://issues.apache.org/jira/browse/SPARK-13585
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.0
> Reporter: Santhosh Gorantla Ramakrishna
> Priority: Minor
>
> addPyFile in earlier versions would remove the .py file if it already
> existed. In 1.6, it throws an exception "__.py exists and does not match
> contents of __.py".
> This might be because the underlying scala code needs an overwrite parameter,
> and this is being defaulted to false when called from python.
> private def copyFile(
> url: String,
> sourceFile: File,
> destFile: File,
> fileOverwrite: Boolean,
> removeSourceFile: Boolean = false): Unit = {
> Would be good if addPyFile takes a parameter to set the overwrite and default
> to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
