Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/2824#issuecomment-59688035
Hi @JoshRosen , I just set `transferToEnabled` to false as default value,
unless users explicitly set it to true, `transferTo` will not be enabled.
Currently, only `ExternalSorter` use this API as file to file copying and
this is controlled by configuration `spark.file.transferTo`, other uses of
`copyStream` in Spark code are all not file to file copying, so this parameter
will not take effect.
If future uses of `copyStream`, user have to get `transferToEnabled` from
configuration, I add some usage notes here. Still user can bypass
`spark.file.transferTo` and directly set this parameter to true, but they have
to be responsible for the correctness of usage.
The reason I didn't take `SparkConf` as a parameter to control the behavior
is that it should modify lots of the current codes to get `SparkConf` in which
it calls `copyStream`.
So what is your opinion? Thanks a lot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
