Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21426
@vanzin, for
https://github.com/apache/spark/pull/21426#discussion_r191010819, mind if we
proceed in a separate ticket? From my look, it needs some changes to verify
this to address this comment. I think we can't simply raise an exception since
we can't recognise if that file is downloaded or not in `deploy.PythonRunner`'s
perspective.
The most appropriate place seems to be in `SparkSubmit` and
`DependencyUtils.downloadFile`. seems we should inject some codes in
`DependencyUtils.downloadFile` since that's where we know the original path and
where we download the file into local when needed, and I would like to avoid
add such changes here. It probably needs another review iteration and the
current change doesn't actually target or change the previous behaviour, really.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
