Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5478#issuecomment-93718724
Hey guys, after discussion with @Sephiroth-Lin offline, we have some
questions:
* If we try to ship the zip file to executer backend with --py-files:
* User must use --py-file to point to the xxx.zip file we provide;
* We must change format of directories(use xxx.zip to replace python dir)
in our release.
* Even the zip file is not very big, shipping it everytime still cost
time.
* Inspired by the former solution, I thought we might add some config like
`spark.exectuor.extraClassPath` in PySpark side, which points to the files
needed by user's application so that user could use them without shipping.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
