Github user takaomag commented on the pull request:
https://github.com/apache/spark/pull/5408#issuecomment-90817956
Note:
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala should be
modified since it adds all pyfile paths to the environment variable
"PYTHONPATH" before creating python (driver ?) process.
But, I could not modify it since I am not familiar with scala/java and
Spark internal details.
Basically, I think the followings are required.
* Add any python package file paths except for .whl (that is, all .egg,
.zip and .jar) to PYTHONPATH
* Install .whl files by invoking subprocess with the command
```
pythonExec -m pip install --quiet --upgrade --no-deps --no-index --target
TEMPORARY_SITE_PACKAGES_DIRECTORY SPACE_SEPARATED_WHL_FILE_PATHS...
```
* Then, add the _TEMPORARY_SITE_PACKAGES_DIRECTORY_ to PYTHONPATH.
Is there someone to modify it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
