Semet created TOREE-337:
---------------------------
Summary: %AddPythonDeps
Key: TOREE-337
URL: https://issues.apache.org/jira/browse/TOREE-337
Project: TOREE
Issue Type: Improvement
Reporter: Semet
I would like to volunteer to add two "magic" to Toree, related to Python
dependency managements:
{code}
%AddPythonDeps
Usage: %AddPythonDeps pypi_package_name [version]
{code}
Download and install Python dependency from pypi.python.org and install it on
the Spark cluster using pip. Transitive dependencies will be automatically
retrieved.
---
{code}
%AddPythonDist
Usage: %AddPythonDist<url to dist>
{code}
Download the distribution package (source distribution package or binary
distribution package or wheel) and install it on the master and workers using
pip.
---
Example:
One would be able to specify something like
{code}
%AddPythonDeps numpy 1.1.1
%AddPythonDeps requests 0.1.1
{code}
I am working to bring pip and virtualenv support to Spark since we need to work
on clean virtualenv. Here is a [proposal on this
subject|http://www.great-a-blog.co/wheel-deployment-for-pyspark/] and the [pull
request|https://github.com/apache/spark/pull/14180].
Let me know if it would make sense within Apache Toree to have such feature (I
think it's cool to let each job describes its python dependencies and have
Spark automatically handle them properly!)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
