Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
Hello. Been a long time, it probably needs a full rework. Maybe we need to
take a step back and have a talk between several person interested in this
feature to see what is the more suitable for the Spark project. I work a lot on
Python packaging nowdays, so I have a pretty good idea on different
distribution solutions we have for python (anaconda, pip/virtualenv, now
Pipfile), and not only barely generating a python package and throwing it in
the wild, I mean ensuring my package work in the targetted environment:
pyexecutable is also a solution eventhough it is more complex, wheelhouse +
some tricks might also do the job for Spark. Ultimately, the goal is to have
something cool and easy to use for PySpark users willing to distribute any kind
of work without having to ask the IT guys to install this numpy version on the
cluster.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
