Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/14180
I was going through the old PySpark JIRAs and there is one about an
unexpected pandas failure, which could be semi-related (e.g. good virtual env
support with a reasonable requirements file could
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
Rebased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
I have written a blog post about this pull request to explain what we can
do with it: http://www.great-a-blog.co/wheel-deployment-for-pyspark/
---
If your project is set up for it, you can reply
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
Hello. Can someone help to review this PR? I find the current way Spark
handle Python programs really problematic, with this proposal (based on top of
#13599), jobs deployment becomes so much
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
Status for test 'standalone install, 'client' deployment":
- virtualenv create and pip install Pypi repository: ok (1 min 30 exec)
- wheelhouse (Pypi repositoy): ko, because 'cffi' refuses
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
This new version is meant to be rebased after #13599 is merged.
Here is my current state:
- only Standalone and YARN supported. Mesos not supported
- only tested with
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
It makes sense. If you manage to get this merged I can rebase with only my
diff.
Too bad we cannot stack pull requests on github :(
---
If your project is set up for it, you can reply to
Github user zjffdu commented on the issue:
https://github.com/apache/spark/pull/14180
@Stibbons After second review your PR, I have one concern that may be
support wheelhouse and virtualenv in one PR is too big to review, it might be
better to do it in 2 PRs. I will try to get some
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
We are implementing Mesos here (may take a while). While not so many people
use it, on the paper it looks great ;)
Please mail me at gaetan[a t]xeberon.net if is easier for you (it is for
Github user zjffdu commented on the issue:
https://github.com/apache/spark/pull/14180
I can help if you have any question regarding spark on yarn. For mesos,
since not so many people use it, we may put it another ticket.
---
If your project is set up for it, you can reply to this
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
Actually I was waiting for #14567 to be reviewed and merged :(
I might have some questions on how Spark deploys Python script on YARN or
Mesos if you know how it works
---
If your
Github user zjffdu commented on the issue:
https://github.com/apache/spark/pull/14180
@Stibbons Do you have time to continue this work ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user Stibbons commented on the issue:
https://github.com/apache/spark/pull/14180
Rebased, without import reorg and editorconfig files. Still not fully
validated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
13 matches
Mail list logo
