Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58429854
We don't support IPython 1.0 anymore, so it seems reasonable to make Python
2.7 the default when using IPython (since IPython 2.0 requires at least Python
2.7). It
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58444611
Updated based on offline discussion. The key changes:
- Introduce PYSPARK_DRIVER_PYTHON to allow the driver's Python executable
to be different than the
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58445007
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21490/consoleFull)
for PR 2651 at commit
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58445558
LGTM, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58449456
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21490/consoleFull)
for PR 2651 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58449460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58273093
@davies Good points; we should definitely discuss this before 1.2. I guess
that the script also loads `spark-env.sh`, but this is less of an issue since
we moved most
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-58276170
The current patch looks good to me.
One question: should we figure out the version of IPython, then use the
correct of python in worker? In the most common cases,
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-57927210
Before 1.2 release, maybe it's time to rethink how to run pyspark shell or
scripts, using bin/pyspark or spark-submit is not so friendly for user, maybe
we could simplify
GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/2651
[SPARK-3772] Allow `ipython` to be used by Pyspark workers; IPython fixes:
This pull request addresses a few issues related to PySpark's IPython
support:
- Fix the remaining uses of the
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-57891560
/cc @davies @cocotomo @robbles for reviews / feedback.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-57891571
/cc @davies @cocoatomo @robbles for reviews / feedback.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-57891695
[QA tests have
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21284/consoleFull)
for PR 2651 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-57893128
[QA tests have
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21284/consoleFull)
for PR 2651 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2651#issuecomment-57893129
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
15 matches
Mail list logo
