GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/294
[SPARK-1134] Fix and document passing of arguments to IPython
This is based on @dianacaroll's previous pull request
https://github.com/apache/spark/pull/227, and @joshrosen's comments on
https://github.com/apache/spark/pull/38. Since we do want to allow passing
arguments to IPython, this does the following:
* It documents that IPython can't be used with standalone jobs for now.
(Later versions of IPython will deal with PYTHONSTARTUP properly and enable
this, see https://github.com/ipython/ipython/pull/5226, but no released version
has that fix.)
* If you run `pyspark` with `IPYTHON=1`, it passes your command-line
arguments to it. This way you can do stuff like `IPYTHON=1 bin/pyspark
notebook`.
* The old `IPYTHON_OPTS` remains, but I've removed it from the
documentation. This is in case people read an old tutorial that uses it.
This is not a perfect solution and I'd also be okay with keeping things as
they are today (ignoring `$@` for IPython and using IPYTHON_OPTS), and only
doing the doc change. With this change though, when IPython fixes
https://github.com/ipython/ipython/pull/5226, people will immediately be able
to do `IPYTHON=1 bin/pyspark myscript.py` to run a standalone script and get
all the benefits of running scripts in IPython (presumably better debugging and
such). Without it, there will be no way to run scripts in IPython.
@joshrosen you should probably take the final call on this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark spark-1134
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/294.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #294
----
commit 747bb1347e09d4bd6c86d24d452b569b9071622f
Author: Diana Carroll <[email protected]>
Date: 2014-02-27T22:39:43Z
SPARK-1134 bug with ipython prevents non-interactive use with spark; only
call ipython if no command line arguments were supplied
commit 87de70ba0a5b804059f02b2f0ac4af55743b95b2
Author: Matei Zaharia <[email protected]>
Date: 2014-04-02T02:53:50Z
Pass command-line args to IPython, and update docs to describe this
commit cd2036bff469ca048837bed13ae6f81a9c8d1c5a
Author: Matei Zaharia <[email protected]>
Date: 2014-04-02T02:59:36Z
Doc tweak
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
