GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/17298
[SPARK-19094][WIP][PySpark] Plumb through logging for IJupyter notebooks
## What changes were proposed in this pull request?
Plumb through logging from the JVM into Jupyter notebooks to make debugging
easier. This is a WIP PR
## How was this patch tested?
Modified standalone_tests to allow testing with Python launch first in
addition to the JVM launch first default.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark
SPARK-19094-plumb-through-logging-for-ijupyter-notebooks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17298.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17298
----
commit aebf7a01196b7aa063d261f8816014ac17d6ea95
Author: Holden Karau <[email protected]>
Date: 2017-01-06T01:18:19Z
Plumb through the jvm output, we could test this by overloading sys.stdout
and launching an instance of spark and verifying we got the log info (TODO)
commit 3e8d7eb2e33dbddbf7bcae2e97baaad7a6bc1908
Author: Holden Karau <[email protected]>
Date: 2017-01-06T06:59:39Z
Python3 support, isn't necessarily a file type. Instead just look for
overriden stdout
commit ca308b15c1b0b0571c3dee61d049739d84216c32
Author: Holden Karau <[email protected]>
Date: 2017-01-12T19:01:20Z
Only redirect if in ZMQInteractiveShell (e.g. notebook) or StringIO (e.g.
testing) and leave things like doctest mode alone [otherwise we'd have to
rewrite all of the doctests]
commit 011863189c301d2e7673d906e238f3e99d8d28bb
Author: Holden Karau <[email protected]>
Date: 2017-01-12T19:02:24Z
We can't test this in the normal test classes since the JVM is already
started, but we can verify it if we are launched without PYSAPRK_GATEWAY_PORT
set
commit dc03ee6ba544ead179d6de9ed72717bfbe35bfd2
Author: Holden Karau <[email protected]>
Date: 2017-02-04T16:08:59Z
Merge in master
commit 733d5b955f28c77925c29ce5f2e39f2ce6359fbc
Author: Holden Karau <[email protected]>
Date: 2017-02-04T16:13:30Z
Update the sanity check to standalone tests
commit 4aea95541bad8df89cc5b6dd13552ba0c4a6dca8
Author: Holden Karau <[email protected]>
Date: 2017-02-04T16:37:28Z
Quick first pass at setting up standalone tests as well
commit 25e9867f5ffa32145d23b44fed162ac61cb9c99b
Author: Holden Karau <[email protected]>
Date: 2017-03-14T23:10:10Z
Merge branch 'master' into
SPARK-19094-plumb-through-logging-for-ijupyter-notebooks
commit 15d999bb901aa0a0eef73ff50f2ba3d24c4d3f72
Author: Holden Karau <[email protected]>
Date: 2017-03-14T23:42:20Z
pep8 fixes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
