GitHub user felixcheung opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/178
[ZEPPELIN-185] ZeppelinContext methods like z.show are not working with
DataFrame in pyspark
(opening a new PR to have a start history)
It turns out the error is caused by Py4J trying to auto convert the
DataFrame, which fails since it can only do that for simple types.
Instead of getting conversion to work, the better approach is to pass along
the inner java object instead. To do that we intercept the call on the python
side with a wrapper object instead of letting Py4J handle it.
As per comment, adding container/dictionary methods to allow for string
passing using ZeppelinContext
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/felixcheung/incubator-zeppelin zpyspark
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/178.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #178
----
commit df6588a3b747b844dcaccc2c5a1e1f204cc8d055
Author: Felix Cheung <[email protected]>
Date: 2015-08-03T06:48:26Z
[ZEPPELIN-185] ZeppelinContext methods like z.show are not working with
DataFrame in pyspark
It turns out the error is caused by Py4J trying to auto convert the
DataFrame, which fails since it can only do that for simple types.
Instead of getting conversion to work, the better approach is to pass along
the inner java object instead. To do that we intercept the call on the python
side with a wrapper object instead of letting Py4J handle it.
As per comment, adding container/dictionary methods to allow for string
passing using ZeppelinContext
commit ddc5bb2cd099ee49826210bcd78134a64a4801b9
Author: Felix Cheung <[email protected]>
Date: 2015-08-03T08:20:45Z
small fixes to python script
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
