Github user dwmclary commented on the pull request:
https://github.com/apache/spark/pull/3213#issuecomment-63152791
I pushed up a Jackson version, which cuts down the size quite a bit. At
present we're not handling complex types, correct?
What I'm a bit stuck on is getting the results of the scala method back
into pyspark. If I call:
newJsonRDD = someSchemaRDD._jschema_rdd.baseSchemaRDD().toJSON()
I'm not sure how do deserialize it back on the python side.
My intuition would be that I'd just do
RDD(newJsonRDD, sc), but that doesn't seem to give me back a valid RDD.
-D
On Fri, Nov 14, 2014 at 4:41 PM, Nathan Howell <[email protected]>
wrote:
> Another approach is to use a JsonGenerator instead of an ObjectMapper.
> This is the implementation I've been using for a while:
> https://gist.github.com/NathanHowell/0a15f0bd23cd940becb3
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3213#issuecomment-63151280>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
