GitHub user jasoncl opened a pull request:
https://github.com/apache/spark/pull/8969
[SPARK-10847] [SQL] [PySpark] Pyspark - DataFrame - Optional Metadata with
`None` triggers cryptic failure
The error message is now changed from "Do not support type class
scala.Tuple2." to "Do not support type class org.json4s.JsonAST$JNull$" to be
more informative about what is not supported. Also, StructType metadata now
handles JNull correctly, i.e., {'a': None}. test_metadata_null is added to
tests.py to show the fix works.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jasoncl/spark SPARK-10847
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8969.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8969
----
commit 362778c2f9fd5a30959e6451017a1bdf428cb272
Author: Jason Lee <[email protected]>
Date: 2015-10-02T18:14:53Z
Added support for JNull in Metadata. Error message now displays the actual
unsupported type in the tuple instead of just tuple2
commit 6bb4fcc09c85ba1e039113d5a0172d87ea1d590a
Author: Jason Lee <[email protected]>
Date: 2015-10-02T21:51:05Z
fix scala style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
