GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/2716
[SPARK-3594] [PySpark] [SQL] take more rows to infer schema or sampling
This patch will try to infer schema for RDD which has empty value (None,
[], {}) in the first row. It will try first 100 rows and merge the types into
schema. If there is still NullType in schema, then it will show an warning,
tell user to try with sampling.
If sampling is presented, it will infer schema from all the rows after
sampling.
Also, add samplingRatio for jsonFile() and jsonRDD()
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark infer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2716.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2716
----
commit 3603e00852f94568523bc641c756cde881616017
Author: Davies Liu <[email protected]>
Date: 2014-10-08T19:29:00Z
take more rows to infer schema, or infer the schema by sampling the RDD
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
