GitHub user stephend-realitymine opened a pull request:
https://github.com/apache/spark/pull/9249
[SPARK-10947] [SQL] With schema inference from JSON into a Dataframe, add
option to infer all primitive object types as strings
Currently, when a schema is inferred from a JSON file using
sqlContext.read.json, the primitive object types are inferred as string, long,
boolean, etc.
However, if the inferred type is too specific (JSON obviously does not
enforce types itself), this can cause issues with merging dataframe schemas.
This pull request adds the option "primitivesAsString" to the JSON
DataFrameReader which when true (defaults to false if not set) will infer all
primitives as strings.
Below is an example usage of this new functionality.
```
val jsonDf = sqlContext.read.option("primitivesAsString",
"true").json(sampleJsonFile)
scala> jsonDf.printSchema()
root
|-- bigInteger: string (nullable = true)
|-- boolean: string (nullable = true)
|-- double: string (nullable = true)
|-- integer: string (nullable = true)
|-- long: string (nullable = true)
|-- null: string (nullable = true)
|-- string: string (nullable = true)
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/RealityMineLtd/spark stephend-primitives
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9249.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9249
----
commit a718b8658cda4d849c9057dc9cd601bd6d31503e
Author: Stephen De Gennaro <steph...@realitymine.com>
Date: 2015-10-23T09:03:22Z
SPARK-10947 Added option to json schema primitivesAsString when true will
infer primative types as strings
commit 9e6411425546400ae94fd67dc143d2b68c6243aa
Author: Stephen De Gennaro <steph...@realitymine.com>
Date: 2015-10-23T09:32:38Z
SPARK-10947 adding missed bracket
commit 3989c6aa33acb0af3e151fcd26737cb295de550e
Author: Stephen De Gennaro <steph...@realitymine.com>
Date: 2015-10-23T09:59:06Z
SPARK-10947 removing duplicate line
commit 18d28619264dbaf10f1e27576f5c4275cbc4ef72
Author: Stephen De Gennaro <steph...@realitymine.com>
Date: 2015-10-23T10:01:30Z
SPARK-10947 removing extra bracket
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
