[ 
https://issues.apache.org/jira/browse/SPARK-21651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119484#comment-16119484
 ] 

Takeshi Yamamuro commented on SPARK-21651:
------------------------------------------

Specifying a schema by yourself is not enough for your case?
{code}
json data:
{"id":"0001", "relations": {"r1": {"x": "7", "z": "1"}, "r2": {"y": "8"} } }

scala> spark.read.json("xxx").printSchema
root
 |-- id: string (nullable = true)
 |-- value: struct (nullable = true)
 |    |-- a: struct (nullable = true)
 |    |    |-- x: string (nullable = true)
 |    |    |-- z: string (nullable = true)
 |    |-- b: struct (nullable = true)
 |    |    |-- y: string (nullable = true)

scala> spark.read.schema("id STRING, relations MAP<STRING, STRUCT<x: STRING, y: 
STRING, z: STRING>>").json("xxx").printSchema
root
 |-- id: string (nullable = true)
 |-- value: map (nullable = true)
 |    |-- key: string
 |    |-- value: struct (valueContainsNull = true)
 |    |    |-- x: string (nullable = true)
 |    |    |-- y: string (nullable = true)
 |    |    |-- z: string (nullable = true)
{code}

I feel this optimization is a little domain-specific and any other runtime 
supporting this kind of optimization?


> Detect MapType in Json InferSchema
> ----------------------------------
>
>                 Key: SPARK-21651
>                 URL: https://issues.apache.org/jira/browse/SPARK-21651
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.1.0, 2.1.1, 2.2.0
>            Reporter: Jochen Niebuhr
>            Priority: Minor
>
> When loading Json Files which include a map with very variable keys, the 
> current schema infer logic might create a very large schema. This will lead 
> to long load times and possibly out of memory errors. 
> I've already submitted a pull request to the mongo spark driver which had the 
> same problem. Should I port this logic over to the json schema infer class?
> The MongoDB Spark pull request mentioned is: 
> https://github.com/mongodb/mongo-spark/pull/24



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to