[
https://issues.apache.org/jira/browse/SPARK-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198591#comment-14198591
]
Yin Huai commented on SPARK-4100:
---------------------------------
[~kul] Yes, it is expected behavior. Since fields and value types appearing in
JSON records can be different, we decided to start from the most verbose way to
infer the schema (by passing the whole dataset). We will provide alternative
ways in future.
> JSON RDD schema inference causes whole RDD to be realized
> ---------------------------------------------------------
>
> Key: SPARK-4100
> URL: https://issues.apache.org/jira/browse/SPARK-4100
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.0
> Reporter: Kuldeep
>
> Registering a JSON RDD as temp table causes each RDD row to be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
