[ https://issues.apache.org/jira/browse/SPARK-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198591#comment-14198591 ]
Yin Huai commented on SPARK-4100: --------------------------------- [~kul] Yes, it is expected behavior. Since fields and value types appearing in JSON records can be different, we decided to start from the most verbose way to infer the schema (by passing the whole dataset). We will provide alternative ways in future. > JSON RDD schema inference causes whole RDD to be realized > --------------------------------------------------------- > > Key: SPARK-4100 > URL: https://issues.apache.org/jira/browse/SPARK-4100 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.1.0 > Reporter: Kuldeep > > Registering a JSON RDD as temp table causes each RDD row to be read. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org