[
https://issues.apache.org/jira/browse/SPARK-4185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193620#comment-14193620
]
Yin Huai commented on SPARK-4185:
---------------------------------
[~willbenton] yeah, agreed. We need to have a better story on handling this
issue. Let me think about it more.
How about we resolve this match error first and then have another JIRA for the
improvement? So, this match error will not prevent users from querying other
fields.
> JSON schema inference failed when dealing with type conflicts in arrays
> -----------------------------------------------------------------------
>
> Key: SPARK-4185
> URL: https://issues.apache.org/jira/browse/SPARK-4185
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.1.0
> Reporter: Yin Huai
> Assignee: Yin Huai
>
> {code}
> val sqlContext = new org.apache.spark.sql.SQLContext(sparkContext)
> val diverging = sparkContext.parallelize(List("""{"branches": ["foo"]}""",
> """{"branches": [{"foo":42}]}"""))
> sqlContext.jsonRDD(diverging) // throws a MatchError
> {code}
> The case is from http://chapeau.freevariable.com/2014/10/fedmsg-and-spark.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
