[
https://issues.apache.org/jira/browse/SPARK-23772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418280#comment-16418280
]
Apache Spark commented on SPARK-23772:
--------------------------------------
User 'maropu' has created a pull request for this issue:
https://github.com/apache/spark/pull/20929
> Provide an option to ignore column of all null values or empty map/array
> during JSON schema inference
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-23772
> URL: https://issues.apache.org/jira/browse/SPARK-23772
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Xiangrui Meng
> Priority: Major
>
> It is common that we convert data from JSON source to structured format
> periodically. In the initial batch of JSON data, if a field's values are
> always null, Spark infers this field as StringType. However, in the second
> batch, one non-null value appears in this field and its type turns out to be
> not StringType. Then merge schema failed because schema inconsistency.
> This also applies to empty arrays and empty objects. My proposal is providing
> an option in Spark JSON source to omit those fields until we see a non-null
> value.
> This is similar to SPARK-12436 but the proposed solution is different.
> cc: [~rxin] [~smilegator]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
