[
https://issues.apache.org/jira/browse/SPARK-25669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxim Gekk updated SPARK-25669:
-------------------------------
Summary: Check CSV header only when it exists (was: Check header only when
it exists)
> Check CSV header only when it exists
> ------------------------------------
>
> Key: SPARK-25669
> URL: https://issues.apache.org/jira/browse/SPARK-25669
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Maxim Gekk
> Priority: Minor
>
> Currently, Spark checks the header in CSV files to fields names in provided
> or inferred schema. The check is bypassed if the header doesn't exists and
> CSV content is read from files. In the case, when input CSV comes as dataset
> of strings, Spark always compares the first row to the user specified or
> inferred schema. For example, parsing the following dataset:
> {code:scala}
> val input = Seq("1,2").toDS()
> spark.read.option("enforceSchema", false).csv(input)
> {code}
> throws the exception:
> {code:java}
> java.lang.IllegalArgumentException: CSV header does not conform to the schema.
> Header: 1, 2
> Schema: _c0, _c1
> Expected: _c0 but found: 1
> {code}
> Need to prevent comparison of the first row (if it is not a header) to
> specific or inferred schema.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
