[
https://issues.apache.org/jira/browse/SPARK-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447775#comment-15447775
]
Liwei Lin commented on SPARK-17290:
-----------------------------------
Oh [~hyukjin.kwon] you are so devoted :-D
> Spark CSVInferSchema does not always respect nullValue settings
> ---------------------------------------------------------------
>
> Key: SPARK-17290
> URL: https://issues.apache.org/jira/browse/SPARK-17290
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Teng Yutong
>
> When loading a csv-formated data file into a table which has boolean type
> column, if the boolean value is not given and the nullValue has been set,
> CSVInferSchema will fail to parse the data.
> e.g.:
> table schema: create table test(id varchar(10), flag boolean) USING
> com.databricks.spark.csv OPTIONS (path "test.csv", header "false", nullValue
> '')
> csv data example:
> aa,
> bb,true
> cc,false
> After some investigation, I found that CSVInferSchema will not check wether
> the current string match the nullValue or not if the target data type is
> Boolean、Timestamp、Date。
> I am wondering that this logic is coded by purpose or not
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
