[ https://issues.apache.org/jira/browse/SPARK-17290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447755#comment-15447755 ]
Hyukjin Kwon commented on SPARK-17290: -------------------------------------- This should be a duplicate of SPARK-16462, SPARK-16460, SPARK-15144 and SPARK-16903 > Spark CSVInferSchema does not always respect nullValue settings > --------------------------------------------------------------- > > Key: SPARK-17290 > URL: https://issues.apache.org/jira/browse/SPARK-17290 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0 > Reporter: Teng Yutong > > When loading a csv-formated data file into a table which has boolean type > column, if the boolean value is not given and the nullValue has been set, > CSVInferSchema will fail to parse the data. > e.g.: > table schema: create table test(id varchar(10), flag boolean) USING > com.databricks.spark.csv OPTIONS (path "test.csv", header "false", nullValue > '') > csv data example: > aa, > bb,true > cc,false > After some investigation, I found that CSVInferSchema will not check wether > the current string match the nullValue or not if the target data type is > Boolean、Timestamp、Date。 > I am wondering that this logic is coded by purpose or not -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org