Re: csv date/timestamp type inference in spark 2.0.1

Hi Koert, I am curious about your case. I guess the purpose of timestampFormat and dateFormat is to infer timestamps/dates when parsing/inferring but not to exclude the type inference/parsing. Actually, it does try to infer/parse in 2.0.0 as well (but it fails) so actually I guess there

Re: csv date/timestamp type inference in spark 2.0.1

i tried setting both dateFormat and timestampFormat to impossible values (e.g. "~|.G~z~a|wW") and it still detected my data to be TimestampType On Wed, Oct 26, 2016 at 1:15 PM, Koert Kuipers wrote: > we had the inference of dates/timestamps when reading csv files disabled >

csv date/timestamp type inference in spark 2.0.1

we had the inference of dates/timestamps when reading csv files disabled in spark 2.0.0 by always setting dateFormat to something impossible (e.g. dateFormat "~|.G~z~a|wW") i noticed in spark 2.0.1 that setting this impossible dateFormat does not stop spark from inferring it is a date or