Hi, In pyspark when if I read a json file using sqlcontext I find that the date field is not infered as date instead it is converted to string. And when I try to convert it to date using df.withColumn(df.DateCol.cast("timestamp")) it does not parse it successfuly and adds a null instead there. Should I use UDF to convert the date ? Is this expected behaviour (not throwing an error after failure to cast all fields)?
-- Regards, Suraj