Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/20140#discussion_r166264802
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -90,7 +90,10 @@ private[csv] object CSVInferSchema {
// DecimalTypes have different precisions and scales, so we try
to find the common type.
findTightestCommonType(typeSoFar, tryParseDecimal(field,
options)).getOrElse(StringType)
case DoubleType => tryParseDouble(field, options)
- case TimestampType => tryParseTimestamp(field, options)
+ case DateType => tryParseDate(field, options)
+ case TimestampType =>
+ findTightestCommonType(typeSoFar, tryParseTimestamp(field,
options)).getOrElse(
--- End diff --
Sorry, your question is not really clear for me.
We have to try parse object as DateType first, because date always can be
parsed as date and as timestamp (begin of day).
Current implementation of spark ignores dates and it is always parsing them
as timestamps
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
