Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145597432
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -42,6 +43,13 @@ object ArrowUtils {
case StringType => ArrowType.Utf8.INSTANCE
case BinaryType => ArrowType.Binary.INSTANCE
case DecimalType.Fixed(precision, scale) => new
ArrowType.Decimal(precision, scale)
+ case DateType => new ArrowType.Date(DateUnit.DAY)
+ case TimestampType =>
+ timeZoneId match {
+ case Some(id) => new ArrowType.Timestamp(TimeUnit.MICROSECOND, id)
--- End diff --
Here we are mapping Spark SQL timestamp type to Arrow timestamp with
timezone type. Does Arrow have tz-naive timestamp type?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
