This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 972b507214fe [SPARK-60540][SQL] Floor Arrow nanosecond timestamps to 
the column precision on read
972b507214fe is described below

commit 972b507214fe6949058d7c08876e2fa0b9da5f84
Author: Stevo Mitric <[email protected]>
AuthorDate: Mon Oct 26 11:17:33 2026 +0000

    [SPARK-60540][SQL] Floor Arrow nanosecond timestamps to the column 
precision on read
    
    ### What changes were proposed in this pull request?
    
    `ArrowColumnVector`'s `TimestampNTZNanosAccessor` / 
`TimestampLTZNanosAccessor` now floor the decoded `timestamp[ns]` value to the 
vector's declared precision (recovered from the 
`SPARK::timestampNanos::precision` field metadata by 
`ArrowUtils.fromArrowField`). Below p = 9 they call the shared 
`DateTimeUtils.epochNanosToTimestampNanos(nanos, precision)` helper, the same 
one the Parquet and Avro readers use; p = 9 keeps the existing decode 
unchanged. Nested array elements, map keys / v [...]
    
    ### Why are the changes needed?
    
    Spark keeps nanosecond timestamp values floored to the column precision (p 
= 7 is a 100 ns grid, p = 8 a 10 ns grid); casts, Parquet, Avro and the text 
parsers all floor on the way in. The Arrow reader did not, so an Arrow batch 
from a foreign producer with more sub-microsecond digits than the column allows 
kept them. Values that are equal at the declared precision then compared, 
hashed and grouped as different, and failed equality with a literal of the same 
type. Reachable today, for [...]
    
    ```python
    pdf = pd.DataFrame({"ts": pd.to_datetime(["2020-01-02 03:04:05.123456789", 
"2020-01-02 03:04:05.123456712"])})
    df = spark.createDataFrame(pdf, "ts timestamp_ntz(7)")
    df.distinct().count()  # 2 before, 1 after
    df.where("ts = CAST('2020-01-02 03:04:05.1234567' AS 
TIMESTAMP_NTZ(7))").count()  # 0 before, 2 after
    ```
    
    The same happens for `createDataFrame` from a `pyarrow.Table`, over Spark 
Connect (`LocalRelation` with Arrow data), and for Arrow-optimized Python UDTF 
results.
    
    This covers vectors whose Arrow field carries the precision tag. The 
PySpark worker currently drops the tag on the top-level fields of UDF result 
batches, so `mapInPandas`, `mapInArrow`, `applyInPandas`, `applyInArrow` and 
Python data sources still read such a column as p = 9, and a scalar 
`pandas_udf` or Arrow-optimized UDF with a p < 9 return type fails with 
`ARROW_TYPE_MISMATCH`. That is a separate PySpark fix, left for a follow-up.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, for the unreleased nanosecond timestamp types (behind 
`spark.sql.timestampNanosTypes.enabled`): Arrow input to a `TIMESTAMP_NTZ(p)` / 
`TIMESTAMP_LTZ(p)` column with p < 9 is now floored to p, matching every other 
ingestion path. p = 9 is unchanged.
    
    ### How was this patch tested?
    
    - `ArrowColumnVectorSuite`: new tests reading off-grid int64 epoch-nanos 
(including pre-epoch values and `Long.MinValue`/`MaxValue`) at p = 7/8/9 for 
NTZ and LTZ, from a nested struct field, and from vectors whose class and time 
zone disagree; `ArrowConvertersSuite`: batches written at p = 9 and read with a 
declared p = 7/8 schema (the `createDataFrame` path); `ArrowWriterSuite`: the p 
= 7 round-trip now expects the floored value, plus array elements and map keys 
/ values.
    - PySpark `test_types.py`: `createDataFrame` from pandas and pyarrow with 
NTZ(7), LTZ(8) and pre-epoch values, run in classic and via the Connect parity 
suite.
    - Existing `ArrowUtilsSuite`, `DateTimeUtilsSuite`, 
`org.apache.spark.sql.execution.arrow.*`, `org.apache.spark.sql.vectorized.*` 
and `ColumnarBatchSuite` pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59780 from stevomitric/stevomitric/nanos-arrow-floor.
    
    Authored-by: Stevo Mitric <[email protected]>
    Signed-off-by: Uros Bojanic <[email protected]>
---
 python/pyspark/sql/tests/test_types.py             | 42 ++++++++++
 python/pyspark/sql/types.py                        |  3 +-
 .../spark/sql/vectorized/ArrowColumnVector.java    | 30 +++++--
 .../sql/execution/arrow/ArrowConvertersSuite.scala | 35 ++++++++
 .../sql/execution/arrow/ArrowWriterSuite.scala     | 47 ++++++++---
 .../sql/vectorized/ArrowColumnVectorSuite.scala    | 95 +++++++++++++++++++++-
 6 files changed, 232 insertions(+), 20 deletions(-)

diff --git a/python/pyspark/sql/tests/test_types.py 
b/python/pyspark/sql/tests/test_types.py
index 0f2e64f88b27..797872efb76e 100644
--- a/python/pyspark/sql/tests/test_types.py
+++ b/python/pyspark/sql/tests/test_types.py
@@ -2900,6 +2900,48 @@ class TypesTestsMixin:
                 self.assertTrue(row.sa.startswith("2020-01-02 
03:04:05.1234567"), row.sa)
                 self.assertTrue(row.mv.startswith("2020-01-02 
03:04:05.1234567"), row.mv)
 
+    def test_timestamp_nanos_type_arrow_conversion_floors_to_precision(self):
+        # Arrow timestamp[ns] carries nine fractional digits; a lower declared 
precision must floor
+        # them, or values equal at that precision compare and group as 
different.
+        import pandas as pd
+        import pyarrow as pa
+
+        cases = [
+            (
+                TimestampNTZNanosType(7),
+                ["2020-01-02 03:04:05.123456789", "2020-01-02 
03:04:05.123456712"],
+                "2020-01-02 03:04:05.1234567",
+            ),
+            (
+                TimestampLTZNanosType(8),
+                ["2020-01-02 03:04:05.123456789", "2020-01-02 
03:04:05.123456781"],
+                "2020-01-02 03:04:05.12345678",
+            ),
+            # Pre-epoch values floor toward -inf, not toward zero.
+            (
+                TimestampNTZNanosType(7),
+                ["1969-12-31 23:59:59.999999999", "1969-12-31 
23:59:59.999999901"],
+                "1969-12-31 23:59:59.9999999",
+            ),
+        ]
+        with self.sql_conf(
+            {
+                "spark.sql.timestampNanosTypes.enabled": True,
+                "spark.sql.session.timeZone": "UTC",
+                "spark.sql.execution.arrow.pyspark.enabled": True,
+            }
+        ):
+            for nanos_type, values, floored in cases:
+                pdf = pd.DataFrame({"ts": pd.to_datetime(pd.Series(values))})
+                table = pa.table({"ts": pa.array(pdf["ts"], 
type=pa.timestamp("ns"))})
+                schema = StructType([StructField("ts", nanos_type)])
+                floored_literal = f"CAST('{floored}' AS 
{nanos_type.simpleString()})"
+                for data in (pdf, table):
+                    df = self.spark.createDataFrame(data, schema)
+                    self.assertEqual(1, df.distinct().count())
+                    self.assertEqual(2, df.where(f"ts = 
{floored_literal}").count())
+                    self.assertEqual([pd.Timestamp(floored)] * 2, 
list(df.toPandas()["ts"]))
+
     def test_timestamp_nanos_type_arrow_udf(self):
         # SPARK-57462 follow-up: with the eager Arrow-conversion reject 
removed, an Arrow-optimized
         # Python UDF (useArrow=True) can take and return the nanosecond 
timestamp types through the
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index a291ea5481c1..5e045ac16bd2 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -617,7 +617,8 @@ class TimestampNTZNanosType(AnyTimestampNanoType):
     :meth:`SparkSession.createDataFrame` from a pandas ``DataFrame`` with 
Arrow enabled
     (``spark.sql.execution.arrow.pyspark.enabled``), including the Spark 
Connect data path --
     carries the value as an Arrow ``timestamp[ns]`` and preserves full 
nanosecond precision
-    (pandas ``datetime64[ns]``). Because that Arrow encoding counts 
nanoseconds since the epoch in
+    (pandas ``datetime64[ns]``); input with more digits than the declared 
precision is floored to
+    it. Because that Arrow encoding counts nanoseconds since the epoch in
     a 64-bit integer, values outside the ``datetime64[ns]`` range (roughly the 
years 1677 to 2262)
     cannot be carried on this path. With Arrow disabled, 
:meth:`DataFrame.toPandas` falls back to
     :meth:`DataFrame.collect` and, like it, truncates to microseconds.
diff --git 
a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
 
b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
index e0cc93df5ea2..ac511cd4c2ce 100644
--- 
a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
+++ 
b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
@@ -30,6 +30,7 @@ import org.apache.arrow.vector.holders.NullableVarCharHolder;
 import org.apache.spark.SparkUnsupportedOperationException;
 import org.apache.spark.annotation.DeveloperApi;
 import org.apache.spark.sql.catalyst.util.DateTimeConstants;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
 import org.apache.spark.sql.catalyst.util.STUtils;
 import org.apache.spark.sql.util.ArrowUtils;
 import org.apache.spark.sql.types.*;
@@ -232,9 +233,9 @@ public class ArrowColumnVector extends ColumnVector {
     } else if (vector instanceof TimeNanoVector timeNanoVector) {
       accessor = new TimeNanoAccessor(timeNanoVector);
     } else if (vector instanceof TimeStampNanoTZVector timeStampNanoTZVector) {
-      accessor = new TimestampLTZNanosAccessor(timeStampNanoTZVector);
+      accessor = new TimestampLTZNanosAccessor(timeStampNanoTZVector, 
nanosPrecision(type));
     } else if (vector instanceof TimeStampNanoVector timeStampNanoVector) {
-      accessor = new TimestampNTZNanosAccessor(timeStampNanoVector);
+      accessor = new TimestampNTZNanosAccessor(timeStampNanoVector, 
nanosPrecision(type));
     } else if (vector instanceof MapVector mapVector) {
       accessor = new MapAccessor(mapVector);
     } else if (vector instanceof ListVector listVector) {
@@ -662,9 +663,20 @@ public class ArrowColumnVector extends ColumnVector {
     }
   }
 
+  // The vector class and its field's time zone can disagree, so accept either 
nanos type.
+  private static int nanosPrecision(DataType type) {
+    if (type instanceof TimestampNTZNanosType t) return t.precision();
+    if (type instanceof TimestampLTZNanosType t) return t.precision();
+    return TimestampNTZNanosType.MAX_PRECISION();
+  }
+
   // Decodes a single int64 of epoch-nanoseconds back into the (epochMicros, 
nanosWithinMicro)
   // pair. floorDiv/floorMod keep nanosWithinMicro in [0, 999] for pre-epoch 
(negative) values too.
-  private static TimestampNanosVal decodeEpochNanos(long nanos) {
+  // Below p = 9 it floors to the column precision, as foreign producers may 
send extra digits.
+  private static TimestampNanosVal decodeEpochNanos(long nanos, int precision) 
{
+    if (precision < TimestampNTZNanosType.MAX_PRECISION()) {
+      return DateTimeUtils.epochNanosToTimestampNanos(nanos, precision);
+    }
     return TimestampNanosVal.fromTrustedRowBytes(
       Math.floorDiv(nanos, DateTimeConstants.NANOS_PER_MICROS),
       (short) Math.floorMod(nanos, DateTimeConstants.NANOS_PER_MICROS));
@@ -673,30 +685,34 @@ public class ArrowColumnVector extends ColumnVector {
   static class TimestampNTZNanosAccessor extends ArrowVectorAccessor {
 
     private final TimeStampNanoVector accessor;
+    private final int precision;
 
-    TimestampNTZNanosAccessor(TimeStampNanoVector vector) {
+    TimestampNTZNanosAccessor(TimeStampNanoVector vector, int precision) {
       super(vector);
       this.accessor = vector;
+      this.precision = precision;
     }
 
     @Override
     final TimestampNanosVal getTimestampNanos(int rowId) {
-      return decodeEpochNanos(accessor.get(rowId));
+      return decodeEpochNanos(accessor.get(rowId), precision);
     }
   }
 
   static class TimestampLTZNanosAccessor extends ArrowVectorAccessor {
 
     private final TimeStampNanoTZVector accessor;
+    private final int precision;
 
-    TimestampLTZNanosAccessor(TimeStampNanoTZVector vector) {
+    TimestampLTZNanosAccessor(TimeStampNanoTZVector vector, int precision) {
       super(vector);
       this.accessor = vector;
+      this.precision = precision;
     }
 
     @Override
     final TimestampNanosVal getTimestampNanos(int rowId) {
-      return decodeEpochNanos(accessor.get(rowId));
+      return decodeEpochNanos(accessor.get(rowId), precision);
     }
   }
 
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
index d5fcbfdaa33b..50ead64700ed 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
@@ -1476,6 +1476,41 @@ class ArrowConvertersSuite extends SharedSparkSession {
     }
   }
 
+  test("nanosecond timestamps read with a declared lower precision are 
floored") {
+    // Mirrors createDataFrame from pandas: timestamp[ns] batches read at a 
lower precision.
+    withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+      Seq(7, 8).foreach { p =>
+        Seq[(DataType, DataType, String)](
+          (TimestampNTZNanosType(9), TimestampNTZNanosType(p), null),
+          (TimestampLTZNanosType(9), TimestampLTZNanosType(p), "UTC")).foreach 
{
+          case (writeType, readType, timeZoneId) =>
+            val grid = if (p == 7) 100 else 10
+            val values = Seq(
+              TimestampNanosVal.fromParts(1234567L, 789.toShort),
+              TimestampNanosVal.fromParts(1234567L, 712.toShort),
+              TimestampNanosVal.fromParts(-1234567L, 999.toShort))
+            val ctx = TaskContext.empty()
+            // A trailing null exercises the null path.
+            val batchIter = ArrowConverters.toBatchIterator(
+              (values.map(v => InternalRow(v)) :+ InternalRow(null)).iterator,
+              StructType(Seq(StructField("value", writeType))), 5, timeZoneId, 
true, false, ctx)
+            val got = ArrowConverters.fromBatchIterator(
+              batchIter, StructType(Seq(StructField("value", readType))), 
timeZoneId, true, false,
+              ctx).map { row =>
+              if (row.isNullAt(0)) null else readType match {
+                case _: TimestampNTZNanosType => row.getTimestampNTZNanos(0)
+                case _: TimestampLTZNanosType => row.getTimestampLTZNanos(0)
+              }
+            }.toSeq
+            assert(got === values.map { v =>
+              TimestampNanosVal.fromParts(
+                v.epochMicros, (v.nanosWithinMicro / grid * grid).toShort)
+            } :+ null)
+        }
+      }
+    }
+  }
+
   test("ArrowBatchStreamWriter roundtrip") {
     val inputRows = (0 until 9).map(InternalRow(_)) :+ InternalRow(null)
 
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
index 964a5767bd8b..7f27c1d0e7aa 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.arrow
 
 import scala.jdk.CollectionConverters._
 
-import org.apache.arrow.vector.{VectorSchemaRoot, ViewVarBinaryVector, 
ViewVarCharVector}
+import org.apache.arrow.vector.{TimeStampVector, VectorSchemaRoot, 
ViewVarBinaryVector, ViewVarCharVector}
 import org.apache.arrow.vector.complex.ListViewVector
 import org.apache.arrow.vector.types.pojo.{ArrowType, FieldType}
 
@@ -196,7 +196,7 @@ class ArrowWriterSuite extends SparkFunSuite {
       fromEpochNanos(Long.MaxValue),
       fromEpochNanos(Long.MinValue + 1))
 
-    def check(dt: DataType, timeZoneId: String): Unit = {
+    def check(dt: DataType, timeZoneId: String, grid: Int): Unit = {
       val schema = new StructType().add("value", dt, nullable = true)
       val writer = ArrowWriter.create(schema, timeZoneId)
       assert(writer.schema === schema)
@@ -206,26 +206,51 @@ class ArrowWriterSuite extends SparkFunSuite {
       }
       writer.finish()
 
-      val reader = new ArrowColumnVector(writer.root.getFieldVectors().get(0))
+      val vector = 
writer.root.getFieldVectors().get(0).asInstanceOf[TimeStampVector]
+      val reader = new ArrowColumnVector(vector)
       values.zipWithIndex.foreach { case (v, rowId) =>
+        assert(vector.get(rowId) === v.epochMicros * 1000L + 
v.nanosWithinMicro)
         val got = dt match {
           case _: TimestampNTZNanosType => reader.getTimestampNTZNanos(rowId)
           case _: TimestampLTZNanosType => reader.getTimestampLTZNanos(rowId)
         }
         assert(got.epochMicros === v.epochMicros)
-        assert(got.nanosWithinMicro === v.nanosWithinMicro)
+        assert(got.nanosWithinMicro === v.nanosWithinMicro / grid * grid)
       }
       assert(reader.isNullAt(values.length))
       writer.root.close()
     }
 
-    check(TimestampNTZNanosType(9), null)
-    check(TimestampLTZNanosType(9), "UTC")
-    // The value path packs the full nanosecond value regardless of the column 
precision (precision
-    // is carried in the Arrow field metadata, not the value), so p=7 
round-trips identically to
-    // p=9; exercising it guards the value path against a future 
precision-enforcing change.
-    check(TimestampNTZNanosType(7), null)
-    check(TimestampLTZNanosType(7), "UTC")
+    check(TimestampNTZNanosType(9), null, grid = 1)
+    check(TimestampLTZNanosType(9), "UTC", grid = 1)
+    // The writer packs the full value regardless of precision; the reader 
floors it.
+    check(TimestampNTZNanosType(7), null, grid = 100)
+    check(TimestampLTZNanosType(7), "UTC", grid = 100)
+  }
+
+  test("timestamp nanos array elements and map keys / values are floored on 
read") {
+    def ts(nanosWithinMicro: Int): TimestampNanosVal =
+      TimestampNanosVal.fromParts(123456L, nanosWithinMicro.toShort)
+    val schema = new StructType()
+      .add("arr", ArrayType(TimestampNTZNanosType(7)))
+      .add("map", MapType(TimestampNTZNanosType(7), TimestampLTZNanosType(8)))
+    val writer = ArrowWriter.create(schema, "UTC")
+    writer.write(InternalRow(
+      new GenericArrayData(Array[Any](ts(789), null)),
+      new ArrayBasedMapData(
+        new GenericArrayData(Array[Any](ts(789), ts(812))),
+        new GenericArrayData(Array[Any](ts(789), null)))))
+    writer.finish()
+
+    val arr = new 
ArrowColumnVector(writer.root.getFieldVectors().get(0)).getArray(0)
+    assert(arr.getTimestampNTZNanos(0) === ts(700))
+    assert(arr.isNullAt(1))
+    val map = new 
ArrowColumnVector(writer.root.getFieldVectors().get(1)).getMap(0)
+    assert(map.keyArray().getTimestampNTZNanos(0) === ts(700))
+    assert(map.keyArray().getTimestampNTZNanos(1) === ts(800))
+    assert(map.valueArray().getTimestampLTZNanos(0) === ts(780))
+    assert(map.valueArray().isNullAt(1))
+    writer.root.close()
   }
 
   test("timestamp nanos out of range raises DATETIME_OVERFLOW") {
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
index 1e69555d0ede..ba64df16611b 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
@@ -25,6 +25,7 @@ import scala.util.{Failure, Random, Success, Try, Using}
 import org.apache.arrow.memory.RootAllocator
 import org.apache.arrow.vector._
 import org.apache.arrow.vector.complex._
+import org.apache.arrow.vector.types.TimeUnit
 import org.apache.arrow.vector.types.pojo.{ArrowType, FieldType}
 
 import org.apache.spark.SparkFunSuite
@@ -33,7 +34,7 @@ import 
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.ArrowUtils
-import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
 
 class ArrowColumnVectorSuite extends SparkFunSuite {
 
@@ -878,6 +879,98 @@ class ArrowColumnVectorSuite extends SparkFunSuite {
     allocator.close()
   }
 
+  test("timestamp nanos are floored to the column precision") {
+    // (epochNanos, epochMicros, nanosWithinMicro), with more digits than p = 
7 / 8 allow.
+    val values = Seq(
+      (123456789L, 123456L, 789),
+      (123456712L, 123456L, 712),
+      (123456700L, 123456L, 700),
+      // Pre-epoch values floor toward -inf, not toward zero.
+      (-1L, -1L, 999),
+      (-123456211L, -123457L, 789),
+      (Long.MaxValue, 9223372036854775L, 807),
+      (Long.MinValue, -9223372036854776L, 192))
+    Seq(7, 8, 9).foreach { p =>
+      val grid = Map(7 -> 100, 8 -> 10, 9 -> 1)(p)
+      Seq[(DataType, String)](
+        (TimestampNTZNanosType(p), null),
+        (TimestampLTZNanosType(p), "UTC")).foreach { case (dt, timeZoneId) =>
+        val allocator = ArrowUtils.rootAllocator.newChildAllocator(s"$dt", 0, 
Long.MaxValue)
+        val vector = ArrowUtils.toArrowField("ts", dt, nullable = true, 
timeZoneId)
+          .createVector(allocator).asInstanceOf[TimeStampVector]
+        vector.allocateNew()
+        values.zipWithIndex.foreach { case ((epochNanos, _, _), i) =>
+          vector.setSafe(i, epochNanos)
+        }
+        vector.setNull(values.length)
+        vector.setValueCount(values.length + 1)
+
+        val columnVector = new ArrowColumnVector(vector)
+        assert(columnVector.dataType === dt)
+        values.zipWithIndex.foreach { case ((_, epochMicros, 
nanosWithinMicro), i) =>
+          val got = dt match {
+            case _: TimestampNTZNanosType => 
columnVector.getTimestampNTZNanos(i)
+            case _: TimestampLTZNanosType => 
columnVector.getTimestampLTZNanos(i)
+          }
+          assert(got.epochMicros === epochMicros)
+          assert(got.nanosWithinMicro === nanosWithinMicro / grid * grid)
+        }
+        assert(columnVector.isNullAt(values.length))
+
+        columnVector.close()
+        allocator.close()
+      }
+    }
+  }
+
+  test("timestamp nanos struct fields are floored to the field precision") {
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("struct", 0, 
Long.MaxValue)
+    val schema = new StructType().add("ts", TimestampNTZNanosType(7))
+    val vector = ArrowUtils.toArrowField("struct", schema, nullable = true, 
null)
+      .createVector(allocator).asInstanceOf[StructVector]
+    vector.allocateNew()
+    val timestampVector = 
vector.getChildByOrdinal(0).asInstanceOf[TimeStampNanoVector]
+
+    vector.setIndexDefined(0)
+    timestampVector.setSafe(0, 123456789L)
+
+    timestampVector.setValueCount(1)
+    vector.setValueCount(1)
+
+    val columnVector = new ArrowColumnVector(vector)
+    assert(columnVector.dataType === schema)
+    assert(columnVector.getStruct(0).getTimestampNTZNanos(0) ===
+      TimestampNanosVal.fromParts(123456L, 700.toShort))
+
+    columnVector.close()
+    allocator.close()
+  }
+
+  test("timestamp nanos vectors whose time zone disagrees with the vector 
class") {
+    val allocator = ArrowUtils.rootAllocator.newChildAllocator("mismatch", 0, 
Long.MaxValue)
+    // A TZ vector without a time zone reads as NTZ; a plain vector with one 
reads as LTZ.
+    val tzVector = new TimeStampNanoTZVector("ts", allocator, null)
+    val ntzVector = new TimeStampNanoVector(
+      "ts", FieldType.nullable(new ArrowType.Timestamp(TimeUnit.NANOSECOND, 
"UTC")), allocator)
+    Seq(tzVector, ntzVector).foreach { vector =>
+      vector.allocateNew()
+      vector.setSafe(0, 123456789L)
+      vector.setValueCount(1)
+    }
+    val expected = TimestampNanosVal.fromParts(123456L, 789.toShort)
+
+    val tzColumn = new ArrowColumnVector(tzVector)
+    assert(tzColumn.dataType === TimestampNTZNanosType(9))
+    assert(tzColumn.getTimestampNTZNanos(0) === expected)
+    val ntzColumn = new ArrowColumnVector(ntzVector)
+    assert(ntzColumn.dataType === TimestampLTZNanosType(9))
+    assert(ntzColumn.getTimestampLTZNanos(0) === expected)
+
+    tzColumn.close()
+    ntzColumn.close()
+    allocator.close()
+  }
+
   private def withDecimalVector(precision: Int, scale: Int)(f: DecimalVector 
=> Unit): Unit = {
     Using.resource(new RootAllocator()) { allocator =>
       Using.resource(new DecimalVector("decimal", allocator, precision, 
scale))(f)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to