This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 972b507214fe [SPARK-60540][SQL] Floor Arrow nanosecond timestamps to
the column precision on read
972b507214fe is described below
commit 972b507214fe6949058d7c08876e2fa0b9da5f84
Author: Stevo Mitric <[email protected]>
AuthorDate: Mon Oct 26 11:17:33 2026 +0000
[SPARK-60540][SQL] Floor Arrow nanosecond timestamps to the column
precision on read
### What changes were proposed in this pull request?
`ArrowColumnVector`'s `TimestampNTZNanosAccessor` /
`TimestampLTZNanosAccessor` now floor the decoded `timestamp[ns]` value to the
vector's declared precision (recovered from the
`SPARK::timestampNanos::precision` field metadata by
`ArrowUtils.fromArrowField`). Below p = 9 they call the shared
`DateTimeUtils.epochNanosToTimestampNanos(nanos, precision)` helper, the same
one the Parquet and Avro readers use; p = 9 keeps the existing decode
unchanged. Nested array elements, map keys / v [...]
### Why are the changes needed?
Spark keeps nanosecond timestamp values floored to the column precision (p
= 7 is a 100 ns grid, p = 8 a 10 ns grid); casts, Parquet, Avro and the text
parsers all floor on the way in. The Arrow reader did not, so an Arrow batch
from a foreign producer with more sub-microsecond digits than the column allows
kept them. Values that are equal at the declared precision then compared,
hashed and grouped as different, and failed equality with a literal of the same
type. Reachable today, for [...]
```python
pdf = pd.DataFrame({"ts": pd.to_datetime(["2020-01-02 03:04:05.123456789",
"2020-01-02 03:04:05.123456712"])})
df = spark.createDataFrame(pdf, "ts timestamp_ntz(7)")
df.distinct().count() # 2 before, 1 after
df.where("ts = CAST('2020-01-02 03:04:05.1234567' AS
TIMESTAMP_NTZ(7))").count() # 0 before, 2 after
```
The same happens for `createDataFrame` from a `pyarrow.Table`, over Spark
Connect (`LocalRelation` with Arrow data), and for Arrow-optimized Python UDTF
results.
This covers vectors whose Arrow field carries the precision tag. The
PySpark worker currently drops the tag on the top-level fields of UDF result
batches, so `mapInPandas`, `mapInArrow`, `applyInPandas`, `applyInArrow` and
Python data sources still read such a column as p = 9, and a scalar
`pandas_udf` or Arrow-optimized UDF with a p < 9 return type fails with
`ARROW_TYPE_MISMATCH`. That is a separate PySpark fix, left for a follow-up.
### Does this PR introduce _any_ user-facing change?
Yes, for the unreleased nanosecond timestamp types (behind
`spark.sql.timestampNanosTypes.enabled`): Arrow input to a `TIMESTAMP_NTZ(p)` /
`TIMESTAMP_LTZ(p)` column with p < 9 is now floored to p, matching every other
ingestion path. p = 9 is unchanged.
### How was this patch tested?
- `ArrowColumnVectorSuite`: new tests reading off-grid int64 epoch-nanos
(including pre-epoch values and `Long.MinValue`/`MaxValue`) at p = 7/8/9 for
NTZ and LTZ, from a nested struct field, and from vectors whose class and time
zone disagree; `ArrowConvertersSuite`: batches written at p = 9 and read with a
declared p = 7/8 schema (the `createDataFrame` path); `ArrowWriterSuite`: the p
= 7 round-trip now expects the floored value, plus array elements and map keys
/ values.
- PySpark `test_types.py`: `createDataFrame` from pandas and pyarrow with
NTZ(7), LTZ(8) and pre-epoch values, run in classic and via the Connect parity
suite.
- Existing `ArrowUtilsSuite`, `DateTimeUtilsSuite`,
`org.apache.spark.sql.execution.arrow.*`, `org.apache.spark.sql.vectorized.*`
and `ColumnarBatchSuite` pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59780 from stevomitric/stevomitric/nanos-arrow-floor.
Authored-by: Stevo Mitric <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
---
python/pyspark/sql/tests/test_types.py | 42 ++++++++++
python/pyspark/sql/types.py | 3 +-
.../spark/sql/vectorized/ArrowColumnVector.java | 30 +++++--
.../sql/execution/arrow/ArrowConvertersSuite.scala | 35 ++++++++
.../sql/execution/arrow/ArrowWriterSuite.scala | 47 ++++++++---
.../sql/vectorized/ArrowColumnVectorSuite.scala | 95 +++++++++++++++++++++-
6 files changed, 232 insertions(+), 20 deletions(-)
diff --git a/python/pyspark/sql/tests/test_types.py
b/python/pyspark/sql/tests/test_types.py
index 0f2e64f88b27..797872efb76e 100644
--- a/python/pyspark/sql/tests/test_types.py
+++ b/python/pyspark/sql/tests/test_types.py
@@ -2900,6 +2900,48 @@ class TypesTestsMixin:
self.assertTrue(row.sa.startswith("2020-01-02
03:04:05.1234567"), row.sa)
self.assertTrue(row.mv.startswith("2020-01-02
03:04:05.1234567"), row.mv)
+ def test_timestamp_nanos_type_arrow_conversion_floors_to_precision(self):
+ # Arrow timestamp[ns] carries nine fractional digits; a lower declared
precision must floor
+ # them, or values equal at that precision compare and group as
different.
+ import pandas as pd
+ import pyarrow as pa
+
+ cases = [
+ (
+ TimestampNTZNanosType(7),
+ ["2020-01-02 03:04:05.123456789", "2020-01-02
03:04:05.123456712"],
+ "2020-01-02 03:04:05.1234567",
+ ),
+ (
+ TimestampLTZNanosType(8),
+ ["2020-01-02 03:04:05.123456789", "2020-01-02
03:04:05.123456781"],
+ "2020-01-02 03:04:05.12345678",
+ ),
+ # Pre-epoch values floor toward -inf, not toward zero.
+ (
+ TimestampNTZNanosType(7),
+ ["1969-12-31 23:59:59.999999999", "1969-12-31
23:59:59.999999901"],
+ "1969-12-31 23:59:59.9999999",
+ ),
+ ]
+ with self.sql_conf(
+ {
+ "spark.sql.timestampNanosTypes.enabled": True,
+ "spark.sql.session.timeZone": "UTC",
+ "spark.sql.execution.arrow.pyspark.enabled": True,
+ }
+ ):
+ for nanos_type, values, floored in cases:
+ pdf = pd.DataFrame({"ts": pd.to_datetime(pd.Series(values))})
+ table = pa.table({"ts": pa.array(pdf["ts"],
type=pa.timestamp("ns"))})
+ schema = StructType([StructField("ts", nanos_type)])
+ floored_literal = f"CAST('{floored}' AS
{nanos_type.simpleString()})"
+ for data in (pdf, table):
+ df = self.spark.createDataFrame(data, schema)
+ self.assertEqual(1, df.distinct().count())
+ self.assertEqual(2, df.where(f"ts =
{floored_literal}").count())
+ self.assertEqual([pd.Timestamp(floored)] * 2,
list(df.toPandas()["ts"]))
+
def test_timestamp_nanos_type_arrow_udf(self):
# SPARK-57462 follow-up: with the eager Arrow-conversion reject
removed, an Arrow-optimized
# Python UDF (useArrow=True) can take and return the nanosecond
timestamp types through the
diff --git a/python/pyspark/sql/types.py b/python/pyspark/sql/types.py
index a291ea5481c1..5e045ac16bd2 100644
--- a/python/pyspark/sql/types.py
+++ b/python/pyspark/sql/types.py
@@ -617,7 +617,8 @@ class TimestampNTZNanosType(AnyTimestampNanoType):
:meth:`SparkSession.createDataFrame` from a pandas ``DataFrame`` with
Arrow enabled
(``spark.sql.execution.arrow.pyspark.enabled``), including the Spark
Connect data path --
carries the value as an Arrow ``timestamp[ns]`` and preserves full
nanosecond precision
- (pandas ``datetime64[ns]``). Because that Arrow encoding counts
nanoseconds since the epoch in
+ (pandas ``datetime64[ns]``); input with more digits than the declared
precision is floored to
+ it. Because that Arrow encoding counts nanoseconds since the epoch in
a 64-bit integer, values outside the ``datetime64[ns]`` range (roughly the
years 1677 to 2262)
cannot be carried on this path. With Arrow disabled,
:meth:`DataFrame.toPandas` falls back to
:meth:`DataFrame.collect` and, like it, truncates to microseconds.
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
index e0cc93df5ea2..ac511cd4c2ce 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java
@@ -30,6 +30,7 @@ import org.apache.arrow.vector.holders.NullableVarCharHolder;
import org.apache.spark.SparkUnsupportedOperationException;
import org.apache.spark.annotation.DeveloperApi;
import org.apache.spark.sql.catalyst.util.DateTimeConstants;
+import org.apache.spark.sql.catalyst.util.DateTimeUtils;
import org.apache.spark.sql.catalyst.util.STUtils;
import org.apache.spark.sql.util.ArrowUtils;
import org.apache.spark.sql.types.*;
@@ -232,9 +233,9 @@ public class ArrowColumnVector extends ColumnVector {
} else if (vector instanceof TimeNanoVector timeNanoVector) {
accessor = new TimeNanoAccessor(timeNanoVector);
} else if (vector instanceof TimeStampNanoTZVector timeStampNanoTZVector) {
- accessor = new TimestampLTZNanosAccessor(timeStampNanoTZVector);
+ accessor = new TimestampLTZNanosAccessor(timeStampNanoTZVector,
nanosPrecision(type));
} else if (vector instanceof TimeStampNanoVector timeStampNanoVector) {
- accessor = new TimestampNTZNanosAccessor(timeStampNanoVector);
+ accessor = new TimestampNTZNanosAccessor(timeStampNanoVector,
nanosPrecision(type));
} else if (vector instanceof MapVector mapVector) {
accessor = new MapAccessor(mapVector);
} else if (vector instanceof ListVector listVector) {
@@ -662,9 +663,20 @@ public class ArrowColumnVector extends ColumnVector {
}
}
+ // The vector class and its field's time zone can disagree, so accept either
nanos type.
+ private static int nanosPrecision(DataType type) {
+ if (type instanceof TimestampNTZNanosType t) return t.precision();
+ if (type instanceof TimestampLTZNanosType t) return t.precision();
+ return TimestampNTZNanosType.MAX_PRECISION();
+ }
+
// Decodes a single int64 of epoch-nanoseconds back into the (epochMicros,
nanosWithinMicro)
// pair. floorDiv/floorMod keep nanosWithinMicro in [0, 999] for pre-epoch
(negative) values too.
- private static TimestampNanosVal decodeEpochNanos(long nanos) {
+ // Below p = 9 it floors to the column precision, as foreign producers may
send extra digits.
+ private static TimestampNanosVal decodeEpochNanos(long nanos, int precision)
{
+ if (precision < TimestampNTZNanosType.MAX_PRECISION()) {
+ return DateTimeUtils.epochNanosToTimestampNanos(nanos, precision);
+ }
return TimestampNanosVal.fromTrustedRowBytes(
Math.floorDiv(nanos, DateTimeConstants.NANOS_PER_MICROS),
(short) Math.floorMod(nanos, DateTimeConstants.NANOS_PER_MICROS));
@@ -673,30 +685,34 @@ public class ArrowColumnVector extends ColumnVector {
static class TimestampNTZNanosAccessor extends ArrowVectorAccessor {
private final TimeStampNanoVector accessor;
+ private final int precision;
- TimestampNTZNanosAccessor(TimeStampNanoVector vector) {
+ TimestampNTZNanosAccessor(TimeStampNanoVector vector, int precision) {
super(vector);
this.accessor = vector;
+ this.precision = precision;
}
@Override
final TimestampNanosVal getTimestampNanos(int rowId) {
- return decodeEpochNanos(accessor.get(rowId));
+ return decodeEpochNanos(accessor.get(rowId), precision);
}
}
static class TimestampLTZNanosAccessor extends ArrowVectorAccessor {
private final TimeStampNanoTZVector accessor;
+ private final int precision;
- TimestampLTZNanosAccessor(TimeStampNanoTZVector vector) {
+ TimestampLTZNanosAccessor(TimeStampNanoTZVector vector, int precision) {
super(vector);
this.accessor = vector;
+ this.precision = precision;
}
@Override
final TimestampNanosVal getTimestampNanos(int rowId) {
- return decodeEpochNanos(accessor.get(rowId));
+ return decodeEpochNanos(accessor.get(rowId), precision);
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
index d5fcbfdaa33b..50ead64700ed 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
@@ -1476,6 +1476,41 @@ class ArrowConvertersSuite extends SharedSparkSession {
}
}
+ test("nanosecond timestamps read with a declared lower precision are
floored") {
+ // Mirrors createDataFrame from pandas: timestamp[ns] batches read at a
lower precision.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ Seq(7, 8).foreach { p =>
+ Seq[(DataType, DataType, String)](
+ (TimestampNTZNanosType(9), TimestampNTZNanosType(p), null),
+ (TimestampLTZNanosType(9), TimestampLTZNanosType(p), "UTC")).foreach
{
+ case (writeType, readType, timeZoneId) =>
+ val grid = if (p == 7) 100 else 10
+ val values = Seq(
+ TimestampNanosVal.fromParts(1234567L, 789.toShort),
+ TimestampNanosVal.fromParts(1234567L, 712.toShort),
+ TimestampNanosVal.fromParts(-1234567L, 999.toShort))
+ val ctx = TaskContext.empty()
+ // A trailing null exercises the null path.
+ val batchIter = ArrowConverters.toBatchIterator(
+ (values.map(v => InternalRow(v)) :+ InternalRow(null)).iterator,
+ StructType(Seq(StructField("value", writeType))), 5, timeZoneId,
true, false, ctx)
+ val got = ArrowConverters.fromBatchIterator(
+ batchIter, StructType(Seq(StructField("value", readType))),
timeZoneId, true, false,
+ ctx).map { row =>
+ if (row.isNullAt(0)) null else readType match {
+ case _: TimestampNTZNanosType => row.getTimestampNTZNanos(0)
+ case _: TimestampLTZNanosType => row.getTimestampLTZNanos(0)
+ }
+ }.toSeq
+ assert(got === values.map { v =>
+ TimestampNanosVal.fromParts(
+ v.epochMicros, (v.nanosWithinMicro / grid * grid).toShort)
+ } :+ null)
+ }
+ }
+ }
+ }
+
test("ArrowBatchStreamWriter roundtrip") {
val inputRows = (0 until 9).map(InternalRow(_)) :+ InternalRow(null)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
index 964a5767bd8b..7f27c1d0e7aa 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.arrow
import scala.jdk.CollectionConverters._
-import org.apache.arrow.vector.{VectorSchemaRoot, ViewVarBinaryVector,
ViewVarCharVector}
+import org.apache.arrow.vector.{TimeStampVector, VectorSchemaRoot,
ViewVarBinaryVector, ViewVarCharVector}
import org.apache.arrow.vector.complex.ListViewVector
import org.apache.arrow.vector.types.pojo.{ArrowType, FieldType}
@@ -196,7 +196,7 @@ class ArrowWriterSuite extends SparkFunSuite {
fromEpochNanos(Long.MaxValue),
fromEpochNanos(Long.MinValue + 1))
- def check(dt: DataType, timeZoneId: String): Unit = {
+ def check(dt: DataType, timeZoneId: String, grid: Int): Unit = {
val schema = new StructType().add("value", dt, nullable = true)
val writer = ArrowWriter.create(schema, timeZoneId)
assert(writer.schema === schema)
@@ -206,26 +206,51 @@ class ArrowWriterSuite extends SparkFunSuite {
}
writer.finish()
- val reader = new ArrowColumnVector(writer.root.getFieldVectors().get(0))
+ val vector =
writer.root.getFieldVectors().get(0).asInstanceOf[TimeStampVector]
+ val reader = new ArrowColumnVector(vector)
values.zipWithIndex.foreach { case (v, rowId) =>
+ assert(vector.get(rowId) === v.epochMicros * 1000L +
v.nanosWithinMicro)
val got = dt match {
case _: TimestampNTZNanosType => reader.getTimestampNTZNanos(rowId)
case _: TimestampLTZNanosType => reader.getTimestampLTZNanos(rowId)
}
assert(got.epochMicros === v.epochMicros)
- assert(got.nanosWithinMicro === v.nanosWithinMicro)
+ assert(got.nanosWithinMicro === v.nanosWithinMicro / grid * grid)
}
assert(reader.isNullAt(values.length))
writer.root.close()
}
- check(TimestampNTZNanosType(9), null)
- check(TimestampLTZNanosType(9), "UTC")
- // The value path packs the full nanosecond value regardless of the column
precision (precision
- // is carried in the Arrow field metadata, not the value), so p=7
round-trips identically to
- // p=9; exercising it guards the value path against a future
precision-enforcing change.
- check(TimestampNTZNanosType(7), null)
- check(TimestampLTZNanosType(7), "UTC")
+ check(TimestampNTZNanosType(9), null, grid = 1)
+ check(TimestampLTZNanosType(9), "UTC", grid = 1)
+ // The writer packs the full value regardless of precision; the reader
floors it.
+ check(TimestampNTZNanosType(7), null, grid = 100)
+ check(TimestampLTZNanosType(7), "UTC", grid = 100)
+ }
+
+ test("timestamp nanos array elements and map keys / values are floored on
read") {
+ def ts(nanosWithinMicro: Int): TimestampNanosVal =
+ TimestampNanosVal.fromParts(123456L, nanosWithinMicro.toShort)
+ val schema = new StructType()
+ .add("arr", ArrayType(TimestampNTZNanosType(7)))
+ .add("map", MapType(TimestampNTZNanosType(7), TimestampLTZNanosType(8)))
+ val writer = ArrowWriter.create(schema, "UTC")
+ writer.write(InternalRow(
+ new GenericArrayData(Array[Any](ts(789), null)),
+ new ArrayBasedMapData(
+ new GenericArrayData(Array[Any](ts(789), ts(812))),
+ new GenericArrayData(Array[Any](ts(789), null)))))
+ writer.finish()
+
+ val arr = new
ArrowColumnVector(writer.root.getFieldVectors().get(0)).getArray(0)
+ assert(arr.getTimestampNTZNanos(0) === ts(700))
+ assert(arr.isNullAt(1))
+ val map = new
ArrowColumnVector(writer.root.getFieldVectors().get(1)).getMap(0)
+ assert(map.keyArray().getTimestampNTZNanos(0) === ts(700))
+ assert(map.keyArray().getTimestampNTZNanos(1) === ts(800))
+ assert(map.valueArray().getTimestampLTZNanos(0) === ts(780))
+ assert(map.valueArray().isNullAt(1))
+ writer.root.close()
}
test("timestamp nanos out of range raises DATETIME_OVERFLOW") {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
index 1e69555d0ede..ba64df16611b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala
@@ -25,6 +25,7 @@ import scala.util.{Failure, Random, Success, Try, Using}
import org.apache.arrow.memory.RootAllocator
import org.apache.arrow.vector._
import org.apache.arrow.vector.complex._
+import org.apache.arrow.vector.types.TimeUnit
import org.apache.arrow.vector.types.pojo.{ArrowType, FieldType}
import org.apache.spark.SparkFunSuite
@@ -33,7 +34,7 @@ import
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.sql.util.ArrowUtils
-import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
class ArrowColumnVectorSuite extends SparkFunSuite {
@@ -878,6 +879,98 @@ class ArrowColumnVectorSuite extends SparkFunSuite {
allocator.close()
}
+ test("timestamp nanos are floored to the column precision") {
+ // (epochNanos, epochMicros, nanosWithinMicro), with more digits than p =
7 / 8 allow.
+ val values = Seq(
+ (123456789L, 123456L, 789),
+ (123456712L, 123456L, 712),
+ (123456700L, 123456L, 700),
+ // Pre-epoch values floor toward -inf, not toward zero.
+ (-1L, -1L, 999),
+ (-123456211L, -123457L, 789),
+ (Long.MaxValue, 9223372036854775L, 807),
+ (Long.MinValue, -9223372036854776L, 192))
+ Seq(7, 8, 9).foreach { p =>
+ val grid = Map(7 -> 100, 8 -> 10, 9 -> 1)(p)
+ Seq[(DataType, String)](
+ (TimestampNTZNanosType(p), null),
+ (TimestampLTZNanosType(p), "UTC")).foreach { case (dt, timeZoneId) =>
+ val allocator = ArrowUtils.rootAllocator.newChildAllocator(s"$dt", 0,
Long.MaxValue)
+ val vector = ArrowUtils.toArrowField("ts", dt, nullable = true,
timeZoneId)
+ .createVector(allocator).asInstanceOf[TimeStampVector]
+ vector.allocateNew()
+ values.zipWithIndex.foreach { case ((epochNanos, _, _), i) =>
+ vector.setSafe(i, epochNanos)
+ }
+ vector.setNull(values.length)
+ vector.setValueCount(values.length + 1)
+
+ val columnVector = new ArrowColumnVector(vector)
+ assert(columnVector.dataType === dt)
+ values.zipWithIndex.foreach { case ((_, epochMicros,
nanosWithinMicro), i) =>
+ val got = dt match {
+ case _: TimestampNTZNanosType =>
columnVector.getTimestampNTZNanos(i)
+ case _: TimestampLTZNanosType =>
columnVector.getTimestampLTZNanos(i)
+ }
+ assert(got.epochMicros === epochMicros)
+ assert(got.nanosWithinMicro === nanosWithinMicro / grid * grid)
+ }
+ assert(columnVector.isNullAt(values.length))
+
+ columnVector.close()
+ allocator.close()
+ }
+ }
+ }
+
+ test("timestamp nanos struct fields are floored to the field precision") {
+ val allocator = ArrowUtils.rootAllocator.newChildAllocator("struct", 0,
Long.MaxValue)
+ val schema = new StructType().add("ts", TimestampNTZNanosType(7))
+ val vector = ArrowUtils.toArrowField("struct", schema, nullable = true,
null)
+ .createVector(allocator).asInstanceOf[StructVector]
+ vector.allocateNew()
+ val timestampVector =
vector.getChildByOrdinal(0).asInstanceOf[TimeStampNanoVector]
+
+ vector.setIndexDefined(0)
+ timestampVector.setSafe(0, 123456789L)
+
+ timestampVector.setValueCount(1)
+ vector.setValueCount(1)
+
+ val columnVector = new ArrowColumnVector(vector)
+ assert(columnVector.dataType === schema)
+ assert(columnVector.getStruct(0).getTimestampNTZNanos(0) ===
+ TimestampNanosVal.fromParts(123456L, 700.toShort))
+
+ columnVector.close()
+ allocator.close()
+ }
+
+ test("timestamp nanos vectors whose time zone disagrees with the vector
class") {
+ val allocator = ArrowUtils.rootAllocator.newChildAllocator("mismatch", 0,
Long.MaxValue)
+ // A TZ vector without a time zone reads as NTZ; a plain vector with one
reads as LTZ.
+ val tzVector = new TimeStampNanoTZVector("ts", allocator, null)
+ val ntzVector = new TimeStampNanoVector(
+ "ts", FieldType.nullable(new ArrowType.Timestamp(TimeUnit.NANOSECOND,
"UTC")), allocator)
+ Seq(tzVector, ntzVector).foreach { vector =>
+ vector.allocateNew()
+ vector.setSafe(0, 123456789L)
+ vector.setValueCount(1)
+ }
+ val expected = TimestampNanosVal.fromParts(123456L, 789.toShort)
+
+ val tzColumn = new ArrowColumnVector(tzVector)
+ assert(tzColumn.dataType === TimestampNTZNanosType(9))
+ assert(tzColumn.getTimestampNTZNanos(0) === expected)
+ val ntzColumn = new ArrowColumnVector(ntzVector)
+ assert(ntzColumn.dataType === TimestampLTZNanosType(9))
+ assert(ntzColumn.getTimestampLTZNanos(0) === expected)
+
+ tzColumn.close()
+ ntzColumn.close()
+ allocator.close()
+ }
+
private def withDecimalVector(precision: Int, scale: Int)(f: DecimalVector
=> Unit): Unit = {
Using.resource(new RootAllocator()) { allocator =>
Using.resource(new DecimalVector("decimal", allocator, precision,
scale))(f)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]