This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new da555037cf Spec: add nanosecond timestamp types (#8683)
da555037cf is described below
commit da555037cf0f19a635927df043fd02af0193d49d
Author: Jacob Marble <[email protected]>
AuthorDate: Tue Oct 31 09:15:24 2023 -0700
Spec: add nanosecond timestamp types (#8683)
This change embodies this design doc:
https://docs.google.com/document/d/1bE1DcEGNzZAMiVJSZ0X1wElKLNkT9kRkk0hDlfkXzvU/edit
Co-authored-by: Ryan Blue <[email protected]>
---
format/spec.md | 98 +++++++++++++++++++++++++++++++++++++---------------------
1 file changed, 62 insertions(+), 36 deletions(-)
diff --git a/format/spec.md b/format/spec.md
index 60c0f99c3f..855db29f56 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -167,29 +167,33 @@ A **`map`** is a collection of key-value pairs with a key
type and a value type.
#### Primitive Types
-| Primitive type | Description
| Requirements |
-|--------------------|--------------------------------------------------------------------------|--------------------------------------------------|
-| **`boolean`** | True or false
| |
-| **`int`** | 32-bit signed integers
| Can promote to `long` |
-| **`long`** | 64-bit signed integers
| |
-| **`float`** | [32-bit IEEE
754](https://en.wikipedia.org/wiki/IEEE_754) floating point | Can promote to
double |
-| **`double`** | [64-bit IEEE
754](https://en.wikipedia.org/wiki/IEEE_754) floating point |
|
-| **`decimal(P,S)`** | Fixed-point decimal; precision P, scale S
| Scale is fixed [1], precision must be 38 or less |
-| **`date`** | Calendar date without timezone or time
| |
-| **`time`** | Time of day without date, timezone
| Microsecond precision [2] |
-| **`timestamp`** | Timestamp without timezone
| Microsecond precision [2] |
-| **`timestamptz`** | Timestamp with timezone
| Stored as UTC [2] |
-| **`string`** | Arbitrary-length character sequences
| Encoded with UTF-8 [3] |
-| **`uuid`** | Universally unique identifiers
| Should use 16-byte fixed |
-| **`fixed(L)`** | Fixed-length byte array of length L
| |
-| **`binary`** | Arbitrary-length byte array
| |
+Supported primitive types are defined in the table below. Primitive types
added after v1 have an "added by" version that is the first spec version in
which the type is allowed. For example, nanosecond-precision timestamps are
part of the v3 spec; using v3 types in v1 or v2 tables can break forward
compatibility.
+
+| Added by verison | Primitive type | Description
| Requirements
|
+|------------------|--------------------|--------------------------------------------------------------------------|--------------------------------------------------|
+| | **`boolean`** | True or false
|
|
+| | **`int`** | 32-bit signed integers
| Can promote to `long`
|
+| | **`long`** | 64-bit signed integers
|
|
+| | **`float`** | [32-bit IEEE
754](https://en.wikipedia.org/wiki/IEEE_754) floating point | Can promote to
double |
+| | **`double`** | [64-bit IEEE
754](https://en.wikipedia.org/wiki/IEEE_754) floating point |
|
+| | **`decimal(P,S)`** | Fixed-point decimal; precision P,
scale S | Scale is fixed [1], precision must be
38 or less |
+| | **`date`** | Calendar date without timezone or
time |
|
+| | **`time`** | Time of day without date, timezone
| Microsecond precision [2]
|
+| | **`timestamp`** | Timestamp, microsecond precision,
without timezone | [2]
|
+| | **`timestamptz`** | Timestamp, microsecond precision,
with timezone | [2]
|
+| [v3](#version-3) | **`timestamp_ns`** | Timestamp, nanosecond precision,
without timezone | [2]
|
+| [v3](#version-3) | **`timestamptz_ns`** | Timestamp, nanosecond precision,
with timezone | [2]
|
+| | **`string`** | Arbitrary-length character sequences
| Encoded with UTF-8 [3]
|
+| | **`uuid`** | Universally unique identifiers
| Should use 16-byte fixed
|
+| | **`fixed(L)`** | Fixed-length byte array of length L
|
|
+| | **`binary`** | Arbitrary-length byte array
|
|
Notes:
1. Decimal scale is fixed and cannot be changed by schema evolution. Precision
can only be widened.
-2. All time and timestamp values are stored with microsecond precision.
- - Timestamps _with time zone_ represent a point in time: values are stored
as UTC and do not retain a source time zone (`2017-11-16 17:10:34 PST` is
stored/retrieved as `2017-11-17 01:10:34 UTC` and these values are considered
identical).
- - Timestamps _without time zone_ represent a date and time of day
regardless of zone: the time value is independent of zone adjustments
(`2017-11-16 17:10:34` is always retrieved as `2017-11-16 17:10:34`). Timestamp
values are stored as a long that encodes microseconds from the unix epoch.
+2. `time`, `timestamp`, and `timestamptz` values are represented with
_microsecond precision_. `timestamp_ns` and `timstamptz_ns` values are
represented with _nanosecond precision_.
+ - Timestamp values _with time zone_ represent a point in time: values are
stored as UTC and do not retain a source time zone (`2017-11-16 17:10:34 PST`
is stored/retrieved as `2017-11-17 01:10:34 UTC` and these values are
considered identical).
+ - Timestamp values _without time zone_ represent a date and time of day
regardless of zone: the time value is independent of zone adjustments
(`2017-11-16 17:10:34` is always retrieved as `2017-11-16 17:10:34`).
3. Character strings must be stored as UTF-8 encoded byte arrays.
For details on how to serialize a schema to JSON, see Appendix C.
@@ -307,12 +311,12 @@ Partition specs capture the transform from table data to
partition values. This
| Transform name | Description
| Source types
| Result type |
|-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------|
| **`identity`** | Source value, unmodified
| Any
| Source type |
-| **`bucket[N]`** | Hash of value, mod `N` (see below)
| `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`,
`string`, `uuid`, `fixed`, `binary` | `int` |
+| **`bucket[N]`** | Hash of value, mod `N` (see below)
| `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`,
`timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int`
|
| **`truncate[W]`** | Value truncated to width `W` (see below)
| `int`, `long`, `decimal`, `string`
| Source type |
-| **`year`** | Extract a date or timestamp year, as years from 1970
| `date`, `timestamp`, `timestamptz`
| `int` |
-| **`month`** | Extract a date or timestamp month, as months from
1970-01-01 | `date`, `timestamp`, `timestamptz`
| `int` |
-| **`day`** | Extract a date or timestamp day, as days from 1970-01-01
| `date`, `timestamp`, `timestamptz`
| `int` |
-| **`hour`** | Extract a timestamp hour, as hours from 1970-01-01
00:00:00 | `timestamp`, `timestamptz`
| `int` |
+| **`year`** | Extract a date or timestamp year, as years from 1970
| `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`
| `int` |
+| **`month`** | Extract a date or timestamp month, as months from
1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`,
`timestamptz_ns` | `int` |
+| **`day`** | Extract a date or timestamp day, as days from 1970-01-01
| `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`
| `int` |
+| **`hour`** | Extract a timestamp hour, as hours from 1970-01-01
00:00:00 | `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`
| `int` |
| **`void`** | Always produces `null`
| Any
| Source type or `int` |
All transforms must return `null` for a `null` input value.
@@ -862,10 +866,12 @@ Maps with non-string keys must use an array
representation with the `map` logica
|**`float`**|`float`||
|**`double`**|`double`||
|**`decimal(P,S)`**|`{ "type": "fixed",`<br /> `"size":
minBytesRequired(P),`<br /> `"logicalType": "decimal",`<br
/> `"precision": P,`<br /> `"scale": S }`|Stored as fixed
using the minimum number of bytes for the given precision.|
-|**`date`**|`{ "type": "int",`<br /> `"logicalType": "date"
}`|Stores days from the 1970-01-01.|
+|**`date`**|`{ "type": "int",`<br /> `"logicalType": "date"
}`|Stores days from 1970-01-01.|
|**`time`**|`{ "type": "long",`<br /> `"logicalType": "time-micros"
}`|Stores microseconds from midnight.|
-|**`timestamp`**|`{ "type": "long",`<br /> `"logicalType":
"timestamp-micros",`<br /> `"adjust-to-utc": false }`|Stores
microseconds from 1970-01-01 00:00:00.000000.|
-|**`timestamptz`**|`{ "type": "long",`<br /> `"logicalType":
"timestamp-micros",`<br /> `"adjust-to-utc": true }`|Stores
microseconds from 1970-01-01 00:00:00.000000 UTC.|
+|**`timestamp`** | `{ "type": "long",`<br /> `"logicalType":
"timestamp-micros",`<br /> `"adjust-to-utc": false }` | Stores
microseconds from 1970-01-01 00:00:00.000000. [1] |
+|**`timestamptz`** | `{ "type": "long",`<br /> `"logicalType":
"timestamp-micros",`<br /> `"adjust-to-utc": true }` | Stores
microseconds from 1970-01-01 00:00:00.000000 UTC. [1] |
+|**`timestamp_ns`** | `{ "type": "long",`<br /> `"logicalType":
"timestamp-nanos",`<br /> `"adjust-to-utc": false }` | Stores
nanoseconds from 1970-01-01 00:00:00.000000000. [1], [2] |
+|**`timestamptz_ns`** | `{ "type": "long",`<br /> `"logicalType":
"timestamp-nanos",`<br /> `"adjust-to-utc": true }` | Stores
nanoseconds from 1970-01-01 00:00:00.000000000 UTC. [1], [2] |
|**`string`**|`string`||
|**`uuid`**|`{ "type": "fixed",`<br /> `"size": 16,`<br
/> `"logicalType": "uuid" }`||
|**`fixed(L)`**|`{ "type": "fixed",`<br /> `"size": L }`||
@@ -874,6 +880,11 @@ Maps with non-string keys must use an array representation
with the `map` logica
|**`list`**|`array`||
|**`map`**|`array` of key-value records, or `map` when keys are strings
(optional).|Array storage must use logical type name `map` and must store
elements that are 2-field records. The first field is a non-null key and the
second field is the value.|
+Notes:
+
+1. Avro type annotation `adjust-to-utc` is an Iceberg convention; default
value is `false` if not present.
+2. Avro logical type `timestamp-nanos` is an Iceberg convention; the Avro
specification does not define this type.
+
**Field IDs**
@@ -908,10 +919,12 @@ Lists must use the [3-level
representation](https://github.com/apache/parquet-fo
| **`float`** | `float`
| |
|
| **`double`** | `double`
| |
|
| **`decimal(P,S)`** | `P <= 9`: `int32`,<br />`P <= 18`: `int64`,<br
/>`fixed` otherwise | `DECIMAL(P,S)` | Fixed must
use the minimum number of bytes that can store `P`. |
-| **`date`** | `int32`
| `DATE` | Stores days from the
1970-01-01. |
+| **`date`** | `int32`
| `DATE` | Stores days from
1970-01-01. |
| **`time`** | `int64`
| `TIME_MICROS` with `adjustToUtc=false` | Stores microseconds
from midnight. |
| **`timestamp`** | `int64`
| `TIMESTAMP_MICROS` with `adjustToUtc=false` | Stores microseconds
from 1970-01-01 00:00:00.000000. |
| **`timestamptz`** | `int64`
| `TIMESTAMP_MICROS` with `adjustToUtc=true` | Stores microseconds
from 1970-01-01 00:00:00.000000 UTC. |
+| **`timestamp_ns`** | `int64`
| `TIMESTAMP_NANOS` with `adjustToUtc=false` | Stores nanoseconds
from 1970-01-01 00:00:00.000000000. |
+| **`timestamptz_ns`** | `int64`
| `TIMESTAMP_NANOS` with `adjustToUtc=true` | Stores nanoseconds
from 1970-01-01 00:00:00.000000000 UTC. |
| **`string`** | `binary`
| `UTF8` | Encoding must be
UTF-8. |
| **`uuid`** | `fixed_len_byte_array[16]`
| `UUID` |
|
| **`fixed(L)`** | `fixed_len_byte_array[L]`
| |
|
@@ -935,8 +948,10 @@ Lists must use the [3-level
representation](https://github.com/apache/parquet-fo
| **`decimal(P,S)`** | `decimal` |
|
|
| **`date`** | `date` |
|
|
| **`time`** | `long` | `iceberg.long-type`=`TIME`
| Stores microseconds from midnight.
|
-| **`timestamp`** | `timestamp` |
| [1]
|
-| **`timestamptz`** | `timestamp_instant` |
| [1]
|
+| **`timestamp`** | `timestamp` |
| Stores microseconds from 2015-01-01 00:00:00.000000. [1],
[2] |
+| **`timestamptz`** | `timestamp_instant` |
| Stores microseconds from 2015-01-01 00:00:00.000000 UTC.
[1], [2] |
+| **`timestamp_ns`** | `timestamp` |
| Stores nanoseconds from 2015-01-01 00:00:00.000000000. [1]
|
+| **`timestamptz_ns`** | `timestamp_instant` |
| Stores nanoseconds from 2015-01-01 00:00:00.000000000 UTC.
[1] |
| **`string`** | `string` |
| ORC `varchar` and `char` would also map to **`string`**.
|
| **`uuid`** | `binary` | `iceberg.binary-type`=`UUID`
|
|
| **`fixed(L)`** | `binary` | `iceberg.binary-type`=`FIXED` &
`iceberg.length`=`L` | The length would not be checked by the ORC reader and
should be checked by the adapter. |
@@ -948,6 +963,7 @@ Lists must use the [3-level
representation](https://github.com/apache/parquet-fo
Notes:
1. ORC's
[TimestampColumnVector](https://orc.apache.org/api/hive-storage-api/org/apache/hadoop/hive/ql/exec/vector/TimestampColumnVector.html)
consists of a time field (milliseconds since epoch) and a nanos field
(nanoseconds within the second). Hence the milliseconds within the second are
reported twice; once in the time field and again in the nanos field. The read
adapter should only use milliseconds within the second from one of these
fields. The write adapter should also report milli [...]
+2. ORC `timestamp` and `timestamp_instant` values store nanosecond precision.
Iceberg ORC writers for Iceberg types `timestamp` and `timestamptz` **must**
truncate nanoseconds to microseconds.
One of the interesting challenges with this is how to map Iceberg’s schema
evolution (id based) on to ORC’s (name based). In theory, we could use
Iceberg’s column ids as the column and field names, but that would be
inconvenient.
@@ -971,8 +987,10 @@ The 32-bit hash implementation is 32-bit Murmur3 hash, x86
variant, seeded with
| **`decimal(P,S)`** | `hashBytes(minBigEndian(unscaled(v)))`[2] | `14.20` ￫
`-500754589` |
| **`date`** | `hashInt(daysFromUnixEpoch(v))` |
`2017-11-16` ￫ `-653330422` |
| **`time`** | `hashLong(microsecsFromMidnight(v))` | `22:31:08`
￫ `-662762989` |
-| **`timestamp`** | `hashLong(microsecsFromUnixEpoch(v))` |
`2017-11-16T22:31:08` ￫ `-2047944441` |
-| **`timestamptz`** | `hashLong(microsecsFromUnixEpoch(v))` |
`2017-11-16T14:31:08-08:00`￫ `-2047944441` |
+| **`timestamp`** | `hashLong(microsecsFromUnixEpoch(v))` |
`2017-11-16T22:31:08` ￫ `-2047944441`<br />`2017-11-16T22:31:08.000001` ￫
`-1207196810` |
+| **`timestamptz`** | `hashLong(microsecsFromUnixEpoch(v))` |
`2017-11-16T14:31:08-08:00` ￫ `-2047944441`<br
/>`2017-11-16T14:31:08.000001-08:00` ￫ `-1207196810` |
+| **`timestamp_ns`** | `hashLong(nanosecsFromUnixEpoch(v))` |
`2017-11-16T22:31:08` ￫ `-737750069`<br />`2017-11-16T22:31:08.000001` ￫
`-976603392`<br />`2017-11-16T22:31:08.000000001` ￫ `-160215926` |
+| **`timestamptz_ns`** | `hashLong(nanosecsFromUnixEpoch(v))` |
`2017-11-16T14:31:08-08:00` ￫ `-737750069`<br
/>`2017-11-16T14:31:08.000001-08:00` ￫ `-976603392`<br
/>`2017-11-16T14:31:08.000000001-08:00` ￫ `-160215926` |
| **`string`** | `hashBytes(utf8Bytes(v))` | `iceberg` ￫
`1210000089` |
| **`uuid`** | `hashBytes(uuidBytes(v))` [3] |
`f79c3e09-677c-4bbd-a479-3f349cb785e7` ￫ `1488055340` |
| **`fixed(L)`** | `hashBytes(v)` | `00 01 02
03` ￫ `-188683207` |
@@ -1018,8 +1036,10 @@ Types are serialized according to this table:
|**`double`**|`JSON string: "double"`|`"double"`|
|**`date`**|`JSON string: "date"`|`"date"`|
|**`time`**|`JSON string: "time"`|`"time"`|
-|**`timestamp without zone`**|`JSON string: "timestamp"`|`"timestamp"`|
-|**`timestamp with zone`**|`JSON string: "timestamptz"`|`"timestamptz"`|
+|**`timestamp, microseconds, without zone`**|`JSON string:
"timestamp"`|`"timestamp"`|
+|**`timestamp, microseconds, with zone`**|`JSON string:
"timestamptz"`|`"timestamptz"`|
+|**`timestamp, nanoseconds, without zone`**|`JSON string:
"timestamp_ns"`|`"timestamp_ns"`|
+|**`timestamp, nanoseconds, with zone`**|`JSON string:
"timestamptz_ns"`|`"timestamptz_ns"`|
|**`string`**|`JSON string: "string"`|`"string"`|
|**`uuid`**|`JSON string: "uuid"`|`"uuid"`|
|**`fixed(L)`**|`JSON string: "fixed[<L>]"`|`"fixed[16]"`|
@@ -1179,8 +1199,10 @@ This serialization scheme is for storing single values
as individual binary valu
| **`double`** | Stored as 8-byte little-endian
|
| **`date`** | Stores days from the 1970-01-01 in an 4-byte
little-endian int |
| **`time`** | Stores microseconds from midnight in an
8-byte little-endian long |
-| **`timestamp without zone`** | Stores microseconds from 1970-01-01
00:00:00.000000 in an 8-byte little-endian long |
-| **`timestamp with zone`** | Stores microseconds from 1970-01-01
00:00:00.000000 UTC in an 8-byte little-endian long |
+| **`timestamp`** | Stores microseconds from 1970-01-01
00:00:00.000000 in an 8-byte little-endian long |
+| **`timestamptz`** | Stores microseconds from 1970-01-01
00:00:00.000000 UTC in an 8-byte little-endian long |
+| **`timestamp_ns`** | Stores nanoseconds from 1970-01-01
00:00:00.000000000 in an 8-byte little-endian long |
+| **`timestamptz_ns`** | Stores nanoseconds from 1970-01-01
00:00:00.000000000 UTC in an 8-byte little-endian long |
| **`string`** | UTF-8 bytes (without length)
|
| **`uuid`** | 16-byte big-endian value, see example in
Appendix B |
| **`fixed(L)`** | Binary value
|
@@ -1206,6 +1228,8 @@ This serialization scheme is for storing single values as
individual binary valu
| **`time`** | **`JSON string`** |
`"22:31:08.123456"` | Stores ISO-8601 standard time with
microsecond precision |
| **`timestamp`** | **`JSON string`** |
`"2017-11-16T22:31:08.123456"` | Stores ISO-8601 standard timestamp
with microsecond precision; must not include a zone offset |
| **`timestamptz`** | **`JSON string`** |
`"2017-11-16T22:31:08.123456+00:00"` | Stores ISO-8601 standard timestamp
with microsecond precision; must include a zone offset and it must be '+00:00' |
+| **`timestamp_ns`** | **`JSON string`** |
`"2017-11-16T22:31:08.123456789"` | Stores ISO-8601 standard timestamp
with nanosecond precision; must not include a zone offset |
+| **`timestamptz_ns`** | **`JSON string`** |
`"2017-11-16T22:31:08.123456789+00:00"` | Stores ISO-8601 standard timestamp
with nanosecond precision; must include a zone offset and it must be '+00:00' |
| **`string`** | **`JSON string`** | `"iceberg"`
| |
| **`uuid`** | **`JSON string`** |
`"f79c3e09-677c-4bbd-a479-3f349cb785e7"` | Stores the lowercase uuid string |
| **`fixed(L)`** | **`JSON string`** |
`"000102ff"` | Stored as a hexadecimal string |
@@ -1223,6 +1247,8 @@ Default values are added to struct fields in v3.
* The `write-default` is a forward-compatible change because it is only used
at write time. Old writers will fail because the field is missing.
* Tables with `initial-default` will be read correctly by older readers if
`initial-default` is always null for optional fields. Otherwise, old readers
will default optional columns with null. Old readers will fail to read required
fields which are populated by `initial-default` because that default is not
supported.
+Types `timestamp_ns` and `timestamptz_ns` are added in v3.
+
### Version 2
Writing v1 metadata:
