+1 On May 15, 2026, at 6:07 PM, Stefan Kandic via dev <[email protected]> wrote:
+1 From: Max Gekk <[email protected]> Date: Friday, 15 May 2026 at 09:53 To: dev <[email protected]> Subject: Re: [VOTE] SPIP: Timestamps with nanosecond precision +1 On Fri, May 15, 2026 at 9:15 AM Max Gekk <[email protected]<mailto:[email protected]>> wrote: Hi Spark devs, I would like to call a vote on adopting the following SPIP as an official Spark SPIP, following the discussion on the list. Motivation Spark SQL today exposes TIMESTAMP / TIMESTAMP_NTZ / TIMESTAMP_LTZ at microsecond precision. Nanosecond timestamps are increasingly common in ORC and in ecosystems such as Oracle, MS SQL Server, Trino, Snowflake, and DuckDB. Spark often rejects Parquet TIMESTAMP(NANOS, …) or, with legacy flags, reads it as LongType, which drops timestamp semantics and time-zone behavior. Users are forced to pre-normalize to micros or maintain custom conversion logic. This SPIP aims to add first-class nanosecond-capable timestamps in SQL and APIs, with a clear fractional precision parameter, while keeping a practical migration path for existing microsecond workloads. Proposal (summary) * SQL surface: TIMESTAMP_NTZ(n), TIMESTAMP_LTZ(n), and TIMESTAMP(n) (including WITHOUT TIME ZONE / WITH LOCAL TIME ZONE spellings), with optional n in [0, 9]. The scope for the initial milestone focuses on 7 <= n <= 9; unparameterized types keep today’s defaults. * New Catalyst types: additive parameterized types (e.g. TimestampNTZNanosType(n) / companion for LTZ) rather than silently widening existing singleton types. * Logical value model: epochMicros (long) + nanosOfMicro in [0, 999] (short) with a single normalization rule, so Spark continues to build on the existing microsecond datetime “currency” while adding a bounded sub-micro correction. * Scope boundaries: explicit non-goals (e.g. not redefining NTZ/LTZ session semantics, not promising every connector end-to-end in v1), risks/mitigations, rough ~25 person-week estimate and success “exams” are documented in the SPIP and JIRA. Full detail, API tables, rejected alternatives, and test/migration criteria are in the SPIP document and JIRA description. Relevant links * SPIP (Google Doc): https://docs.google.com/document/d/1DeW15QueI4PdRyPm6C6jsTZFmIjbXX2j4h-Ja5W_fsg/edit?usp=sharing * Discussion thread: https://lists.apache.org/thread/xvv4qt9dpnb1kszxdqlxyv9b46749ypo * JIRA: https://issues.apache.org/jira/browse/SPARK-56822 Vote Please vote on accepting this proposal as an official SPIP (the SPIP text above; implementation and follow-up JIRAs can land incrementally after acceptance). [ ] +1: Accept the proposal as an official SPIP [ ] +0: No opinion [ ] -1: I do not think we should adopt this SPIP (please explain why) The vote will remain open for at least 72 hours. Thanks to everyone who commented on the discussion thread and helped refine the design. Best regards, Max Gekk
