+1 On Fri, May 15, 2026 at 9:15 AM Max Gekk <[email protected]> wrote:
> Hi Spark devs, > > I would like to call a vote on adopting the following SPIP as an official > Spark SPIP, following the discussion on the list. > Motivation > > Spark SQL today exposes TIMESTAMP / TIMESTAMP_NTZ / TIMESTAMP_LTZ at > microsecond precision. Nanosecond timestamps are increasingly common in ORC > and in ecosystems such as Oracle, MS SQL Server, Trino, Snowflake, and > DuckDB. Spark often rejects Parquet TIMESTAMP(NANOS, …) or, with legacy > flags, reads it as LongType, which drops timestamp semantics and time-zone > behavior. Users are forced to pre-normalize to micros or maintain custom > conversion logic. > > This SPIP aims to add first-class nanosecond-capable timestamps in SQL and > APIs, with a clear fractional precision parameter, while keeping a > practical migration path for existing microsecond workloads. > Proposal (summary) > > - > > SQL surface: TIMESTAMP_NTZ(n), TIMESTAMP_LTZ(n), and TIMESTAMP(n) > (including WITHOUT TIME ZONE / WITH LOCAL TIME ZONE spellings), with > optional n in [0, 9]. The scope for the initial milestone focuses on 7 <= n > <= 9; unparameterized types keep today’s defaults. > - > > New Catalyst types: additive parameterized types (e.g. > TimestampNTZNanosType(n) / companion for LTZ) rather than silently widening > existing singleton types. > - > > Logical value model: epochMicros (long) + nanosOfMicro in [0, 999] > (short) with a single normalization rule, so Spark continues to build on > the existing microsecond datetime “currency” while adding a bounded > sub-micro correction. > - > > Scope boundaries: explicit non-goals (e.g. not redefining NTZ/LTZ > session semantics, not promising every connector end-to-end in v1), > risks/mitigations, rough ~25 person-week estimate and success “exams” are > documented in the SPIP and JIRA. > > Full detail, API tables, rejected alternatives, and test/migration > criteria are in the SPIP document and JIRA description. > > Relevant links > > - > > SPIP (Google Doc): > > https://docs.google.com/document/d/1DeW15QueI4PdRyPm6C6jsTZFmIjbXX2j4h-Ja5W_fsg/edit?usp=sharing > - > > Discussion thread: > https://lists.apache.org/thread/xvv4qt9dpnb1kszxdqlxyv9b46749ypo > - > > JIRA: https://issues.apache.org/jira/browse/SPARK-56822 > > Vote > > Please vote on accepting this proposal as an official SPIP (the SPIP text > above; implementation and follow-up JIRAs can land incrementally after > acceptance). > > [ ] +1: Accept the proposal as an official SPIP > > [ ] +0: No opinion > > [ ] -1: I do not think we should adopt this SPIP (please explain why) > > The vote will remain open for at least 72 hours. > > Thanks to everyone who commented on the discussion thread and helped > refine the design. > > Best regards, > Max Gekk >
