Hi Divjot, I've looked and left comments. The spec changes look reasonable. Unless there are any objections I would propose we make a second implementation for both, and try vote on the changes.
Thanks, Micah On Thu, Aug 6, 2026 at 8:18 AM Divjot Arora via dev <[email protected]> wrote: > Hi, > > After discussing with Micah, I've split out the spec change to describe how > unsupported logical/physical type > combinations should be handled by readers (TLDR the logical type and stats > should be dropped). I've opened > a parquet-format PR [1] and parquet-java PR [2] for the spec change and > reference implementation. The previous > spec change [3] and implementation are still open and ready for review. > Please take a look if interested. > > Best, > Div > > [1] https://github.com/apache/parquet-format/pull/606 > [2] https://github.com/apache/parquet-java/pull/3711 > [3] https://github.com/apache/parquet-format/pull/601 > [4] https://github.com/apache/parquet-java/pull/3680 > > On Tue, Jul 21, 2026 at 7:42 PM Divjot Arora <[email protected]> > wrote: > > > Hi folks, > > > > I've opened PRs for parquet-format [1] and parquet-java [2] to support > the > > FLBA(12) physical type for the TimestampType logical annotation. Please > > take a look if interested and comment. > > > > Best, > > Div Arora > > > > [1] https://github.com/apache/parquet-format/pull/601 > > [2] https://github.com/apache/parquet-java/pull/3680 > > > > On Thu, Jul 16, 2026 at 4:16 PM Russell Spitzer < > [email protected]> > > wrote: > > > >> I'd lean toward choosing a bit width and going forward for similar > >> reasons. > >> I don't think a lot > >> of engines/users are going to custom-define their bit-width ahead of > time. > >> It seems more > >> likely if we wanted to benefit readers, we would just store the max > >> used-bit width as a stat. > >> > >> On Thu, Jul 16, 2026 at 12:30 AM Micah Kornfield <[email protected] > > > >> wrote: > >> > >> > Being able to use compact data types is a good thing, no? :-) > >> > > >> > > >> > Generally yes, but I think there is a difference here between Decimal > >> and > >> > Timestamp. Decimal a user is specifying precision and scale up front, > >> so we > >> > can calculate the exact FLBA width. The closest thing we have with > >> > timestamp is for SQL purposes it has to cover a 10K year range. But in > >> > general, we won't know the exact min and max value up front. > >> > > >> > For existing granularities of milliseconds and microseconds there is > not > >> > any benefit to using FLBA (users should use existing physical integer > >> > values). Extra engineering effort for these granularities doesn't > seem > >> > worthwhile. > >> > > >> > For nanoseconds, I believe we could use the FLBA<8> type. However, > some > >> > systems are starting to move toward picoseconds [1][2] (and I've heard > >> > femtoseconds discussed) so we will eventually want to support all of > >> those > >> > ranges. FLBA<9> allows us to do it all with the same code. > >> > > >> > I think extending the spec to support different size types is > certainly > >> > something we could do in the future, but doing it now seems like it is > >> just > >> > more code to maintain without a concrete use-case. > >> > > >> > Cheers, > >> > Micah > >> > > >> > [1] > >> > > >> > > >> > https://docs.cloud.google.com/bigquery/docs/reference/storage/rpc/google.cloud.bigquery.storage.v1?content_ref=read%20api%20will%20return%20full%20precision%20picosecond%20value%20the%20value%20will%20be%20encoded%20as%20a%20string%20which%20conforms%20to%20iso%208601%20format#picostimestampprecision > >> > [2] https://trino.io/docs/current/language/types.html#timestamp-p > >> > > >> > On Fri, Jul 10, 2026 at 2:38 AM Antoine Pitrou <[email protected]> > >> wrote: > >> > > >> > > > >> > > Le 10/07/2026 à 11:27, Divjot Arora via dev a écrit : > >> > > >> Yeah, well, regardless, this is derailing the discussion quite a > >> bit. > >> > > > This proposal is not about extending DBP to FLBA. Can we keep it > on > >> > > topic? > >> > > > > >> > > > Agreed, let's keep this discussion focused to extended-precision > >> > > > timestamps. We can have separate discussions on encodings.. > >> > > > > >> > > >> And conversely, I do not see what we get out of mandating > FLBA<12>. > >> > :-) > >> > > > > >> > > > FLBA<N> works, but I would be concerned about added complexity on > >> the > >> > > > reader side as I assume engines will want to use compact data > types > >> to > >> > > > represent timestamps as much as possible, so they'd have to manage > >> > > > different data types for different values of N. > >> > > > >> > > Being able to use compact data types is a good thing, no? :-) > >> > > > >> > > The good news is that some readers already have similar code for > >> reading > >> > > Decimals from arbitrary FLBA<N>. For example, Parquet C++ can read > >> > > Decimal-annotated FLBA<N> into Arrow Decimal128 or Decimal256, > >> depending > >> > > on the advertised precision. > >> > > > >> > > (and the code to do that is reasonably simple: you're really > >> > > sign-extending from one integer width to another) > >> > > > >> > > Regards > >> > > > >> > > Antoine. > >> > > > >> > > > >> > > > >> > > >> > > >
