Sent too early. Here are the numbers from our experiments:
dbp bss entropy
Scenario 64+32 +zstd bound
sorted 1µs gaps 1.64 2.03 1.56
sorted 1ms gaps 2.89 3.09 2.80
sorted 1s gaps 4.14 4.28 4.05
1ms, ms-quant 2.88 1.89 0.24
1s, s-quant 4.13 2.27 0.24
batched 1k/s 0.92 0.013 0.005
unsorted 1h 5.39 5.37 5.32
DBPs gap on quantized timestamps can be solved by either the writer
choosing MS or S precision at write time or by proposing an additional
encoding for timestamp encoding to trim the low zeros.
On Fri, Jul 10, 2026 at 9:37 AM Alkis Evlogimenos <
[email protected]> wrote:
> The 12-byte ints were chosen because they are enough to cover the SQL
> range with up to femtosecs precision. This saves 4 bytes per timestamp if
> they are encoded PLAIN.
>
> Internally we have done some testing with BYTE_STREAM_SPLIT and DBP for
> compressing timestamps.
>
> BYTE_STREAM_SPLIT is unimpressive. It improves a general compressor's
> compression ratio but general compression is generally very slow. The
> surprising findings is how good DBP is for timestamps.
>
> DBP (64 bottom, 32 top) unmodified is SOTA for some distributions.
> DBP modified (trimming low zeros) is SOTA for all distributions we tried.
>
> Considering the above the next logical proposal is to define how FLBA can
> be encoded with DBP and keep timestamps to 12-bytes to avoid bloat when
> engines choose PLAIN.
>
>
> On Fri, Jul 10, 2026 at 9:15 AM Antoine Pitrou <[email protected]> wrote:
>
>>
>> Regarding the choice of 12-byte ints rather than 16-byte: this is purely
>> a storage footprint optimization, right? Or do some readers also plan to
>> read these as 12-byte ints in memory?
>>
>> If it's purely a footprint optimization, I'll point that 16-byte ints
>> with 0s in the top 4 bytes should compress pretty well, especially using
>> BYTE_STREAM_SPLIT.
>>
>>
>> By the way, DECIMAL can already annotate a FLBA<N> with an open-ended N.
>> Perhaps that should be the solution here too, rather than mandating a
>> particular byte width?
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 10/07/2026 à 07:28, Micah Kornfield a écrit :
>> > Hi Divjot,
>> > I reviewed the doc and it seems like a pragmatic solution to a long
>> > standing gap in parquet.
>> >
>> > Let's maybe what until next week to collect feedback and if there are no
>> > objections we can hopefully start on POCs.
>> >
>> > Cheers,
>> > Micah
>> >
>> > On Wednesday, July 8, 2026, Divjot Arora via dev <
>> [email protected]>
>> > wrote:
>> >
>> >> Hi folks,
>> >>
>> >> We're interested in supporting the full SQL timestamp range (year 0001
>> to
>> >> 9999) in nanosecond precision. Parquet currently cannot do this via
>> INT64 +
>> >> the TimestampType logical type, so I'm sharing a proposal to allow the
>> >> TimestampType annotation on FIXED_LEN_BYTE_ARRAY of size 12 (96-bits):
>> >> https://docs.google.com/document/d/1H43RQZhWKcg9c4tJO5W87YxhbUxN7
>> >> qGSFxJBrrwWOfY/edit?tab=t.0.
>> >> Please take a look if interested and feel free to leave comments on the
>> >> document.
>> >>
>> >> Best,
>> >> Divjot Arora
>> >>
>> >
>>
>>
>>
