Hi Thomas,
Thank you for the detailed response. I think we are close to agreement on
the requirements. The main remaining question is whether canonicality
should be represented as a property of the logical type or as a property of
the encoded data.
We agree on the following:
- Precision should be parameterized rather than fixed at 38, allowing
narrower encodings and a defined path to wider precisions.
- The type should cover the complete finite decimal128 range.
- At minimum, positive and negative infinity and a canonical NaN should be
supported.
- Logical equality, hashing, Bloom filters, ordering, and statistics must
remain consistent regardless of the stored representation.
- The physical representation should be selected only after comparing
candidates under equivalent semantics through public benchmarks and
validation vectors.
We do not think normalization should parameterize the logical type solely
to expose an optimization opportunity. Members of the same finite cohort,
such as 1.5 and 1.50, have the same numerical identity and must behave
consistently for equality, hashing, grouping, ordering, and Bloom filters.
At the same time, a Parquet writer that is supplied with a particular
coefficient and exponent should preserve that representation.
These requirements do not conflict. Implementations can derive a
cohort-invariant numerical key for equality and hashing without rewriting
the stored value. Canonicalizing that derived key is always safe; replacing
the stored value 1.50 with 1.5 is not safe when quantum must round-trip.
An engine may expose only numerical semantics and therefore supply
already-canonical values to the writer, but that should not create a second
Parquet logical type. Once a DECFLOAT value with a particular coefficient
and exponent is supplied, a conforming writer must preserve it.
At the same time, we agree with your concern that preserving cohorts can
add work to equality and hash-based operations. That work can be avoided
when the encoded data is known to satisfy stronger invariants, without
making those invariants part of the logical type.
We therefore suggest defining content-derived properties that writers can
record at the file or row-group level (or another agreed physical scope).
These could indicate, for example, that:
1. every finite value uses the defined cohort-canonical representation
2. no negative zero occurs;
3. no signaling NaN occurs; or
4. only the canonical NaN occurs.
Implementations could inspect these properties before decoding the column
data and select an optimized path when safe. For example, if every finite
value has a unique canonical encoding and the relevant special cases are
absent, equality and hashing may avoid per-value canonicalization. If the
properties are false, absent or unknown, implementations must use the
general numerical path performing canonicalization. The properties only
inform optimization; they do not change the values’ logical behavior.
There remains an important distinction between observing that a particular
file or row group contains only canonical values and enforcing that a
column may never contain anything else. A mandatory canonical mode would
prevent representational round-tripping even when the source engine
retained the necessary information. A content-derived property provides the
optimized path whenever the data already satisfies the invariant without
changing the logical type or authorizing information loss.
This requires defining new metadata or statistics, including their precise
meaning, scope, and writer requirements. Concretely, a writer could compute
the property as follows:
allValuesCanonical = true
For each value:
if value is not canonical:
allValuesCanonical = false
This flag at the end, stores one of three states:
1. true: every value is normalized
2. false: at least one value is not normalized
3. absent/unknown: the writer did not establish the property
For example:
File A: 1.5, 2.7, 3.9
allValuesCanonical = true
File B: 1.5, 1.50, 3.9
allValuesCanonical = false
This could be stored in one of these places:
1. Table/column: one property for the entire table
2. File: each Parquet file reports its own property
3. Row group: different parts of a file can report different properties
4. Page: even finer-grained, but with greater metadata overhead
We suggest capturing the remaining requirements in a small decision matrix:
1. Supported precision values and range.
2. The representation and round-trip contract for finite cohorts and
special values.
3. Numerical equality and normative canonical-key semantics.
4. Content-derived canonicality properties, including their scope and
missing/unknown behavior.
5. Whether canonicality is an observed property or an enforced invariant.
6. Statistics and numerical ordering.
7. Invalid, reserved, and non-canonical encoding handling.
8. Required benchmarks, validation vectors, and physical-format artifacts.
Once that matrix converges, we would be happy to work jointly on the
benchmark and validation suite. At that point, the candidate physical
representations can be evaluated against the same requirements rather than
embedding different semantics in each comparison.
Best,
Costas
On Tue, Aug 25, 2026 at 6:50 AM Thomas Kissinger via dev <
[email protected]> wrote:
> Hi Costas,
>
> Thank you for the thoughtful reply and for separating the requirements from
> the physical representation. Starting with the requirements is the right
> approach, and we are confident that we can find a solution covering the
> important use cases on both sides.
>
> Thank you also for sharing the Spark SPIP. Seeing the proposed engine
> semantics alongside the Parquet representation is very helpful. Although
> Spark and Parquet have separate decision processes, both projects would
> benefit if these efforts converge on shared requirements rather than
> develop alongside each other with different assumptions. In particular,
> aligning on precision and range, which numerical and representational
> distinctions must survive a round-trip, and which special values are
> supported before either design is finalized would avoid unnecessary
> conversion and interoperability gaps.
>
> Our current view is the following.
> Precision and range
>
> We agree that the type should not impose a permanent 38-digit ceiling and
> should cover at least the complete finite decimal128 range. Narrower and
> wider precisions should be possible, with the exact supported set decided
> when evaluating the physical representations.
>
> This requirement also exposes a mismatch with the generalized IEEE
> interchange layout. Its precision ladder moves from 34 to 43 digits around
> the established 38-digit SQL boundary, and from 70 to 79 around the
> 76-digit boundary. Current Intel and Boost implementations stop at
> decimal128, and we are not aware of an interoperable production decimal160
> codec. Signed 128- and 256-bit integer significands support precisions 38
> and 76 directly, aligning more naturally with existing database decimal
> representations.
> Quantum, equality, and hashing
>
> We agree that some applications need to preserve the original quantum, and
> FHIR is a useful example. The question is not whether this capability
> should exist, but whether it should be mandatory for every column. For
> workloads that require only ordinary numerical semantics, preserving
> cohorts adds work to equality and hash-based operations without providing
> additional value.
>
> When cohort members are retained, physical identity and numerical identity
> diverge. Equality checks must handle multiple encodings of the same value,
> while hash joins, grouping, DISTINCT, and partitioning must derive a
> cohort- invariant numerical key for every processed value or maintain an
> additional canonical key. That adds work and state to fundamental database
> paths. Because Parquet Bloom filters currently hash the PLAIN encoding, a
> quantum-preserving mode would require a normative cohort-invariant key
> shared by writers and readers; any disagreement could cause incorrect
> pruning.
>
> Preserving cohort members also increases physical cardinality: numerically
> equal values may be stored as different byte sequences. This can create
> additional dictionary entries and reduce repeated byte patterns available
> to compression. The actual size impact will depend on the data distribution
> and should be included in the benchmark.
>
> We therefore propose making quantum preservation an explicit parameter of
> the logical type, with canonical representation as the default. A canonical
> mode would require one representation per finite numerical value, while a
> quantum- preserving mode would retain the supplied exponent for
> applications requiring faithful round-tripping. Both modes would use
> ordinary numerical equality; the parameter would define the representation
> round-trip contract. This makes the semantic and performance tradeoff
> explicit rather than imposing either choice on every consumer.
> Special values
>
> We agree on positive and negative infinity and a canonical NaN. We are less
> convinced that signed zero, signaling NaNs, and NaN payloads belong in an
> interoperable Parquet type, since many important target type systems cannot
> represent them reliably.
>
> Signaling NaN is difficult to reconcile with an interoperable columnar
> format because its behavior depends on how a value is inspected. IEEE
> numerical comparisons signal invalid operation for an sNaN, while
> totalOrder, which is appropriate for statistics, does not. Parquet uses
> values through several paths—statistics and pruning, dictionary processing,
> and row-level predicate evaluation—that may therefore treat the same sNaN
> differently. With traps enabled, observable query behavior could even
> depend on whether an optimization path is used.
>
> Supporting sNaN would consequently require precise rules for statistics,
> pruning, predicates, dictionaries, quieting, and round-tripping. That is
> substantial complexity for an execution mechanism without a demonstrated
> Parquet interoperability requirement. In the exchange-rate example, NULL or
> an explicit state column would express missing or invalid input more
> predictably across engines.
>
> Signed zero creates a similar interoperability problem to signaling NaN. It
> introduces two encodings for the same numerical zero and therefore requires
> special treatment in equality, hashing, Bloom filters, ordering, and
> statistics. Many decimal type systems and conversion paths do not preserve
> the distinction and silently turn negative zero into ordinary zero. A
> downstream consumer relying on the sign may then observe different behavior
> without any indication that information was lost. This is misleading in an
> interoperable format, so unless a compelling use case is identified, we
> prefer one canonical zero.
>
> We do not think NaN payloads should be included. IEEE defines their
> representation but not interoperable payload semantics, and does not fully
> specify payload selection when multiple NaNs participate in an operation.
> Parquet could preserve the bytes, but that would impose a round-trip
> obligation for opaque metadata that receiving engines cannot interpret
> consistently. Diagnostic information is more portable and queryable in a
> separate column.
> Performance and benchmarks
>
> We agree that, once the requirements are finalized, we should work together
> on a public benchmark comparing equivalent semantics and documenting
> distributions, rounding behavior, fast paths, fallbacks, compiler settings,
> and hardware. It should cover fundamental database operations, including
> encoding, decoding, cohort reduction and canonical-key derivation,
> equality, and arithmetic.
>
> Until the existing measurements can be reproduced under equivalent
> semantics, they should be treated as directional evidence rather than a
> representation-level result. Please feel free to propose an initial
> benchmark design already; it could provide a useful foundation for the
> joint version.
> Storage and runtime representation
>
> We agree that engines can decode the Parquet representation once and use a
> different runtime representation. This amortizes conversion across long
> pipelines but does not eliminate representation-dependent work. Values
> surviving pruning must still be interpreted or decoded for row-level
> filters, numerical hashing, statistics construction on write, or runtime
> materialization. The physical representation therefore remains relevant to
> scan-heavy workloads.
> Validation
>
> We likewise agree that, once the requirements are finalized, we should
> jointly develop a public validation suite independently of the performance
> benchmark. Please feel free to propose an initial set of test vectors
> already as a foundation. It should eventually cover canonical and
> quantum-preserving values, precision boundaries, special values,
> non-canonical, reserved, and invalid encodings as applicable, equality,
> hashing, ordering and statistics, and cross-implementation round-trips.
>
> Thanks again for moving the discussion toward explicit requirements. We
> already agree on many important points: support for at least 38 digits with
> a path to wider precisions, the decimal128 finite range, infinities and a
> canonical NaN, and public benchmarks and test vectors. The remaining
> discussion is focused on whether quantum preservation should be optional,
> which additional IEEE distinctions belong in an interoperable database
> type, and which physical representation best satisfies the agreed
> requirements. We believe an extensible type with an explicit
> quantum-preservation choice offers a practical path forward.
>
> Best, Thomas
>
>
> On Fri, Aug 14, 2026 at 10:38 PM Costas Zarifis via dev <
> [email protected]> wrote:
>
> > Hi Thomas,
> >
> > Thank you for laying out the requirements explicitly. We agree that
> > requirements should come before selecting a physical representation.
> > There is substantial agreement, but we see several requirements
> > differently.
> >
> > For context on engine integration, we have also published a draft
> > Spark Project Improvement Proposal (SPIP): Add the DECFLOAT data type
> > (
> >
> https://docs.google.com/document/d/1mTSOoCvX7yYMSa6CpI6OPS-MQZocPxUtrrFOEiZjCPk/edit
> > ).
> > The SPIP covers the proposed Spark SQL type and execution semantics,
> > while this discussion focuses on the portable Parquet representation.
> >
> > Before choosing BID or a new 18-byte format, we suggest first agreeing
> > on the following:
> >
> > - Precision: We agree that the format must losslessly support 38-digit
> > values. Your proposal targets exactly 38 digits, while we think the
> > format should provide a defined path to both narrower and wider
> > precisions.
> > - Range: We agree that the format should cover at least the complete
> > finite decimal128 range.
> > - Cohorts: Your proposal requires one canonical representation per
> > numerical value. We think the original coefficient, exponent, and
> > quantum should round-trip.
> > - Special values: We agree on infinity and NaN. We additionally think
> > engines supporting signed zero, signaling NaNs, and NaN payloads
> > should be able to round-trip them.
> > - Hashing and equality: We agree that numerically equal values must
> > behave consistently in equality and hash-based operations. We do not
> > think that requires canonicalizing the stored representation.
> > - Performance: We take the reported results seriously, but think the
> > candidate representations should be compared using equivalent
> > semantics, public benchmarks, and independent implementations.
> > - Validation: Correctness and interoperability should be validated
> > using public test vectors, separately from performance evaluation.
> >
> > The sections below provide the rationale for each position.
> >
> > Precision and range
> > ---------------------------
> >
> > We agree that a Parquet decimal-floating-point type must losslessly
> > represent every numerical value in the DECIMAL(38,s) domain.
> >
> > We do not think the format should necessarily stop at exactly 38
> > digits, however. Precision 38 is an important interoperability
> > boundary established by an original crop of DBMS, which does not
> > establish that present or future requirements stop at 38 digits.
> >
> > Many open source or major newer DBMS have higher limits:
> >
> > - PostgreSQL
> > Type: NUMERIC unconstrained
> > Max precision: 131,072 before decimal / 16,383 after; declared
> > NUMERIC(p,s) up to p=1000
> > Source: https://www.postgresql.org/docs/current/datatype-numeric.html
> >
> > - MySQL / MariaDB / SingleStore
> > Type: DECIMAL
> > Max precision: 65
> > Source: https://dev.mysql.com/doc/refman/8.0/en/fixed-point-types.html
> >
> > - BigQuery
> > Type: BIGNUMERIC
> > Max precision: ~76.8 (scale 38)
> > Source:
> > https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types
> >
> > - ClickHouse
> > Type: Decimal256
> > Max precision: 76
> > Source: https://clickhouse.com/docs/sql-reference/data-types/decimal
> >
> > - Oracle
> > Type: unconstrained NUMBER
> > Max precision: ~40 (parameterized 38)
> > Source:
> >
> https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlqr/Data-Types.html
> >
> > - Trino ≥480
> > Type: NUMBER
> > Max precision: >= 50
> > Source: https://trino.io/docs/current/language/types.html#number
> >
> > While we can opine on the value of precision past a certain limit, we
> > cannot pass judgement on whether the cut-off is 34, 38, 43 or
> > something higher.
> >
> > Therefore Parquet should avoid making any such precision a permanent
> > semantic ceiling.
> >
> > Faithful round-tripping and cohorts
> > ----------------------------------------------
> >
> > We do not agree that Parquet should require one cohort-canonical
> > stored representation for every finite numerical value.
> >
> > Although 1.5, 1.50, and 1.500 are numerically equal, their different
> > quantum can carry information that some applications must preserve.
> > Engines can provide numerical operations alongside IEEE-defined
> > cohort-sensitive operations such as sameQuantum and totalOrder, the
> > latter often exposed as compareTotal. These operations are only
> > possible if the storage format retains the original representation.
> > Canonicalizing on write permanently removes that option.
> >
> > A concrete example is HL7 FHIR healthcare data
> > (https://hl7.org/fhir/datatypes.html). FHIR explicitly specifies that
> > decimal precision is significant: 0.010 is different from 0.01, and
> > implementations must preserve that distinction. This requirement
> > concerns faithful interchange and presentation rather than numerical
> > equality: 0.010 and 0.01 have the same numerical value, but FHIR
> > requires their represented precision to survive a round-trip. A
> > healthcare data lake using Parquet as an interchange or archival
> > format must therefore retain the original quantum to reconstruct
> > standards-compliant FHIR resources. Canonicalizing on write would make
> > that round-trip impossible.
> >
> > Beyond FHIR, this behavior appears in other decimal arithmetic
> > ecosystems. ISO/IEC TS 18661-2
> > (https://open-std.org/JTC1/SC22/WG14/www/docs/n1912.pdf), which
> > specifies C language support for decimal floating-point arithmetic,
> > explicitly states that 1.0 and 1.00 are distinguishable. The
> > language-independent General Decimal Arithmetic specification
> > (https://speleotrove.com/decimal/daops.html), which underpins several
> > decimal arithmetic libraries, preserves trailing zeros unless the
> > caller explicitly requests `reduce`. The same principle appears
> > independently in measurement and reporting standards. ISO 80000-1
> > (
> >
> https://www.iso.org/home.isoDocumentsDownload.do?t=0bc30Dw3Fis0Kmeb_a6O_bzO2QP14DxJAxwtmaRLmv-WwspRuGQ-iHB6VgCGRJnU
> > ),
> > NIST SP 811 (
> >
> https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-7-rules-and-style-conventions-expressing-values
> > ),
> > and ASTM E29 (
> > https://www.galvanizeit.com/uploads/resources/ASTM-E-29-yr-13.pdf)
> > all treat trailing zeros after a decimal point as significant in
> > reported values.
> >
> > The Bloom-filter example identifies a real correctness concern, but
> > optional Bloom-filter support should not require writers to discard
> > quantum information from stored values. If Bloom-filter support is
> > deemed necessary, Parquet could canonicalize only the numerical hash
> > key.
> >
> > For example:
> >
> > Stored representation Numerical Bloom-filter key
> > 1.5 = (15, -1) (15, -1)
> > 1.50 = (150, -2) (15, -1)
> > 1.500 = (1500, -3) (15, -1)
> >
> > Writers and readers would hash the same normalized numerical form.
> > This keeps numerical-equality Bloom filters correct without changing
> > the stored representation. Otherwise, readers could disable
> > Bloom-filter pruning for DECFLOAT equality.
> >
> > Special values
> > -------------------
> >
> > We agree that positive and negative infinity and NaN should be
> > representable. We think the same faithful-round-trip principle should
> > apply to signed zero, signaling NaNs, and NaN payloads.
> >
> > An engine does not need to expose every distinction to its users. It
> > may normalize or reject some values, or provide traps, similar to
> > Firebird’s SET DECFLOAT TRAPS
> > (
> >
> https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref50-management.html
> > ),
> > controlling whether division by zero, overflow, invalid operations, or
> > signaling NaN produce a value or raise an error. Those are execution
> > policies.
> >
> > A concrete example of signaling NaN’s automatic fail-on-use behavior
> > is a bank preparing its daily foreign-exchange rates. Each currency
> > pair must receive a rate from a market-data provider before trades can
> > be valued. The bank could initialize rates that have not yet arrived
> > to signaling NaN and periodically checkpoint the table to Parquet:
> >
> > EUR → USD = 1.17
> > GBP → USD = 1.35
> > JPY → USD = sNaN
> >
> > The valuation job enables the InvalidOperation trap. If it
> > accidentally attempts to convert a JPY payment before the rate has
> > arrived, multiplying the payment by sNaN fails immediately at the
> > point of use. Merely copying, checkpointing, or inspecting the table
> > does not trigger the exception. A NaN payload could additionally
> > retain a diagnostic code identifying the missing or invalid data
> > source.
> >
> > Parquet should reproduce the values written to it rather than
> > normalize away supported distinctions. If it converts sNaN to quiet
> > NaN, removes NaN payloads, or collapses -0 into +0, engines that
> > support those values can no longer round-trip them faithfully. Whether
> > an engine exposes or uses these distinctions should remain an
> > engine-level decision; the storage format should preserve them rather
> > than make an irreversible decision for every consumer.
> >
> > Performance
> > ------------------
> >
> > We take the reported performance differences seriously, but the
> > current results do not isolate the physical representation. They
> > compare Intel libbid with an optimized Snowflake implementation while
> > potentially also varying precision, semantics, algorithms, input
> > distributions, and fast-path behavior.
> >
> > Our own benchmarks reinforce this concern: libbid and Boost.Decimal
> > both use BID, yet differ by several times on some operations. Large
> > performance gaps can therefore arise from implementation choices even
> > when the representation is identical. This does not rule out an
> > inherent advantage in Snowflake’s representation, but the current
> > comparison does not isolate it.
> >
> > Could you share the implementation or enough of the benchmark harness
> > to reproduce the results, including:
> >
> > - input and exponent distributions
> > - precision and rounding configuration
> > - treatment of cohorts and special values
> > - the percentage of operations using the optimized path versus fallback
> > - compiler and optimization settings
> > - results on both ARM and x86?
> >
> > We should compare equivalent value domains and semantics. For Parquet,
> > we should also measure encoding, decoding, comparisons, numerical
> > hashing, statistics, predicate evaluation, compression, and end-to-end
> > scan and aggregation performance, not only arithmetic.
> >
> > We should also separate the Parquet encoding from an engine’s runtime
> > representation. An engine could decode BID once into its preferred
> > coefficient/exponent structure, execute an expression pipeline using
> > that representation, and encode only at the output boundary. A
> > benchmark comparing that approach would help determine whether the
> > reported cost is inherent to the interchange encoding or to using
> > packed BID for every intermediate result.
> >
> > Once those requirements are settled, we can evaluate the candidate
> > representations against the same semantics using reproducible
> > benchmarks.
> >
> > Best,
> > Costas Zarifis
> >
>
>
> --
> THOMAS KISSINGER
> Staff Software Engineer
>
> MOBILE +49 174-2195270 <+49%20174%202195270>
> EMAIL [email protected]
>
>
> Snowflake Inc.
> 135 Constitution Drive
> Menlo Park, CA 94025, USA
>
