It would be good to move this conversation along. I think it is reasonable to potentially make the normalization part of statistics.
To move this conversation forward I would suggest the following: 1. Given the amount of back and forth, I think we should translate the discussion so far into a formal proposal ( https://github.com/apache/parquet-format/blob/master/proposals/README.md) with a proper design doc. Costas, could you do this? 2. Identify open issues and how to close on them. I think reading through the thread the two main ones that pop out are: a. Normalization and how to represent it (hopefully we can close on this by using statistics). b. Representation of the value in Parquet. In order to close on I think we need to solidify any alternatives ieee754 and how they would work. Then we can run appropriate benchmarks to help make a decision. Thomas if you still want to consider an alternative please write up a design that we can include in the proposal. Best, Micah On Fri, Sep 4, 2026 at 8:34 PM Costas Zarifis via dev < [email protected]> wrote: > Hi Thomas, > > Thank you for the detailed response. I think we are close to agreement on > the requirements. The main remaining question is whether canonicality > should be represented as a property of the logical type or as a property of > the encoded data. > > We agree on the following: > - Precision should be parameterized rather than fixed at 38, allowing > narrower encodings and a defined path to wider precisions. > - The type should cover the complete finite decimal128 range. > - At minimum, positive and negative infinity and a canonical NaN should be > supported. > - Logical equality, hashing, Bloom filters, ordering, and statistics must > remain consistent regardless of the stored representation. > - The physical representation should be selected only after comparing > candidates under equivalent semantics through public benchmarks and > validation vectors. > > We do not think normalization should parameterize the logical type solely > to expose an optimization opportunity. Members of the same finite cohort, > such as 1.5 and 1.50, have the same numerical identity and must behave > consistently for equality, hashing, grouping, ordering, and Bloom filters. > At the same time, a Parquet writer that is supplied with a particular > coefficient and exponent should preserve that representation. > > These requirements do not conflict. Implementations can derive a > cohort-invariant numerical key for equality and hashing without rewriting > the stored value. Canonicalizing that derived key is always safe; replacing > the stored value 1.50 with 1.5 is not safe when quantum must round-trip. > > An engine may expose only numerical semantics and therefore supply > already-canonical values to the writer, but that should not create a second > Parquet logical type. Once a DECFLOAT value with a particular coefficient > and exponent is supplied, a conforming writer must preserve it. > > At the same time, we agree with your concern that preserving cohorts can > add work to equality and hash-based operations. That work can be avoided > when the encoded data is known to satisfy stronger invariants, without > making those invariants part of the logical type. > > We therefore suggest defining content-derived properties that writers can > record at the file or row-group level (or another agreed physical scope). > These could indicate, for example, that: > > 1. every finite value uses the defined cohort-canonical representation > 2. no negative zero occurs; > 3. no signaling NaN occurs; or > 4. only the canonical NaN occurs. > > Implementations could inspect these properties before decoding the column > data and select an optimized path when safe. For example, if every finite > value has a unique canonical encoding and the relevant special cases are > absent, equality and hashing may avoid per-value canonicalization. If the > properties are false, absent or unknown, implementations must use the > general numerical path performing canonicalization. The properties only > inform optimization; they do not change the values’ logical behavior. > > There remains an important distinction between observing that a particular > file or row group contains only canonical values and enforcing that a > column may never contain anything else. A mandatory canonical mode would > prevent representational round-tripping even when the source engine > retained the necessary information. A content-derived property provides the > optimized path whenever the data already satisfies the invariant without > changing the logical type or authorizing information loss. > > This requires defining new metadata or statistics, including their precise > meaning, scope, and writer requirements. Concretely, a writer could compute > the property as follows: > > allValuesCanonical = true > For each value: > if value is not canonical: > allValuesCanonical = false > > This flag at the end, stores one of three states: > > 1. true: every value is normalized > 2. false: at least one value is not normalized > 3. absent/unknown: the writer did not establish the property > > For example: > > File A: 1.5, 2.7, 3.9 > allValuesCanonical = true > File B: 1.5, 1.50, 3.9 > allValuesCanonical = false > > This could be stored in one of these places: > > 1. Table/column: one property for the entire table > 2. File: each Parquet file reports its own property > 3. Row group: different parts of a file can report different properties > 4. Page: even finer-grained, but with greater metadata overhead > > We suggest capturing the remaining requirements in a small decision matrix: > > 1. Supported precision values and range. > 2. The representation and round-trip contract for finite cohorts and > special values. > 3. Numerical equality and normative canonical-key semantics. > 4. Content-derived canonicality properties, including their scope and > missing/unknown behavior. > 5. Whether canonicality is an observed property or an enforced invariant. > 6. Statistics and numerical ordering. > 7. Invalid, reserved, and non-canonical encoding handling. > 8. Required benchmarks, validation vectors, and physical-format artifacts. > > Once that matrix converges, we would be happy to work jointly on the > benchmark and validation suite. At that point, the candidate physical > representations can be evaluated against the same requirements rather than > embedding different semantics in each comparison. > > Best, > Costas > > On Tue, Aug 25, 2026 at 6:50 AM Thomas Kissinger via dev < > [email protected]> wrote: > > > Hi Costas, > > > > Thank you for the thoughtful reply and for separating the requirements > from > > the physical representation. Starting with the requirements is the right > > approach, and we are confident that we can find a solution covering the > > important use cases on both sides. > > > > Thank you also for sharing the Spark SPIP. Seeing the proposed engine > > semantics alongside the Parquet representation is very helpful. Although > > Spark and Parquet have separate decision processes, both projects would > > benefit if these efforts converge on shared requirements rather than > > develop alongside each other with different assumptions. In particular, > > aligning on precision and range, which numerical and representational > > distinctions must survive a round-trip, and which special values are > > supported before either design is finalized would avoid unnecessary > > conversion and interoperability gaps. > > > > Our current view is the following. > > Precision and range > > > > We agree that the type should not impose a permanent 38-digit ceiling and > > should cover at least the complete finite decimal128 range. Narrower and > > wider precisions should be possible, with the exact supported set decided > > when evaluating the physical representations. > > > > This requirement also exposes a mismatch with the generalized IEEE > > interchange layout. Its precision ladder moves from 34 to 43 digits > around > > the established 38-digit SQL boundary, and from 70 to 79 around the > > 76-digit boundary. Current Intel and Boost implementations stop at > > decimal128, and we are not aware of an interoperable production > decimal160 > > codec. Signed 128- and 256-bit integer significands support precisions 38 > > and 76 directly, aligning more naturally with existing database decimal > > representations. > > Quantum, equality, and hashing > > > > We agree that some applications need to preserve the original quantum, > and > > FHIR is a useful example. The question is not whether this capability > > should exist, but whether it should be mandatory for every column. For > > workloads that require only ordinary numerical semantics, preserving > > cohorts adds work to equality and hash-based operations without providing > > additional value. > > > > When cohort members are retained, physical identity and numerical > identity > > diverge. Equality checks must handle multiple encodings of the same > value, > > while hash joins, grouping, DISTINCT, and partitioning must derive a > > cohort- invariant numerical key for every processed value or maintain an > > additional canonical key. That adds work and state to fundamental > database > > paths. Because Parquet Bloom filters currently hash the PLAIN encoding, a > > quantum-preserving mode would require a normative cohort-invariant key > > shared by writers and readers; any disagreement could cause incorrect > > pruning. > > > > Preserving cohort members also increases physical cardinality: > numerically > > equal values may be stored as different byte sequences. This can create > > additional dictionary entries and reduce repeated byte patterns available > > to compression. The actual size impact will depend on the data > distribution > > and should be included in the benchmark. > > > > We therefore propose making quantum preservation an explicit parameter of > > the logical type, with canonical representation as the default. A > canonical > > mode would require one representation per finite numerical value, while a > > quantum- preserving mode would retain the supplied exponent for > > applications requiring faithful round-tripping. Both modes would use > > ordinary numerical equality; the parameter would define the > representation > > round-trip contract. This makes the semantic and performance tradeoff > > explicit rather than imposing either choice on every consumer. > > Special values > > > > We agree on positive and negative infinity and a canonical NaN. We are > less > > convinced that signed zero, signaling NaNs, and NaN payloads belong in an > > interoperable Parquet type, since many important target type systems > cannot > > represent them reliably. > > > > Signaling NaN is difficult to reconcile with an interoperable columnar > > format because its behavior depends on how a value is inspected. IEEE > > numerical comparisons signal invalid operation for an sNaN, while > > totalOrder, which is appropriate for statistics, does not. Parquet uses > > values through several paths—statistics and pruning, dictionary > processing, > > and row-level predicate evaluation—that may therefore treat the same sNaN > > differently. With traps enabled, observable query behavior could even > > depend on whether an optimization path is used. > > > > Supporting sNaN would consequently require precise rules for statistics, > > pruning, predicates, dictionaries, quieting, and round-tripping. That is > > substantial complexity for an execution mechanism without a demonstrated > > Parquet interoperability requirement. In the exchange-rate example, NULL > or > > an explicit state column would express missing or invalid input more > > predictably across engines. > > > > Signed zero creates a similar interoperability problem to signaling NaN. > It > > introduces two encodings for the same numerical zero and therefore > requires > > special treatment in equality, hashing, Bloom filters, ordering, and > > statistics. Many decimal type systems and conversion paths do not > preserve > > the distinction and silently turn negative zero into ordinary zero. A > > downstream consumer relying on the sign may then observe different > behavior > > without any indication that information was lost. This is misleading in > an > > interoperable format, so unless a compelling use case is identified, we > > prefer one canonical zero. > > > > We do not think NaN payloads should be included. IEEE defines their > > representation but not interoperable payload semantics, and does not > fully > > specify payload selection when multiple NaNs participate in an operation. > > Parquet could preserve the bytes, but that would impose a round-trip > > obligation for opaque metadata that receiving engines cannot interpret > > consistently. Diagnostic information is more portable and queryable in a > > separate column. > > Performance and benchmarks > > > > We agree that, once the requirements are finalized, we should work > together > > on a public benchmark comparing equivalent semantics and documenting > > distributions, rounding behavior, fast paths, fallbacks, compiler > settings, > > and hardware. It should cover fundamental database operations, including > > encoding, decoding, cohort reduction and canonical-key derivation, > > equality, and arithmetic. > > > > Until the existing measurements can be reproduced under equivalent > > semantics, they should be treated as directional evidence rather than a > > representation-level result. Please feel free to propose an initial > > benchmark design already; it could provide a useful foundation for the > > joint version. > > Storage and runtime representation > > > > We agree that engines can decode the Parquet representation once and use > a > > different runtime representation. This amortizes conversion across long > > pipelines but does not eliminate representation-dependent work. Values > > surviving pruning must still be interpreted or decoded for row-level > > filters, numerical hashing, statistics construction on write, or runtime > > materialization. The physical representation therefore remains relevant > to > > scan-heavy workloads. > > Validation > > > > We likewise agree that, once the requirements are finalized, we should > > jointly develop a public validation suite independently of the > performance > > benchmark. Please feel free to propose an initial set of test vectors > > already as a foundation. It should eventually cover canonical and > > quantum-preserving values, precision boundaries, special values, > > non-canonical, reserved, and invalid encodings as applicable, equality, > > hashing, ordering and statistics, and cross-implementation round-trips. > > > > Thanks again for moving the discussion toward explicit requirements. We > > already agree on many important points: support for at least 38 digits > with > > a path to wider precisions, the decimal128 finite range, infinities and a > > canonical NaN, and public benchmarks and test vectors. The remaining > > discussion is focused on whether quantum preservation should be optional, > > which additional IEEE distinctions belong in an interoperable database > > type, and which physical representation best satisfies the agreed > > requirements. We believe an extensible type with an explicit > > quantum-preservation choice offers a practical path forward. > > > > Best, Thomas > > > > > > On Fri, Aug 14, 2026 at 10:38 PM Costas Zarifis via dev < > > [email protected]> wrote: > > > > > Hi Thomas, > > > > > > Thank you for laying out the requirements explicitly. We agree that > > > requirements should come before selecting a physical representation. > > > There is substantial agreement, but we see several requirements > > > differently. > > > > > > For context on engine integration, we have also published a draft > > > Spark Project Improvement Proposal (SPIP): Add the DECFLOAT data type > > > ( > > > > > > https://docs.google.com/document/d/1mTSOoCvX7yYMSa6CpI6OPS-MQZocPxUtrrFOEiZjCPk/edit > > > ). > > > The SPIP covers the proposed Spark SQL type and execution semantics, > > > while this discussion focuses on the portable Parquet representation. > > > > > > Before choosing BID or a new 18-byte format, we suggest first agreeing > > > on the following: > > > > > > - Precision: We agree that the format must losslessly support 38-digit > > > values. Your proposal targets exactly 38 digits, while we think the > > > format should provide a defined path to both narrower and wider > > > precisions. > > > - Range: We agree that the format should cover at least the complete > > > finite decimal128 range. > > > - Cohorts: Your proposal requires one canonical representation per > > > numerical value. We think the original coefficient, exponent, and > > > quantum should round-trip. > > > - Special values: We agree on infinity and NaN. We additionally think > > > engines supporting signed zero, signaling NaNs, and NaN payloads > > > should be able to round-trip them. > > > - Hashing and equality: We agree that numerically equal values must > > > behave consistently in equality and hash-based operations. We do not > > > think that requires canonicalizing the stored representation. > > > - Performance: We take the reported results seriously, but think the > > > candidate representations should be compared using equivalent > > > semantics, public benchmarks, and independent implementations. > > > - Validation: Correctness and interoperability should be validated > > > using public test vectors, separately from performance evaluation. > > > > > > The sections below provide the rationale for each position. > > > > > > Precision and range > > > --------------------------- > > > > > > We agree that a Parquet decimal-floating-point type must losslessly > > > represent every numerical value in the DECIMAL(38,s) domain. > > > > > > We do not think the format should necessarily stop at exactly 38 > > > digits, however. Precision 38 is an important interoperability > > > boundary established by an original crop of DBMS, which does not > > > establish that present or future requirements stop at 38 digits. > > > > > > Many open source or major newer DBMS have higher limits: > > > > > > - PostgreSQL > > > Type: NUMERIC unconstrained > > > Max precision: 131,072 before decimal / 16,383 after; declared > > > NUMERIC(p,s) up to p=1000 > > > Source: > https://www.postgresql.org/docs/current/datatype-numeric.html > > > > > > - MySQL / MariaDB / SingleStore > > > Type: DECIMAL > > > Max precision: 65 > > > Source: > https://dev.mysql.com/doc/refman/8.0/en/fixed-point-types.html > > > > > > - BigQuery > > > Type: BIGNUMERIC > > > Max precision: ~76.8 (scale 38) > > > Source: > > > > https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types > > > > > > - ClickHouse > > > Type: Decimal256 > > > Max precision: 76 > > > Source: https://clickhouse.com/docs/sql-reference/data-types/decimal > > > > > > - Oracle > > > Type: unconstrained NUMBER > > > Max precision: ~40 (parameterized 38) > > > Source: > > > > > > https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlqr/Data-Types.html > > > > > > - Trino ≥480 > > > Type: NUMBER > > > Max precision: >= 50 > > > Source: https://trino.io/docs/current/language/types.html#number > > > > > > While we can opine on the value of precision past a certain limit, we > > > cannot pass judgement on whether the cut-off is 34, 38, 43 or > > > something higher. > > > > > > Therefore Parquet should avoid making any such precision a permanent > > > semantic ceiling. > > > > > > Faithful round-tripping and cohorts > > > ---------------------------------------------- > > > > > > We do not agree that Parquet should require one cohort-canonical > > > stored representation for every finite numerical value. > > > > > > Although 1.5, 1.50, and 1.500 are numerically equal, their different > > > quantum can carry information that some applications must preserve. > > > Engines can provide numerical operations alongside IEEE-defined > > > cohort-sensitive operations such as sameQuantum and totalOrder, the > > > latter often exposed as compareTotal. These operations are only > > > possible if the storage format retains the original representation. > > > Canonicalizing on write permanently removes that option. > > > > > > A concrete example is HL7 FHIR healthcare data > > > (https://hl7.org/fhir/datatypes.html). FHIR explicitly specifies that > > > decimal precision is significant: 0.010 is different from 0.01, and > > > implementations must preserve that distinction. This requirement > > > concerns faithful interchange and presentation rather than numerical > > > equality: 0.010 and 0.01 have the same numerical value, but FHIR > > > requires their represented precision to survive a round-trip. A > > > healthcare data lake using Parquet as an interchange or archival > > > format must therefore retain the original quantum to reconstruct > > > standards-compliant FHIR resources. Canonicalizing on write would make > > > that round-trip impossible. > > > > > > Beyond FHIR, this behavior appears in other decimal arithmetic > > > ecosystems. ISO/IEC TS 18661-2 > > > (https://open-std.org/JTC1/SC22/WG14/www/docs/n1912.pdf), which > > > specifies C language support for decimal floating-point arithmetic, > > > explicitly states that 1.0 and 1.00 are distinguishable. The > > > language-independent General Decimal Arithmetic specification > > > (https://speleotrove.com/decimal/daops.html), which underpins several > > > decimal arithmetic libraries, preserves trailing zeros unless the > > > caller explicitly requests `reduce`. The same principle appears > > > independently in measurement and reporting standards. ISO 80000-1 > > > ( > > > > > > https://www.iso.org/home.isoDocumentsDownload.do?t=0bc30Dw3Fis0Kmeb_a6O_bzO2QP14DxJAxwtmaRLmv-WwspRuGQ-iHB6VgCGRJnU > > > ), > > > NIST SP 811 ( > > > > > > https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-7-rules-and-style-conventions-expressing-values > > > ), > > > and ASTM E29 ( > > > https://www.galvanizeit.com/uploads/resources/ASTM-E-29-yr-13.pdf) > > > all treat trailing zeros after a decimal point as significant in > > > reported values. > > > > > > The Bloom-filter example identifies a real correctness concern, but > > > optional Bloom-filter support should not require writers to discard > > > quantum information from stored values. If Bloom-filter support is > > > deemed necessary, Parquet could canonicalize only the numerical hash > > > key. > > > > > > For example: > > > > > > Stored representation Numerical Bloom-filter key > > > 1.5 = (15, -1) (15, -1) > > > 1.50 = (150, -2) (15, -1) > > > 1.500 = (1500, -3) (15, -1) > > > > > > Writers and readers would hash the same normalized numerical form. > > > This keeps numerical-equality Bloom filters correct without changing > > > the stored representation. Otherwise, readers could disable > > > Bloom-filter pruning for DECFLOAT equality. > > > > > > Special values > > > ------------------- > > > > > > We agree that positive and negative infinity and NaN should be > > > representable. We think the same faithful-round-trip principle should > > > apply to signed zero, signaling NaNs, and NaN payloads. > > > > > > An engine does not need to expose every distinction to its users. It > > > may normalize or reject some values, or provide traps, similar to > > > Firebird’s SET DECFLOAT TRAPS > > > ( > > > > > > https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref50-management.html > > > ), > > > controlling whether division by zero, overflow, invalid operations, or > > > signaling NaN produce a value or raise an error. Those are execution > > > policies. > > > > > > A concrete example of signaling NaN’s automatic fail-on-use behavior > > > is a bank preparing its daily foreign-exchange rates. Each currency > > > pair must receive a rate from a market-data provider before trades can > > > be valued. The bank could initialize rates that have not yet arrived > > > to signaling NaN and periodically checkpoint the table to Parquet: > > > > > > EUR → USD = 1.17 > > > GBP → USD = 1.35 > > > JPY → USD = sNaN > > > > > > The valuation job enables the InvalidOperation trap. If it > > > accidentally attempts to convert a JPY payment before the rate has > > > arrived, multiplying the payment by sNaN fails immediately at the > > > point of use. Merely copying, checkpointing, or inspecting the table > > > does not trigger the exception. A NaN payload could additionally > > > retain a diagnostic code identifying the missing or invalid data > > > source. > > > > > > Parquet should reproduce the values written to it rather than > > > normalize away supported distinctions. If it converts sNaN to quiet > > > NaN, removes NaN payloads, or collapses -0 into +0, engines that > > > support those values can no longer round-trip them faithfully. Whether > > > an engine exposes or uses these distinctions should remain an > > > engine-level decision; the storage format should preserve them rather > > > than make an irreversible decision for every consumer. > > > > > > Performance > > > ------------------ > > > > > > We take the reported performance differences seriously, but the > > > current results do not isolate the physical representation. They > > > compare Intel libbid with an optimized Snowflake implementation while > > > potentially also varying precision, semantics, algorithms, input > > > distributions, and fast-path behavior. > > > > > > Our own benchmarks reinforce this concern: libbid and Boost.Decimal > > > both use BID, yet differ by several times on some operations. Large > > > performance gaps can therefore arise from implementation choices even > > > when the representation is identical. This does not rule out an > > > inherent advantage in Snowflake’s representation, but the current > > > comparison does not isolate it. > > > > > > Could you share the implementation or enough of the benchmark harness > > > to reproduce the results, including: > > > > > > - input and exponent distributions > > > - precision and rounding configuration > > > - treatment of cohorts and special values > > > - the percentage of operations using the optimized path versus fallback > > > - compiler and optimization settings > > > - results on both ARM and x86? > > > > > > We should compare equivalent value domains and semantics. For Parquet, > > > we should also measure encoding, decoding, comparisons, numerical > > > hashing, statistics, predicate evaluation, compression, and end-to-end > > > scan and aggregation performance, not only arithmetic. > > > > > > We should also separate the Parquet encoding from an engine’s runtime > > > representation. An engine could decode BID once into its preferred > > > coefficient/exponent structure, execute an expression pipeline using > > > that representation, and encode only at the output boundary. A > > > benchmark comparing that approach would help determine whether the > > > reported cost is inherent to the interchange encoding or to using > > > packed BID for every intermediate result. > > > > > > Once those requirements are settled, we can evaluate the candidate > > > representations against the same semantics using reproducible > > > benchmarks. > > > > > > Best, > > > Costas Zarifis > > > > > > > > > -- > > THOMAS KISSINGER > > Staff Software Engineer > > > > MOBILE +49 174-2195270 <+49%20174%202195270> <+49%20174%202195270> > > EMAIL [email protected] > > > > > > Snowflake Inc. > > 135 Constitution Drive > > Menlo Park, CA 94025, USA > > >
