Hi all, Following Micah’s suggestion, we consolidated the discussion into a formal proposal:
Public Proposal: Extensible Decimal Floating-Point Type: https://docs.google.com/document/d/1cHzlsYH_-JBkcvHLXKSJXMAZ0zwRz3jcnIl4izE1UFQ/edit?usp=sharing The proposal takes a clear position in favor of: - the complete IEEE 754 decimal value model; - one logical type that preserves the supplied representation; - content-derived canonicality metadata at the column-chunk level; - numerical equality and canonical keys for hashing and Bloom filters; and - IEEE `totalOrder` for representation-sensitive ordering and statistics. The physical representation remains to be selected. The document specifies generalized BID as one candidate, including decimal64, decimal128, and a 43-digit BID160 profile. It also lists the revisions needed for the existing significand-and-exponent proposal to become a complete, comparable alternative. Once we agree on the semantic contract, the remaining candidates can be evaluated under equivalent semantics using public conformance tests, validation vectors, and benchmarks. Please review the proposal and share any correctness, interoperability, or implementation concerns. We are sharing it now so everyone has time to review it before our planned discussion. Best, Costas On Thu, Sep 24, 2026 at 10:25 AM Micah Kornfield <[email protected]> wrote: > Hi Thomas, > > - An empty table has a schema but no existing values from which to derive >> that policy. > > > The engine can choose the policy; my argument is that the shared schema >> should communicate it across independent engines and connectors, rather >> than require separate configuration. A logical-type parameter would express >> that write-time obligation, while statistics would describe the actual data >> for read optimization. > > > I think the misalignment we have is that Parquet is not a table format. > An "empty" table means the absence of a parquet file so no schema would be > available to coordinate writes. This is partially what I meant by a higher > level. I could see the argument for documenting the contract at the table > level, but I don't think it needs to flow down to parquet, as an individual > file/schema is not something we really care about once the file is written. > > As an analogy, there are a lot of other properties we want to specify > about how writes to a particular column happen (e.g. compression and > encoding). These are configured separately at the parquet level via an API > and don't end up in the schema. Table formats have conventions for > persistently configuring them. > > Your point that Parquet cannot prevent these conversions is precisely why >> we favor excluding these special values from the interoperable type. >> Documenting an engine's limitation does not alert a downstream consumer >> that a safety-relevant distinction has already been lost. > > > It isn't Parquet's job to place arbitrary limits on a type for the safety > of the end-users of a particular engine. Again, I can see higher level > systems consuming/writing parquet making choices about limiting values, but > arbitrarily limiting the type does not make sense to me at the file layer. > > As an example from traditional floating point numbers. Some systems (e.g. > postgres) decide to collapse -0.0 and 0.0 to the same bucket on > aggregation but still persist the values exactly. > > Cheers, > Micah > > > > > > On Thu, Sep 24, 2026 at 8:44 AM Thomas Kissinger < > [email protected]> wrote: > >> Hi Micah, >> >> Thank you. >> *1. Canonicalization: statistics describe data, not writer intent* >> >> I think the distinction between read-time optimization and the write-time >> contract is central here. I agree that statistics can identify safe read >> optimizations. However, they cannot tell an independent writer whether it >> should canonicalize incoming values or preserve their quantum: >> >> - An empty table has a schema but no existing values from which to derive >> that policy. >> - Two columns may both contain only 1.5 and 2.7, yet one requires >> canonical values and the other requires quantum preservation. Their >> canonicality statistics are identical. When appending 1.50, the required >> behavior differs: store 1.5 in the first case and retain 1.50 in the second. >> >> A writer must therefore not interpret “all existing values are canonical” >> as permission to canonicalize new values. Always preserving the supplied >> representation avoids that mistake, but leaves the canonical-only contract >> entirely outside Parquet. The engine can choose the policy; my argument is >> that the shared schema should communicate it across independent engines and >> connectors, rather than require separate configuration. A logical-type >> parameter would express that write-time obligation, while statistics would >> describe the actual data for read optimization. >> >> *2. Special values: explicit state is safer for interoperability* >> >> We support broad value coverage as a general design principle. However, >> we do not support including signaling NaN and negative zero in this case. >> Your examples motivating them encode sensitive application state, not >> merely numerical values, and that state can disappear silently when passing >> through engines that do not preserve these distinctions. >> >> Consider your financial example of initializing unavailable exchange >> rates to signaling NaN. If an intermediate engine quietly converts those >> values to quiet NaN, downstream arithmetic no longer triggers the expected >> InvalidOperation trap. Likewise, converting -0 to +0 removes information a >> downstream application may rely on. The rewritten Parquet file can be >> valid, with accurate statistics, without revealing either loss. >> >> Your point that Parquet cannot prevent these conversions is precisely why >> we favor excluding these special values from the interoperable type. >> Documenting an engine's limitation does not alert a downstream consumer >> that a safety-relevant distinction has already been lost. >> >> We would instead represent such state explicitly in auxiliary columns—for >> example, a nullable exchange rate alongside a status such as VALID, >> MISSING, or INVALID, with an optional diagnostic code. An application >> relying on the sign of zero could likewise store that distinction >> explicitly. These fields still require application-level validation, but >> use ordinary, widely supported types and make the state directly >> inspectable, rather than depend on special-value behavior that may change >> across engines. >> >> Best, >> Thomas >> >> quick note: won't be able to reply before next Thu. >> >> On Wed, Sep 23, 2026 at 6:08 PM Micah Kornfield <[email protected]> >> wrote: >> >>> Hi Thomas, >>> >>>> We expect quantum-preserving round-trips to serve a minority of use >>>> cases, while most analytical workloads need only numerical values. >>>> Requiring cohort handling in these common cases risks substantial overhead >>>> in equality and hashing without adding value. Statistics can identify fast >>>> paths, but describe what was written rather than prescribe whether a column >>>> must be canonicalized. A logical-type parameter would explicitly define the >>>> writer's obligation to canonicalize or preserve quantum; statistics alone >>>> do not establish that contract. >>> >>> >>> I agree this can be useful but it seems like a decision taken at a >>> higher level? From a parquet persistence perspective I think this breaks >>> down as follows: >>> 1. An engine/connector decides to hand values to parquet. If we are in >>> the regime that canonicalization is required the engine/connector is making >>> that decision and only providing canonical values. This will show up in >>> statistics. For optimizing stats collection it seems like we could expose >>> APIs in parquet to assume values are canonicalized if the overhead of >>> checking is deemed too expensive. But that seems like an API level concern >>> and not a schema level concern. >>> 2. When reading, an engine/connector can look at the statistics to >>> determine if everything is canocilized. Having the type parameter seems >>> superfluous? >>> >>> Given this model could you expand on why you think the parameterization >>> has to happen at the parquet schema level? >>> >>> Our concern is silent information loss when data passes through a >>>> non-supporting engine. Such an engine may quiet a signaling NaN or turn >>>> negative zero into ordinary zero when reading and rewriting the data. A >>>> downstream supporting reader could then rely on the result as a faithful >>>> representation of the original values, without knowing that these >>>> meaningful distinctions were discarded. Metadata describing the rewritten >>>> values would not reveal that loss. >>> >>> >>> I think this is outside the scope of Parquet. We've always taken the >>> stance that Parquet should be able to support taking in a full range of >>> values, and be able to return them losslessly. If an engine that doesn't >>> support the full range chooses to coerce the values, that is its choice and >>> it should document the limitation. >>> >>> Best, >>> Micah >>> >>> >>> >>> >>> >>> On Mon, Sep 21, 2026 at 8:00 AM Thomas Kissinger < >>> [email protected]> wrote: >>> >>>> Hi Costas, >>>> >>>> Sorry for the late reply. I was on vacation last week. >>>> >>>> Thank you for the thoughtful reply. The statistics proposal sounds >>>> interesting, and I would like to focus on two remaining points. >>>> >>>> *1. Canonical representation* >>>> We expect quantum-preserving round-trips to serve a minority of use >>>> cases, while most analytical workloads need only numerical values. >>>> Requiring cohort handling in these common cases risks substantial overhead >>>> in equality and hashing without adding value. Statistics can identify fast >>>> paths, but describe what was written rather than prescribe whether a column >>>> must be canonicalized. A logical-type parameter would explicitly define the >>>> writer's obligation to canonicalize or preserve quantum; statistics alone >>>> do not establish that contract. >>>> >>>> Another option would be to always store a canonical numerical component >>>> alongside compact quantum information, such as the number of trailing zeros >>>> removed during canonicalization. Equality and hashing would use the >>>> canonical component, while the additional information would allow the >>>> original representation to round-trip. This could reconcile efficient >>>> numerical operations with quantum preservation at a small storage overhead. >>>> Would you consider this approach? >>>> >>>> *2. Signaling NaNs and negative zero* >>>> Our concern is silent information loss when data passes through a >>>> non-supporting engine. Such an engine may quiet a signaling NaN or turn >>>> negative zero into ordinary zero when reading and rewriting the data. A >>>> downstream supporting reader could then rely on the result as a faithful >>>> representation of the original values, without knowing that these >>>> meaningful distinctions were discarded. Metadata describing the rewritten >>>> values would not reveal that loss. >>>> >>>> How would the proposed round-trip contract prevent this loss from going >>>> unnoticed and misleading downstream readers? >>>> >>>> >>>> Micah, thank you for helping move this forward; since the requirements >>>> seem to be converging, we would prefer to resolve these remaining points >>>> here first, then work together on a joint proposal, if that works for >>>> everyone. >>>> >>>> Best, >>>> Thomas >>>> >>>> On Tue, Sep 15, 2026 at 7:18 PM Micah Kornfield <[email protected]> >>>> wrote: >>>> >>>>> It would be good to move this conversation along. I think it is >>>>> reasonable to potentially make the normalization part of statistics. >>>>> >>>>> To move this conversation forward I would suggest the following: >>>>> 1. Given the amount of back and forth, I think we should translate >>>>> the discussion so far into a formal proposal ( >>>>> https://github.com/apache/parquet-format/blob/master/proposals/README.md) >>>>> with a proper design doc. Costas, could you do this? >>>>> 2. Identify open issues and how to close on them. I think reading >>>>> through the thread the two main ones that pop out are: >>>>> a. Normalization and how to represent it (hopefully we can close >>>>> on this by using statistics). >>>>> b. Representation of the value in Parquet. In order to close on >>>>> I think we need to solidify any alternatives ieee754 and how they would >>>>> work. Then we can run appropriate benchmarks to help make a decision. >>>>> Thomas if you still want to consider an alternative please write up a >>>>> design that we can include in the proposal. >>>>> >>>>> Best, >>>>> Micah >>>>> >>>>> On Fri, Sep 4, 2026 at 8:34 PM Costas Zarifis via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Thomas, >>>>>> >>>>>> Thank you for the detailed response. I think we are close to >>>>>> agreement on >>>>>> the requirements. The main remaining question is whether canonicality >>>>>> should be represented as a property of the logical type or as a >>>>>> property of >>>>>> the encoded data. >>>>>> >>>>>> We agree on the following: >>>>>> - Precision should be parameterized rather than fixed at 38, allowing >>>>>> narrower encodings and a defined path to wider precisions. >>>>>> - The type should cover the complete finite decimal128 range. >>>>>> - At minimum, positive and negative infinity and a canonical NaN >>>>>> should be >>>>>> supported. >>>>>> - Logical equality, hashing, Bloom filters, ordering, and statistics >>>>>> must >>>>>> remain consistent regardless of the stored representation. >>>>>> - The physical representation should be selected only after comparing >>>>>> candidates under equivalent semantics through public benchmarks and >>>>>> validation vectors. >>>>>> >>>>>> We do not think normalization should parameterize the logical type >>>>>> solely >>>>>> to expose an optimization opportunity. Members of the same finite >>>>>> cohort, >>>>>> such as 1.5 and 1.50, have the same numerical identity and must behave >>>>>> consistently for equality, hashing, grouping, ordering, and Bloom >>>>>> filters. >>>>>> At the same time, a Parquet writer that is supplied with a particular >>>>>> coefficient and exponent should preserve that representation. >>>>>> >>>>>> These requirements do not conflict. Implementations can derive a >>>>>> cohort-invariant numerical key for equality and hashing without >>>>>> rewriting >>>>>> the stored value. Canonicalizing that derived key is always safe; >>>>>> replacing >>>>>> the stored value 1.50 with 1.5 is not safe when quantum must >>>>>> round-trip. >>>>>> >>>>>> An engine may expose only numerical semantics and therefore supply >>>>>> already-canonical values to the writer, but that should not create a >>>>>> second >>>>>> Parquet logical type. Once a DECFLOAT value with a particular >>>>>> coefficient >>>>>> and exponent is supplied, a conforming writer must preserve it. >>>>>> >>>>>> At the same time, we agree with your concern that preserving cohorts >>>>>> can >>>>>> add work to equality and hash-based operations. That work can be >>>>>> avoided >>>>>> when the encoded data is known to satisfy stronger invariants, without >>>>>> making those invariants part of the logical type. >>>>>> >>>>>> We therefore suggest defining content-derived properties that writers >>>>>> can >>>>>> record at the file or row-group level (or another agreed physical >>>>>> scope). >>>>>> These could indicate, for example, that: >>>>>> >>>>>> 1. every finite value uses the defined cohort-canonical representation >>>>>> 2. no negative zero occurs; >>>>>> 3. no signaling NaN occurs; or >>>>>> 4. only the canonical NaN occurs. >>>>>> >>>>>> Implementations could inspect these properties before decoding the >>>>>> column >>>>>> data and select an optimized path when safe. For example, if every >>>>>> finite >>>>>> value has a unique canonical encoding and the relevant special cases >>>>>> are >>>>>> absent, equality and hashing may avoid per-value canonicalization. If >>>>>> the >>>>>> properties are false, absent or unknown, implementations must use the >>>>>> general numerical path performing canonicalization. The properties >>>>>> only >>>>>> inform optimization; they do not change the values’ logical behavior. >>>>>> >>>>>> There remains an important distinction between observing that a >>>>>> particular >>>>>> file or row group contains only canonical values and enforcing that a >>>>>> column may never contain anything else. A mandatory canonical mode >>>>>> would >>>>>> prevent representational round-tripping even when the source engine >>>>>> retained the necessary information. A content-derived property >>>>>> provides the >>>>>> optimized path whenever the data already satisfies the invariant >>>>>> without >>>>>> changing the logical type or authorizing information loss. >>>>>> >>>>>> This requires defining new metadata or statistics, including their >>>>>> precise >>>>>> meaning, scope, and writer requirements. Concretely, a writer could >>>>>> compute >>>>>> the property as follows: >>>>>> >>>>>> allValuesCanonical = true >>>>>> For each value: >>>>>> if value is not canonical: >>>>>> allValuesCanonical = false >>>>>> >>>>>> This flag at the end, stores one of three states: >>>>>> >>>>>> 1. true: every value is normalized >>>>>> 2. false: at least one value is not normalized >>>>>> 3. absent/unknown: the writer did not establish the property >>>>>> >>>>>> For example: >>>>>> >>>>>> File A: 1.5, 2.7, 3.9 >>>>>> allValuesCanonical = true >>>>>> File B: 1.5, 1.50, 3.9 >>>>>> allValuesCanonical = false >>>>>> >>>>>> This could be stored in one of these places: >>>>>> >>>>>> 1. Table/column: one property for the entire table >>>>>> 2. File: each Parquet file reports its own property >>>>>> 3. Row group: different parts of a file can report different >>>>>> properties >>>>>> 4. Page: even finer-grained, but with greater metadata overhead >>>>>> >>>>>> We suggest capturing the remaining requirements in a small decision >>>>>> matrix: >>>>>> >>>>>> 1. Supported precision values and range. >>>>>> 2. The representation and round-trip contract for finite cohorts and >>>>>> special values. >>>>>> 3. Numerical equality and normative canonical-key semantics. >>>>>> 4. Content-derived canonicality properties, including their scope and >>>>>> missing/unknown behavior. >>>>>> 5. Whether canonicality is an observed property or an enforced >>>>>> invariant. >>>>>> 6. Statistics and numerical ordering. >>>>>> 7. Invalid, reserved, and non-canonical encoding handling. >>>>>> 8. Required benchmarks, validation vectors, and physical-format >>>>>> artifacts. >>>>>> >>>>>> Once that matrix converges, we would be happy to work jointly on the >>>>>> benchmark and validation suite. At that point, the candidate physical >>>>>> representations can be evaluated against the same requirements rather >>>>>> than >>>>>> embedding different semantics in each comparison. >>>>>> >>>>>> Best, >>>>>> Costas >>>>>> >>>>>> On Tue, Aug 25, 2026 at 6:50 AM Thomas Kissinger via dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>> > Hi Costas, >>>>>> > >>>>>> > Thank you for the thoughtful reply and for separating the >>>>>> requirements from >>>>>> > the physical representation. Starting with the requirements is the >>>>>> right >>>>>> > approach, and we are confident that we can find a solution covering >>>>>> the >>>>>> > important use cases on both sides. >>>>>> > >>>>>> > Thank you also for sharing the Spark SPIP. Seeing the proposed >>>>>> engine >>>>>> > semantics alongside the Parquet representation is very helpful. >>>>>> Although >>>>>> > Spark and Parquet have separate decision processes, both projects >>>>>> would >>>>>> > benefit if these efforts converge on shared requirements rather than >>>>>> > develop alongside each other with different assumptions. In >>>>>> particular, >>>>>> > aligning on precision and range, which numerical and >>>>>> representational >>>>>> > distinctions must survive a round-trip, and which special values are >>>>>> > supported before either design is finalized would avoid unnecessary >>>>>> > conversion and interoperability gaps. >>>>>> > >>>>>> > Our current view is the following. >>>>>> > Precision and range >>>>>> > >>>>>> > We agree that the type should not impose a permanent 38-digit >>>>>> ceiling and >>>>>> > should cover at least the complete finite decimal128 range. >>>>>> Narrower and >>>>>> > wider precisions should be possible, with the exact supported set >>>>>> decided >>>>>> > when evaluating the physical representations. >>>>>> > >>>>>> > This requirement also exposes a mismatch with the generalized IEEE >>>>>> > interchange layout. Its precision ladder moves from 34 to 43 digits >>>>>> around >>>>>> > the established 38-digit SQL boundary, and from 70 to 79 around the >>>>>> > 76-digit boundary. Current Intel and Boost implementations stop at >>>>>> > decimal128, and we are not aware of an interoperable production >>>>>> decimal160 >>>>>> > codec. Signed 128- and 256-bit integer significands support >>>>>> precisions 38 >>>>>> > and 76 directly, aligning more naturally with existing database >>>>>> decimal >>>>>> > representations. >>>>>> > Quantum, equality, and hashing >>>>>> > >>>>>> > We agree that some applications need to preserve the original >>>>>> quantum, and >>>>>> > FHIR is a useful example. The question is not whether this >>>>>> capability >>>>>> > should exist, but whether it should be mandatory for every column. >>>>>> For >>>>>> > workloads that require only ordinary numerical semantics, preserving >>>>>> > cohorts adds work to equality and hash-based operations without >>>>>> providing >>>>>> > additional value. >>>>>> > >>>>>> > When cohort members are retained, physical identity and numerical >>>>>> identity >>>>>> > diverge. Equality checks must handle multiple encodings of the same >>>>>> value, >>>>>> > while hash joins, grouping, DISTINCT, and partitioning must derive a >>>>>> > cohort- invariant numerical key for every processed value or >>>>>> maintain an >>>>>> > additional canonical key. That adds work and state to fundamental >>>>>> database >>>>>> > paths. Because Parquet Bloom filters currently hash the PLAIN >>>>>> encoding, a >>>>>> > quantum-preserving mode would require a normative cohort-invariant >>>>>> key >>>>>> > shared by writers and readers; any disagreement could cause >>>>>> incorrect >>>>>> > pruning. >>>>>> > >>>>>> > Preserving cohort members also increases physical cardinality: >>>>>> numerically >>>>>> > equal values may be stored as different byte sequences. This can >>>>>> create >>>>>> > additional dictionary entries and reduce repeated byte patterns >>>>>> available >>>>>> > to compression. The actual size impact will depend on the data >>>>>> distribution >>>>>> > and should be included in the benchmark. >>>>>> > >>>>>> > We therefore propose making quantum preservation an explicit >>>>>> parameter of >>>>>> > the logical type, with canonical representation as the default. A >>>>>> canonical >>>>>> > mode would require one representation per finite numerical value, >>>>>> while a >>>>>> > quantum- preserving mode would retain the supplied exponent for >>>>>> > applications requiring faithful round-tripping. Both modes would use >>>>>> > ordinary numerical equality; the parameter would define the >>>>>> representation >>>>>> > round-trip contract. This makes the semantic and performance >>>>>> tradeoff >>>>>> > explicit rather than imposing either choice on every consumer. >>>>>> > Special values >>>>>> > >>>>>> > We agree on positive and negative infinity and a canonical NaN. We >>>>>> are less >>>>>> > convinced that signed zero, signaling NaNs, and NaN payloads belong >>>>>> in an >>>>>> > interoperable Parquet type, since many important target type >>>>>> systems cannot >>>>>> > represent them reliably. >>>>>> > >>>>>> > Signaling NaN is difficult to reconcile with an interoperable >>>>>> columnar >>>>>> > format because its behavior depends on how a value is inspected. >>>>>> IEEE >>>>>> > numerical comparisons signal invalid operation for an sNaN, while >>>>>> > totalOrder, which is appropriate for statistics, does not. Parquet >>>>>> uses >>>>>> > values through several paths—statistics and pruning, dictionary >>>>>> processing, >>>>>> > and row-level predicate evaluation—that may therefore treat the >>>>>> same sNaN >>>>>> > differently. With traps enabled, observable query behavior could >>>>>> even >>>>>> > depend on whether an optimization path is used. >>>>>> > >>>>>> > Supporting sNaN would consequently require precise rules for >>>>>> statistics, >>>>>> > pruning, predicates, dictionaries, quieting, and round-tripping. >>>>>> That is >>>>>> > substantial complexity for an execution mechanism without a >>>>>> demonstrated >>>>>> > Parquet interoperability requirement. In the exchange-rate example, >>>>>> NULL or >>>>>> > an explicit state column would express missing or invalid input more >>>>>> > predictably across engines. >>>>>> > >>>>>> > Signed zero creates a similar interoperability problem to signaling >>>>>> NaN. It >>>>>> > introduces two encodings for the same numerical zero and therefore >>>>>> requires >>>>>> > special treatment in equality, hashing, Bloom filters, ordering, and >>>>>> > statistics. Many decimal type systems and conversion paths do not >>>>>> preserve >>>>>> > the distinction and silently turn negative zero into ordinary zero. >>>>>> A >>>>>> > downstream consumer relying on the sign may then observe different >>>>>> behavior >>>>>> > without any indication that information was lost. This is >>>>>> misleading in an >>>>>> > interoperable format, so unless a compelling use case is >>>>>> identified, we >>>>>> > prefer one canonical zero. >>>>>> > >>>>>> > We do not think NaN payloads should be included. IEEE defines their >>>>>> > representation but not interoperable payload semantics, and does >>>>>> not fully >>>>>> > specify payload selection when multiple NaNs participate in an >>>>>> operation. >>>>>> > Parquet could preserve the bytes, but that would impose a round-trip >>>>>> > obligation for opaque metadata that receiving engines cannot >>>>>> interpret >>>>>> > consistently. Diagnostic information is more portable and queryable >>>>>> in a >>>>>> > separate column. >>>>>> > Performance and benchmarks >>>>>> > >>>>>> > We agree that, once the requirements are finalized, we should work >>>>>> together >>>>>> > on a public benchmark comparing equivalent semantics and documenting >>>>>> > distributions, rounding behavior, fast paths, fallbacks, compiler >>>>>> settings, >>>>>> > and hardware. It should cover fundamental database operations, >>>>>> including >>>>>> > encoding, decoding, cohort reduction and canonical-key derivation, >>>>>> > equality, and arithmetic. >>>>>> > >>>>>> > Until the existing measurements can be reproduced under equivalent >>>>>> > semantics, they should be treated as directional evidence rather >>>>>> than a >>>>>> > representation-level result. Please feel free to propose an initial >>>>>> > benchmark design already; it could provide a useful foundation for >>>>>> the >>>>>> > joint version. >>>>>> > Storage and runtime representation >>>>>> > >>>>>> > We agree that engines can decode the Parquet representation once >>>>>> and use a >>>>>> > different runtime representation. This amortizes conversion across >>>>>> long >>>>>> > pipelines but does not eliminate representation-dependent work. >>>>>> Values >>>>>> > surviving pruning must still be interpreted or decoded for row-level >>>>>> > filters, numerical hashing, statistics construction on write, or >>>>>> runtime >>>>>> > materialization. The physical representation therefore remains >>>>>> relevant to >>>>>> > scan-heavy workloads. >>>>>> > Validation >>>>>> > >>>>>> > We likewise agree that, once the requirements are finalized, we >>>>>> should >>>>>> > jointly develop a public validation suite independently of the >>>>>> performance >>>>>> > benchmark. Please feel free to propose an initial set of test >>>>>> vectors >>>>>> > already as a foundation. It should eventually cover canonical and >>>>>> > quantum-preserving values, precision boundaries, special values, >>>>>> > non-canonical, reserved, and invalid encodings as applicable, >>>>>> equality, >>>>>> > hashing, ordering and statistics, and cross-implementation >>>>>> round-trips. >>>>>> > >>>>>> > Thanks again for moving the discussion toward explicit >>>>>> requirements. We >>>>>> > already agree on many important points: support for at least 38 >>>>>> digits with >>>>>> > a path to wider precisions, the decimal128 finite range, infinities >>>>>> and a >>>>>> > canonical NaN, and public benchmarks and test vectors. The remaining >>>>>> > discussion is focused on whether quantum preservation should be >>>>>> optional, >>>>>> > which additional IEEE distinctions belong in an interoperable >>>>>> database >>>>>> > type, and which physical representation best satisfies the agreed >>>>>> > requirements. We believe an extensible type with an explicit >>>>>> > quantum-preservation choice offers a practical path forward. >>>>>> > >>>>>> > Best, Thomas >>>>>> > >>>>>> > >>>>>> > On Fri, Aug 14, 2026 at 10:38 PM Costas Zarifis via dev < >>>>>> > [email protected]> wrote: >>>>>> > >>>>>> > > Hi Thomas, >>>>>> > > >>>>>> > > Thank you for laying out the requirements explicitly. We agree >>>>>> that >>>>>> > > requirements should come before selecting a physical >>>>>> representation. >>>>>> > > There is substantial agreement, but we see several requirements >>>>>> > > differently. >>>>>> > > >>>>>> > > For context on engine integration, we have also published a draft >>>>>> > > Spark Project Improvement Proposal (SPIP): Add the DECFLOAT data >>>>>> type >>>>>> > > ( >>>>>> > > >>>>>> > >>>>>> https://docs.google.com/document/d/1mTSOoCvX7yYMSa6CpI6OPS-MQZocPxUtrrFOEiZjCPk/edit >>>>>> > > ). >>>>>> > > The SPIP covers the proposed Spark SQL type and execution >>>>>> semantics, >>>>>> > > while this discussion focuses on the portable Parquet >>>>>> representation. >>>>>> > > >>>>>> > > Before choosing BID or a new 18-byte format, we suggest first >>>>>> agreeing >>>>>> > > on the following: >>>>>> > > >>>>>> > > - Precision: We agree that the format must losslessly support >>>>>> 38-digit >>>>>> > > values. Your proposal targets exactly 38 digits, while we think >>>>>> the >>>>>> > > format should provide a defined path to both narrower and wider >>>>>> > > precisions. >>>>>> > > - Range: We agree that the format should cover at least the >>>>>> complete >>>>>> > > finite decimal128 range. >>>>>> > > - Cohorts: Your proposal requires one canonical representation per >>>>>> > > numerical value. We think the original coefficient, exponent, and >>>>>> > > quantum should round-trip. >>>>>> > > - Special values: We agree on infinity and NaN. We additionally >>>>>> think >>>>>> > > engines supporting signed zero, signaling NaNs, and NaN payloads >>>>>> > > should be able to round-trip them. >>>>>> > > - Hashing and equality: We agree that numerically equal values >>>>>> must >>>>>> > > behave consistently in equality and hash-based operations. We do >>>>>> not >>>>>> > > think that requires canonicalizing the stored representation. >>>>>> > > - Performance: We take the reported results seriously, but think >>>>>> the >>>>>> > > candidate representations should be compared using equivalent >>>>>> > > semantics, public benchmarks, and independent implementations. >>>>>> > > - Validation: Correctness and interoperability should be validated >>>>>> > > using public test vectors, separately from performance evaluation. >>>>>> > > >>>>>> > > The sections below provide the rationale for each position. >>>>>> > > >>>>>> > > Precision and range >>>>>> > > --------------------------- >>>>>> > > >>>>>> > > We agree that a Parquet decimal-floating-point type must >>>>>> losslessly >>>>>> > > represent every numerical value in the DECIMAL(38,s) domain. >>>>>> > > >>>>>> > > We do not think the format should necessarily stop at exactly 38 >>>>>> > > digits, however. Precision 38 is an important interoperability >>>>>> > > boundary established by an original crop of DBMS, which does not >>>>>> > > establish that present or future requirements stop at 38 digits. >>>>>> > > >>>>>> > > Many open source or major newer DBMS have higher limits: >>>>>> > > >>>>>> > > - PostgreSQL >>>>>> > > Type: NUMERIC unconstrained >>>>>> > > Max precision: 131,072 before decimal / 16,383 after; declared >>>>>> > > NUMERIC(p,s) up to p=1000 >>>>>> > > Source: >>>>>> https://www.postgresql.org/docs/current/datatype-numeric.html >>>>>> > > >>>>>> > > - MySQL / MariaDB / SingleStore >>>>>> > > Type: DECIMAL >>>>>> > > Max precision: 65 >>>>>> > > Source: >>>>>> https://dev.mysql.com/doc/refman/8.0/en/fixed-point-types.html >>>>>> > > >>>>>> > > - BigQuery >>>>>> > > Type: BIGNUMERIC >>>>>> > > Max precision: ~76.8 (scale 38) >>>>>> > > Source: >>>>>> > > >>>>>> https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types >>>>>> > > >>>>>> > > - ClickHouse >>>>>> > > Type: Decimal256 >>>>>> > > Max precision: 76 >>>>>> > > Source: >>>>>> https://clickhouse.com/docs/sql-reference/data-types/decimal >>>>>> > > >>>>>> > > - Oracle >>>>>> > > Type: unconstrained NUMBER >>>>>> > > Max precision: ~40 (parameterized 38) >>>>>> > > Source: >>>>>> > > >>>>>> > >>>>>> https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlqr/Data-Types.html >>>>>> > > >>>>>> > > - Trino ≥480 >>>>>> > > Type: NUMBER >>>>>> > > Max precision: >= 50 >>>>>> > > Source: >>>>>> https://trino.io/docs/current/language/types.html#number >>>>>> > > >>>>>> > > While we can opine on the value of precision past a certain >>>>>> limit, we >>>>>> > > cannot pass judgement on whether the cut-off is 34, 38, 43 or >>>>>> > > something higher. >>>>>> > > >>>>>> > > Therefore Parquet should avoid making any such precision a >>>>>> permanent >>>>>> > > semantic ceiling. >>>>>> > > >>>>>> > > Faithful round-tripping and cohorts >>>>>> > > ---------------------------------------------- >>>>>> > > >>>>>> > > We do not agree that Parquet should require one cohort-canonical >>>>>> > > stored representation for every finite numerical value. >>>>>> > > >>>>>> > > Although 1.5, 1.50, and 1.500 are numerically equal, their >>>>>> different >>>>>> > > quantum can carry information that some applications must >>>>>> preserve. >>>>>> > > Engines can provide numerical operations alongside IEEE-defined >>>>>> > > cohort-sensitive operations such as sameQuantum and totalOrder, >>>>>> the >>>>>> > > latter often exposed as compareTotal. These operations are only >>>>>> > > possible if the storage format retains the original >>>>>> representation. >>>>>> > > Canonicalizing on write permanently removes that option. >>>>>> > > >>>>>> > > A concrete example is HL7 FHIR healthcare data >>>>>> > > (https://hl7.org/fhir/datatypes.html). FHIR explicitly specifies >>>>>> that >>>>>> > > decimal precision is significant: 0.010 is different from 0.01, >>>>>> and >>>>>> > > implementations must preserve that distinction. This requirement >>>>>> > > concerns faithful interchange and presentation rather than >>>>>> numerical >>>>>> > > equality: 0.010 and 0.01 have the same numerical value, but FHIR >>>>>> > > requires their represented precision to survive a round-trip. A >>>>>> > > healthcare data lake using Parquet as an interchange or archival >>>>>> > > format must therefore retain the original quantum to reconstruct >>>>>> > > standards-compliant FHIR resources. Canonicalizing on write would >>>>>> make >>>>>> > > that round-trip impossible. >>>>>> > > >>>>>> > > Beyond FHIR, this behavior appears in other decimal arithmetic >>>>>> > > ecosystems. ISO/IEC TS 18661-2 >>>>>> > > (https://open-std.org/JTC1/SC22/WG14/www/docs/n1912.pdf), which >>>>>> > > specifies C language support for decimal floating-point >>>>>> arithmetic, >>>>>> > > explicitly states that 1.0 and 1.00 are distinguishable. The >>>>>> > > language-independent General Decimal Arithmetic specification >>>>>> > > (https://speleotrove.com/decimal/daops.html), which underpins >>>>>> several >>>>>> > > decimal arithmetic libraries, preserves trailing zeros unless the >>>>>> > > caller explicitly requests `reduce`. The same principle appears >>>>>> > > independently in measurement and reporting standards. ISO 80000-1 >>>>>> > > ( >>>>>> > > >>>>>> > >>>>>> https://www.iso.org/home.isoDocumentsDownload.do?t=0bc30Dw3Fis0Kmeb_a6O_bzO2QP14DxJAxwtmaRLmv-WwspRuGQ-iHB6VgCGRJnU >>>>>> > > ), >>>>>> > > NIST SP 811 ( >>>>>> > > >>>>>> > >>>>>> https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-7-rules-and-style-conventions-expressing-values >>>>>> > > ), >>>>>> > > and ASTM E29 ( >>>>>> > > https://www.galvanizeit.com/uploads/resources/ASTM-E-29-yr-13.pdf >>>>>> ) >>>>>> > > all treat trailing zeros after a decimal point as significant in >>>>>> > > reported values. >>>>>> > > >>>>>> > > The Bloom-filter example identifies a real correctness concern, >>>>>> but >>>>>> > > optional Bloom-filter support should not require writers to >>>>>> discard >>>>>> > > quantum information from stored values. If Bloom-filter support is >>>>>> > > deemed necessary, Parquet could canonicalize only the numerical >>>>>> hash >>>>>> > > key. >>>>>> > > >>>>>> > > For example: >>>>>> > > >>>>>> > > Stored representation Numerical Bloom-filter key >>>>>> > > 1.5 = (15, -1) (15, -1) >>>>>> > > 1.50 = (150, -2) (15, -1) >>>>>> > > 1.500 = (1500, -3) (15, -1) >>>>>> > > >>>>>> > > Writers and readers would hash the same normalized numerical form. >>>>>> > > This keeps numerical-equality Bloom filters correct without >>>>>> changing >>>>>> > > the stored representation. Otherwise, readers could disable >>>>>> > > Bloom-filter pruning for DECFLOAT equality. >>>>>> > > >>>>>> > > Special values >>>>>> > > ------------------- >>>>>> > > >>>>>> > > We agree that positive and negative infinity and NaN should be >>>>>> > > representable. We think the same faithful-round-trip principle >>>>>> should >>>>>> > > apply to signed zero, signaling NaNs, and NaN payloads. >>>>>> > > >>>>>> > > An engine does not need to expose every distinction to its users. >>>>>> It >>>>>> > > may normalize or reject some values, or provide traps, similar to >>>>>> > > Firebird’s SET DECFLOAT TRAPS >>>>>> > > ( >>>>>> > > >>>>>> > >>>>>> https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref50-management.html >>>>>> > > ), >>>>>> > > controlling whether division by zero, overflow, invalid >>>>>> operations, or >>>>>> > > signaling NaN produce a value or raise an error. Those are >>>>>> execution >>>>>> > > policies. >>>>>> > > >>>>>> > > A concrete example of signaling NaN’s automatic fail-on-use >>>>>> behavior >>>>>> > > is a bank preparing its daily foreign-exchange rates. Each >>>>>> currency >>>>>> > > pair must receive a rate from a market-data provider before >>>>>> trades can >>>>>> > > be valued. The bank could initialize rates that have not yet >>>>>> arrived >>>>>> > > to signaling NaN and periodically checkpoint the table to Parquet: >>>>>> > > >>>>>> > > EUR → USD = 1.17 >>>>>> > > GBP → USD = 1.35 >>>>>> > > JPY → USD = sNaN >>>>>> > > >>>>>> > > The valuation job enables the InvalidOperation trap. If it >>>>>> > > accidentally attempts to convert a JPY payment before the rate has >>>>>> > > arrived, multiplying the payment by sNaN fails immediately at the >>>>>> > > point of use. Merely copying, checkpointing, or inspecting the >>>>>> table >>>>>> > > does not trigger the exception. A NaN payload could additionally >>>>>> > > retain a diagnostic code identifying the missing or invalid data >>>>>> > > source. >>>>>> > > >>>>>> > > Parquet should reproduce the values written to it rather than >>>>>> > > normalize away supported distinctions. If it converts sNaN to >>>>>> quiet >>>>>> > > NaN, removes NaN payloads, or collapses -0 into +0, engines that >>>>>> > > support those values can no longer round-trip them faithfully. >>>>>> Whether >>>>>> > > an engine exposes or uses these distinctions should remain an >>>>>> > > engine-level decision; the storage format should preserve them >>>>>> rather >>>>>> > > than make an irreversible decision for every consumer. >>>>>> > > >>>>>> > > Performance >>>>>> > > ------------------ >>>>>> > > >>>>>> > > We take the reported performance differences seriously, but the >>>>>> > > current results do not isolate the physical representation. They >>>>>> > > compare Intel libbid with an optimized Snowflake implementation >>>>>> while >>>>>> > > potentially also varying precision, semantics, algorithms, input >>>>>> > > distributions, and fast-path behavior. >>>>>> > > >>>>>> > > Our own benchmarks reinforce this concern: libbid and >>>>>> Boost.Decimal >>>>>> > > both use BID, yet differ by several times on some operations. >>>>>> Large >>>>>> > > performance gaps can therefore arise from implementation choices >>>>>> even >>>>>> > > when the representation is identical. This does not rule out an >>>>>> > > inherent advantage in Snowflake’s representation, but the current >>>>>> > > comparison does not isolate it. >>>>>> > > >>>>>> > > Could you share the implementation or enough of the benchmark >>>>>> harness >>>>>> > > to reproduce the results, including: >>>>>> > > >>>>>> > > - input and exponent distributions >>>>>> > > - precision and rounding configuration >>>>>> > > - treatment of cohorts and special values >>>>>> > > - the percentage of operations using the optimized path versus >>>>>> fallback >>>>>> > > - compiler and optimization settings >>>>>> > > - results on both ARM and x86? >>>>>> > > >>>>>> > > We should compare equivalent value domains and semantics. For >>>>>> Parquet, >>>>>> > > we should also measure encoding, decoding, comparisons, numerical >>>>>> > > hashing, statistics, predicate evaluation, compression, and >>>>>> end-to-end >>>>>> > > scan and aggregation performance, not only arithmetic. >>>>>> > > >>>>>> > > We should also separate the Parquet encoding from an engine’s >>>>>> runtime >>>>>> > > representation. An engine could decode BID once into its preferred >>>>>> > > coefficient/exponent structure, execute an expression pipeline >>>>>> using >>>>>> > > that representation, and encode only at the output boundary. A >>>>>> > > benchmark comparing that approach would help determine whether the >>>>>> > > reported cost is inherent to the interchange encoding or to using >>>>>> > > packed BID for every intermediate result. >>>>>> > > >>>>>> > > Once those requirements are settled, we can evaluate the candidate >>>>>> > > representations against the same semantics using reproducible >>>>>> > > benchmarks. >>>>>> > > >>>>>> > > Best, >>>>>> > > Costas Zarifis >>>>>> > > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > THOMAS KISSINGER >>>>>> > Staff Software Engineer >>>>>> > >>>>>> > MOBILE +49 174-2195270 <+49%20174%202195270> <+49%20174%202195270> >>>>>> > EMAIL [email protected] >>>>>> > >>>>>> > >>>>>> > Snowflake Inc. >>>>>> > 135 Constitution Drive >>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>> >>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g>> >>>>>> Menlo Park, CA 94025, USA >>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>> >>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>>> > >>>>>> >>>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>>> >>>>> >>>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>> >>>> >>>> >>>> -- >>>> THOMAS KISSINGER >>>> Staff Software Engineer >>>> >>>> MOBILE +49 174-2195270 <+49%20174%202195270> >>>> EMAIL [email protected] >>>> >>>> >>>> Snowflake Inc. >>>> 135 Constitution Drive >>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>> Menlo Park, CA 94025, USA >>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>>> >>>> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >>> >>> >> >> -- >> THOMAS KISSINGER >> Staff Software Engineer >> >> MOBILE +49 174-2195270 <+49%20174%202195270> >> EMAIL [email protected] >> >> >> Snowflake Inc. >> 135 Constitution Drive >> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >> Menlo Park, CA 94025, USA >> <https://www.google.com/maps/search/135+Constitution+Drive+%0D%0A+Menlo+Park,+CA+94025,+USA?entry=gmail&source=g> >> >
