Hi Costas, Thank you for the thoughtful reply and for separating the requirements from the physical representation. Starting with the requirements is the right approach, and we are confident that we can find a solution covering the important use cases on both sides.
Thank you also for sharing the Spark SPIP. Seeing the proposed engine semantics alongside the Parquet representation is very helpful. Although Spark and Parquet have separate decision processes, both projects would benefit if these efforts converge on shared requirements rather than develop alongside each other with different assumptions. In particular, aligning on precision and range, which numerical and representational distinctions must survive a round-trip, and which special values are supported before either design is finalized would avoid unnecessary conversion and interoperability gaps. Our current view is the following. Precision and range We agree that the type should not impose a permanent 38-digit ceiling and should cover at least the complete finite decimal128 range. Narrower and wider precisions should be possible, with the exact supported set decided when evaluating the physical representations. This requirement also exposes a mismatch with the generalized IEEE interchange layout. Its precision ladder moves from 34 to 43 digits around the established 38-digit SQL boundary, and from 70 to 79 around the 76-digit boundary. Current Intel and Boost implementations stop at decimal128, and we are not aware of an interoperable production decimal160 codec. Signed 128- and 256-bit integer significands support precisions 38 and 76 directly, aligning more naturally with existing database decimal representations. Quantum, equality, and hashing We agree that some applications need to preserve the original quantum, and FHIR is a useful example. The question is not whether this capability should exist, but whether it should be mandatory for every column. For workloads that require only ordinary numerical semantics, preserving cohorts adds work to equality and hash-based operations without providing additional value. When cohort members are retained, physical identity and numerical identity diverge. Equality checks must handle multiple encodings of the same value, while hash joins, grouping, DISTINCT, and partitioning must derive a cohort- invariant numerical key for every processed value or maintain an additional canonical key. That adds work and state to fundamental database paths. Because Parquet Bloom filters currently hash the PLAIN encoding, a quantum-preserving mode would require a normative cohort-invariant key shared by writers and readers; any disagreement could cause incorrect pruning. Preserving cohort members also increases physical cardinality: numerically equal values may be stored as different byte sequences. This can create additional dictionary entries and reduce repeated byte patterns available to compression. The actual size impact will depend on the data distribution and should be included in the benchmark. We therefore propose making quantum preservation an explicit parameter of the logical type, with canonical representation as the default. A canonical mode would require one representation per finite numerical value, while a quantum- preserving mode would retain the supplied exponent for applications requiring faithful round-tripping. Both modes would use ordinary numerical equality; the parameter would define the representation round-trip contract. This makes the semantic and performance tradeoff explicit rather than imposing either choice on every consumer. Special values We agree on positive and negative infinity and a canonical NaN. We are less convinced that signed zero, signaling NaNs, and NaN payloads belong in an interoperable Parquet type, since many important target type systems cannot represent them reliably. Signaling NaN is difficult to reconcile with an interoperable columnar format because its behavior depends on how a value is inspected. IEEE numerical comparisons signal invalid operation for an sNaN, while totalOrder, which is appropriate for statistics, does not. Parquet uses values through several paths—statistics and pruning, dictionary processing, and row-level predicate evaluation—that may therefore treat the same sNaN differently. With traps enabled, observable query behavior could even depend on whether an optimization path is used. Supporting sNaN would consequently require precise rules for statistics, pruning, predicates, dictionaries, quieting, and round-tripping. That is substantial complexity for an execution mechanism without a demonstrated Parquet interoperability requirement. In the exchange-rate example, NULL or an explicit state column would express missing or invalid input more predictably across engines. Signed zero creates a similar interoperability problem to signaling NaN. It introduces two encodings for the same numerical zero and therefore requires special treatment in equality, hashing, Bloom filters, ordering, and statistics. Many decimal type systems and conversion paths do not preserve the distinction and silently turn negative zero into ordinary zero. A downstream consumer relying on the sign may then observe different behavior without any indication that information was lost. This is misleading in an interoperable format, so unless a compelling use case is identified, we prefer one canonical zero. We do not think NaN payloads should be included. IEEE defines their representation but not interoperable payload semantics, and does not fully specify payload selection when multiple NaNs participate in an operation. Parquet could preserve the bytes, but that would impose a round-trip obligation for opaque metadata that receiving engines cannot interpret consistently. Diagnostic information is more portable and queryable in a separate column. Performance and benchmarks We agree that, once the requirements are finalized, we should work together on a public benchmark comparing equivalent semantics and documenting distributions, rounding behavior, fast paths, fallbacks, compiler settings, and hardware. It should cover fundamental database operations, including encoding, decoding, cohort reduction and canonical-key derivation, equality, and arithmetic. Until the existing measurements can be reproduced under equivalent semantics, they should be treated as directional evidence rather than a representation-level result. Please feel free to propose an initial benchmark design already; it could provide a useful foundation for the joint version. Storage and runtime representation We agree that engines can decode the Parquet representation once and use a different runtime representation. This amortizes conversion across long pipelines but does not eliminate representation-dependent work. Values surviving pruning must still be interpreted or decoded for row-level filters, numerical hashing, statistics construction on write, or runtime materialization. The physical representation therefore remains relevant to scan-heavy workloads. Validation We likewise agree that, once the requirements are finalized, we should jointly develop a public validation suite independently of the performance benchmark. Please feel free to propose an initial set of test vectors already as a foundation. It should eventually cover canonical and quantum-preserving values, precision boundaries, special values, non-canonical, reserved, and invalid encodings as applicable, equality, hashing, ordering and statistics, and cross-implementation round-trips. Thanks again for moving the discussion toward explicit requirements. We already agree on many important points: support for at least 38 digits with a path to wider precisions, the decimal128 finite range, infinities and a canonical NaN, and public benchmarks and test vectors. The remaining discussion is focused on whether quantum preservation should be optional, which additional IEEE distinctions belong in an interoperable database type, and which physical representation best satisfies the agreed requirements. We believe an extensible type with an explicit quantum-preservation choice offers a practical path forward. Best, Thomas On Fri, Aug 14, 2026 at 10:38 PM Costas Zarifis via dev < [email protected]> wrote: > Hi Thomas, > > Thank you for laying out the requirements explicitly. We agree that > requirements should come before selecting a physical representation. > There is substantial agreement, but we see several requirements > differently. > > For context on engine integration, we have also published a draft > Spark Project Improvement Proposal (SPIP): Add the DECFLOAT data type > ( > https://docs.google.com/document/d/1mTSOoCvX7yYMSa6CpI6OPS-MQZocPxUtrrFOEiZjCPk/edit > ). > The SPIP covers the proposed Spark SQL type and execution semantics, > while this discussion focuses on the portable Parquet representation. > > Before choosing BID or a new 18-byte format, we suggest first agreeing > on the following: > > - Precision: We agree that the format must losslessly support 38-digit > values. Your proposal targets exactly 38 digits, while we think the > format should provide a defined path to both narrower and wider > precisions. > - Range: We agree that the format should cover at least the complete > finite decimal128 range. > - Cohorts: Your proposal requires one canonical representation per > numerical value. We think the original coefficient, exponent, and > quantum should round-trip. > - Special values: We agree on infinity and NaN. We additionally think > engines supporting signed zero, signaling NaNs, and NaN payloads > should be able to round-trip them. > - Hashing and equality: We agree that numerically equal values must > behave consistently in equality and hash-based operations. We do not > think that requires canonicalizing the stored representation. > - Performance: We take the reported results seriously, but think the > candidate representations should be compared using equivalent > semantics, public benchmarks, and independent implementations. > - Validation: Correctness and interoperability should be validated > using public test vectors, separately from performance evaluation. > > The sections below provide the rationale for each position. > > Precision and range > --------------------------- > > We agree that a Parquet decimal-floating-point type must losslessly > represent every numerical value in the DECIMAL(38,s) domain. > > We do not think the format should necessarily stop at exactly 38 > digits, however. Precision 38 is an important interoperability > boundary established by an original crop of DBMS, which does not > establish that present or future requirements stop at 38 digits. > > Many open source or major newer DBMS have higher limits: > > - PostgreSQL > Type: NUMERIC unconstrained > Max precision: 131,072 before decimal / 16,383 after; declared > NUMERIC(p,s) up to p=1000 > Source: https://www.postgresql.org/docs/current/datatype-numeric.html > > - MySQL / MariaDB / SingleStore > Type: DECIMAL > Max precision: 65 > Source: https://dev.mysql.com/doc/refman/8.0/en/fixed-point-types.html > > - BigQuery > Type: BIGNUMERIC > Max precision: ~76.8 (scale 38) > Source: > https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types > > - ClickHouse > Type: Decimal256 > Max precision: 76 > Source: https://clickhouse.com/docs/sql-reference/data-types/decimal > > - Oracle > Type: unconstrained NUMBER > Max precision: ~40 (parameterized 38) > Source: > https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlqr/Data-Types.html > > - Trino ≥480 > Type: NUMBER > Max precision: >= 50 > Source: https://trino.io/docs/current/language/types.html#number > > While we can opine on the value of precision past a certain limit, we > cannot pass judgement on whether the cut-off is 34, 38, 43 or > something higher. > > Therefore Parquet should avoid making any such precision a permanent > semantic ceiling. > > Faithful round-tripping and cohorts > ---------------------------------------------- > > We do not agree that Parquet should require one cohort-canonical > stored representation for every finite numerical value. > > Although 1.5, 1.50, and 1.500 are numerically equal, their different > quantum can carry information that some applications must preserve. > Engines can provide numerical operations alongside IEEE-defined > cohort-sensitive operations such as sameQuantum and totalOrder, the > latter often exposed as compareTotal. These operations are only > possible if the storage format retains the original representation. > Canonicalizing on write permanently removes that option. > > A concrete example is HL7 FHIR healthcare data > (https://hl7.org/fhir/datatypes.html). FHIR explicitly specifies that > decimal precision is significant: 0.010 is different from 0.01, and > implementations must preserve that distinction. This requirement > concerns faithful interchange and presentation rather than numerical > equality: 0.010 and 0.01 have the same numerical value, but FHIR > requires their represented precision to survive a round-trip. A > healthcare data lake using Parquet as an interchange or archival > format must therefore retain the original quantum to reconstruct > standards-compliant FHIR resources. Canonicalizing on write would make > that round-trip impossible. > > Beyond FHIR, this behavior appears in other decimal arithmetic > ecosystems. ISO/IEC TS 18661-2 > (https://open-std.org/JTC1/SC22/WG14/www/docs/n1912.pdf), which > specifies C language support for decimal floating-point arithmetic, > explicitly states that 1.0 and 1.00 are distinguishable. The > language-independent General Decimal Arithmetic specification > (https://speleotrove.com/decimal/daops.html), which underpins several > decimal arithmetic libraries, preserves trailing zeros unless the > caller explicitly requests `reduce`. The same principle appears > independently in measurement and reporting standards. ISO 80000-1 > ( > https://www.iso.org/home.isoDocumentsDownload.do?t=0bc30Dw3Fis0Kmeb_a6O_bzO2QP14DxJAxwtmaRLmv-WwspRuGQ-iHB6VgCGRJnU > ), > NIST SP 811 ( > https://www.nist.gov/pml/special-publication-811/nist-guide-si-chapter-7-rules-and-style-conventions-expressing-values > ), > and ASTM E29 ( > https://www.galvanizeit.com/uploads/resources/ASTM-E-29-yr-13.pdf) > all treat trailing zeros after a decimal point as significant in > reported values. > > The Bloom-filter example identifies a real correctness concern, but > optional Bloom-filter support should not require writers to discard > quantum information from stored values. If Bloom-filter support is > deemed necessary, Parquet could canonicalize only the numerical hash > key. > > For example: > > Stored representation Numerical Bloom-filter key > 1.5 = (15, -1) (15, -1) > 1.50 = (150, -2) (15, -1) > 1.500 = (1500, -3) (15, -1) > > Writers and readers would hash the same normalized numerical form. > This keeps numerical-equality Bloom filters correct without changing > the stored representation. Otherwise, readers could disable > Bloom-filter pruning for DECFLOAT equality. > > Special values > ------------------- > > We agree that positive and negative infinity and NaN should be > representable. We think the same faithful-round-trip principle should > apply to signed zero, signaling NaNs, and NaN payloads. > > An engine does not need to expose every distinction to its users. It > may normalize or reject some values, or provide traps, similar to > Firebird’s SET DECFLOAT TRAPS > ( > https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref50-management.html > ), > controlling whether division by zero, overflow, invalid operations, or > signaling NaN produce a value or raise an error. Those are execution > policies. > > A concrete example of signaling NaN’s automatic fail-on-use behavior > is a bank preparing its daily foreign-exchange rates. Each currency > pair must receive a rate from a market-data provider before trades can > be valued. The bank could initialize rates that have not yet arrived > to signaling NaN and periodically checkpoint the table to Parquet: > > EUR → USD = 1.17 > GBP → USD = 1.35 > JPY → USD = sNaN > > The valuation job enables the InvalidOperation trap. If it > accidentally attempts to convert a JPY payment before the rate has > arrived, multiplying the payment by sNaN fails immediately at the > point of use. Merely copying, checkpointing, or inspecting the table > does not trigger the exception. A NaN payload could additionally > retain a diagnostic code identifying the missing or invalid data > source. > > Parquet should reproduce the values written to it rather than > normalize away supported distinctions. If it converts sNaN to quiet > NaN, removes NaN payloads, or collapses -0 into +0, engines that > support those values can no longer round-trip them faithfully. Whether > an engine exposes or uses these distinctions should remain an > engine-level decision; the storage format should preserve them rather > than make an irreversible decision for every consumer. > > Performance > ------------------ > > We take the reported performance differences seriously, but the > current results do not isolate the physical representation. They > compare Intel libbid with an optimized Snowflake implementation while > potentially also varying precision, semantics, algorithms, input > distributions, and fast-path behavior. > > Our own benchmarks reinforce this concern: libbid and Boost.Decimal > both use BID, yet differ by several times on some operations. Large > performance gaps can therefore arise from implementation choices even > when the representation is identical. This does not rule out an > inherent advantage in Snowflake’s representation, but the current > comparison does not isolate it. > > Could you share the implementation or enough of the benchmark harness > to reproduce the results, including: > > - input and exponent distributions > - precision and rounding configuration > - treatment of cohorts and special values > - the percentage of operations using the optimized path versus fallback > - compiler and optimization settings > - results on both ARM and x86? > > We should compare equivalent value domains and semantics. For Parquet, > we should also measure encoding, decoding, comparisons, numerical > hashing, statistics, predicate evaluation, compression, and end-to-end > scan and aggregation performance, not only arithmetic. > > We should also separate the Parquet encoding from an engine’s runtime > representation. An engine could decode BID once into its preferred > coefficient/exponent structure, execute an expression pipeline using > that representation, and encode only at the output boundary. A > benchmark comparing that approach would help determine whether the > reported cost is inherent to the interchange encoding or to using > packed BID for every intermediate result. > > Once those requirements are settled, we can evaluate the candidate > representations against the same semantics using reproducible > benchmarks. > > Best, > Costas Zarifis > -- THOMAS KISSINGER Staff Software Engineer MOBILE +49 174-2195270 EMAIL [email protected] Snowflake Inc. 135 Constitution Drive Menlo Park, CA 94025, USA
