Hi Serge, Let me briefly introduce myself. I co-authored last year’s Parquet DECFLOAT proposal and currently participate in the renewed discussion of a similar type on the Parquet dev list:
https://lists.apache.org/thread/5kp1bl2czz45wflydq2qzs3nld518lox https://lists.apache.org/thread/p62ns0qmyko331crhnxxdoy25mdm4bnz I do not think this is a chicken-and-egg problem. As I understood the feedback last year, the concern was not that an open-source engine had to implement DECFLOAT before Parquet could proceed. The concern was that DECFLOAT appeared to be a niche type, and standardizing a new Parquet type required stronger evidence of its importance and broader demand. That concern is much less applicable today, and this Spark SPIP provides further evidence of the type’s relevance. IEEE standardization is valuable, but it should not be confused with broad implementation support. Direct support for IEEE decimal floating-point remains limited across mainstream language runtimes. Decimal128 itself is limited to 34 digits. Wider formats derived from the generalized layout currently lack the same implementation ecosystem; the cited Intel and Boost implementations stop at decimal128. The SPIP is perhaps the clearest demonstration of this gap in Java. Java has no standard value type implementing complete IEEE decimal128 semantics, and the SPIP does not identify an established pure-Java implementation. It therefore leans toward JNI calls into libbid for arithmetic, introducing platform-dependent native binaries into Spark’s arithmetic path. Java already provides the mature and extensively used java.math.BigDecimal, with arbitrary-precision finite decimal arithmetic. It is an important portable implementation that should be considered. It supports precision of 38 digits and beyond, aligning more naturally with the established SQL numeric ecosystem than a 34-digit type. The latter can cause problems in areas such as type coercion for joins, as Shrirang already pointed out during the discussion. I agree that Spark’s runtime representation need not match Parquet’s physical encoding or cover its complete domain. However, their semantic models and conversion contract should be well aligned. Precision, range, cohorts, special values, and round-trip guarantees should not be decided independently and reconciled afterward. My recommendation would therefore be to reach agreement on the Parquet type and its requirements first, and then base the Spark SPIP’s storage mapping on that stable contract. I, of course, leave it to you and the other SPIP authors to decide whether that warrants pausing or restructuring the current proposal. Please let us know how you plan to proceed. Best, Thomas > From: Serge Rielau <[email protected]> > To: "[email protected]" <[email protected]> > Cc: Stefan Kandic <[email protected]> > Bcc: > Date: Tue, 25 Aug 2026 16:44:08 +0000 > Subject: Re: [VOTE] SPIP: Add the DECFLOAT data type > Russell, > > I think we are facing a chicken egg problem here. One of the objections to > the Parquet proposal from last year was that there were no stated needs > from Open Source engines. > > That being said, I think runtime engines and storage play different roles > here. > Where our discussion on Parquet centers around maximizing what the storage > type can accommodate (cohorts, special values, maximum precision, range, > etc). The Spark runtime engine should focus on what its users are likely to > use, general directive to prefer standards, and a preference to rely on > existing, battle hardened libraries. > > It goes without saying that this SPIP shall rely on the Parquet type. > However, it will not need to cover its domain completely, or match its > encoding. > > Cheers > Serge > > > On Aug 25, 2026, at 6:56 AM, Russell Spitzer <[email protected]> > wrote: > > The discussion thread and design doc seem to only have two people > discussing the topic, are we sure there aren't any other comments? I know > of some folks who are chiming in on the similar Parquet proposal, and I was > wondering if we should get alignment there before adding the type to Spark? > > On Tue, Aug 25, 2026 at 8:35 AM Shrirang Mhalgi <[email protected]> > wrote: > >> +1 (non binding) Thank you Uroš. >> >> Regards, >> Shrirang. >> >> On Tue, Aug 25, 2026 at 6:24 AM Serge Rielau <[email protected]> wrote: >> >>> +1 (non binding) >>> >>> >>> On Aug 25, 2026, at 5:20 AM, Stefan Kandic via dev <[email protected]> >>> wrote: >>> >>> + 1 >>> >>> *From: *Uroš Bojanić <[email protected]> >>> *Date: *Tuesday, 25 August 2026 at 13:20 >>> *To: *[email protected] <[email protected]> >>> *Subject: *[VOTE] SPIP: Add the DECFLOAT data type >>> >>> Hi all, >>> >>> Discussion regarding DECFLOAT data type has settled, so I'm calling a >>> vote on the SPIP "Add the DECFLOAT data type". The SPIP proposes to add a >>> new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for >>> base-10 floating-point decimals with per-value exponents. The SPIP document >>> has been updated along the way to address the feedback raised in the doc & >>> discussion thread. >>> >>> DISCUSS thread: >>> https://lists.apache.org/thread/wyfnsj45s3cn95thrgw4oho74zbcc5vm >>> Spark JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820 >>> SPIP document: >>> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E >>> >>> The vote is open for at least the next 72 hours. >>> >>> [ ] +1: Accept the proposal as an official SPIP >>> [ ] +0 >>> [ ] -1: I don't think this is a good idea because... >>> >>> Thank you! >>> >>> Best, >>> Uroš >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> >>> >>> >
