Thomas, Dongjoon, I stand corrected on last year's feedback. You (Thomas) were there, I was not. So my information was second hand.
I agree with Dongjoon, that Spark should have a pure Java solution which does not depend on third party libraries. It is my understanding that, while BigDecimal is portable, it has a restrictive license. I have taken the liberty of porting the open source libbid library to Java - to the extend needed to do benchmarking - and am attaching the findings. For kicks, I also extended support to DECFLOAT(70), so we can project performance for larger precision. I do not want to overstate the effort. Consider it a prototype to gauge complexity and estimate performance, I’ll happily share the code and harness. It is my understanding that Spark thrives to be an open system which does not have favorite datasources. As such Spark should, wherever possible, bend towards being least restrictive. Adopting the semantic limitations of BigDecimal constitutes a one-way door. It means Spark cannot, by design, operate one some numerical data from specific datasources or specific applications. Cheers Serge [Screenshot 2026-08-26 at 11.36.59 PM.png] On Aug 25, 2026, at 2:06 PM, Thomas Kissinger via dev <[email protected]> wrote: Hi Serge, Let me briefly introduce myself. I co-authored last year’s Parquet DECFLOAT proposal and currently participate in the renewed discussion of a similar type on the Parquet dev list: https://lists.apache.org/thread/5kp1bl2czz45wflydq2qzs3nld518lox https://lists.apache.org/thread/p62ns0qmyko331crhnxxdoy25mdm4bnz I do not think this is a chicken-and-egg problem. As I understood the feedback last year, the concern was not that an open-source engine had to implement DECFLOAT before Parquet could proceed. The concern was that DECFLOAT appeared to be a niche type, and standardizing a new Parquet type required stronger evidence of its importance and broader demand. That concern is much less applicable today, and this Spark SPIP provides further evidence of the type’s relevance. IEEE standardization is valuable, but it should not be confused with broad implementation support. Direct support for IEEE decimal floating-point remains limited across mainstream language runtimes. Decimal128 itself is limited to 34 digits. Wider formats derived from the generalized layout currently lack the same implementation ecosystem; the cited Intel and Boost implementations stop at decimal128. The SPIP is perhaps the clearest demonstration of this gap in Java. Java has no standard value type implementing complete IEEE decimal128 semantics, and the SPIP does not identify an established pure-Java implementation. It therefore leans toward JNI calls into libbid for arithmetic, introducing platform-dependent native binaries into Spark’s arithmetic path. Java already provides the mature and extensively used java.math.BigDecimal, with arbitrary-precision finite decimal arithmetic. It is an important portable implementation that should be considered. It supports precision of 38 digits and beyond, aligning more naturally with the established SQL numeric ecosystem than a 34-digit type. The latter can cause problems in areas such as type coercion for joins, as Shrirang already pointed out during the discussion. I agree that Spark’s runtime representation need not match Parquet’s physical encoding or cover its complete domain. However, their semantic models and conversion contract should be well aligned. Precision, range, cohorts, special values, and round-trip guarantees should not be decided independently and reconciled afterward. My recommendation would therefore be to reach agreement on the Parquet type and its requirements first, and then base the Spark SPIP’s storage mapping on that stable contract. I, of course, leave it to you and the other SPIP authors to decide whether that warrants pausing or restructuring the current proposal. Please let us know how you plan to proceed. Best, Thomas From: Serge Rielau <[email protected]<mailto:[email protected]>> To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: Stefan Kandic <[email protected]<mailto:[email protected]>> Bcc: Date: Tue, 25 Aug 2026 16:44:08 +0000 Subject: Re: [VOTE] SPIP: Add the DECFLOAT data type Russell, I think we are facing a chicken egg problem here. One of the objections to the Parquet proposal from last year was that there were no stated needs from Open Source engines. That being said, I think runtime engines and storage play different roles here. Where our discussion on Parquet centers around maximizing what the storage type can accommodate (cohorts, special values, maximum precision, range, etc). The Spark runtime engine should focus on what its users are likely to use, general directive to prefer standards, and a preference to rely on existing, battle hardened libraries. It goes without saying that this SPIP shall rely on the Parquet type. However, it will not need to cover its domain completely, or match its encoding. Cheers Serge On Aug 25, 2026, at 6:56 AM, Russell Spitzer <[email protected]<mailto:[email protected]>> wrote: The discussion thread and design doc seem to only have two people discussing the topic, are we sure there aren't any other comments? I know of some folks who are chiming in on the similar Parquet proposal, and I was wondering if we should get alignment there before adding the type to Spark? On Tue, Aug 25, 2026 at 8:35 AM Shrirang Mhalgi <[email protected]<mailto:[email protected]>> wrote: +1 (non binding) Thank you Uroš. Regards, Shrirang. On Tue, Aug 25, 2026 at 6:24 AM Serge Rielau <[email protected]<mailto:[email protected]>> wrote: +1 (non binding) On Aug 25, 2026, at 5:20 AM, Stefan Kandic via dev <[email protected]<mailto:[email protected]>> wrote: + 1 From: Uroš Bojanić <[email protected]<mailto:[email protected]>> Date: Tuesday, 25 August 2026 at 13:20 To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: [VOTE] SPIP: Add the DECFLOAT data type Hi all, Discussion regarding DECFLOAT data type has settled, so I'm calling a vote on the SPIP "Add the DECFLOAT data type". The SPIP proposes to add a new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 floating-point decimals with per-value exponents. The SPIP document has been updated along the way to address the feedback raised in the doc & discussion thread. DISCUSS thread: https://lists.apache.org/thread/wyfnsj45s3cn95thrgw4oho74zbcc5vm Spark JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820 SPIP document: https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E The vote is open for at least the next 72 hours. [ ] +1: Accept the proposal as an official SPIP [ ] +0 [ ] -1: I don't think this is a good idea because... Thank you! Best, Uroš --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]<mailto:[email protected]>
