Russell, I think we are facing a chicken egg problem here. One of the objections to the Parquet proposal from last year was that there were no stated needs from Open Source engines.
That being said, I think runtime engines and storage play different roles here. Where our discussion on Parquet centers around maximizing what the storage type can accommodate (cohorts, special values, maximum precision, range, etc). The Spark runtime engine should focus on what its users are likely to use, general directive to prefer standards, and a preference to rely on existing, battle hardened libraries. It goes without saying that this SPIP shall rely on the Parquet type. However, it will not need to cover its domain completely, or match its encoding. Cheers Serge On Aug 25, 2026, at 6:56 AM, Russell Spitzer <[email protected]> wrote: The discussion thread and design doc seem to only have two people discussing the topic, are we sure there aren't any other comments? I know of some folks who are chiming in on the similar Parquet proposal, and I was wondering if we should get alignment there before adding the type to Spark? On Tue, Aug 25, 2026 at 8:35 AM Shrirang Mhalgi <[email protected]<mailto:[email protected]>> wrote: +1 (non binding) Thank you Uroš. Regards, Shrirang. On Tue, Aug 25, 2026 at 6:24 AM Serge Rielau <[email protected]<mailto:[email protected]>> wrote: +1 (non binding) On Aug 25, 2026, at 5:20 AM, Stefan Kandic via dev <[email protected]<mailto:[email protected]>> wrote: + 1 From: Uroš Bojanić <[email protected]<mailto:[email protected]>> Date: Tuesday, 25 August 2026 at 13:20 To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: [VOTE] SPIP: Add the DECFLOAT data type Hi all, Discussion regarding DECFLOAT data type has settled, so I'm calling a vote on the SPIP "Add the DECFLOAT data type". The SPIP proposes to add a new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 floating-point decimals with per-value exponents. The SPIP document has been updated along the way to address the feedback raised in the doc & discussion thread. DISCUSS thread: https://lists.apache.org/thread/wyfnsj45s3cn95thrgw4oho74zbcc5vm Spark JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820 SPIP document: https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E The vote is open for at least the next 72 hours. [ ] +1: Accept the proposal as an official SPIP [ ] +0 [ ] -1: I don't think this is a good idea because... Thank you! Best, Uroš --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]<mailto:[email protected]>
