Narrower Decimal types has come up in the past as something that is desirable (for instance parquet supports using ints of similar precision/scale). IIRC I think the main blocker has been people willing to work on two compatible implementations.
FWIW, we decided for Decimal256 to be conservative on the bounds I believe (e.g. max precision 76 instead of 77). Thanks, Micah On Thu, Nov 9, 2023 at 8:30 AM Curt Hagenlocher <c...@hagenlocher.org> wrote: > (But yes, I think this would be more commonly described as having a > precision of 18 and my writeup was probably influenced by looking at the > SQL Server and Postgres descriptions of the "money" type, both of which > allow the full range of the underlying 64-bit value to be used.) > > On Thu, Nov 9, 2023 at 8:23 AM Curt Hagenlocher <c...@hagenlocher.org> > wrote: > > > Obviously the limits don't match up exactly, so it depends on whether you > > want to express the decimal bounds in a way that captures all the > possible > > underlying values or in a way that allows all the decimal values to be > > represented as the underlying type. Or more succinctly, > > "111,111,111,111,111.1111" will fit into a decimal64; would you prevent > it > > from being stored in one so that you can describe the column as > > "decimal(18, 4)"? In any event, all scaled integer representations of > > decimal values have the same question and it's more-or-less orthogonal to > > having a decimal64 type. > > > > On Thu, Nov 9, 2023 at 8:17 AM Raphael Taylor-Davies > > <r.taylordav...@googlemail.com.invalid> wrote: > > > >> Perhaps my maths is incorrect, but a decimal64 would have a maximum > >> precision of 18, not 19? log(9223372036854775807) = 18.9? > >> > >> On 09/11/2023 16:01, Curt Hagenlocher wrote: > >> > Recently, someone opened an issue on GitHub ([C++] Decimal64/32 > >> support? · > >> > Issue #38622 · apache/arrow (github.com) > >> > <https://github.com/apache/arrow/issues/38622>) asking for support > for > >> > narrower decimal types. They were advised to start a thread on the > >> mailing > >> > list, and as they haven't done so yet I will start. > >> > > >> > It's fairly common to store currency in databases in a type that's > >> > compatible with decimal64. Both PostgreSQL and Microsoft SQL Server > have > >> > "money" data types; for Postgres, this is a decimal(19, 2) and for SQL > >> > Server it's decimal(19, 4). Microsoft Analysis Services also uses > >> > decimal(19, 4) as one of its core data types. If you search the > internet > >> > for suggestions on the database type to use for money, the vast > majority > >> > recommend a decimal type with a precision <= 19. Currency is something > >> > stored very frequently as data, and it makes sense to have a type > that's > >> > optimized for this purpose. I submit that it's a far more common type > >> than > >> > float16, and that even if it's not as hip as the AI scenarios which > >> > popularized float16, the ultimate goal of those scenarios is, after > >> all, to > >> > make more "money". > >> > > >> > decimal64 is considerably easier to work with on modern CPUs and in > >> common > >> > programming languages than decimal128, and requires half the amount of > >> > storage space. And while adding new types to Arrow obviously needs to > be > >> > done very sparingly, it's harder to imagine a new type for which > support > >> > would be easier to implement than this one. > >> > > >> > I think decimal32 is much harder to justify. MS SQL Server has a > >> > "smallmoney" (decimal(10, 4)), but I suspect it's not that heavily > used. > >> > Maybe others have more feedback on this one. > >> > > >> > > >> > -Curt > >> > > >> > > >
