Perhaps my maths is incorrect, but a decimal64 would have a maximum
precision of 18, not 19? log(9223372036854775807) = 18.9?
On 09/11/2023 16:01, Curt Hagenlocher wrote:
Recently, someone opened an issue on GitHub ([C++] Decimal64/32 support? ·
Issue #38622 · apache/arrow (github.com)
<https://github.com/apache/arrow/issues/38622>) asking for support for
narrower decimal types. They were advised to start a thread on the mailing
list, and as they haven't done so yet I will start.
It's fairly common to store currency in databases in a type that's
compatible with decimal64. Both PostgreSQL and Microsoft SQL Server have
"money" data types; for Postgres, this is a decimal(19, 2) and for SQL
Server it's decimal(19, 4). Microsoft Analysis Services also uses
decimal(19, 4) as one of its core data types. If you search the internet
for suggestions on the database type to use for money, the vast majority
recommend a decimal type with a precision <= 19. Currency is something
stored very frequently as data, and it makes sense to have a type that's
optimized for this purpose. I submit that it's a far more common type than
float16, and that even if it's not as hip as the AI scenarios which
popularized float16, the ultimate goal of those scenarios is, after all, to
make more "money".
decimal64 is considerably easier to work with on modern CPUs and in common
programming languages than decimal128, and requires half the amount of
storage space. And while adding new types to Arrow obviously needs to be
done very sparingly, it's harder to imagine a new type for which support
would be easier to implement than this one.
I think decimal32 is much harder to justify. MS SQL Server has a
"smallmoney" (decimal(10, 4)), but I suspect it's not that heavily used.
Maybe others have more feedback on this one.
-Curt
