Have you considered using fixed-length binary values for these?
Crypto algorithms might logically be defined in terms of mathematical
operations on integers, but their efficient implementation tends to feature
inlined operations at the machine word level instead of generic add, div,
mod, mul operations at the big-integer level. Having these as logical
integers in Arrow would be a lot of work and they wouldn’t be adequate
building blocks for efficient crypto algorithms.
For instance, compare the pseuso-code for Poly1305
clamp(r): r &= 0x0ffffffc0ffffffc0ffffffc0fffffff
poly1305_mac(msg, key):
r = (le_bytes_to_num(key[0..15])
clamp(r)
s = le_num(key[16..31])
accumulator = 0
p = (1<<130)-5
for i=1 upto ceil(msg length in bytes / 16)
n = le_bytes_to_num(msg[((i-1)*16)..(i*16)] | [0x01])
a += n
a = (r * a) % p
end
a += s
return num_to_16_le_bytes(a)
end
with an implementation using only 128-bit and 64-bit math operations
https://github.com/hacl-star/hacl-star/blob/main/dist/gcc-compatible/Hacl_Poly1305_128.c
What crypto algorithms are you trying to implement using Arrow data?
—
Felipe
On Tue, 23 May 2023 at 14:13 Ian Joiner <[email protected]> wrote:
> Hi,
>
> We need to have really large integers (with 128, 256 and 512 bits) as well
> as decimals (up to at least decimal1024) because they do actually exist in
> crypto / web3 space.
>
> See https://docs.rs/primitive-types/latest/primitive_types/ for an example
> of what needs to be supported.
>
> If accepted we can implement the types for C++/Python and Rust.
>
> Thanks,
> Ian
>
