jorgecarleitao commented on pull request #9232: URL: https://github.com/apache/arrow/pull/9232#issuecomment-766611408
> Arrow is used inside DF, which is used to build databases on to of it. If the user defines `DECIMAL(2,2)` it's a bit overhead to use i128 as representation for it. @ovr, maybe there is some misunderstanding of the Arrow format and the scope of this crate. Arrow format's goal is to address a common problem in performing analytics: interoperability of data between implementations. For example, the internal memory representation of data in spark is different from Python's numpy, which means that using some numpy function against that data is expensive because it incurs a complete serialization-deserialization roundtrip to handle spark datum. The goal of this crate is to offer the necessary implementation in Rust to allow handling the arrow format in memory. If we start adding extra types to the crate, we fall to the exact same problem that we were trying to solve in the first place: other implementations will need to implement a custom serialize and deserialize and therefore they will have to both implement it and incur the serialization-deserialization roundtrip cost. This is the reason for @alamb 's comment that, if you think that arrow would benefit from other decimal types, then the right place to do it is on a broader forum that includes all other implementations (and the discussion of the spec itself). Does this make sense? > Memory usage, DecimalArray<Decimal128> will use 16 bytes for each element. The argument is not about memory usage, but about memory layout and how it is consistent with a specification. Arrow specification is very specific about how memory is lay out so that it enables a [stable ABI](https://arrow.apache.org/docs/format/CDataInterface.html) to share data within the same process via foreign interfaces as well as across processes via the [flight protocol](https://arrow.apache.org/docs/format/Flight.html). ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
