Hi Wes, I'm +1 on this. As part of the PR it might good to beef up the documentation in general: 1. Encoding representation expected for bytes. 2. Add a clarification that the only accepted lengths are 16 for the time being.
Thanks, Micah On Mon, Jun 1, 2020 at 3:48 PM Wes McKinney <[email protected]> wrote: > I mentioned this on the recent sync call and opened > > https://issues.apache.org/jira/browse/ARROW-8985 > > I believe at some point that Arrow may need to be used to transport > decimal widths different from 128 bits. For example systems like > Apache Kudu have 32-bit and 64-bit decimals. Computational code may > immediately promote small decimals, but it's valuable to be able to > transfer and represent the data as is rather than forcing an > up-promotion even for low-precision decimal data. > > In order to allow for this work to possibly happen in the future > without requiring a new value be added to the "Type" Flatbuffers > union, I propose to add a "byteWidth" field with default value 16 to > the existing Decimal type. Here is a patch with this change: > > https://github.com/apache/arrow/pull/7321 > > To make the forward compatibility issue clear, if this field is not > added then current library versions would not be able to perceive the > absence of the field, this making it unsafe for future library > versions to annotate anything other than 16-byte decimals with this > metadata. > > As part of adopting this change, we would want to add assertions to > the existing libraries to check that the byteWidth is indeed 16 and > either throwing an exception or passing through the data as > FixedSizeBinary otherwise. > > Thanks, > Wes >
