Do we have a good definition of what is necessary to add a new data type? Adding a type but not pulling it through most of the code seems less than ideal since it means one part of Arrow doesn't work with another (providing a less optimal end-user experience).
For example, would this work include making Gandiva and all the kernels support this new data type where appropriate? On Wed, Aug 5, 2020 at 12:01 PM Wes McKinney <wesmck...@gmail.com> wrote: > Sounds fine to me. I guess one question is what needs to be formalized > in the Schema.fbs files or elsewhere in the columnar format > documentation (and we will need to hold an associated vote for that I > think) > > On Mon, Aug 3, 2020 at 11:30 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > Given no objections, we'll go ahead and start implementing support for > 256-bit decimals. > > > > I'm considering setting up another branch to develop all the components > so they can be merged to master atomically. > > > > Thanks, > > Micah > > > > On Tue, Jul 28, 2020 at 6:39 AM Wes McKinney <wesmck...@gmail.com> > wrote: > >> > >> Generally this sounds fine to me. At some point it would be good to > >> add 32-bit and 64-bit decimal support but this can be done in the > >> future. > >> > >> On Tue, Jul 28, 2020 at 7:28 AM Fan Liya <liya.fa...@gmail.com> wrote: > >> > > >> > Hi Micah, > >> > > >> > Thanks for opening the discussion. > >> > I am aware of some scenarios where decimal requires more than 16 > bytes, so > >> > I think it would be beneficial to support this in Arrow. > >> > > >> > Best, > >> > Liya Fan > >> > > >> > > >> > On Tue, Jul 28, 2020 at 11:12 AM Micah Kornfield < > emkornfi...@gmail.com> > >> > wrote: > >> > > >> > > Hi Arrow Dev, > >> > > ZetaSQL (Google's open source standard SQL library) recently > introduced a > >> > > BigNumeric [1] type which requires a 256 bit width to properly > support it. > >> > > I'd like to add support (possibly in collaboration with some of my > >> > > colleagues) to add support for 256 bit width Decimals in Arrow to > support a > >> > > type corresponding to BigNumeric. > >> > > > >> > > In past discussions on this, I don't think we established a minimum > bar for > >> > > supporting additional bit-widths within Arrow. > >> > > > >> > > I'd like to propose the following requirements: > >> > > 1. A vote agreeing on adding support for a new bitwidth (we can > discuss > >> > > any objections here). > >> > > 2. Support in Java and C++ for integration tests verifying the > ability to > >> > > round-trip the value. > >> > > 3. Support in Java for conversion to/from BigDecimal [2] > >> > > 4. Support in Python converting to/from Decimal [3] > >> > > > >> > > Is there anything else that people feel like is a requirement for > basic > >> > > support of an additional bit width for Decimal's? > >> > > > >> > > Thanks, > >> > > Micah > >> > > > >> > > > >> > > [1] > >> > > > >> > > > https://github.com/google/zetasql/blob/1aefaa7c62fc7a50def879bb7c4225ec6974b7ef/zetasql/public/numeric_value.h#L486 > >> > > [2] > https://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html > >> > > [3] https://docs.python.org/3/library/decimal.html > >> > > >
