On Mon, 22 Jul 2019 08:40:08 -0700 Brian Hulette <hulet...@gmail.com> wrote: > To me, the most important aspect of this proposal is the addition of sparse > encodings, and I'm curious if there are any more objections to that > specifically. So far I believe the only one is that it will make > computation libraries more complicated. This is absolutely true, but I > think it's worth that cost.
It's not just computation libraries, it's any library peeking inside Arrow data. Currently, the Arrow data types are simple, which makes it easy and non-intimidating to build data processing utilities around them. If we start adding sophisticated encodings, we also raise the cost of supporting Arrow for third-party libraries. Regards Antoine.
