To add to this, I'm not suggesting to change Iceberg writers to support writing non-optional unions. The motivation for this is to support legacy datasets [not written by Iceberg].
On Mon, May 13, 2019 at 10:05 AM RD <rdsr...@gmail.com> wrote: > Iceberg today does not support non optional unions and that is the right > behaviour, but we do have a lot of datasets which have non-optional union > fields. I'm wondering whether Iceberg should allow reading these datasets > as long as the user does not project the union field. > > I tried it out and today is throws an exception during column pruning. > > If we think this should be support, I'll create an issue for this. > > -Best, > R. >
