Are you talking about Avro data? I think Parquet data would work fine because unions are represented as a struct of optionals.
I think this makes sense. Maybe we could also allow projecting the contents of unions by representing them as structs of optionals and materializing them that way. I'd be up for reviewing this. On Mon, May 13, 2019 at 1:48 PM RD <rdsr...@gmail.com> wrote: > To add to this, I'm not suggesting to change Iceberg writers to support > writing non-optional unions. The motivation for this is to support legacy > datasets [not written by Iceberg]. > > On Mon, May 13, 2019 at 10:05 AM RD <rdsr...@gmail.com> wrote: > >> Iceberg today does not support non optional unions and that is the right >> behaviour, but we do have a lot of datasets which have non-optional union >> fields. I'm wondering whether Iceberg should allow reading these datasets >> as long as the user does not project the union field. >> >> I tried it out and today is throws an exception during column pruning. >> >> If we think this should be support, I'll create an issue for this. >> >> -Best, >> R. >> > -- Ryan Blue Software Engineer Netflix
