Re: [DISCUSS] Limitation of storing UnknownType in MapType

Thanks for updating the PR, Fokko! If we are okay with the approach of failing these unsupported cases, then I think we need to talk about when to fail. The PR currently changes what is allowed by Iceberg, but I think that Iceberg should allow these cases (which work with Avro) and should fail whe

Re: [DISCUSS] Limitation of storing UnknownType in MapType

Hey everyone, Thanks for jumping in here. To reiterate Bart, Having a UnknownType in a StructType, or a nested Struct is not an issue AFAIK, except when the struct solely contains Unknowns. Another scenario I would be concerned about is a table that contains only > UnknownType top-level fields.

Re: [DISCUSS] Limitation of storing UnknownType in MapType

I think it's reasonable to fail in cases where the underlying format can't represent a type, like the element of a list. We can go back and fix this by adding support for using Parquet's UNKNOWN type annotation

RE: Re: [DISCUSS] Limitation of storing UnknownType in MapType

Another scenario I would be concerned about is a table that contains only UnknownType top-level fields. What will happen in such a scenario? Will the Parquet format tolerate zero-column files? I think it's likely that even if Parquet-java supports this, it might be an untested and unsupported corne

Re: [DISCUSS] Limitation of storing UnknownType in MapType

On Sat, Jul 26, 2025 at 6:09 PM Kevin Liu wrote: > > My initial idea was to disallow the use of UnknownType as the element > in ListType and not allow the UnknownType as either a Key or Value of a > MapType. Any thoughts or concerns? > > That makes sense. I would also include `StructType` here to

Re: [DISCUSS] Limitation of storing UnknownType in MapType

> My initial idea was to disallow the use of UnknownType as the element in ListType and not allow the UnknownType as either a Key or Value of a MapType. Any thoughts or concerns? That makes sense. I would also include `StructType` here too. `StructType` is another "complex type" (extends NestedTy

[DISCUSS] Limitation of storing UnknownType in MapType

Hi everyone, Recently I took a stab at implementing reading UknownType in the Java implementation. I thought it would make sense to add this to the reference implementation first. However, I ran into a limitation with the current definition in the spe