dannycjones commented on issue #3258:
URL: https://github.com/apache/iceberg-rust/issues/3258#issuecomment-5767544202
> > For example, we can implement variant type handling in table metadata
(as we have done already) and block both read and write path until we've fully
tested each one. Reads would block if projections or predicates are including
the type, in both table scanner and scan task to surface the problem as early
as possible.
>
> Do we need anything special to do here beyond conditionals? The configs
for variant have only been in shredding. The rest should be enabled or disabled
based on whether a codepath supports or doesn't.
I think the risk for me is that it can be hard to understand what changes
there are required for future data types. For example, I think it would have
been hard to predict that geospatial types are both a primitive but that we
cannot compare them byte-wise.
> > As we implement the V3 data types, a lot of hazards are found as we have
to carefully understand each part of the read or write path to ensure we don't
silently write corrupt data.
>
> Could we enumerate these hazards somewhere so we can update the interfaces
and enable or disable accordingly for types especially? This would help us make
specific changes to the code. If this heads to let's add configs to enable
disable features, then that would be a larger discussion.
Yes, I do think it's a great idea to document these better in either in the
type system / APIs themselves or at the very least the doc comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
