nssalian commented on issue #3258: URL: https://github.com/apache/iceberg-rust/issues/3258#issuecomment-5766830113
> For example, we can implement variant type handling in table metadata (as we have done already) and block both read and write path until we've fully tested each one. Reads would block if projections or predicates are including the type, in both table scanner and scan task to surface the problem as early as possible. Do we need anything special to do here beyond conditionals? The configs for variant have only been in shredding. The rest are enabled or disabled based on a codepath supports or doesn't. > As we implement the V3 data types, a lot of hazards are found as we have to carefully understand each part of the read or write path to ensure we don't silently write corrupt data. Could we enumerate these hazards somewhere so we can update the interfaces and enable or disable accordingly for types especially? This would help us make specific changes to the code. If this heads to let's add configs to enable disable features, then that would be a larger discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
