RussellSpitzer commented on pull request #2496: URL: https://github.com/apache/iceberg/pull/2496#issuecomment-1036626170
> > I think the default issue could be issued by scanning through the schemas created since the file was created and only using the first available default. This seems kind of expensive to me though. General algorithm is > > ``` > > For missing column A in file written at Snapshot X > > Find the first Schema created after X which applies a default value to A > > If such a schema exists > > return the schema's default > > else > > return null > > ``` > > that would do. Or, if we can detect if we are coming from rewriting path, we skip reading the default value for the missing/unmaterialized columns. Looking into that option I think this would be tricky as every engine would have to customize their rewrite pathway to accommodate it. Additionally what would you do when rewriting a row with a required column that you default'd previously? You cannot write a new file where only some records are missing an entry for a column (null vs empty issue again) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
