RussellSpitzer commented on pull request #2877: URL: https://github.com/apache/iceberg/pull/2877#issuecomment-887882956
I added another test and found an issue with my fix. We create a StructInternalRow with a "type" and it's data "struct" When we actually prune the projected record's "type" does not match the struct layout, this is an issue for Map, Array, Struct types. For example, if we make a BaseFile with a projection we end up with a mapping of ordinal to physical entry in the row, but the StructInternal row does *not* do the same mapping when looking up types. So although my fix works for all non parameterized lookups, it fails on parameterized types since the parameterized types are looked up based on the original layout and not the pruned one. Ie StructInternalRow(TableLayout) stores ``` Types = ( int, string, int, list) data = ([1,2], "hello") with Map ( 0 -> 3, 2 -> 1) ``` Since the type lookup doesn't know about the projection it is incorrect ... trying to figure out a fix without breaking everything -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
