mhatipog opened a new pull request, #19065:
URL: https://github.com/apache/iceberg/pull/19065

   Fixes #19064.
   
   ORC only stores a struct field where all of its ancestors are present, and 
`hasNull()` covers only the field's own nulls, so a field that is null because 
its parent is null was treated as non-null:
   
   - `OrcMetrics` now derives the null count as `rowCount - numberOfValues` for 
fields outside lists and maps, instead of 0 when `hasNull()` is false. Trino's 
Iceberg writer computes it the same way.
   - `ExpressionToSearchArgument` no longer pushes `isNull` to ORC for a field 
with an optional ancestor struct. `notNull` and fields inside lists and maps 
are unchanged.
   
   Both are needed: with only the metrics change, ORC still skips the row 
group. Files written before this change keep their metrics until they are 
rewritten.
   
   Testing: new cases in `TestMetrics` (Parquet and ORC) and 
`TestExpressionToSearchArgument`, which fail without the change, plus the 
updated expectation in `testNestedPrimitives`. I also checked the repro from 
the issue on 1.12.0 with Spark 4.0: the row is returned and deleted.
   
   AI tools helped draft the change and tests; I reviewed and tested them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to