mhatipog opened a new pull request, #19065: URL: https://github.com/apache/iceberg/pull/19065
Fixes #19064. ORC only stores a struct field where all of its ancestors are present, and `hasNull()` covers only the field's own nulls, so a field that is null because its parent is null was treated as non-null: - `OrcMetrics` now derives the null count as `rowCount - numberOfValues` for fields outside lists and maps, instead of 0 when `hasNull()` is false. Trino's Iceberg writer computes it the same way. - `ExpressionToSearchArgument` no longer pushes `isNull` to ORC for a field with an optional ancestor struct. `notNull` and fields inside lists and maps are unchanged. Both are needed: with only the metrics change, ORC still skips the row group. Files written before this change keep their metrics until they are rewritten. Testing: new cases in `TestMetrics` (Parquet and ORC) and `TestExpressionToSearchArgument`, which fail without the change, plus the updated expectation in `testNestedPrimitives`. I also checked the repro from the issue on 1.12.0 with Spark 4.0: the row is returned and deleted. AI tools helped draft the change and tests; I reviewed and tested them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
