Hi, Xiening - What you brought up is an important subtlety. I view it a bit differently though: what matters is *effective* nullability along the whole path from the root, not the leaf field's own *required* flag. In your specific example, `s.a` is declared NOT NULL, but its parent `s` is optional. So effectively `s.a` is nullable. A field is only truly non-nullable if it is required and every ancestor struct up to the root is also required. A genuinely non-nullable field like this does not require `null_value_count` (because it will always be 0).
So `s.a` here is not a required field in the spec perspective, and null_value_count *should* be recorded for it. I believe the current Iceberg stats collector and evaluator handles this correctly. e.g. the InclusiveMetricsEvaluator relies <https://github.com/apache/iceberg/blob/89e2f887491c1b5fa9f8b9de81b3aa8b31fa6974/api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java#L100-L102> on the explicit count and not the field-level required flag. @Override protected boolean mayContainNull(int id) { return nullCounts == null || !nullCounts.containsKey(id) || nullCounts.get(id) != 0; } Overall, I agree with the spirit of your suggestion. We could clarify the spec wording so "optional fields" reads as *effectively required*, and make explicit that a field is non-nullable only when it and all of its ancestors are required. Best, Anoop On Mon, Aug 10, 2026 at 10:45 AM Xiening Dai <[email protected]> wrote: > Hi all, > > I see this in V4 spec: "null_value_count is only used for optional fields" > ( > https://github.com/apache/iceberg/blob/89e2f887491c1b5fa9f8b9de81b3aa8b31fa6974/format/spec.md?plain=1#L838 > ) > > This only makes sense if required field never contains nulls. But that > actually is not true. In table schema, we can have a struct field which is > optional and has a child field that is required. In that case, when the > parent struct is null, the required field values is treated as NULL too. > > For example, below works in Spark: > > CREATE TABLE t (i INT, s STRUCT<a: INT NOT NULL, b: STRING>) USING iceberg; > INSERT INTO t VALUES (1, NULL); > SELECT COUNT (*) WHERE s.a IS NULL; -- returns 1 > > Because of this, I don't think the query engine can optimize the execution > plan and assume a field doesn't contain null when it's marked as required. > It would need to rely on the null_value_count explicitly being 0. If you > agree on this, then we should always record the null_value_count no matter > if it's required or optionals field. > > Thoughts? >
