Hello,
I agree that the various "num_values", "num_nulls" (or "null_count") and
"null_rows" need to be made less ambiguous, especially when max_level > 1.
I would intuitively say the following:
- "num_rows" is equal to the number of repetition levels equal to 0 (or,
if no repetition levels, equal to "num_values")
- "num_nulls" is equal to the number of definition levels smaller than
the max def level (or, if no definition levels, equal to 0)
- "num_values" is actually the number of *levels*
- "null_count" and "num_nulls" should be equal, when both present
Regards
Antoine.
Le 21/10/2026 à 22:43, Adam Reeve a écrit :
Hi Andrew
My understanding is that these values may be different when there is a
schema element with repetition_type=repeated, like a list typed column for
example. num_values is the total number of leaf-level values, including
nulls, and matches the field with the same name in the V1 header. num_rows
is the number of logical rows, so would be the number of lists, including
nulls, for a list typed column. V2 data pages are guaranteed to be split
across row boundaries, so a page will always contain an integer number of
rows. V1 pages in comparison are allowed to be split in the middle of a row.
Cheers,
Adam
On Thu, 22 Oct 2026 at 05:12, Andrew Bell <[email protected]> wrote:
Hi,
Can someone please explain the difference between "num_values" and
"num_rows" in the DataPageHeaderV2 metadata structure?
Thanks,
--
Andrew Bell
[email protected]
