emkornfield commented on code in PR #197:
URL: https://github.com/apache/parquet-format/pull/197#discussion_r1313652871
##########
src/main/thrift/parquet.thrift:
##########
@@ -191,6 +191,74 @@ enum FieldRepetitionType {
REPEATED = 2;
}
+/**
+ * A histogram of repetition and definition levels for either a page or column
+ * chunk.
+ *
+ * This is useful for:
+ * 1. Estimating the size of the data when materialized in
+ * memory
+ *
+ * 2. For filter push-down on nulls at various levels of nested
+ * structures and list lengths.
+ */
+struct RepetitionDefinitionLevelHistogram {
+ /**
+ * When present, there is expected to be one element corresponding to each
+ * repetition (i.e. size=max repetition_level+1) where each element
+ * represents the number of times the repetition level was observed in the
+ * data.
+ *
+ * This value should not be written if max_repetition_level is 0.
+ **/
+ 1: optional list<i64> repetition_level_histogram;
+ /**
+ * Same as repetition_level_histogram except for definition levels.
+ *
+ * This value should not be written if max_definition_level is 0 or 1.
Review Comment:
> Now that I think about it some, I understand the first element of the
histogram for page p will be the same as null_counts[p]
Technically, `null_counts[p] = sum(histogram[0..max_depth_level-1)`
> The second element of the histogram is num_values[p] - def_hist[p][0], but
num_values only lives on the page header. It's not in the page indexes.
Yeah, so I was originally considering this only from a filtering perspective
and not reconstructing sizes per page. For filtering we really care about
three states per page: 1. All null, 2. Some null, 3. no nulls. This is all
reconstructable from the page index (which is why I was ok removing the
histogram for the first case).
> Oh, but in the case where max_rep == 0, num_rows == num_values, and the
former can be recovered from the offset index. If max_rep > 0, then I think
max_def has to be at least 1. So never mind.
Technically max_def_level could be zero and it requires all lists have at
least one non-null element. However, even in this case (and when
max_defintion_level=1 for 3-level list encoding) we have enough information to
understand total number values written and the number of lists from the
rep_level histogram and null_count
Right non-repeated cases with max_depth_level=1, we need to subtract
`OffsetIndex[p].first_row_index - Offsetindex[p-1].first_row_index` which is a
bit awkward to get the actual number of values written but the information is
there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
