[
https://issues.apache.org/jira/browse/PARQUET-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705655#comment-17705655
]
ASF GitHub Bot commented on PARQUET-2261:
-----------------------------------------
emkornfield commented on code in PR #197:
URL: https://github.com/apache/parquet-format/pull/197#discussion_r1149697363
##########
src/main/thrift/parquet.thrift:
##########
@@ -190,6 +190,35 @@ enum FieldRepetitionType {
/** The field is repeated and can contain 0 or more values */
REPEATED = 2;
}
+/**
+ * A structure for capturing metadata for estimating the unencoded,
uncompressed size
+ * of data.
+ */
+struct SizeEstimationStatistics {
+ /**
+ * The number of logic bytes needed to store present/non-null values.
+ * Unless specified below, the computed size is the size it would take to
plain-encode the underlying
+ * physical type.
+ * Special calculations:
+ * - Enum: plain-encoded BYTE_ARRAY size
+ * - Integers (same size used for signed and unsigned): int8 - 1 bytes,
int16 - 2
+ * - Decimal - Each value is assumed to take the minimal number of bytes
necessary to encode
+ * the precision of the decimal value.
+ * - Nested types (lists, nested groups and maps) - No additional size for
these structures
+ * are accounted for in this field, instead the histogram fields below
can be
+ * be used to estimate overhead to recreate these structures.
+ */
+ 1: optional i64 logical_value_byte_storage;
+ /**
+ * When present there is expected to be one element corresponding to each
repetition (i.e. size=max repetition_level+1)
+ * where each element represens the number of time the repetition level
was observed in the data.
+ */
+ 2: optional list<i64> repetition_level_histogram;
Review Comment:
Made a proposal for these questions.
> [Format] Add statistics that reflect decoded size to metadata
> -------------------------------------------------------------
>
> Key: PARQUET-2261
> URL: https://issues.apache.org/jira/browse/PARQUET-2261
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-format
> Reporter: Micah Kornfield
> Assignee: Micah Kornfield
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
