Lars Volker created PARQUET-826: ----------------------------------- Summary: parquet.thrift comments for Statistics are not consistent with parquet-mr and Hive implementations Key: PARQUET-826 URL: https://issues.apache.org/jira/browse/PARQUET-826 Project: Parquet Issue Type: Bug Components: parquet-format Reporter: Lars Volker Assignee: Lars Volker
I'm currently working on adding support for writing min/max statistics to Parquet files to Impala ([IMPALA-3909|https://issues.cloudera.org/browse/IMPALA-3909]). I noticed, that the comments in [parquet.thrift#L201|https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L201] don't seem to match the implementations in parquet-mr and Hive. The comments ask for min/max statistics to be "encoded in PLAIN encoding". For strings (BYTE_ARRAY), this should be "4 byte length stored as little endian, followed by bytes". Looking at [BinaryStatistics.java#L61|https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/statistics/BinaryStatistics.java#L61], it seems to return the bytes without a length-prefix. Writing a parquet file with Hive also shows this behavior. Similarly, but less ambiguous, PLAIN encoding for booleans uses bit-packing. It seems to be implied that for a single bit (min/max of a boolean column) it means setting the least significant bit of a single byte. This could be made more clear in the parquet.thrift file, too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)