Gabor Szadovszky created PARQUET-1217:
-----------------------------------------
Summary: Incorrect check for null min/max values in
StatisticsFilter
Key: PARQUET-1217
URL: https://issues.apache.org/jira/browse/PARQUET-1217
Project: Parquet
Issue Type: Bug
Affects Versions: 1.8.0, 1.7.0, 1.6.0, 1.5.0, 1.9.0, 1.10.0
Reporter: Gabor Szadovszky
Assignee: Gabor Szadovszky
As per the parquet-format specs the min/max values in statistics are optional.
Therefore, it is possible to have {{numNulls}} in {{Statistics}} while we don't
have min/max values. In {{StatisticsFilter}} we rely on the method
[StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90]
to handle the case of {{null}} min/max values which is not correct due to the
described scenario.
We shall check {{Statistics.hasNonNullValue()}} any time before using the
actual min/max values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
