[
https://issues.apache.org/jira/browse/PARQUET-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1217:
--------------------------------------
Summary: Incorrect handling of missing values in Statistics (was: Missing
check for null min/max values in StatisticsFilter)
> Incorrect handling of missing values in Statistics
> --------------------------------------------------
>
> Key: PARQUET-1217
> URL: https://issues.apache.org/jira/browse/PARQUET-1217
> Project: Parquet
> Issue Type: Bug
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0
> Reporter: Gabor Szadovszky
> Assignee: Gabor Szadovszky
> Priority: Major
>
> As per the parquet-format specs the min/max values in statistics are
> optional. Therefore, it is possible to have {{numNulls}} in {{Statistics}}
> while we don't have min/max values. In {{StatisticsFilter}} we rely on the
> method
> [StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90]
> to handle the case of {{null}} min/max values which is not correct due to
> the described scenario.
> We shall check {{Statistics.hasNonNullValue()}} any time before using the
> actual min/max values.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
