[
https://issues.apache.org/jira/browse/PARQUET-1217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Szadovszky updated PARQUET-1217:
--------------------------------------
Description:
As per the parquet-format specs the min/max values in statistics are optional.
Therefore, it is possible to have {{numNulls}} in {{Statistics}} while we don't
have min/max values. In {{StatisticsFilter}} we rely on the method
[StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90]
to handle the case of {{null}} min/max values which is not correct due to the
described scenario.
We shall check {{Statistics.hasNonNullValue()}} any time before using the
actual min/max values.
In addition we don't check if the {{null_count}} is set or not when reading
from the parquet file. We simply use the value which is {{0}} in case of unset.
In the parquet-mr side the {{Statistics}} object uses the value {{0}} to sign
that the {{num_nulls}} is unset. It is incorrect if we are searching for null
values and we falsely drop a column chunk thinking there are no null values but
the field in the statistics was simply unset.
was:
As per the parquet-format specs the min/max values in statistics are optional.
Therefore, it is possible to have {{numNulls}} in {{Statistics}} while we don't
have min/max values. In {{StatisticsFilter}} we rely on the method
[StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90]
to handle the case of {{null}} min/max values which is not correct due to the
described scenario.
We shall check {{Statistics.hasNonNullValue()}} any time before using the
actual min/max values.
> Incorrect handling of missing values in Statistics
> --------------------------------------------------
>
> Key: PARQUET-1217
> URL: https://issues.apache.org/jira/browse/PARQUET-1217
> Project: Parquet
> Issue Type: Bug
> Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0
> Reporter: Gabor Szadovszky
> Assignee: Gabor Szadovszky
> Priority: Major
>
> As per the parquet-format specs the min/max values in statistics are
> optional. Therefore, it is possible to have {{numNulls}} in {{Statistics}}
> while we don't have min/max values. In {{StatisticsFilter}} we rely on the
> method
> [StatisticsFilter.isAllNulls(ColumnChunkMetaData)|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/filter2/statisticslevel/StatisticsFilter.java#L90]
> to handle the case of {{null}} min/max values which is not correct due to
> the described scenario.
> We shall check {{Statistics.hasNonNullValue()}} any time before using the
> actual min/max values.
> In addition we don't check if the {{null_count}} is set or not when reading
> from the parquet file. We simply use the value which is {{0}} in case of
> unset. In the parquet-mr side the {{Statistics}} object uses the value {{0}}
> to sign that the {{num_nulls}} is unset. It is incorrect if we are searching
> for null values and we falsely drop a column chunk thinking there are no null
> values but the field in the statistics was simply unset.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
