[
https://issues.apache.org/jira/browse/IMPALA-15640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy resolved IMPALA-15640.
----------------------------------------
Resolution: Fixed
> Parquet writer accumulates BOOLEAN column statistics across files
> -----------------------------------------------------------------
>
> Key: IMPALA-15640
> URL: https://issues.apache.org/jira/browse/IMPALA-15640
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Critical
> Labels: correctness
> Fix For: Impala 5.0.0
>
>
> {{HdfsParquetTableWriter::BoolColumnWriter}} doesn't override {{Reset()}}, so
> its page and row group statistics are not reset when the writer starts a new
> file. When a fragment instance writes several files, e.g. because of
> PARQUET_FILE_SIZE, the statistics of a BOOLEAN column also include the values
> of the previous files: min/max is their union, and null_count is cumulative.
> The Parquet scanner skips row groups where null_count equals num_values for
> min/max predicates, so queries return wrong results. Iceberg tables also
> record these statistics in their manifests, so Iceberg can skip such files
> too.
> {code:sql}
> set parquet_file_size=8m;
> set num_nodes=1;
> set compression_codec=none;
> create table t (b boolean, s string) stored as parquet;
> insert into t
> select if(id <= 8095, null, true), lpad(cast(id as string), 1000, 'x')
> from (select row_number() over (order by a.id) as id
> from functional.alltypes a, functional.alltypes b limit 16190) v
> order by id limit 16190;
> set num_nodes=0;
> select count(*) from t where b = true;
> -- returns 0, expected 8095
> {code}
> Each file has 8095 rows here. The second file only has TRUE values, but its
> null_count is 8095, the number of NULLs in the first file.
> Found while working on IMPALA-15629, which makes Iceberg use these statistics
> for any predicate.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
