Zoltán Borók-Nagy created IMPALA-15640:
------------------------------------------
Summary: Parquet writer accumulates BOOLEAN column statistics
across files
Key: IMPALA-15640
URL: https://issues.apache.org/jira/browse/IMPALA-15640
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
{{HdfsParquetTableWriter::BoolColumnWriter}} doesn't override {{Reset()}}, so
its page and row group statistics are not reset when the writer starts a new
file. When a fragment instance writes several files, e.g. because of
PARQUET_FILE_SIZE, the statistics of a BOOLEAN column also include the values
of the previous files: min/max is their union, and null_count is cumulative.
The Parquet scanner skips row groups where null_count equals num_values for
min/max predicates, so queries return wrong results. Iceberg tables also record
these statistics in their manifests, so Iceberg can skip such files too.
{code:sql}
set parquet_file_size=8m;
set num_nodes=1;
set compression_codec=none;
create table t (b boolean, s string) stored as parquet;
insert into t
select if(id <= 8095, null, true), lpad(cast(id as string), 1000, 'x')
from (select row_number() over (order by a.id) as id
from functional.alltypes a, functional.alltypes b limit 16190) v
order by id limit 16190;
set num_nodes=0;
select count(*) from t where b = true;
-- returns 0, expected 8095
{code}
Each file has 8095 rows here. The second file only has TRUE values, but its
null_count is 8095, the number of NULLs in the first file.
Found while working on IMPALA-15629, which makes Iceberg use these statistics
for any predicate.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
