Zoltán Borók-Nagy created IMPALA-15515:
------------------------------------------
Summary: Parquet late materialization does not detect a non-filter
column with too few values (corrupt file)
Key: IMPALA-15515
URL: https://issues.apache.org/jira/browse/IMPALA-15515
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
HdfsParquetScanner::FillScratchMicroBatches() only compares each column
reader's tuple count with the first reader's (c != 0 && last_num_tuples[r] !=
*num_tuples).
- Filter pass: this is correct, because the first filter reader decides how
many rows the scratch batch has.
- Non-filter pass: the expected count is already known. Every micro batch must
produce min(length, scratch_batch_->num_tuples - start) tuples. If there is
only one non-filter reader, or the first one is the short one, nothing compares
it to
anything. So a column chunk with fewer values than the row group goes
undetected.
ValidateEndOfRowGroup() doesn't catch it either. It takes the row count from
the filter readers, compares each scalar reader only with its own column
metadata, and skips complex readers.
Without late materialization, the same file fails with a corrupt-file error.
With late materialization it should fail the same way: "Corrupt Parquet file
'…': column '…' had N remaining values but expected M", or the existing
PARQUET_ROWS_SKIPPING error.
Steps to reproduce:
1. Write a Parquet file with one row group of 10 rows and two INT columns a
(0..9) and b, with one uncompressed PLAIN V1 data page per column.
2. Change DataPageHeader.num_values and ColumnMetaData.num_values of b from 10
to 9. This is a same-length varint patch.
3. Run SELECT a, b FROM t WHERE a >= 5. a is the only filter reader and b the
only non-filter reader, and the last row survives.
- With parquet_late_materialization_threshold=-1, the query fails with a
corrupt-file error.
- With the default threshold, no error is reported. From reading the code, the
last row likely gets a zero or garbage value for b.
{*}Proposed fix{*}: in the non-filter pass, also check the first reader's tuple
count against the expected count per micro batch, and use the existing mismatch
error path. Both non-filter call sites in AssembleRows() would pass the scratch
batch's
row count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
