Gabor Kaszab created IMPALA-12620:
-------------------------------------
Summary: Missing field ID in the eq-delete file could filter out
rows with null values
Key: IMPALA-12620
URL: https://issues.apache.org/jira/browse/IMPALA-12620
Project: IMPALA
Issue Type: Sub-task
Reporter: Gabor Kaszab
If a malformed equality delete file doesn't have some of the equality field IDs
then Parquet schema resolver will identify these ase missing fields but won't
fail the query. Missing fields instead are filled with NULL values. But when
some of the columns in the equality delete tuples are NULLs then when
anti-joining them with the data rows, they will match the NULL values from the
data rows. As a result a malformed equality delete file could cause the rows
being ommitted from the result where the field ID of the data row contains NULL
and the field ID in the equality delete file is missing.
E.g.
Test data is
i, s
(1, "str1")
(NULL, "str2")
and equality field ID is 1 (corresponding to column i).
When an equality delete file doesn't have column i and doesn't have field ID 1
then it will make the second row missing from the result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
