Zoltán Borók-Nagy created IMPALA-15470:
------------------------------------------
Summary: Iceberg: fields of a NULL struct read from an Avro data
file are not NULL (wrong results)
Key: IMPALA-15470
URL: https://issues.apache.org/jira/browse/IMPALA-15470
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
h3. Problem
When an Iceberg table has a STRUCT column and Avro data files, Impala returns
wrong values for the fields of a NULL struct: {{s.a}} and {{s.b}} are not NULL
when {{s}} is NULL. Predicates and aggregates on the fields give wrong results
too
({{s.b IS NULL}}, {{count(s.a)}}). NULL fields inside a non-NULL struct are
read correctly.
Plain Avro tables can't have this problem, because the planner rejects scans of
Avro tables with complex-typed columns. Iceberg tables can: the planner only
checks the table's {{write.format.default}}, but a table can have Avro data
files
next to Parquet ones, e.g. after {{write.format.default}} was changed.
With codegen, the same queries crash impalad instead (IMPALA-4061). By default
the planner doesn't disable codegen for Iceberg tables without statistics, even
small ones, so these queries crash with default options and return wrong
results
with {{DISABLE_CODEGEN=true}}. If IMPALA-4061 is fixed by falling back to the
interpreted code, every such query returns wrong results, so this bug should be
fixed before or together with it.
h3. Steps to reproduce
Spark 4.1.3 + Iceberg 1.11.0 ({{hc}} is a Hadoop catalog):
{code:sql}
CREATE TABLE hc.db.null_record (id INT, s STRUCT<a: INT, b: STRING>) USING
iceberg
TBLPROPERTIES ('format-version'='2', 'write.format.default'='avro');
INSERT INTO hc.db.null_record VALUES
(1, named_struct('a', 10, 'b', 'x1')),
(2, NULL),
(3, named_struct('a', CAST(NULL AS INT), 'b', 'x3')),
(4, named_struct('a', 40, 'b', CAST(NULL AS STRING)));
-- Otherwise the planner rejects the scan because of the STRUCT column.
ALTER TABLE hc.db.null_record SET TBLPROPERTIES
('write.format.default'='parquet');
{code}
Impala:
{code:sql}
CREATE EXTERNAL TABLE null_record STORED AS ICEBERG
LOCATION '<warehouse>/db/null_record' TBLPROPERTIES
('iceberg.catalog'='hadoop.tables');
-- With codegen the queries crash impalad (IMPALA-4061).
SET DISABLE_CODEGEN=true;
SELECT id, s.a, s.b, s.a IS NULL FROM null_record ORDER BY id;
SELECT count(s.a), count(s.b) FROM null_record;
SELECT id FROM null_record WHERE s.b IS NULL ORDER BY id;
{code}
h3. Expected result
What Spark returns:
{noformat}
1, 10, x1, false
2, NULL, NULL, true
3, NULL, x3, true
4, 40, NULL, false
count(s.a) = 2, count(s.b) = 2
WHERE s.b IS NULL: 2, 4
{noformat}
h3. Actual result (master, DISABLE_CODEGEN=true)
{noformat}
1, 10, x1, false
2, 0, '', false
3, NULL, x3, true
4, 40, NULL, false
count(s.a) = 3, count(s.b) = 3
WHERE s.b IS NULL: 4
{noformat}
No warning is reported.
{{SELECT id, s FROM null_record}} (the whole struct) returns no rows instead,
with the warning "Unresolvable types for column 's': declared column type:
STRUCT, table's Avro schema type: record". This is the same kind of silent skip
as
IMPALA-15445.
h3. Analysis (master, be/src/exec/avro/hdfs-avro-scanner.cc)
* {{ResolveSchemas()}} (:226) attaches slot descriptors only to the fields that
the slots' column paths end at (:288). For {{s.a}} that is field {{a}}; the
record field {{s}} has no slot descriptor.
* {{MaterializeTuple()}} (:602) reads the union of a nullable field, and for a
NULL value only sets the null indicator of the field's own slot ({{case
AVRO_NULL}}, :629-631). For a NULL record the field has no slot, so nothing is
set, and the
slots of the record's fields are not written at all. They keep what
{{InitTuple()}} left in the tuple: the template tuple's values, or, if there is
no template tuple, memory that was not initialized for this row
({{InitTuple()}} only clears
the null bits, {{hdfs-scanner.h:614-621}}). In our runs this was 0 and an empty
string.
* The codegen'd version has the same gap: the {{null_block}} in
{{CodegenReadRecord()}} only sets the null indicator of the field's own slot
(:994-997).
* By code reading, the fields of nested records inside a NULL record have the
same problem.
h3. Possible fix
When a record is NULL, set the null indicator of every materialized slot whose
column path goes through the record. For example, {{ResolveSchemas()}} could
store these slots in the record's {{AvroSchemaElement}}, and
{{MaterializeTuple()}}
and the codegen'd {{null_block}} could set them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
