Zoltán Borók-Nagy created IMPALA-15705:
------------------------------------------
Summary: NPE in HdfsScanNode.initOverlapPredicate() for joins on
Iceberg scans without Parquet files when the table's default format is Parquet
Key: IMPALA-15705
URL: https://issues.apache.org/jira/browse/IMPALA-15705
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 5.0.0
Reporter: Zoltán Borók-Nagy
Planning fails with a NullPointerException when a min/max runtime filter
targets an Iceberg scan, the scanned data files contain no Parquet file, and
the table's write.format.default is Parquet. This hits plain SELECT joins, not
only DML. It happens, for example, with tables written in Avro by Hive or Spark
and later switched to Parquet, or when partition pruning leaves only Avro files.
Repro on master with an existing test table (partition int_col=1 holds only an
Avro file):
{code:sql}
select m.string_col
from functional_parquet.iceberg_mixed_file_format_part m
join functional.alltypestiny a on m.double_col = a.double_col
where m.int_col = 1;
{code}
{noformat}
ERROR: RuntimeException: java.lang.NullPointerException
CAUSED BY: NullPointerException: null
{noformat}
{noformat}
Caused by: java.lang.NullPointerException
at
org.apache.impala.planner.HdfsScanNode.initOverlapPredicate(HdfsScanNode.java:971)
at
org.apache.impala.planner.RuntimeFilterGenerator.assignRuntimeFilters(RuntimeFilterGenerator.java:1297)
at
org.apache.impala.planner.RuntimeFilterGenerator.generateFiltersRecursive(RuntimeFilterGenerator.java:1183)
...
at
org.apache.impala.planner.Planner.createPlanFragments(Planner.java:176)
{noformat}
The same query on int_col=2 (ORC only) fails with SET
ORC_READ_STATISTICS=false. MERGE on such a table fails the same way. The query
works with RUNTIME_FILTER_MODE=OFF.
*Cause*
HdfsScanNode.computeScanRangeLocations() derives allParquet_ from
partition.getFileFormat(). For an Iceberg table that is the table's default
write format, not the format of the scanned files. HdfsScanNode.init() creates
statsTuple_ only if fileFormats_, the formats of the scanned files, contain
Parquet (or ORC with ORC_READ_STATISTICS). So allParquet_ is true while
statsTuple_ is null, and the Preconditions.checkNotNull(statsTuple_) in
initOverlapPredicate() throws. IMPALA-13467 fixed only the case of an Iceberg
scan without any files.
*Suggested fix*
For Iceberg scans, compute allParquet_ and allColumnarFormat_ from the formats
of the scanned file descriptors (fileFormats_) instead of the partition format.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
