Noémi Pap-Takács created IMPALA-15527:
-----------------------------------------
Summary: INSERT OVERWRITE into an unpartitioned Iceberg table
creates empty files
Key: IMPALA-15527
URL: https://issues.apache.org/jira/browse/IMPALA-15527
Project: IMPALA
Issue Type: Bug
Reporter: Noémi Pap-Takács
INSERT OVERWRITE into an unpartitioned Iceberg table commits one zero-row
Parquet file for every table sink instance that received no rows. This happens
even when the statement does write data. For example, with mt_dop=4 on a 3-node
cluster (12 sink instances), a 1-row overwrite committed 1 real file and 11
empty ones. These extra files show up in table metadata and add work to scan
planning.
The empty files are also what makes an empty-result overwrite (INSERT OVERWRITE
t SELECT ... WHERE false) clear the table. The zero-row file makes
ReplacePartitions replace the table's only partition. Removing the empty files
alone would turn that into a no-op, so the fix also has to change how the
commit works.
Cause
- HdfsTableSink::FlushFinal() always opens the root output partition when there
are no dynamic partition expressions. For Iceberg, that means unpartitioned
tables, since static partition clauses are rejected.
- TableSinkBase::InitOutputPartition() normally doesn't create a writer for an
empty partition, except when is_overwrite() && IsTransactional().
IsTransactional() is IsHiveAcid() || IsIceberg(). The exception exists so Hive
ACID can create an empty base_N directory, but it also applies to Iceberg. The
result is a writer and a 0-row file in every sink instance that got no rows.
Steps to reproduce
Run on a 3-node minicluster:
{code:java}
create table ow_probe (id int) stored as iceberg;
-- 1 row from a source spread over 24 files
set mt_dop=4;
insert overwrite ow_probe select id from functional_parquet.alltypes where id =
1;
select record_count, count(*) from <db>.ow_probe.`files` group by record_count;
-- Actual: record_count=1 -> 1 file, record_count=0 -> 11 files
-- Expected: record_count=1 -> 1 file only
-- Empty result
set mt_dop=0;
insert overwrite ow_probe select id from functional_parquet.alltypes where id <
0;
select record_count, count(*) from <db>.ow_probe.`files` group by record_count;
-- Actual: record_count=0 -> 3 files (table looks empty)
-- Expected: no data files, table empty
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
