[
https://issues.apache.org/jira/browse/HIVE-30246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30246:
----------------------------------
Description:
*Problem*
With {{hive.merge.tezfiles=true}}, a DELETE on a format-version 3 Iceberg
table fails in the File Merge vertex. The merge stage for delete files
(HIVE-28267) takes every delete file the DELETE wrote as merge
input. On V3 these are deletion vectors in Puffin files, and
{{IcebergMergeRecordReader}} reads only Avro, ORC and Parquet.
{noformat}
TezRuntimeException: Vertex failed, vertexName=File Merge, ...
Caused by: java.lang.UnsupportedOperationException: Cannot read PUFFIN file:
file:/.../warehouse/v3_source/data/00000-0-...
at
org.apache.iceberg.mr.mapreduce.IcebergMergeRecordReader.openGeneric(IcebergMergeRecordReader.java:108)
at
org.apache.iceberg.mr.mapreduce.IcebergMergeRecordReader.nextTask(IcebergMergeRecordReader.java:59)
at
org.apache.iceberg.mr.mapreduce.IcebergMergeRecordReader.initialize(IcebergMergeRecordReader.java:55)
...
at
org.apache.iceberg.mr.hive.HiveIcebergInputFormat.getRecordReader(HiveIcebergInputFormat.java:232)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:104)
{noformat}
Only DELETE is affected: UPDATE and MERGE never get a merge stage.
*Reproducer* (TestIcebergLlapLocalCliDriver; iceberg_merge_delete_files.q
with format-version 3)
{code:sql}
--! qt:dataset:src
set hive.merge.mapredfiles=true;
set hive.merge.mapfiles=true;
set hive.merge.tezfiles=true;
set hive.optimize.sort.dynamic.partition.threshold=-1;
set mapred.reduce.tasks=5;
set hive.blobstore.supported.schemes=hdfs,file;
create table v3_source(key string) stored by iceberg stored as parquet
tblproperties('format-version'='3');
insert overwrite table v3_source select key from src;
insert into table v3_source select key from src;
insert into table v3_source select key from src;
insert into table v3_source select key from src;
delete from v3_source where key in (select key from src where key < '200');
{code}
*Expected*
The DELETE succeeds. A DV already holds every deleted position of its data
file, so DVs have nothing to gain from the row-based merge.
*Proposed fix*
Skip DVs ({{ContentFileUtil.isDV}}) when
{{HiveIcebergOutputCommitter.getOutputFiles}} lists the merge input. For a V3
DELETE the list is then empty, and {{ConditionalResolverMergeFiles}} runs only
the move
task. V2 position-delete files and data files are merged as before.
> Iceberg: V3 DELETE fails with hive.merge.tezfiles=true: the merge task can't
> read DVs
> -------------------------------------------------------------------------------------
>
> Key: HIVE-30246
> URL: https://issues.apache.org/jira/browse/HIVE-30246
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Reporter: Denys Kuzmenko
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
