[
https://issues.apache.org/jira/browse/HIVE-30253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30253:
----------------------------------
Issue Type: Improvement (was: Bug)
> Iceberg: V2 merge-on-read deletes with write.fanout.enabled=false are
> sorted, but always written by the fanout writer
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30253
> URL: https://issues.apache.org/jira/browse/HIVE-30253
> Project: Hive
> Issue Type: Improvement
> Reporter: Denys Kuzmenko
> Priority: Major
>
> *Problem*
>
>
> On a V2 table with {{write.fanout.enabled=false}} (default: true),
> {{HiveIcebergStorageHandler.acidSortColumns(DELETE)}} sorts the records that
> a merge-on-read DELETE, split UPDATE or MERGE deletes by spec
> ID, partition, file path and position, so that they can be written by the
> clustered position delete writer with bounded memory.
>
>
>
>
> The delete writer is never told that its input is sorted.
> {{FileSinkOperator}} marks the input sorted only when the file sink has a
> {{DynamicPartitionCtx}} with custom partition or sort expressions, and the
> file sink of a merge-on-read delete has none. So
> {{HiveIcebergWriterBase.newDeleteWriter}} always builds a
> {{FanoutPositionOnlyDeleteWriter}}, which buffers the positions of every
> partition in memory.
>
>
>
> As a result, every such statement pays for a reducer stage and a full sort
> that nothing uses, and the {{ClusteredPositionDeleteWriter}} that the
> property selects never runs.
>
>
>
> V3 is not affected: a DV writer needs no order, and {{acidSortColumns}}
> returns none for V3.
>
>
>
>
> *Reproducer*
>
>
> {code:sql}
>
>
> create external table ice_sorted_v2 (id int, name string, amount int)
> partitioned by spec (bucket(2, id))
>
> stored by iceberg tblproperties ('format-version'='2',
> 'write.fanout.enabled'='false');
>
> insert into ice_sorted_v2 values (1, 'a', 10), (2, 'b', 20), (3, 'c', 30),
> (4, 'd', 40);
>
>
>
>
> explain delete from ice_sorted_v2 where id = 1;
>
>
> {code}
>
>
> The plan has Map 1 -> Reducer 2 with a Reduce Output Operator sorted (++++)
> by the spec ID, partition, file path and position, and the delete writer of
> Reducer 2 is the fanout writer.
>
>
>
> *Proposed fix*
>
>
> Either mark the input of the delete writer as sorted when
> {{acidSortColumns(DELETE)}} is non-empty, so that the clustered writer is
> used, or drop the sort and keep the fanout writer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
