[
https://issues.apache.org/jira/browse/HIVE-30253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30253:
----------------------------------
Description:
*Problem*
On a V2 table with {{write.fanout.enabled=false}} (default: true),
{{HiveIcebergStorageHandler.acidSortColumns(DELETE)}} sorts the records that a
merge-on-read DELETE, split UPDATE or MERGE deletes by spec
ID, partition, file path and position, so that they can be written by the
clustered position delete writer with bounded memory.
The delete writer is never told that its input is sorted.
{{FileSinkOperator}} marks the input sorted only when the file sink has a
{{DynamicPartitionCtx}} with custom partition or sort expressions, and the
file sink of a merge-on-read delete has none. So
{{HiveIcebergWriterBase.newDeleteWriter}} always builds a
{{FanoutPositionOnlyDeleteWriter}}, which buffers the positions of every
partition in memory.
As a result, every such statement pays for a reducer stage and a full sort
that nothing uses, and the {{ClusteredPositionDeleteWriter}} that the property
selects never runs.
V3 is not affected: a DV writer needs no order, and {{acidSortColumns}}
returns none for V3.
*Reproducer*
{code:sql}
create external table ice_sorted_v2 (id int, name string, amount int)
partitioned by spec (bucket(2, id))
stored by iceberg tblproperties ('format-version'='2',
'write.fanout.enabled'='false');
insert into ice_sorted_v2 values (1, 'a', 10), (2, 'b', 20), (3, 'c', 30),
(4, 'd', 40);
explain delete from ice_sorted_v2 where id = 1;
{code}
The plan has Map 1 -> Reducer 2 with a Reduce Output Operator sorted (++++)
by the spec ID, partition, file path and position, and the delete writer of
Reducer 2 is the fanout writer.
*Proposed fix*
Either mark the input of the delete writer as sorted when
{{acidSortColumns(DELETE)}} is non-empty, so that the clustered writer is used,
or drop the sort and keep the fanout writer.
> Iceberg: V2 merge-on-read deletes with write.fanout.enabled=false are
> sorted, but always written by the fanout writer
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30253
> URL: https://issues.apache.org/jira/browse/HIVE-30253
> Project: Hive
> Issue Type: Bug
> Reporter: Denys Kuzmenko
> Priority: Major
>
> *Problem*
>
>
> On a V2 table with {{write.fanout.enabled=false}} (default: true),
> {{HiveIcebergStorageHandler.acidSortColumns(DELETE)}} sorts the records that
> a merge-on-read DELETE, split UPDATE or MERGE deletes by spec
> ID, partition, file path and position, so that they can be written by the
> clustered position delete writer with bounded memory.
>
>
>
>
> The delete writer is never told that its input is sorted.
> {{FileSinkOperator}} marks the input sorted only when the file sink has a
> {{DynamicPartitionCtx}} with custom partition or sort expressions, and the
> file sink of a merge-on-read delete has none. So
> {{HiveIcebergWriterBase.newDeleteWriter}} always builds a
> {{FanoutPositionOnlyDeleteWriter}}, which buffers the positions of every
> partition in memory.
>
>
>
> As a result, every such statement pays for a reducer stage and a full sort
> that nothing uses, and the {{ClusteredPositionDeleteWriter}} that the
> property selects never runs.
>
>
>
> V3 is not affected: a DV writer needs no order, and {{acidSortColumns}}
> returns none for V3.
>
>
>
>
> *Reproducer*
>
>
> {code:sql}
>
>
> create external table ice_sorted_v2 (id int, name string, amount int)
> partitioned by spec (bucket(2, id))
>
> stored by iceberg tblproperties ('format-version'='2',
> 'write.fanout.enabled'='false');
>
> insert into ice_sorted_v2 values (1, 'a', 10), (2, 'b', 20), (3, 'c', 30),
> (4, 'd', 40);
>
>
>
>
> explain delete from ice_sorted_v2 where id = 1;
>
>
> {code}
>
>
> The plan has Map 1 -> Reducer 2 with a Reduce Output Operator sorted (++++)
> by the spec ID, partition, file path and position, and the delete writer of
> Reducer 2 is the fanout writer.
>
>
>
> *Proposed fix*
>
>
> Either mark the input of the delete writer as sorted when
> {{acidSortColumns(DELETE)}} is non-empty, so that the clustered writer is
> used, or drop the sort and keep the fanout writer.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
