Noemi Pap-Takacs created IMPALA-13656:
-----------------------------------------
Summary: MERGE redundantly accumulates memory in HDFS WRITER
Key: IMPALA-13656
URL: https://issues.apache.org/jira/browse/IMPALA-13656
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Noemi Pap-Takacs
Assignee: Noemi Pap-Takacs
Fix For: Impala 4.5.0
When we want to merge Iceberg tables that have lots of partitions, the
execution will use much more memory than needed, possibly resulting in a Memory
Limit Exceeded error.
It happens because when IcebergMergeImpl creates the insert table sink it
doesn't set 'inputIsClustered' to true. Therefore HdfsTableSink expects random
input and keeps the output writers open for every partition, which results in
high memory consumption.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
