Hello Peter Rozsa, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24614
to look at the new patch set (#2).
Change subject: IMPALA-12587: Respect MAX_FS_WRITERS for Iceberg
DELETE/UPDATE/MERGE
......................................................................
IMPALA-12587: Respect MAX_FS_WRITERS for Iceberg DELETE/UPDATE/MERGE
Previously, MAX_FS_WRITERS only capped parallelism for HdfsTableSink
(INSERT). Iceberg DELETE (IcebergBufferedDeleteSink), UPDATE and MERGE
(MultiDataSink) ignored the limit, potentially creating too many small
files.
This patch:
- Adds maxTableSinks_ to IcebergBufferedDeleteSink with a
getNumInstances() method that caps writer instances.
- Adds getNumInstances() to MultiDataSink that delegates to child sinks.
- Introduces the InstanceCountProvidingSink interface to eliminate the
instanceof cascade in PlanFragment.getNumInstances().
- Simplifies MultiDataSink construction with a varargs constructor.
Change-Id: Ice7362cecb8b43fbff43b8ad82827e31e2b4eea3
Assisted-by: Claude Opus 4.6 (Claude Code)
---
M fe/src/main/java/org/apache/impala/analysis/IcebergDeleteImpl.java
M fe/src/main/java/org/apache/impala/analysis/IcebergMergeImpl.java
M fe/src/main/java/org/apache/impala/analysis/IcebergUpdateImpl.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/IcebergBufferedDeleteSink.java
M fe/src/main/java/org/apache/impala/planner/IcebergMergeSink.java
A fe/src/main/java/org/apache/impala/planner/InstanceCountProvidingSink.java
M fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java
M fe/src/main/java/org/apache/impala/planner/MultiDataSink.java
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
A fe/src/test/java/org/apache/impala/planner/IcebergWriterLimitTest.java
11 files changed, 253 insertions(+), 42 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/24614/2
--
To view, visit http://gerrit.cloudera.org:8080/24614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ice7362cecb8b43fbff43b8ad82827e31e2b4eea3
Gerrit-Change-Number: 24614
Gerrit-PatchSet: 2
Gerrit-Owner: Nandor Kollar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
