Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21927 )
Change subject: IMPALA-13445: Ignore num partition for unpartitioned writes ...................................................................... Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21927/4//COMMIT_MSG@18 PS4, Line 18: 1. If the insert is unpartitioned, use the byte-based estimate fully. : Shuffling should only happen if num writers is less than num input : fragment instances. : 2. If the insert is partitioned, try to plan at least one writer for : each shuffling executor nodes, but do not exceed number of : partitions. > Should byte based estimate always be used regardless of partitioned or unpa Scheduling less than num nodes for partitioned insert necessitate inserting exchange in-between. It also risk opening more file handles per writers because each of them need to write to multiple partitions. We also estimate bytes to write from output cardinality of root fragment, which can be an underestimate in certain corner cases. -- To view, visit http://gerrit.cloudera.org:8080/21927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51ab8fc35a5489351a88d372b28642b35449acfc Gerrit-Change-Number: 21927 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Abhishek Rawat <[email protected]> Gerrit-Reviewer: David Rorke <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Fri, 18 Oct 2024 18:46:34 +0000 Gerrit-HasComments: Yes
