Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/19807 )
Change subject: IMPALA-12091: Control scan parallelism by its processing cost ...................................................................... Patch Set 6: (2 comments) Patch set 6 adds override of perThreadIoBuffers to 2 at HdfsScanNode.java, following the constant at scan-node.cc. The impact can be observed in the modified tpcds-processing-cost.test, where several Per-Instance Resources mem-estimate is now fixed at 16MB (2 * 8MB buffer). Consequently, Per-Host Resource Estimate is also lowered in some queries. This might impact scan speed, but I'm not sure if there is any benefit of having more IO buffer per scan instance. http://gerrit.cloudera.org:8080/#/c/19807/3/be/src/exec/scan-node.cc File be/src/exec/scan-node.cc: http://gerrit.cloudera.org:8080/#/c/19807/3/be/src/exec/scan-node.cc@298 PS3, Line 298: * FLAGS_max_queued_row_batches_per_scanner_thread)); > We can and should preserve legacy computations for MT_DOP. Just keep the co Patch set 6 reorganize the calculation into separate blocks. http://gerrit.cloudera.org:8080/#/c/19807/3/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/19807/3/common/thrift/ImpalaService.thrift@793 PS3, Line 793: MAX_FRAGMENT_INSTANCES_PER_NODE = 156 > Still 156 Per PM, 156 is the number corresponding to this in Query.thrift, not related with the upper limit 128. -- To view, visit http://gerrit.cloudera.org:8080/19807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If948e45455275d9a61a6cd5d6a30a8b98a7c729a Gerrit-Change-Number: 19807 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Wenzhe Zhou <[email protected]> Gerrit-Comment-Date: Sat, 06 May 2023 00:06:52 +0000 Gerrit-HasComments: Yes
