Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22047 )
Change subject: IMPALA-2945: Account for duplicate keys on multiple nodes preAgg ...................................................................... Patch Set 5: I did a performance run that roughly compares Impala before IMPALA-13405 vs all patches in this stack up to this IMPALA-2945 over TPC-DS 3TB. Cluster has 10 executors, and I have MT_DOP=12. No significant reduction in query latency, while 3 queries improve by over 10%. +-------+-----------------------+------------------+-----------+----------+ | query | pre-IMPALA-13405 (ms) | IMPALA-2945 (ms) | diff (ms) | diff (%) | +-------+-----------------------+------------------+-----------+----------+ | 74 | 29255 | 22507 | -6748 | -23.07% | | 11 | 36088 | 30412 | -5676 | -15.73% | | 4 | 58216 | 50195 | -8021 | -13.78% | +-------+-----------------------+------------------+-----------+----------+ Memory-wise, I look for "Per-Host Resource Estimates" info in query plan. Several queries have few megabytes resource estimate increase, but there are 12 queries that reduce significantly. +-------+-----------------------+------------------+-------------+----------+ | query | pre-IMPALA-13405 (GB) | IMPALA-2945 (GB) | diff (GB) | diff (%) | +-------+-----------------------+------------------+-------------+----------+ | 31 | 8388608 | 23.42 | -8388584.58 | -100.00% | | 74 | 110.55 | 56.55 | -54 | -48.85% | | 11 | 187.15 | 108.81 | -78.34 | -41.86% | | 22 | 81.50 | 48.04 | -33.46 | -41.06% | | 4 | 259.53 | 155.01 | -104.52 | -40.27% | | 37 | 14.80 | 8.86 | -5.94 | -40.14% | | 82 | 18.57 | 11.40 | -7.17 | -38.61% | | 98 | 7.97 | 5.14 | -2.83 | -35.51% | | 50 | 18.88 | 12.88 | -6 | -31.78% | | 66 | 16.42 | 13.07 | -3.35 | -20.40% | | 67 | 282.49 | 250.09 | -32.4 | -11.47% | | 20 | 5.40 | 4.79 | -0.61 | -11.30% | +-------+-----------------------+------------------+-------------+----------+ -- To view, visit http://gerrit.cloudera.org:8080/22047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04c563e59421928875b340cb91654b9d4bc80b55 Gerrit-Change-Number: 22047 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Comment-Date: Tue, 10 Dec 2024 20:09:48 +0000 Gerrit-HasComments: No
