Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15963
Change subject: WIP IMPALA-6692: Trigger sort node run before hitting memory limit. ...................................................................... WIP IMPALA-6692: Trigger sort node run before hitting memory limit. Sorter node works by adding row batches to a sort run. After all batches added to current unsorted run or memory limit is hit, sorter will immediately start the run. If the latter case happen, sorter will spill the sorted run to disk after sort complete, create new unsorted run object, and continue add the next row batches, and so on. This algorithm try to fit as much rows into memory before start sorting. However, in the case of partitioned sort with large number of row batches, fitting too much rows into memory will cause the sort to be slow and block the sorter node for a long time before it can release some memory and continue accepting the next row batch from exchange node. One slow sorter node can block exchange node from sending row batches to other sorter node that is free. This patch speedup the decision to start the sort without waiting it to hit memory limit first by capping the intermediary quicksort run to lower memory limit, determined by query option 'sort_bytes_limit'. If the total used reservation of quicksort has exceed sort_bytes_limit, current unsorted_run_ will be wrapped up, sorted, and then spilled. Thus, overlapping the next sort run with spill from previous sort run. We also add new summary counter 'AddBatchTime' to get summary of how much time spent in Sorter::AddBatch. Max of 'AddBatchTime' indicate the longest time spent in Sorter::AddBatch, presumably busy doing intermediary sort. Testing: - Add new e2e test TestQueryFullSort::test_multiple_sort_bytes_limits - Run data loading test into 3 largest TPC-DS facts table of 300GB scale, using 5 backends, and 4GB mem_limit. sort_bytes_limit is varied between unspecified (not limited) vs 512 MB. The performance result is summarized in the following table. +---------------+---------+--------------+-----------------------+-------------------------+ | Insert table | #Rows | Avg | no sort_bytes_limit | 512 MB sort_bytes_limit | | | | SortDataSize +--------+--------------+---------+---------------+ | | | per Node | Query | Max | Query | Max | | | | | Time | AddBatchTime | Time | AddBatchTime | +---------------+---------+--------------+--------+--------------+---------+---------------+ | store_sales | 864.00M | 15.29 GB | 30m18s | 53s311ms | 20m | 5s634ms | +---------------+---------+--------------+--------+--------------+---------+---------------+ | catalog_sales | 431.97M | 11.34 GB | 23m24s | 31s212ms | 15m27s | 3s603ms | +---------------+---------+--------------+--------+--------------+---------+---------------+ | web_sales | 216.01M | 5.67 GB | 8m16s | 29s250ms | 6m41s | 3s856ms | +---------------+---------+--------------+--------+--------------+---------+---------------+ Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 --- M be/src/runtime/sorter.cc M be/src/runtime/sorter.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/query_test/test_sort.py 8 files changed, 49 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/15963/8 -- To view, visit http://gerrit.cloudera.org:8080/15963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2a0ba7c4bae4f1d300d4d9d7f594f63ced06a240 Gerrit-Change-Number: 15963 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
