Hello Zoltan Borok-Nagy, Mostafa Mokhtar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11081 to look at the new patch set (#2). Change subject: IMPALA-6034: Add scanned bytes limits per query ...................................................................... IMPALA-6034: Add scanned bytes limits per query This adds support for aggregate resource limits at runtime, specified via query options. If a query exceeds a limit it is terminated. The checks are periodic so the query may go somewhat over the limits. SCAN_BYTES_LIMIT is exposed as an advanced query option. CPU_LIMIT_S is hidden as a development query option because it is flawed - the CPU user/sys time is only updated upon thread completion, so in many cases the limit will not take effect until well after the resources have been used. IMPALA-7318 tracks enabling this. Query profile is updated to include query wide and per backend metrics for CPU and scanned bytes. Example from "select count(*) from tpch_parquet.lineitem": Per Node Peak Memory Usage: tarmstrong-box:22000(289.50 KB) tarmstrong-box:22001(249.50 KB) tarmstrong-box:22002(249.50 KB) Per Node Bytes Read: tarmstrong-box:22000(100.00 KB) tarmstrong-box:22001(100.00 KB) tarmstrong-box:22002(100.00 KB) Per Node User Time: tarmstrong-box:22000(40.000ms) tarmstrong-box:22001(32.000ms) tarmstrong-box:22002(24.000ms) Per Node System Time: tarmstrong-box:22000(0.000ns) tarmstrong-box:22001(0.000ns) tarmstrong-box:22002(0.000ns) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns - NumBackends: 3 (3) - NumFragmentInstances: 4 (4) - NumFragments: 2 (2) - TotalBytesRead: 300.00 KB (307200) - TotalCpuTime: 96.000ms Testing: Added tests for various permutations for CPU_LIMIT_S and SCAN_BYTES_LIMIT Based on a previous patch by Mostafa Mokhtar <mmokh...@cloudera.com> Change-Id: I3e85f80b70b3fce47e637e9322ed0316ee84f6a9 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A testdata/workloads/functional-query/queries/QueryTest/query-resource-limits.test M tests/query_test/test_cancellation.py M tests/query_test/test_resource_limits.py M www/query_backends.tmpl 15 files changed, 370 insertions(+), 54 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/11081/2 -- To view, visit http://gerrit.cloudera.org:8080/11081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3e85f80b70b3fce47e637e9322ed0316ee84f6a9 Gerrit-Change-Number: 11081 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Mostafa Mokhtar <mmokh...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Zoltan Borok-Nagy <borokna...@cloudera.com>