Tim Armstrong has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/14538 )
Change subject: IMPALA-4400: aggregate runtime filters locally ...................................................................... IMPALA-4400: aggregate runtime filters locally Move RuntimeFilterBank to QueryState(). Implement fine-grained locking for each filter to mitigate any increased lock contention from the change. Make RuntimeFilterBank handle multiple producers of the same filter, e.g. multiple instances of a partitioned join. It computes the expected number of filters upfront then sends the filter to the coordinator once all the local instances have been merged together. The merging can done in parallel locally to improve latency of filter propagation. Add Or() methods to MinMaxFilter and BloomFilter, since we now need to merge those, not just the thrift versions. Update coordinator filter routing to expect only one instance of a filter from each producer backend and to only send one instance to each consumer backend (instead of sending one per fragment). Update memory reservations and estimates to be lower to account for sharing of filters between fragment instances. mt_dop plans are modified to show these shared and non-shared resources separately. Enable waiting for runtime filters for kudu scanner with mt_dop. Made min/max filters const-correct. Testing * Added unit tests for Or() methods. * Added some additional e2e test coverage for mt_dop queries * Updated planner tests with new estimates and reservation. * Ran a single node 3-impalad stress test with TPC-H kudu and TPC-DS parquet. * Ran exhaustive tests. * Ran core tests with ASAN. Perf * Did a single-node perf run on TPC-H with default settings. No perf change. * Single-node perf run with mt_dop=8 showed significant speedups: +----------+-----------------------+---------+------------+------------+----------------+ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +----------+-----------------------+---------+------------+------------+----------------+ | TPCH(30) | parquet / none / none | 10.14 | -7.29% | 5.05 | -11.68% | +----------+-----------------------+---------+------------+------------+----------------+ +----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+---------+ | Workload | Query | File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval | +----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+---------+ | TPCH(30) | TPCH-Q7 | parquet / none / none | 38.87 | 38.44 | +1.13% | 7.17% | * 10.92% * | 20 | +0.72% | 0.72 | 0.39 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.28 | 4.26 | +0.50% | 1.92% | 1.09% | 20 | +0.03% | 0.31 | 1.01 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.32 | 2.32 | +0.05% | 2.01% | 1.89% | 20 | -0.03% | -0.36 | 0.08 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.73 | 3.75 | -0.42% | 0.84% | 1.05% | 20 | -0.25% | -0.77 | -1.40 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 9.80 | 9.83 | -0.38% | 0.51% | 0.80% | 20 | -0.32% | -1.30 | -1.81 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.98 | 2.00 | -1.32% | 1.74% | 2.81% | 20 | -0.64% | -1.71 | -1.79 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.22 | 1.25 | -2.14% | 2.66% | 4.15% | 20 | -0.96% | -2.00 | -1.95 | | TPCH(30) | TPCH-Q19 | parquet / none / none | 5.13 | 5.22 | -1.65% | 1.20% | 1.40% | 20 | -1.76% | -3.34 | -4.02 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.46 | 2.56 | -4.13% | 2.49% | 1.99% | 20 | -4.31% | -4.04 | -5.94 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 81.63 | 85.07 | -4.05% | 4.94% | 3.06% | 20 | -5.46% | -3.28 | -3.21 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.07 | 5.50 | I -7.92% | 0.96% | 1.33% | 20 | I -8.51% | -5.27 | -22.14 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.00 | 26.24 | I -8.57% | 0.46% | 0.38% | 20 | I -9.34% | -5.27 | -67.47 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 8.66 | 9.50 | I -8.86% | 0.62% | 0.44% | 20 | I -9.75% | -5.27 | -55.17 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 6.01 | 6.70 | I -10.19% | 1.01% | 0.90% | 20 | I -11.25% | -5.27 | -35.76 | | TPCH(30) | TPCH-Q12 | parquet / none / none | 2.98 | 3.39 | I -12.23% | 1.48% | 1.48% | 20 | I -13.56% | -5.27 | -27.75 | | TPCH(30) | TPCH-Q11 | parquet / none / none | 1.69 | 2.00 | I -15.55% | 1.63% | 1.47% | 20 | I -18.09% | -5.27 | -34.60 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.42 | 2.87 | I -15.69% | 1.48% | 1.26% | 20 | I -18.61% | -5.27 | -39.50 | | TPCH(30) | TPCH-Q14 | parquet / none / none | 4.64 | 6.27 | I -26.02% | 1.35% | 0.73% | 20 | I -35.37% | -5.27 | -94.07 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 3.19 | 4.37 | I -27.01% | 1.54% | 0.99% | 20 | I -36.85% | -5.27 | -80.74 | | TPCH(30) | TPCH-Q5 | parquet / none / none | 4.57 | 6.39 | I -28.36% | 1.04% | 0.75% | 20 | I -39.56% | -5.27 | -120.02 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 3.15 | 4.71 | I -33.06% | 1.59% | 1.31% | 20 | I -49.43% | -5.27 | -87.64 | | TPCH(30) | TPCH-Q8 | parquet / none / none | 5.25 | 7.95 | I -33.95% | 0.95% | 0.53% | 20 | I -51.11% | -5.27 | -185.02 | +----------+----------+-----------------------+--------+-------------+------------+-----------+----------------+-------+----------------+---------+---------+ Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc --- M be/src/codegen/gen_ir_descriptions.py M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-util.cc M be/src/exec/kudu-util.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter.cc M be/src/runtime/runtime-filter.h M be/src/runtime/runtime-filter.inline.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/data-stream-service.cc M be/src/util/bloom-filter-test.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/min-max-filter-ir.cc M be/src/util/min-max-filter-test.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M common/protobuf/data_stream_service.proto M common/thrift/Planner.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation-hdfs-num-rows-est-enabled.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/result-spooling.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test A testdata/workloads/functional-query/queries/QueryTest/runtime_filters_mt_dop.test M tests/custom_cluster/test_mt_dop.py 45 files changed, 1,565 insertions(+), 632 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/14538/16 -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]> Gerrit-Reviewer: Thomas Tauber-Marshall <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
