Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16499 )
Change subject: IMPALA-10112: Remove FpRateTooHigh() check for bloom filter ...................................................................... IMPALA-10112: Remove FpRateTooHigh() check for bloom filter FpRateTooHigh() check for bloom filter that can be disabled if the observed false-positive probability (FPP) rate is higher than FLAGS_max_filter_error_rate. This patch remove FpRateTooHigh() check for several reasons: 1. Partition filters are probably still worth evaluating even if there are false positives because it is cheap and beneficial to eliminate a partition. 2. Runtime filters are dynamically disabled on the scan side if they are ineffective. Even an ALWAYS_TRUE_FILTER that substitutes a disabled filter is also still being evaluated and not entirely free. 3. The disabling is less likely to kick in for partitioned joins because it only applied to a small subset of the filters before the Or() operation. 4. FpRateTooHigh() use num_build_rows to approximate the FPP rate of the resulting filter. This can be inaccurate because it does not take account of duplicate values of the filter key on the build side. This patch also removes some tests in test_runtime_filters.py that check cancellation of filters having a high FPP rate. Testing: - Run and pass core tests. - Manually test and verify in a real large cluster (TPC-DS 30TB scale) that there is only a little (< 2.1%) to no performance regression incurred from the removal of high FPP check. The test used TPC-DS queries Q14a, Q50, Q64, Q71, Q84, Q93, and a modified Q93 with the left outer join replaced by an inner join. The query time comparison between having an FPP check versus without having an FPP check are the following: | Query | With check (ms) | Without check (ms) | |-------+-----------------+--------------------| | Q14a | 129801 | 125992 | | Q50 | 72519 | 72652 | | Q64 | 150684 | 145241 | | Q71 | 21009 | 20358 | | Q84 | 29334 | 29944 | | Q93 | 91782 | 91923 | | Q93m | 92897 | 92135 | Change-Id: Id9f8f40764b4f6664cc81b0da428afea8e3588d4 Reviewed-on: http://gerrit.cloudera.org:8080/16499 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test D testdata/workloads/functional-query/queries/QueryTest/bloom_filters_wait.test M tests/query_test/test_runtime_filters.py 6 files changed, 8 insertions(+), 76 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16499 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id9f8f40764b4f6664cc81b0da428afea8e3588d4 Gerrit-Change-Number: 16499 Gerrit-PatchSet: 4 Gerrit-Owner: Riza Suminto <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Riza Suminto <[email protected]> Gerrit-Reviewer: Shant Hovsepian <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
