[
https://issues.apache.org/jira/browse/IMPALA-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17203468#comment-17203468
]
ASF subversion and git services commented on IMPALA-10112:
----------------------------------------------------------
Commit 5508be2b6273edd8074f466a7a64e70ecf304ddf in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5508be2 ]
IMPALA-10112: Remove FpRateTooHigh() check for bloom filter
FpRateTooHigh() check for bloom filter that can be disabled if the
observed false-positive probability (FPP) rate is higher than
FLAGS_max_filter_error_rate. This patch remove FpRateTooHigh() check for
several reasons:
1. Partition filters are probably still worth evaluating even if there
are false positives because it is cheap and beneficial to eliminate a
partition.
2. Runtime filters are dynamically disabled on the scan side if they are
ineffective. Even an ALWAYS_TRUE_FILTER that substitutes a disabled
filter is also still being evaluated and not entirely free.
3. The disabling is less likely to kick in for partitioned joins because
it only applied to a small subset of the filters before the Or()
operation.
4. FpRateTooHigh() use num_build_rows to approximate the FPP rate of the
resulting filter. This can be inaccurate because it does not take
account of duplicate values of the filter key on the build side.
This patch also removes some tests in test_runtime_filters.py that check
cancellation of filters having a high FPP rate.
Testing:
- Run and pass core tests.
- Manually test and verify in a real large cluster (TPC-DS 30TB scale)
that there is only a little (< 2.1%) to no performance regression
incurred from the removal of high FPP check. The test used TPC-DS
queries Q14a, Q50, Q64, Q71, Q84, Q93, and a modified Q93 with the
left outer join replaced by an inner join.
The query time comparison between having an FPP check versus without
having an FPP check are the following:
| Query | With check (ms) | Without check (ms) |
|-------+-----------------+--------------------|
| Q14a | 129801 | 125992 |
| Q50 | 72519 | 72652 |
| Q64 | 150684 | 145241 |
| Q71 | 21009 | 20358 |
| Q84 | 29334 | 29944 |
| Q93 | 91782 | 91923 |
| Q93m | 92897 | 92135 |
Change-Id: Id9f8f40764b4f6664cc81b0da428afea8e3588d4
Reviewed-on: http://gerrit.cloudera.org:8080/16499
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Consider skipping FpRateTooHigh() check for bloom filters
> ---------------------------------------------------------
>
> Key: IMPALA-10112
> URL: https://issues.apache.org/jira/browse/IMPALA-10112
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Riza Suminto
> Priority: Major
> Labels: performance
> Fix For: Impala 4.0
>
>
> This check disables bloom filters on the sender side.
> It is inaccurate in cases where there are duplicate values of the filter key
> on the build side. E.g. many-to-many join or a join with multiple keys. This
> could be fixed with some effort, but is probably not worth it, because:
> * Partition filters are probably still worth evaluating even if there are
> false positives, because it's cheap and eliminating a partition is still
> beneficial.
> * Runtime filters are dynamically disabled on the scan side if they are
> ineffective. I think we still also "evaluate" the always true filter, which
> is cheaper than doing the hashing and bloom evaluation, but still not
> entirely free.
> * The disabling is fairly unlikely to kick in for partitioned joins because
> it's only applied to a small subset of the filter, before the Or() operation.
> So it's potentially harmful and only likely beneficial for broadcast join
> filters, in which case it saves a small amount of scan CPU and, for global
> filters, coordinator RPCs and broadcasting. It's unclear that the complexity
> is worth it for this relatively small and uncertain benefit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
