somu-imply commented on a change in pull request #12195:
URL: https://github.com/apache/druid/pull/12195#discussion_r800291882
##########
File path:
sql/src/main/java/org/apache/druid/sql/calcite/run/NativeQueryMaker.java
##########
@@ -122,6 +127,32 @@ public boolean feature(QueryFeature feature)
);
}
}
+ int numFilters =
plannerContext.getPlannerConfig().getMaxNumericInFilters();
+
+ // special corner case handling for numeric IN filters
+ // in case of query containing IN (v1, v2, v3,...) where Vi is numeric
+ // a BoundFilter is created internally for each of the values
+ // whereas when Vi s are String the Filters are converted as BoundFilter
to SelectorFilter to InFilter
+ // which takes lesser processing for bitmaps
+ // So in a case where user executes a query with multiple numeric INs,
flame graph shows BoundFilter.getBitmapResult
+ // and BoundFilter.match predicate eating up processing time which stalls
a historical for a query with large number
+ // of numeric INs (> 10K). In such cases user should change the query to
specify the IN clauses as String
+ // Instead of IN(v1,v2,v3) user should specify IN('v1','v2','v3')
+ if (numFilters != PlannerConfig.NUM_FILTER_NOT_USED) {
+ if (query.getFilter() instanceof OrDimFilter) {
Review comment:
With ANDs or ORs or ORs of ANDs traversing the filter tree should not be
expensive. Would be O(#filters). With cases of filters going upto 10K or even
100K this can be done. In case of combinations such as
```
dim1=1 AND dim2=2 AND dim3=3 AND dim4 IN (1,2,3)
```
with `maxNumericInFilter=4` it would be difficult to point out which
dimension is at fault. A generic error message can be thrown in that case
saying something like the number of elements in WHERE clause exceeds system
specified limit of K
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
