The idea of optimizer rule failure did not fail the entire query is that rule's failure might only happen in a plan which turn out to be sub-optimal, and we do not want block optimizer from continuing finding the final/optimal plan.
If the case of PruneScanRule, the filters to be pushed might be pushed down from ancestor operators (join, or other filter, etc). That is, PruneScanRule could be fired multiple times, each with different filter condition. If after one filter triggers the failure, Drill stops the entire query, then Drill does not have the chance find the final plan. For issue either (a) or (b), one criteria is whether the same filter evaluation would hit the same failure under the evaluation model as the interpreter model. If the interpreter model hit a failure that is not going to happen in the evaluation model, that means there is a bug which deserves a fix. On the other hand, I understand that providing such option would save user from seeing inconsistent behavior in partition pruning. On Wed, Aug 26, 2015 at 8:51 AM, Vicky Markman <[email protected]> wrote: > There is Jira for (a) as well: > https://issues.apache.org/jira/browse/DRILL-3045 > > On Wed, Aug 26, 2015 at 8:31 AM, Aman Sinha <[email protected]> wrote: > > > We have had some issues where the same query run at different times > > (possibly with other queries running concurrently...not sure about the > > concurrency level) either performed partition pruning or did not. The > > times where it failed happened due to couple of reasons : > > (a) allocateNew() in the PruneScanRule failed with an out of memory > > condition > > (b) the interpreter evaluator encountered an error with a particular > > expression type evaluation > > > > The PruneScanRule currently logs a warning message and does not fail the > > query since this is a performance optimization. While we will address > the > > root cause of (a) and (b) (there's a JIRA open for (b) ) an important > > issue is the inconsistent behavior of a query. > > > > Should we provide a system setting that allows the query to fail in this > > situation ? > > Note that other rules in the optimizer could also fail and some rules > log > > warnings but those failures are very rare, while PruneScan rule is doing > > more complex operations - creating value vectors, doing interpreter > > evaluation - so the chances of something failing increases. > > > > Aman > > >
