Due to an infinite loop occurring in Calcite planning, we had to disable the filter pushdown past the union (SetOps). See https://issues.apache.org/jira/browse/DRILL-3855. Now that we have rebased on Calcite 1.15.0, we should re-enable this and test and if the pushdown works then the partition pruning on both sides of the union should automatically work after that.
Will follow-up on this.. -Aman On Mon, Mar 19, 2018 at 3:02 PM, Kunal Khatua <[email protected]> wrote: > I think Ted's question is 2 fold, with the former being more important. > 1. Can we push filters past a union. > 2. Will Drill push filters down to the source. > > For the latter, it depends on the source. > For the former, it depends primarily on whether Calcite supports this. I > haven't tried it, so I can't say. > > On 3/19/2018 2:22:54 PM, rahul challapalli <[email protected]> > wrote: > First I would suggest to ignore the view and try out a query which has the > required filters as part of the subqueries on both sides of the union (for > both the database and partitioned parquet data). The plan for such a query > should have the answers to your question. If both the subqueries > independently prune out un-necessary data, using partitions or indexes, I > don't think adding a union between them would alter that behavior. > > -Rahul > > On Mon, Mar 19, 2018 at 1:44 PM, Ted Dunning wrote: > > > IF I create a view that is a union of partitioned parquet files and a > > database that has secondary indexes, will Drill be able to properly push > > down query limits into both parts of the union? > > > > In particular, if I have lots of archival data and parquet partitioned by > > time but my query only asks for recent data that is in the database, will > > the query avoid the parquet files entirely (as you would wish)? > > > > Conversely, if the data I am asking for is entirely in the archive, will > > the query make use of the partitioning on my parquet files correctly? > > >
