> On March 20, 2013, 12:42 a.m., Dmitriy Ryaboy wrote: > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/PColFilterExtractor.java, > > line 224 > > <https://reviews.apache.org/r/10035/diff/1/?file=272254#file272254line224> > > > > (A and B) or (C and D) > > > > is impossible if (A or C) is false. We can push this up, while > > retaining the original filter to apply the original filter. > > Rohini Palaniswamy wrote: > So just to confirm, you want to extract A and C from each AND condition > and push (A OR C) as the partition filter for optimization and still leave > ((A AND B) or (C AND D)) to be applied on each tuple?
correct, unless my logic is wrong. I actually think we made a bad decision when we decided that if we can push partitions down, we can drop the filter on the pig side -- this means we can't take advantage of partial filters loaders might support (for example, a bloom filter a loader can consult to return just the rows that "probably" match the condition, as opposed to definitely match. With filter removal, we have to have loaders implement a second-pass filtering on top of such filters). - Dmitriy ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10035/#review18127 ----------------------------------------------------------- On March 20, 2013, 12:16 a.m., Rohini Palaniswamy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/10035/ > ----------------------------------------------------------- > > (Updated March 20, 2013, 12:16 a.m.) > > > Review request for pig. > > > Description > ------- > > 1) Fixed cases where partition pushdown was not happening for AND and OR > construct > 2) Commented out the negative test cases as they were actually not asserting > anything. > > > This addresses bug PIG-3173. > https://issues.apache.org/jira/browse/PIG-3173 > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/PColFilterExtractor.java > 1458047 > > http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestPartitionFilterPushDown.java > 1458047 > > Diff: https://reviews.apache.org/r/10035/diff/ > > > Testing > ------- > > Unit tests added and tested few cases manually with hcat. > > > Thanks, > > Rohini Palaniswamy > >
