Hi Boaz, I looked at DRILL-6606 and have updated the ticket, I should have a fix monday. It looks like a minor logical error.
I'm not clear on why you suspect DRILL-6453 is cause by batch sniffing, perhaps we can discuss offline. Thanks, Tim On Fri, Jul 13, 2018 at 2:01 PM, Arina Yelchiyeva < arina.yelchiy...@gmail.com> wrote: > Two more regressions: > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > apache.org_jira_browse_DRILL-2D6603&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r= > 4eQVr8zB8ZBff-yxTimdOQ&m=F7Ih5Ah_SfOS5fZXFApt88iMe3Vd- > Jq1XDvxPN6b3y4&s=FqDR26vK2kVG-P69NcqoqNxRrdHKZvBCWamYRftPPYg&e= > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > apache.org_jira_browse_DRILL-2D6605&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r= > 4eQVr8zB8ZBff-yxTimdOQ&m=F7Ih5Ah_SfOS5fZXFApt88iMe3Vd-Jq1XDvxPN6b3y4&s= > KzhJcDP4F8FIA7h4GiQE8wg_CSae0CAI0yEMslff52o&e= > > Kind regards, > Arina > > On Fri, Jul 13, 2018 at 11:25 PM Sorabh Hamirwasia <shamirwa...@mapr.com> > wrote: > > > Hi Boaz, > > Couple of updates. > > > > *Merged In:* > > DRILL-6542: (May be Ready2Commit soon) IndexOutOfBounds exception for > > multilevel lateral ((Sorabh / Parth)) > > > > *In Review:* > > > > > > *DRILL-6475: Query with UNNEST causes a Null Pointer . (( Hanumath ))* > > Thanks, > > Sorabh > > > > On Fri, Jul 13, 2018 at 1:17 PM, Parth Chandra <par...@apache.org> > wrote: > > > > > Our (unwritten) rule has been that a commit cannot even go in unless > unit > > > _and_ regression tests pass. > > > Releases are stricter, all tests, longevity tests, UI, are required to > > > pass. In addition, any performance regression needs to be discussed. > > > > > > So far we have not made any exceptions, but that is not to say we > cannot. > > > > > > On Fri, Jul 13, 2018 at 1:03 PM, Vlad Rozov <vro...@apache.org> wrote: > > > > > > > My 2 cents: > > > > > > > > From Apache point of view it is OK to do a release even if unit tests > > do > > > > not pass at all or there is a large number of regression introduced. > > > Apache > > > > release is a source release and as long as it compiles and does not > > have > > > > license issues, it is up to community (PMC) to decide on any other > > > criteria > > > > for a release. > > > > > > > > The issue in DRILL-6453 is not limited to a large number of hash > joins. > > > It > > > > should be possible to reproduce it even with a single hash join as > long > > > as > > > > left and right sides are getting batches from one(many) to many > > exchanges > > > > (broadcast or hash partitioner senders). > > > > > > > > Thank you, > > > > > > > > Vlad > > > > > > > > > > > > On 7/13/18 08:41, Aman Sinha wrote: > > > > > > > >> I would say we have to take a measured approach to this and decide > on > > a > > > >> case-by-case which issue is a show stopper. > > > >> While of course we have to make every effort to avoid regression, we > > > >> cannot > > > >> claim that a particular release will not cause any regression. > > > >> I believe there are 10000+ passing tests, so that should provide a > > > level > > > >> of confidence. The TPC-DS 72 is a 10 table join which in the > hadoop > > > >> world > > > >> of > > > >> denormalized schemas is not relatively common. The main question is > > > does > > > >> the issue reproduce with fewer joins having the same type of > > > distribution > > > >> plan ? > > > >> > > > >> > > > >> Aman > > > >> > > > >> On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva < > > > >> arina.yelchiy...@gmail.com> > > > >> wrote: > > > >> > > > >> We cannot release with existing regressions, especially taking into > > > >>> account > > > >>> the there are not minor issues. > > > >>> As far as I understand reverting is not an option since hash join > > spill > > > >>> feature are extended into several commits + subsequent fixes. > > > >>> I guess we need to consider postponing the release until issues are > > > >>> resolved. > > > >>> > > > >>> Kind regards, > > > >>> Arina > > > >>> > > > >>> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <b...@apache.org> > > wrote: > > > >>> > > > >>> (Guessing ...) It is possible that the root cause for DRILL-6606 is > > > >>>> similar to that in DRILL-6453 -- that is the new "early sniffing" > > in > > > >>>> the > > > >>>> Hash-Join, which repeatedly invokes next() on the two "children" > of > > > the > > > >>>> join *during schema discovery* until non-empty data is returned > (or > > > >>>> NONE, > > > >>>> STOP, etc). Last night Salim, Vlad and I briefly discussed > > > >>>> alternatives, > > > >>>> like postponing the "sniffing" to a later time (beginning of the > > build > > > >>>> > > > >>> for > > > >>> > > > >>>> the right child, and beginning of the probe for the left child). > > > >>>> > > > >>>> However this would require some work time. So what should we do > > about > > > >>>> > > > >>> 1.14 > > > >>> > > > >>>> ? > > > >>>> > > > >>>> Thanks, > > > >>>> > > > >>>> Boaz > > > >>>> > > > >>>> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva < > > > >>>> arina.yelchiy...@gmail.com> wrote: > > > >>>> > > > >>>> During implementing late limit 0 optimization, Bohdan has found > one > > > more > > > >>>>> regression after Hash Join spill to disk. > > > >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > > > apache.org_jira_browse_DRILL-2D6606&d=DwIBaQ&c= > cskdkSMqhcnjZxdQVpwTXg&r= > > > gRpEl0WzXE3EMrwj0KFbZXGXRyadOthF2jlYxvhTlQg&m=TGqnVoxNweQMiHTgP4J- > > > rOnguFThVKShnQqHE_CmySI&s=aHUGrClgE_9UsRpRlNM95TbW91ivkqGdF1hV0EDc > 3xU&e= > > > >>>>> < > > > >>>>> > > > >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > > > >>> apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZ > > > >>> xdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E > > > >>> 90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y > 6XC0ZUqVmSjko&e= > > > >>> > > > >>>> Boaz please take a look. > > > >>>>> > > > >>>>> Kind regards, > > > >>>>> Arina > > > >>>>> > > > >>>>> > > > >>>> > > > > > > > > > >
