Yeah, let me get a few machines going tomorrow and script something together that uses git bisect along with clusterdock to stand up clusters and run ITBLL. In the meantime, should we open a bug to address why the test doesn't fail even when nodes are dropped from other CFs?
-Dima On Wednesday, June 22, 2016, Ted Yu <[email protected]> wrote: > I think Dima's past work would help narrow down the patch(es) which > introduced the regression: > > http://search-hadoop.com/m/YGbbPH6kn1dKe8j2 > > A list of JIRAs which went into 1.3 but not 1.2.2 can be obtained. > The scope of search should be considerably smaller once we have the list. > > FYI > > On Wed, Jun 22, 2016 at 3:15 PM, Elliott Clark <[email protected] > <javascript:;>> wrote: > > > So the errors are pretty sparse. We're running in a pretty reasonable > test > > cluster and only seeing 300 errors out of 39 billion. > > > > I'm going to start things running on two clusters with some logging, if > > anyone has some other hints around what to look for. Or can get a better > > repro that would be really helpful. > > > > On Wed, Jun 22, 2016 at 2:21 PM, Dima Spivak <[email protected] > <javascript:;>> wrote: > > > > > Hey Elliott (et al.), > > > > > > I don’t have insight into the code that might have gone in to scanners, > > but > > > I do have the ability to get clusters with chaos monkeys set up quickly > > > (and some machines in-house to run them), so I’d be happy to help if > > > there’s anything I can do. > > > > > > -Dima > > > > > > On Wed, Jun 22, 2016 at 12:47 PM, Elliott Clark <[email protected] > <javascript:;>> > > wrote: > > > > > > > Could use some help in HBASE-16074 if anyone has a cluster that has > > chaos > > > > monkey set up. Right now it looks like there is some issue with > > scanners > > > > during failures giving corrupt data. > > > > > > > > On Wed, Jun 22, 2016 at 12:28 PM, Mikhail Antonov < > > [email protected] <javascript:;>> > > > > wrote: > > > > > > > > > Hi guys, > > > > > > > > > > as we're stabilizing branch-1.3 builds and I also need to keep > > release > > > > > notes / tag for 1.3 accurate, could you please ping me on jira if > you > > > > > commit something to this branch (I read commit log, but it's easy > to > > > miss > > > > > something in there)? > > > > > > > > > > Thanks! > > > > > Mikhail > > > > > > > > > > On Fri, Jun 17, 2016 at 3:24 PM, Mikhail Antonov < > > [email protected] <javascript:;> > > > > > > > > > wrote: > > > > > > > > > > > Yeah, branch-1.3 was cut some time ago and for a while most of > > > commits > > > > > > going to branch-1 would also go to it, but last few days > > > > > > I'm trying to let only the following things go in: > > > > > > > > > > > > - criticals and blockers > > > > > > - test fixes and other patches stabilizing the branch > > > > > > - cherry-picks that were missed earlier. > > > > > > - oneliners / doc changes etc > > > > > > > > > > > > Appreciate understanding and help :) > > > > > > > > > > > > -Mikhail > > > > > > > > > > > > On Fri, Jun 17, 2016 at 3:07 PM, Enis Söztutar <[email protected] > <javascript:;>> > > > > wrote: > > > > > > > > > > > >> nvm, it is there already. > > > > > >> > > > > > >> On Fri, Jun 17, 2016 at 3:06 PM, Enis Söztutar <[email protected] > <javascript:;>> > > > > wrote: > > > > > >> > > > > > >> > Mikhail, I suggest that we create the branch-1.3 now so that > you > > > can > > > > > >> > control what goes in and what not. branch-1 is free for all > > > usually. > > > > > >> > > > > > > >> > Enis > > > > > >> > > > > > > >> > On Wed, Jun 15, 2016 at 9:24 PM, Mikhail Antonov < > > > > > [email protected] <javascript:;>> > > > > > >> > wrote: > > > > > >> > > > > > > >> >> Suddenly we had kind of a spike in jiras filed with > > > fixVersion=1.3 > > > > > last > > > > > >> >> few > > > > > >> >> days, and I really want to get it out one of this days, so I > > want > > > > to > > > > > >> just > > > > > >> >> stabilize it now. > > > > > >> >> > > > > > >> >> I kicked some jiras labeled as "major" out of 1.3, and if > > there's > > > > > >> >> something > > > > > >> >> affecting branch-1.3 but not "Blocker" or "Critical" let's > > target > > > > it > > > > > >> for > > > > > >> >> 1.3.1 and / or 1.4. > > > > > >> >> > > > > > >> >> Thanks! > > > > > >> >> Mikhail > > > > > >> >> > > > > > >> >> On Mon, Jun 13, 2016 at 11:52 AM, Mikhail Antonov < > > > > > >> [email protected] <javascript:;>> > > > > > >> >> wrote: > > > > > >> >> > > > > > >> >> > I'm not aware of any, and changes made to 1.3 shouldn't > > render > > > > 2.4 > > > > > >> >> > unsupportable. > > > > > >> >> > > > > > > >> >> > On the second thought, if we want to have to maintain less > > > minor > > > > > >> >> releases > > > > > >> >> > in 1.* line and encourage folks to update, > > > > > >> >> > we need to keep maintaining those Hadoop versions, yeah. > > > > > >> >> > > > > > > >> >> > Let's leave 2.4 as supported. > > > > > >> >> > > > > > > >> >> > -Mikhail > > > > > >> >> > > > > > > >> >> > On Mon, Jun 13, 2016 at 7:11 AM, Sean Busbey < > > > > [email protected] <javascript:;>> > > > > > >> >> wrote: > > > > > >> >> > > > > > > >> >> >> On Fri, Jun 10, 2016 at 7:00 PM, Mikhail Antonov < > > > > > >> [email protected] <javascript:;> > > > > > >> >> > > > > > > >> >> >> wrote: > > > > > >> >> >> > > > > > > >> >> >> > I'm thinking to move Hadoop 2.4.* from Supported to Not > > > > Tested, > > > > > to > > > > > >> >> kind > > > > > >> >> >> of > > > > > >> >> >> > encourage people to move and have less versions to test. > > How > > > > > many > > > > > >> >> people > > > > > >> >> >> > want to stick with Hadoop 2.4 yet upgrade to HBase 1.3? > > > > > >> >> >> > > > > > > >> >> >> > > > > > >> >> >> Hadoop 2.4 is still considered a "safe bet" stable release > > for > > > > > those > > > > > >> >> >> in LTM mode. > > > > > >> >> >> > > > > > >> >> >> Our compatibility guidelines say that we won't force an > > > > > incompatible > > > > > >> >> >> dependency > > > > > >> >> >> upgrade in a minor version. Do we know if Hadoop 2.4 -> > 2.5 > > > > > includes > > > > > >> >> any > > > > > >> >> >> documented incompatibilities? > > > > > >> >> >> > > > > > >> >> >> -- > > > > > >> >> >> busbey > > > > > >> >> >> > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > -- > > > > > >> >> > Thanks, > > > > > >> >> > Michael Antonov > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> -- > > > > > >> >> Thanks, > > > > > >> >> Michael Antonov > > > > > >> >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Thanks, > > > > > > Michael Antonov > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Michael Antonov > > > > > > > > > > > > > > >
