So the errors are pretty sparse. We're running in a pretty reasonable test cluster and only seeing 300 errors out of 39 billion.
I'm going to start things running on two clusters with some logging, if anyone has some other hints around what to look for. Or can get a better repro that would be really helpful. On Wed, Jun 22, 2016 at 2:21 PM, Dima Spivak <dspi...@cloudera.com> wrote: > Hey Elliott (et al.), > > I don’t have insight into the code that might have gone in to scanners, but > I do have the ability to get clusters with chaos monkeys set up quickly > (and some machines in-house to run them), so I’d be happy to help if > there’s anything I can do. > > -Dima > > On Wed, Jun 22, 2016 at 12:47 PM, Elliott Clark <ecl...@apache.org> wrote: > > > Could use some help in HBASE-16074 if anyone has a cluster that has chaos > > monkey set up. Right now it looks like there is some issue with scanners > > during failures giving corrupt data. > > > > On Wed, Jun 22, 2016 at 12:28 PM, Mikhail Antonov <olorinb...@gmail.com> > > wrote: > > > > > Hi guys, > > > > > > as we're stabilizing branch-1.3 builds and I also need to keep release > > > notes / tag for 1.3 accurate, could you please ping me on jira if you > > > commit something to this branch (I read commit log, but it's easy to > miss > > > something in there)? > > > > > > Thanks! > > > Mikhail > > > > > > On Fri, Jun 17, 2016 at 3:24 PM, Mikhail Antonov <olorinb...@gmail.com > > > > > wrote: > > > > > > > Yeah, branch-1.3 was cut some time ago and for a while most of > commits > > > > going to branch-1 would also go to it, but last few days > > > > I'm trying to let only the following things go in: > > > > > > > > - criticals and blockers > > > > - test fixes and other patches stabilizing the branch > > > > - cherry-picks that were missed earlier. > > > > - oneliners / doc changes etc > > > > > > > > Appreciate understanding and help :) > > > > > > > > -Mikhail > > > > > > > > On Fri, Jun 17, 2016 at 3:07 PM, Enis Söztutar <e...@apache.org> > > wrote: > > > > > > > >> nvm, it is there already. > > > >> > > > >> On Fri, Jun 17, 2016 at 3:06 PM, Enis Söztutar <e...@apache.org> > > wrote: > > > >> > > > >> > Mikhail, I suggest that we create the branch-1.3 now so that you > can > > > >> > control what goes in and what not. branch-1 is free for all > usually. > > > >> > > > > >> > Enis > > > >> > > > > >> > On Wed, Jun 15, 2016 at 9:24 PM, Mikhail Antonov < > > > olorinb...@gmail.com> > > > >> > wrote: > > > >> > > > > >> >> Suddenly we had kind of a spike in jiras filed with > fixVersion=1.3 > > > last > > > >> >> few > > > >> >> days, and I really want to get it out one of this days, so I want > > to > > > >> just > > > >> >> stabilize it now. > > > >> >> > > > >> >> I kicked some jiras labeled as "major" out of 1.3, and if there's > > > >> >> something > > > >> >> affecting branch-1.3 but not "Blocker" or "Critical" let's target > > it > > > >> for > > > >> >> 1.3.1 and / or 1.4. > > > >> >> > > > >> >> Thanks! > > > >> >> Mikhail > > > >> >> > > > >> >> On Mon, Jun 13, 2016 at 11:52 AM, Mikhail Antonov < > > > >> olorinb...@gmail.com> > > > >> >> wrote: > > > >> >> > > > >> >> > I'm not aware of any, and changes made to 1.3 shouldn't render > > 2.4 > > > >> >> > unsupportable. > > > >> >> > > > > >> >> > On the second thought, if we want to have to maintain less > minor > > > >> >> releases > > > >> >> > in 1.* line and encourage folks to update, > > > >> >> > we need to keep maintaining those Hadoop versions, yeah. > > > >> >> > > > > >> >> > Let's leave 2.4 as supported. > > > >> >> > > > > >> >> > -Mikhail > > > >> >> > > > > >> >> > On Mon, Jun 13, 2016 at 7:11 AM, Sean Busbey < > > bus...@cloudera.com> > > > >> >> wrote: > > > >> >> > > > > >> >> >> On Fri, Jun 10, 2016 at 7:00 PM, Mikhail Antonov < > > > >> olorinb...@gmail.com > > > >> >> > > > > >> >> >> wrote: > > > >> >> >> > > > > >> >> >> > I'm thinking to move Hadoop 2.4.* from Supported to Not > > Tested, > > > to > > > >> >> kind > > > >> >> >> of > > > >> >> >> > encourage people to move and have less versions to test. How > > > many > > > >> >> people > > > >> >> >> > want to stick with Hadoop 2.4 yet upgrade to HBase 1.3? > > > >> >> >> > > > > >> >> >> > > > >> >> >> Hadoop 2.4 is still considered a "safe bet" stable release for > > > those > > > >> >> >> in LTM mode. > > > >> >> >> > > > >> >> >> Our compatibility guidelines say that we won't force an > > > incompatible > > > >> >> >> dependency > > > >> >> >> upgrade in a minor version. Do we know if Hadoop 2.4 -> 2.5 > > > includes > > > >> >> any > > > >> >> >> documented incompatibilities? > > > >> >> >> > > > >> >> >> -- > > > >> >> >> busbey > > > >> >> >> > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > -- > > > >> >> > Thanks, > > > >> >> > Michael Antonov > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> -- > > > >> >> Thanks, > > > >> >> Michael Antonov > > > >> >> > > > >> > > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > Thanks, > > > > Michael Antonov > > > > > > > > > > > > > > > > -- > > > Thanks, > > > Michael Antonov > > > > > >
