Thanks for chiming in Jincheng. Really appreciated. But from this thread, it's apparent that people don't have enough enthusiasm to see this backported in branch-1. So, maybe what would be really nice to instead do is to make sure that MOB is performant / reliable / releasable in the HBASE-2.0 release. As suggested in this thread, let's run YCSB / ITBLL and other such tests to make sure we have enough confidence on this before HBASE-2.0 goes out. Of course, Unit Tests is another area in the master branch that MOB is still flaky in.... ________________________________________ From: Du, Jingcheng <jingcheng...@intel.com> Sent: Thursday, March 03, 2016 9:55 PM To: dev@hbase.apache.org Subject: RE: [DISCUSS] Criteria for including MOB feature backport in branch-1
Sorry for the late chime in and thank you all for the discussion. I agree with Jon's suggestion, to make a new branch for MOB in branch-1.x, and backport it when it is definitely ready. I can start to work on that branch after the distributed mob compaction is finished. I think the distributed mob compaction is not a blocker. The old mob compaction can be disabled by configuration, this would not impact the read/write performance of mob files. Users can count on the TTL cleaner to remove the expired mob files which is very light weighted. And I will keep eyes on the UT results, and get ready to fix the coming up failures. Thanks. Regards, Jingcheng -----Original Message----- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Friday, March 4, 2016 9:53 AM To: dev@hbase.apache.org Subject: Re: [DISCUSS] Criteria for including MOB feature backport in branch-1 Good questions, Enis. If my bandwidth permits, I am planning to collect performance statistics using ycsb against cluster with and without MOB feature enabled. Cheers On Thu, Mar 3, 2016 at 3:49 PM, Enis Söztutar <enis....@gmail.com> wrote: > Regardless of the backport, did we do the same regression analysis for > the master branch merge at the time of the merge? Like make sure that > the latencies and stability of non-mob is affected or not. Sorry, I > was not following the merge vote closely. > > The reason I am asking is that although we can allow some flexibility > in master, we still want to keep master releasable and prevent a > 1-year effort to re-stabilize master when we want to release 2.0. If > we have design / stability / impact questions, when and how they will > be addressed for a 2.0 release? Let's say that we would like to > release 2.0 by summer time as Matteo was quoting, how do we make sure > that the feature as is, is stable and supportable? > > Enis > > On Thu, Mar 3, 2016 at 3:36 PM, Andrew Purtell <apurt...@apache.org> > wrote: > > > Just to be clear, I did not actually oppose a backport of MOB. > > Although > - I > > have to say am sympathetic to Elliott's position that there's a > > project management reason not to. My insistence here is "merely" for > > this change > in > > particular a backport to the branch we are making production > > releases > from > > demands a credible data driven assessment on stability, > > functionality, > and > > performance - both avoidance of regression and affirmative > > indication of the alleged benefit. > > > > On Thu, Mar 3, 2016 at 3:16 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > In the middle of writing response to this thread, I saw subsequent > > comments > > > from Andrew, Jon and Elliott. > > > > > > In light of opposition from the community w.r.t. backporting MOB > > feature, I > > > think it suffices to say that this wouldn't be done now. > > > > > > Thanks everyone for chiming in. > > > > > > On Thu, Mar 3, 2016 at 2:42 PM, Elliott Clark <ecl...@apache.org> > wrote: > > > > > > > I just don't see a why we would back port. We're going to > > > > release a > 2.0 > > > > when things are ready. It will be a major feature release. MOB > > > > is a > > major > > > > feature. Without compelling reason backporting to branch-1 seems > > > > like > > an > > > > end run around what sem ver is supposed to mean (not the api > > guarantees, > > > > the actual meaning). > > > > > > > > Backporting major features is a bad habit that the Hadoop > > > > community > > seems > > > > to have. We shouldn't follow their lead. > > > > > > > > On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote: > > > > > > > > > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > > > > > > > > Hi, > > > > > > As requested by Sean Busbey, I am starting a new thread w.r.t. > > > > > backporting > > > > > > MOB feature to branch-1. > > > > > > > > > > > This is to solicit discussion on the criteria for including > > > > > MOB > > feature > > > > > > backport in branch-1. > > > > > > > > > > > > I can think of the following: > > > > > > 1. whether there is customer interest > > > > > > > > > > > > There is. > > > > > > See Jonathan's response here: > > > > > http://search-hadoop.com/m/YGbbDSqxD1PYXK62 > > > > > > > > > > > > > > > > > You should probably mention that a note requesting if there > > > > > was > > > interest > > > > > got no response here on the public lists. This would seem to > > > > > imply > no > > > > > interest by the community? > > > > > > > > > > Jon's note is pregnant but opaque on the details of these > 'supported' > > > > > deploys. He is in a bit of an awkward spot unable to share > > > > > detail > on > > > > > someone else's deploy. > > > > > > > > > > Would be good to see more interest than Jon's note as evidence > > > > > of > > > > interest > > > > > I'd say. > > > > > > > > > > You know of any? Even if they could be described in outline > > > > > and > > > hopefully > > > > > more than one instance and that MOB makes sense for this deploy. > And > > > they > > > > > need it now in a branch-1? > > > > > > > > > > Which version of hbase are we talking of backporting too? 1.3? 1.4? > > > > > > > > > > > > > > > > > > > > > 2. whether unit test stability can be maintained in branch-1 > > > > > > > > > > > > Inclusion of the backport should keep unit tests (both > > > > > > existing > and > > > > new) > > > > > in > > > > > > branch-1 green. > > > > > > Preliminary test runs showed that MOB / snapshot related > > > > > > tests > > > > > consistently > > > > > > pass. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=151 > 76094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tab > panel#comment-15176094 > > > > > > > > > > > > > > > Hopefully we are aiming for more than just 'test stability'. > > > > > Some > > large > > > > > community users have placed big bets on branch-1 being stable > > > > > at > > scale; > > > > > this is the stability we should be precious of -- not just > > > > > 'test stability'. > > > > > > > > > > Besides, I see failures in your listing. MOB is pervasive. Why > > > > > is > it > > > not > > > > > responsible for the mentioned test failures? (And "... passes > > > > > on my > > > local > > > > > machine... " doesn't cut it; builds.apache.org is our public > > > > > CI > > where > > > > > community comes together on state of branch and master, not > > > > > some > devs > > > > > laptop). > > > > > > > > > > Branch-1 builds have been generally stable after large > > > > > investment > > > fixing, > > > > > tuning and pruning tests. Master branch had been rendered > > > > > stable > but > > > > seems > > > > > to be rotting again though it is the most important of our > > > > > builds > > given > > > > it > > > > > can catch the bad stuff before the commits make it in. During > > > > > the > > > > campaign > > > > > to stabilize Master, MOB tests failed often. I see MOB > > > > > failures in > > > master > > > > > patch build from time to time still (I've been lax of late but > > > > > our > > > > flakies > > > > > list contains a few mentionds, see HBASE-15012). They go > unaddressed > > > > > (though, to be fair, Jingcheng, the original author, addressed > > > > > a > few > > > > > failures early on when asked). Just this morning I note: > > > > > > > > > > > > > > > > > > > > https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Buil > d/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSw > eepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/ > > > > > > > > > > > > > > > (Suggestion: MOB seems to be better in master branch of late > > > > > -- > > Matteo > > > > and > > > > > Heng work I believe (or just my disabling of broke tests) -- > > > > > so a > > > review > > > > > and report of master and patch builds looking for MOB failures > > > > > and > > > fixing > > > > > any found would help address the above concern. Another way to > build > > > > > confidence in a patched branch-1 would be doing the branch > suggested > > up > > > > in > > > > > the backport issue and then running builds on apache for a period. > Or > > > > > better, copy what was done by Jon et al. to build confidence; > > > > > run long-running ITBLLs on a cluster w/ MOB set with obnoxious > > > > > configs > > and > > > > post > > > > > evidence it all holds up). > > > > > > > > > > > > > > > > Comments are welcome > > > > > > > > > > > > <https://issues.apache.org/jira/browse/HBASE-15370#> > > > > > > > > > > > > > > > > > > > > > Is MOB an 'owned' feature. The MOB original author is mostly > > > > > absent > > > (for > > > > > instance, no participation in these conversations and no > > > > > general > > > > follow-up > > > > > on test failures). As has been said before many times, > > > > > features > need > > to > > > > be > > > > > owned. Writing the code is just one part of owning a feature. > > Features > > > > that > > > > > are not owned become a burden on the community to maintain. We > > > > > have > > > > enough > > > > > of this latter type of feature already. > > > > > > > > > > (Jingcheng did show up in the last day though with > > > HBASE-15381"Implement > > > > a > > > > > distributed MOB compaction by procedure" which looks like a > > > > > pretty important issue and begs the question, is MOB finished? > > > > > And if not, shouldn't we wait till its done before we > > > > > backport?) > > > > > > > > > > Finally, MOB backport should be done by someone who is > > > > > familiar > with > > > > MOB. I > > > > > see no evidence of your expertise in MOB other than an odd > > > > > review > nor > > > > even > > > > > that you've run it in other than a unit test mode. Not to > > > > > mention > > that > > > > the > > > > > way you go about the backport, dumping an 800k blob of > > > > > unattributed > > > code > > > > > into the issue for review in HBASE-15370, does not bode well > > > > > for > our > > > > > continued stability. > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein (via Tom White) > > >
