On Thu, Mar 3, 2016 at 1:01 PM, Stack <st...@duboce.net> wrote: > On Thu, Mar 3, 2016 at 9:02 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > > Hi, > > As requested by Sean Busbey, I am starting a new thread w.r.t. > backporting > > MOB feature to branch-1. > > > This is to solicit discussion on the criteria for including MOB feature > > backport in branch-1. > > > > I can think of the following: > > 1. whether there is customer interest > > > > There is. > > See Jonathan's response here: > http://search-hadoop.com/m/YGbbDSqxD1PYXK62 > > > > > You should probably mention that a note requesting if there was interest > got no response here on the public lists. This would seem to imply no > interest by the community? > > Jon's note is pregnant but opaque on the details of these 'supported' > deploys. He is in a bit of an awkward spot unable to share detail on > someone else's deploy. > > Sorry, what I provided is basically the most I could offer.
Would be good to see more interest than Jon's note as evidence of interest > I'd say. > > You know of any? Even if they could be described in outline and hopefully > more than one instance and that MOB makes sense for this deploy. And they > need it now in a branch-1? > > There are some notes from the vote thread staying that Ashutosh from Huawei (at the time) had tested and patched part of mob -- might be interesting to see if he has anything to share. > Which version of hbase are we talking of backporting too? 1.3? 1.4? > > > > > 2. whether unit test stability can be maintained in branch-1 > > > > Inclusion of the backport should keep unit tests (both existing and new) > in > > branch-1 green. > > Preliminary test runs showed that MOB / snapshot related tests > consistently > > pass. > > > > > > > https://issues.apache.org/jira/browse/HBASE-15370?focusedCommentId=15176094&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15176094 > > > Hopefully we are aiming for more than just 'test stability'. Some large > community users have placed big bets on branch-1 being stable at scale; > this is the stability we should be precious of -- not just 'test > stability'. > > Besides, I see failures in your listing. MOB is pervasive. Why is it not > responsible for the mentioned test failures? (And "... passes on my local > machine... " doesn't cut it; builds.apache.org is our public CI where > community comes together on state of branch and master, not some devs > laptop). > I agree with stack here -- unit tests passing is not sufficient. > > Branch-1 builds have been generally stable after large investment fixing, > tuning and pruning tests. Master branch had been rendered stable but seems > to be rotting again though it is the most important of our builds given it > can catch the bad stuff before the commits make it in. During the campaign > to stabilize Master, MOB tests failed often. I see MOB failures in master > patch build from time to time still (I've been lax of late but our flakies > list contains a few mentionds, see HBASE-15012). They go unaddressed > (though, to be fair, Jingcheng, the original author, addressed a few > failures early on when asked). Just this morning I note: > > https://builds.apache.org/view/H-L/view/HBase/job/PreCommit-HBASE-Build/818/testReport/junit/org.apache.hadoop.hbase.mob.mapreduce/TestMobSweepMapper/org_apache_hadoop_hbase_mob_mapreduce_TestMobSweepMapper/ > > > (Suggestion: MOB seems to be better in master branch of late -- Matteo and > Heng work I believe (or just my disabling of broke tests) -- so a review > and report of master and patch builds looking for MOB failures and fixing > any found would help address the above concern. Another way to build > confidence in a patched branch-1 would be doing the branch suggested up in > the backport issue and then running builds on apache for a period. Or > better, copy what was done by Jon et al. to build confidence; run > long-running ITBLLs on a cluster w/ MOB set with obnoxious configs and post > evidence it all holds up). > Before I would +1 a branch merge, I would like to know that several of the fault injection tests ran and performed reasonably well. Before I proposed the hbase-11339 merge to trunk I had tested by adding ITIIngestMob and options to ITBLL, ITAcidGuarantees and LTT and added a monkey that triggered mob compaction operation. This is necessary to build up confidence that the feature works beyond just unit tests. As noted in the original trunk merge discussion thread, branch-1 is a stable branch, and the bar for merge should be higher than it was for trunk. > > > > Comments are welcome > > > > <https://issues.apache.org/jira/browse/HBASE-15370#> > > > > > Is MOB an 'owned' feature. The MOB original author is mostly absent (for > instance, no participation in these conversations and no general follow-up > on test failures). As has been said before many times, features need to be > owned. Writing the code is just one part of owning a feature. Features that > are not owned become a burden on the community to maintain. We have enough > of this latter type of feature already. > > (Jingcheng did show up in the last day though with HBASE-15381"Implement a > distributed MOB compaction by procedure" which looks like a pretty > important issue and begs the question, is MOB finished? And if not, > shouldn't we wait till its done before we backport?) > The proc v2 work necessary to do this is likely very similar to the work necessary to port snapshot to pv2. This isn't done yet and not in branch-1 yet. IMO, because of this, this particular criteria is not a blocker for me. > > Finally, MOB backport should be done by someone who is familiar with MOB. I > see no evidence of your expertise in MOB other than an odd review nor even > that you've run it in other than a unit test mode. Not to mention that the > way you go about the backport, dumping an 800k blob of unattributed code > into the issue for review in HBASE-15370, does not bode well for our > continued stability. > I don't have the cycles to do the backport, but I will do reviews if the procedure I outline was followed. I will also review any merge into branch-1 attempt. > > St.Ack > -- // Jonathan Hsieh (shay) // HBase Tech Lead, Software Engineer, Cloudera // j...@cloudera.com // @jmhsieh
