Hi folks, I had the chance today to run some performance benchmarks on one of Cloudera's 100 node test clusters. I've posted the results to the JIRA, but the summary is that I think we need to do a bit more optimization of the persistBlocks edit log entries before we merge.
That said, I think there are a couple easy wins that should bring performance back in line with trunk, which I expect to complete by early next week. Please continue reviewing the branch so that when the optimizations have been made, we can proceed with a merge. -Todd On Thu, Feb 23, 2012 at 10:27 AM, Todd Lipcon <t...@cloudera.com> wrote: > On Thu, Feb 23, 2012 at 10:23 AM, Suresh Srinivas > <sur...@hortonworks.com> wrote: >> I am not sure any of these issues are serious show stoppers for merging >> into trunk. >> Why not merge into trunk and fix some of these issues? >> >> The reason is, merging is non trivial with two branches changing >> independently. Given that >> Jitendra has posted a merge patch, why not do it earlier? Do we need heads >> up of a week. >> If merging must wait, should we consider creating a merge branch and >> committing the patch >> Jitendra has. This makes other merges more manageable. > > We already have a merge branch - the patch is easy to generate since > we have been merging in the trunk->HA direction daily since its > inception. > > I'm all for merging to trunk earlier if everyone's cool with it, but > we do need to start a vote. Shall I call one? > > -Todd > >> On Wed, Feb 22, 2012 at 6:24 PM, Aaron T. Myers <a...@cloudera.com> wrote: >> >>> Hello HDFS devs, >>> >>> Work has largely stabilized on the HA-branch in the last few weeks. At this >>> point the HA NN project is nearly feature-complete for manual failover. >>> We've been running the full test suite nightly, and all automated tests >>> have been passing, except for one known test failure which should be fixed >>> shortly. >>> >>> I'd like to begin the process of merging this branch back to HDFS trunk. >>> There are still several outstanding sub-JIRAs under the HDFS-1623 and >>> HADOOP-7454 umbrella JIRAs, but most of these are either nice-to-haves or >>> relate to supporting automatic failover. Once the branch is merged to >>> trunk, work on these JIRAs can continue there. >>> >>> I've identified the following JIRAs which I think should be the only >>> remaining blockers for merging to trunk: >>> >>> HDFS-2904 - Client support for getting delegation tokens in an HA cluster >>> HDFS-2920 - Fix remaining TODOs in the code from HA. Mostly little cleanup >>> stuff. >>> HDFS-2958 - Sweep for remaining proxy construction which doesn't go through >>> failover path >>> HDFS-2979 - Balancer should use logical URI for creating failover proxy >>> (will fix the only current test failure) >>> >>> All of these JIRAs should be fixed in the next few days. >>> >>> I propose that, unless more blocker issues are discovered in the interim, >>> we merge this branch to trunk one week from today, i.e. Wednesday, February >>> 29th. During this time we will also execute the test plans described in the >>> test documents attached to HDFS-1623 to try to identify any regressions or >>> performance issues in the branch. If you plan to review the code changes or >>> the test plan, I ask that you please do so as soon as possible. >>> >>> Feedback is certainly welcome on this plan. >>> >>> Thanks a lot, >>> Aaron >>> >>> -- >>> Aaron T. Myers >>> Software Engineer, Cloudera >>> > > > > -- > Todd Lipcon > Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera
