On Jan 30, 2012, at 11:39 AM, Jonathan Hsieh wrote: > Nicolas, > > For point 2 and 3, there definitely is a group of us that have been > informally discussing long term wire and format compatibility in > conversations. There are some documents that should be coming out soon to > the lists from these chats. The initial goal of this some of is to define a > vocabulary, to try to scope and phase-in changes necessary to make this > happen (possibly parts in 0.94, 0.96, 0.98...), and to define possible > goals and non-goals. > > My hope is that a thought-out first cut will come from Jimmy and I in the > next week or so so that we start having discussion to reach a consensus and > then start implementing. > > Some folks from another group (I'll let them announce themselves) should be > producing a doc focused more on the rpc portion of this soon as well. >
Thanks for the hint, Jon (*smile*) BTW I have opened - https://issues.apache.org/jira/browse/HBASE-5305 / HBASE-5306 to keep track of the issues.. We will upload some docs/discussion-points there soon. > Jon. > > On Mon, Jan 30, 2012 at 11:21 AM, Nicolas Spiegelberg > <[email protected]>wrote: > >> Currently, we at Facebook are developing mostly on 89 but also doing some >> significant exploratory work on trunk. I think that most of our >> development will continue to be done on 89 in the near future. We plan to >> release some lower-risk projects on 94. However, we won't even entertain >> a wide-scale upgrade strategy until: >> >> 1) The trunk has more master work. Specifically 89-master related >> features that we implemented for our grid architecture. It sounds like >> most of those features are rising in prominence in the Open Source >> community as well (Ted from eBay & Todd from Cloudera, in particular). >> Hopefully, we can collaborate on porting/implementation. >> 2) Client/Server compatibility. Even if we upgraded the servers to >> 94/96/whatever, we still will have a lot of clients running 89 & we can't >> simply stop/start all of them at the same time. We need 89/trunk RPC >> compat for client/server. >> 3) Durable Server Upgrade. It does bother me that people are very quick >> to consider removing both RPC & persistent data compatibility from 90 to >> trunk (a branch that we're still actively releasing minor upgrades for). >> We will need the ability to mutate the HDFS & ZK persistent data from the >> 89 format to trunk. Specifically, work like HFileV1 reader is critical >> for analysis if the upgrade script fails and we need to debug. >> >> Instead of discussing what classes we can throw away, can we instead think >> about a strategy for long-term, cross-version compatibility that minimally >> hurts trunk development? Is HFileV1's presence severely hindering another >> feature? >> >> On 1/27/12 6:11 PM, "Jonathan Hsieh" <[email protected]> wrote: >> >>> Lars, Matt, >>> >>> I like Matt's suggestion -- as long as it is not a burden let's keep it >>> in. >>> If it becomes one, let's do what is best for the project. >>> >>> If Cloudera needs to do the upgrade path, we'll provide one. If it >>> doesn't >>> go to the apache repo, we'll most likely open source it on github or >>> something, or "keep it on the jira" like some of the repair ruby scripts. >>> >>> Question -- to the Trend/SalesForce/FB folks can you talk about your plans >>> for upgrades? Are you guys moving from 0.89/0.90ish to 0.92 already? or >>> are you guys going to have to do the direct upgrade to 0.94 from 0.90 land >>> as well? >>> >>> Jon. >>> >>> >>> On Fri, Jan 27, 2012 at 12:06 PM, Matt Corgan <[email protected]> >> wrote: >>> >>>> The reason I brought it up is now that Mikhail committed the data block >>>> encoding I was going to take a stab at adding the prefix trie encoding I >>>> was working on this past summer. My plan is to first make a minimally >>>> invasive patch to prove correctness. But, after that there will >>>> probably >>>> be some big performance gains to be had from reworking some of the >>>> things >>>> like KeyValueScanner which I would not have the courage to get working >>>> with >>>> v1. >>>> >>>> So, that was why I asked, but all of that is still more hypothetical >>>> than >>>> real, and I don't even know if the first part will be done before >>>> branching >>>> .94 at the end of February. Makes sense to me to not delete v1 until >>>> there's a good reason to, which it doesn't look like we have yet. If I >>>> get >>>> to the point where v1 is halting progress then we can reevaluate based >>>> on >>>> more specific issues. Maybe none of the prefix trie will even make >>>> .94... >>>> >>>> ..sent from my phone >>>> On Jan 27, 2012 1:55 PM, "lars hofhansl" <[email protected]> wrote: >>>> >>>>> Hey Jon, >>>>> >>>>> understood. Makes 0.94 hard, though. If we decide now to have a 0.90 to >>>>> 0.94 upgrade path and then timing does not work out and nobody signs >>>>> up for >>>>> the testing because it 0.92 is more convenient we'd have gone through >>>>> this >>>>> for nothing. >>>>> >>>>> So... Thinking about this more I am -1 on supporting an official >>>>> upgrade >>>>> path other then from one release to the next. >>>>> That said, we do not have to break things intentionally. >>>>> >>>>> >>>>> I'm fine pushing HBASE-2600 and HFile v1 removal out of 0.94... As long >>>>> as we won't havethe same argument for 0.96 :) >>>>> >>>>> And I am not aware of any file compatibility issues. >>>>> >>>>> >>>>> We can also leave the 0.92 migration code in, but not officially >>>>> support >>>>> it as 0.90 to 0.94 path. >>>>> Then CDH4 can make sure (if needed) that it is all working. >>>>> >>>>> >>>>> Does that work for you guys? >>>>> >>>>> -- Lars >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> From: Jonathan Hsieh <[email protected]> >>>>> To: [email protected]; lars hofhansl <[email protected]> >>>>> Sent: Friday, January 27, 2012 10:10 AM >>>>> Subject: Re: hbase 0.94.0 >>>>> >>>>> >>>>> Lars, >>>>> >>>>> The upcoming CDH4 Beta HBase will be based on the latest hotness, >>>>> 0.92.0. >>>>> A CDH4 GA HBase will have to have an upgrade path from CDH3 HBase. If >>>>> that >>>>> HBase is 0.92 based, we'll test that, and if timing works out and we >>>>> decide >>>>> 0.94, we'll have to have a path (0.90->0.94) for than and will test >>>>> that. >>>>> >>>>> HBASE-2600 is a big change of encoding of meta, while my understanding >>>>> is >>>>> that 0.90->0,92 is a graceful HFile format conversion. Are there are >>>>> things currently in trunk that further break compatiblity of the file >>>>> format? (what jiras?) >>>>> >>>>> Jon. >>>>> >>>>> >>>>> On Fri, Jan 27, 2012 at 9:16 AM, lars hofhansl <[email protected]> >>>>> wrote: >>>>> >>>>> Not removing code for upgrade is fine. >>>>>> >>>>>> Todd, is Cloudera signing up for testing this path (0.90 to 0.94)? >>>>>> >>>>>> >>>>>> Stack, what's your feeling w.r.t. to HBASE-2600, will keeping the 0.90 >>>>> -> 0.92 migration path >>>>>> make the migration code for HBASE-2600 (much) more complicated in >>>>> 0.94? >>>>>> >>>>>> >>>>>> -- Lars >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> From: Stack <[email protected]> >>>>>> To: [email protected] >>>>>> Cc: >>>>>> Sent: Friday, January 27, 2012 9:02 AM >>>>>> Subject: Re: hbase 0.94.0 >>>>>> >>>>>> >>>>>> On Fri, Jan 27, 2012 at 8:42 AM, Stack <[email protected]> wrote: >>>>>>> In this particular case, there was no explicit migration step needed >>>>>>> going 0.90 to 0.92. Upgrading from 0.90 to 0.94 should just be >>>>>>> running the 0.92 to 0.94 migration script (if there is one). >>>>>>> >>>>>> >>>>>> Thinking more, the above only really holds if we keep the .META. >>>>>> migration code that runs in 0.92 on startup on into 0.94 (I have a >>>>>> patch here where I'm removing it... I should put this bit of code >>>>>> back). >>>>>> >>>>>> I see Todd that you vote against removing hfile v1 in 0.94. To make >>>>>> going from CDH3 to CDH4, we should not purge any migrating code or old >>>>>> version support. >>>>>> >>>>>> I'd be fine w/ that. We'll need to work hard to ensure it taking it >>>>>> on as a principal for 0.94. Ok w/ you "Iron Hand" Lars? >>>>>> >>>>>> St.Ack >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> // Jonathan Hsieh (shay) >>>>> // Software Engineer, Cloudera >>>>> >>>>> // [email protected] >>>>> >>>> >>> >>> >>> -- >>> // Jonathan Hsieh (shay) >>> // Software Engineer, Cloudera >>> // [email protected] >> >> > > > -- > // Jonathan Hsieh (shay) > // Software Engineer, Cloudera > // [email protected]
