This is not moving state to zookeeper, we already have this state in zookeeper... The location of meta region is only stored in zookeeper, and if it is cleaned, we will be in trouble...
And I filed HBASE-26193 to fix this problem. PTAL if you have interest. Thanks. Andrew Purtell <[email protected]> 于2021年8月13日周五 下午11:30写道： > > Moving state into zookeeper is the wrong direction. Absolutely not. No, no, > no. We can discuss it but it is very unlikely I would even not veto such a > change. It doesn’t seem like a problem because you have other MUCH better and > more viable options. > > > On Aug 13, 2021, at 7:50 AM, Stack <[email protected]> wrote: > > > > Rough minutes from our meeting of Wed Aug 11 17:04:44 PDT 2021 > > > > Attendees: Francis, Duo, Francis. > > > > We moved to discuss the proposed agenda item, how to implement the > > "ROOT" Region. > > > > Duo had a new proposal that we host meta Regions in zookeeper (either > > all in a single znode or a znode per location). > > > > We talked about it. Mainly, it was noted that zookeeper doesn't scale, > > that a metatable in zk would not be able to satisfy > > "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also > > in violation of the old invariant that we keep no permanent state in > > zookeeper [2]. > > > > Much back and forth on the fact that we violate the invariant currently; > > i.e. > > there is info in zk now that can't be scrubbed; that we need to fix this > > before > > we can go forward -- that the fact that meta location is currently > > persistent > > means we could make split meta Region locations also zk persistent. Counter > > arguments note that the original invariant script allowed that we had > > violations, > > that we have worked to undo, that addressing invariant violations was > > off-topic, > > as well as variations on two wrongs do not make a right, etc. > > > > We tried to reset to talk of previously suggested options of ROOT as a > > new table to host hbase:meta Regions or just use first-Region in > > hbase:meta as special case Region. A new ROOT table was generally > > dismissed as an impediment when small installs in no need of split meta. > > > > The second approach was called 'ugly' -- I think 'ugly' was the term > > used -- and too much if/else and possibly in the way of future evolutions. > > Counter suggestions thought any uglyness could be hidden from client and > > isolated and that this approach seemed to meet requirements. > > Counter-arguments > > to the counter-argument where this approach can not hide everything from > > client, > > as opposed to separate ROOT table or master local region... see [1] for > > detail. > > > > We ran out of time. Will try again next week (meantime on-going > > developments on design doc [1] and in master). > > > > (Would be good if we could get others on the call, other PMCers > > with in an interest in split meta. Could be of help when we hit > > blockage) > > Thanks, > > S > > > > 1. > > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75 > > 2. http://hbase.apache.org/book.html#design.invariants.zk.data > > > > > >> On Tue, Aug 10, 2021 at 6:11 PM Stack <[email protected]> wrote: > >> > >> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets > >> continue our Split Meta Design Reset discussion. > >> > >> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada) > >> > >> Join Zoom Meeting > >> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09 > >> > >> Meeting ID: 739 0014 2989 > >> Passcode: hbase > >> > >> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before > >> the meeting. > >> > >> Topics for discussion: > >> > >> * Reports on progress since last meeting. > >> * How will we actually implement the ROOT table (as a distinct ROOT table, > >> as the first region of hbase:meta, etc.) > >> > >> 1. > >> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle > >> > >> Thanks, > >> S > >> > >> > >>> On Tue, Aug 3, 2021 at 4:13 PM Stack <[email protected]> wrote: > >>> > >>> Any time Yu Li! > >>> > >>> No meeting tomorrow... Lets meet next week, the 10th. > >>> > >>> Thanks, > >>> S > >>> > >>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <[email protected]> wrote: > >>> > >>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the > >>>> progress and latest status of the work! > >>>> > >>>> Best Regards, > >>>> Yu > >>>> > >>>> > >>>> On Wed, 28 Jul 2021 at 12:13, Stack <[email protected]> wrote: > >>>> > >>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of > >>>>> August. > >>>>> Thanks, > >>>>> S > >>>>> > >>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <[email protected]> wrote: > >>>>> > >>>>>> Notes from yesterday's meeting (attendees, please amend if I > >>>> misrepresent > >>>>>> or if you have anything extra to add!) > >>>>>> > >>>>>> Split Meta Design Reset Status > >>>>>> > >>>>>> Wed Jul 21 21:24:38 PDT 2021 > >>>>>> > >>>>>> Attendees: Bharath, Stack, Duo, and Francis > >>>>>> > >>>>>> We went over the new updates to the Brainstorming [1] section under > >>>>>> > >>>>>> Design in the Super Split Meta Design doc [2]. > >>>>>> > >>>>>> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; > >>>>> hide > >>>>>> > >>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might be > >>>>>> > >>>>>> implemented behind the new API in ConnectionRegistry. On option 1., > >>>>>> > >>>>>> replicating master-local Region to RegionServers, options considered > >>>>>> > >>>>>> included > >>>>>> > >>>>>> * Listener on master-local Region WAL to replicate. > >>>>>> > >>>>>> * Perhaps Read-Replica but master-local is not an actual Region > >>>>>> > >>>>>> * Needs to be incremental edits because ROOT could get too big to > >>>> ship > >>>>>> > >>>>>> in a lump; need to visit how... > >>>>>> > >>>>>> * Possibly in-memory-only Regions on RS replicated from master-local > >>>>>> > >>>>>> Region via WAL tailing <= [email protected] > >>>>>> > >>>>>> * Which RegionServers? Those hosting ROOT replicas? > >>>>>> > >>>>>> * How to bootstrap? Failure scenarios. > >>>>>> > >>>>>> * This would be a new replication system alongside current; could > >>>>>> > >>>>>> evolve to replace/improve old? > >>>>>> > >>>>>> Duo offered to look into means of replicating the master-local Region > >>>>>> > >>>>>> out to RegionServers. > >>>>>> > >>>>>> Next up was discussion constrasting ROOT as a standalone table vs > >>>>>> > >>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e. > >>>>>> > >>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came up > >>>>>> > >>>>>> was whether a need to specify one replica count for a ROOT table vs > >>>>>> > >>>>>> another for hbase:meta. If so, then it would be argument for ROOT as > >>>>>> > >>>>>> standalone table (Others of us argued it not a concern of > >>>> consequence). > >>>>>> > >>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, how > >>>>>> > >>>>>> to stop clients reading hbase:meta table if not Master or fronted by > >>>>>> > >>>>>> a ConnectionRegistry request? (Should be able to switch on client > >>>>>> > >>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was > >>>>>> > >>>>>> NOT returning the first Region to the client post-meta split when > >>>>>> > >>>>>> accessing via the simple API. Some concern this would confuse old > >>>>>> > >>>>>> Clients (Francis was going to take a look). > >>>>>> > >>>>>> Moved to discussion how we'd move ConnectionRegistry from > >>>>>> > >>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a > >>>>>> > >>>>>> system came up? Where do clients go? How do they know which > >>>>>> > >>>>>> RegionServers (special regionserver group? Every RS fields > >>>>>> > >>>>>> ConnectionRegistry requests but only designated core serve the ROOT > >>>>>> > >>>>>> lookup APIs?). This was a TODO. > >>>>>> > >>>>>> This led naturally into 4.1.5 System RS group for client meta > >>>> services > >>>>>> > >>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to look > >>>>>> > >>>>>> into feedback. > >>>>>> > >>>>>> On the end of the discussion, group expressed support for adding > >>>>>> > >>>>>> simple API to the ConnectionRegistry to hide ROOT implementation > >>>>>> > >>>>>> detail from client. Support was expressed for moving > >>>> ConnectionRegistry > >>>>>> > >>>>>> from Master to RegionServers. Intent is to move forward on design of > >>>>>> > >>>>>> these pieces: e.g. how client bootstraps. > >>>>>> > >>>>>> Support was expressed for getting at least the bones of a split meta > >>>>>> > >>>>>> into an hbase3 before the RCs. > >>>>>> > >>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a > >>>>>> > >>>>>> "ROOT' would be implemented -- was for our next meeting informed by > >>>>>> > >>>>>> research of the various approaches noted mostly above. It was > >>>>>> > >>>>>> also thought that the new ConnectionRegistry should not preclude > >>>>>> > >>>>>> making progress on the "ROOT" implementation. > >>>>>> > >>>>>> Will post notice of next meeting (next Weds or the one > >>>>>> > >>>>>> following). > >>>>>> > >>>>>> 1. > >>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n > >>>>>> > >>>>>> 2. > >>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq > >>>>>> > >>>>>> 3. > >>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153 > >>>>>> > >>>>>> 4. > >>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle > >>>>>> > >>>>>> 5. > >>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b > >>>>>> > >>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <[email protected]> wrote: > >>>>>> > >>>>>>> Lets meet tomorrow. Please review the design doc > >>>> "Design/Brainstorming" > >>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a refresh > >>>> of > >>>>> the > >>>>>>> requirements section while you are at it). > >>>>>>> > >>>>>>> Topic: Split Meta Design Reset Status > >>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada) > >>>>>>> > >>>>>>> Join Zoom Meeting > >>>>>>> > >>>>> > >>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09 > >>>>>>> > >>>>>>> Meeting ID: 773 1892 0525 > >>>>>>> Passcode: hbase > >>>>>>> > >>>>>>> Thanks, > >>>>>>> S > >>>>>>> > >>>>>>> 1. 1. > >>>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n > >>>>>>> > >>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <[email protected]> wrote: > >>>>>>> > >>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a > >>>> 'waiting > >>>>>>>> room' -- sorry Duo) > >>>>>>>> > >>>>>>>> Split Meta Status Zoom Meeting > >>>>>>>> Wed Jul 7, 2021 @ 5pm for ~90minutes > >>>>>>>> Attendees: Duo, Francis, Stack, and Clay > >>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed in > >>>> [2] > >>>>>>>> and added to the > >>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as > >>>> ROOT' > >>>>>>>> approach) > >>>>>>>> > >>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta > >>>> table > >>>>>>>> special; as a "root" > >>>>>>>> and other comments (see remarks in [2]). > >>>>>>>> > >>>>>>>> Some pushback. When meta is NOT split, all works as it did before > >>>> (big > >>>>>>>> on backward-compatible). > >>>>>>>> > >>>>>>>> Duo suggested just intro a ROOT table altogether rather than treat > >>>> the > >>>>>>>> first Region > >>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the first > >>>>> meta > >>>>>>>> region; if no > >>>>>>>> split, then old clients should just work even though now hbase > >>>> cluster > >>>>>>>> has a ROOT table. > >>>>>>>> Discussion. If no split, what's to do, etc.? > >>>>>>>> > >>>>>>>> Duo expressed concern that if split-meta is not on always -- > >>>> enabled > >>>>>>>> optionally -- then > >>>>>>>> the code path will not see exercise and split-meta will likely > >>>> fail and > >>>>>>>> go the way of > >>>>>>>> prefix tree and distributed log replay -- a feature that failed, > >>>>>>>> cluttered up the > >>>>>>>> code base, and only later was removed. Discussion. Was allowed this > >>>>>>>> could happen. > >>>>>>>> Counter examples: RegionServer Groups. A few of the attendees > >>>>>>>> volunteered they need > >>>>>>>> split meta for their production so would try to see it through. > >>>>>>>> > >>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 > >>>> as > >>>>>>>> ROOT' if we don't > >>>>>>>> split; not much changes. > >>>>>>>> > >>>>>>>> Comment on the PoC that its all > >>>>>>>> > >>>>>>>> if 'first special meta region' do this > >>>>>>>> else do something else... > >>>>>>>> > >>>>>>>> (all over the codebase) but it was suggested that this will be the > >>>> case > >>>>>>>> if ROOT table > >>>>>>>> added also and argued any implementation will have this issue (if > >>>> ROOT > >>>>>>>> then....) and > >>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the > >>>>>>>> 'hbase:meta,,1 as ROOT' > >>>>>>>> approach. > >>>>>>>> > >>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and > >>>>>>>> 'new-to-the-topic' > >>>>>>>> roles). Some discussion around how indeed it is and why we want to > >>>>> split > >>>>>>>> meta at all. > >>>>>>>> > >>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted > >>>>>>>> Registry).... > >>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to > >>>> be > >>>>>>>> hosted on > >>>>>>>> Master -- and Master will return to its background role -- soon. > >>>>>>>> > >>>>>>>> Clay brought up rollback after meta split. Discussion. Some > >>>> agreement > >>>>> it > >>>>>>>> could be done for > >>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root > >>>> table > >>>>> and > >>>>>>>> client had > >>>>>>>> started to use ROOT, it might be harder... > >>>>>>>> > >>>>>>>> Duo suggested that we not change the client at all; that client > >>>> stays > >>>>>>>> same however split > >>>>>>>> meta is implemented (with addition of a root table or using > >>>>>>>> hbase:meta,,1 as 'root'). > >>>>>>>> This sounded attractive. We discussed how this could be done in a > >>>>>>>> backward compatible way; > >>>>>>>> add simple location lookup API to Registry...A write-up on how this > >>>>>>>> might work will be posted > >>>>>>>> in next day or so for others to review (Need to figure getting > >>>> Registry > >>>>>>>> off Master). > >>>>>>>> > >>>>>>>> Clay suggested, as an operator, how he thought the split meta > >>>> should > >>>>>>>> roll out. One of us > >>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach. > >>>>>>>> > >>>>>>>> Duo had to go to work so we called it quits and said we'd meet > >>>> again > >>>>>>>> same time, next week. > >>>>>>>> > >>>>>>>> 1. > >>>>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle > >>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761 > >>>>>>>> > >>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) < > >>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Is it only me? I tried to join the meeting but it always tell me > >>>> that > >>>>> I > >>>>>>>>> need to wait for the presenter to invite me to join... > >>>>>>>>> > >>>>>>>>> Stack <[email protected]> 于2021年7月8日周四 上午1:04写道： > >>>>>>>>> > >>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST > >>>> (We > >>>>>>>>> talked > >>>>>>>>>> of doing it earlier but in the end lets just keep the original > >>>> 5pm > >>>>>>>>> time). > >>>>>>>>>> Thanks, > >>>>>>>>>> S > >>>>>>>>>> > >>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <[email protected]> wrote: > >>>>>>>>>> > >>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta > >>>> project. > >>>>>>>>>>> > >>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761 > >>>>> which > >>>>>>>>>> would > >>>>>>>>>>> be good to review before the meeting if you can. Lets discuss > >>>> this > >>>>>>>>> and > >>>>>>>>>> any > >>>>>>>>>>> other suggestions for moving the project forward. > >>>>>>>>>>> > >>>>>>>>>>> Meeting details below. > >>>>>>>>>>> > >>>>>>>>>>> Thanks, > >>>>>>>>>>> S > >>>>>>>>>>> > >>>>>>>>>>> Topic Split Meta Design Reset Status > >>>>>>>>>>> Description Review one-pager design attached to > >>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761 > >>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada) > >>>>>>>>>>> > >>>>>>>>>>> Meeting ID 736 3907 8852 > >>>>>>>>>>> Security > >>>>>>>>>>> Passcode *hbase* Hide > >>>>>>>>>>> Waiting Room > >>>>>>>>>>> Invite Link > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> > >>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 > >>>>>>>>>> Copy > >>>>>>>>>>> Invitation > >>>>>>>>>>> > >>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <[email protected]> > >>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Now the requirements are in [1], we're going to move to the > >>>> next > >>>>>>>>> stage > >>>>>>>>>> -- > >>>>>>>>>>>> actual design for split-meta -- and have set up a chat for > >>>> this > >>>>>>>>> thursday > >>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the > >>>> ball > >>>>>>>>>> rolling. > >>>>>>>>>>>> Please come if interested. Zoom details are below. > >>>>>>>>>>>> > >>>>>>>>>>>> Yours, > >>>>>>>>>>>> S > >>>>>>>>>>>> 1. > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> > >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Topic: hbase split-meta design warmup chat > >>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada) > >>>>>>>>>>>> > >>>>>>>>>>>> Join Zoom Meeting > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>> > >>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09 > >>>>>>>>>>>> > >>>>>>>>>>>> Meeting ID: 759 8800 3798 > >>>>>>>>>>>> Passcode: hbase > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <[email protected]> > >>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the > >>>>> split > >>>>>>>>> meta > >>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the > >>>>> requirements. > >>>>>>>>> Are > >>>>>>>>>> there > >>>>>>>>>>>>> any others to consider? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next > >>>> phase > >>>>> -- > >>>>>>>>>> design. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>> S > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >>>
