Moving state into zookeeper is the wrong direction. Absolutely not. No, no, no. We can discuss it but it is very unlikely I would even not veto such a change. It doesn’t seem like a problem because you have other MUCH better and more viable options.
> On Aug 13, 2021, at 7:50 AM, Stack <[email protected]> wrote: > > Rough minutes from our meeting of Wed Aug 11 17:04:44 PDT 2021 > > Attendees: Francis, Duo, Francis. > > We moved to discuss the proposed agenda item, how to implement the > "ROOT" Region. > > Duo had a new proposal that we host meta Regions in zookeeper (either > all in a single znode or a znode per location). > > We talked about it. Mainly, it was noted that zookeeper doesn't scale, > that a metatable in zk would not be able to satisfy > "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also > in violation of the old invariant that we keep no permanent state in > zookeeper [2]. > > Much back and forth on the fact that we violate the invariant currently; i.e. > there is info in zk now that can't be scrubbed; that we need to fix this > before > we can go forward -- that the fact that meta location is currently persistent > means we could make split meta Region locations also zk persistent. Counter > arguments note that the original invariant script allowed that we had > violations, > that we have worked to undo, that addressing invariant violations was > off-topic, > as well as variations on two wrongs do not make a right, etc. > > We tried to reset to talk of previously suggested options of ROOT as a > new table to host hbase:meta Regions or just use first-Region in > hbase:meta as special case Region. A new ROOT table was generally > dismissed as an impediment when small installs in no need of split meta. > > The second approach was called 'ugly' -- I think 'ugly' was the term > used -- and too much if/else and possibly in the way of future evolutions. > Counter suggestions thought any uglyness could be hidden from client and > isolated and that this approach seemed to meet requirements. > Counter-arguments > to the counter-argument where this approach can not hide everything from > client, > as opposed to separate ROOT table or master local region... see [1] for > detail. > > We ran out of time. Will try again next week (meantime on-going > developments on design doc [1] and in master). > > (Would be good if we could get others on the call, other PMCers > with in an interest in split meta. Could be of help when we hit > blockage) > Thanks, > S > > 1. > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75 > 2. http://hbase.apache.org/book.html#design.invariants.zk.data > > >> On Tue, Aug 10, 2021 at 6:11 PM Stack <[email protected]> wrote: >> >> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets >> continue our Split Meta Design Reset discussion. >> >> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada) >> >> Join Zoom Meeting >> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09 >> >> Meeting ID: 739 0014 2989 >> Passcode: hbase >> >> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before >> the meeting. >> >> Topics for discussion: >> >> * Reports on progress since last meeting. >> * How will we actually implement the ROOT table (as a distinct ROOT table, >> as the first region of hbase:meta, etc.) >> >> 1. >> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle >> >> Thanks, >> S >> >> >>> On Tue, Aug 3, 2021 at 4:13 PM Stack <[email protected]> wrote: >>> >>> Any time Yu Li! >>> >>> No meeting tomorrow... Lets meet next week, the 10th. >>> >>> Thanks, >>> S >>> >>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <[email protected]> wrote: >>> >>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the >>>> progress and latest status of the work! >>>> >>>> Best Regards, >>>> Yu >>>> >>>> >>>> On Wed, 28 Jul 2021 at 12:13, Stack <[email protected]> wrote: >>>> >>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of >>>>> August. >>>>> Thanks, >>>>> S >>>>> >>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <[email protected]> wrote: >>>>> >>>>>> Notes from yesterday's meeting (attendees, please amend if I >>>> misrepresent >>>>>> or if you have anything extra to add!) >>>>>> >>>>>> Split Meta Design Reset Status >>>>>> >>>>>> Wed Jul 21 21:24:38 PDT 2021 >>>>>> >>>>>> Attendees: Bharath, Stack, Duo, and Francis >>>>>> >>>>>> We went over the new updates to the Brainstorming [1] section under >>>>>> >>>>>> Design in the Super Split Meta Design doc [2]. >>>>>> >>>>>> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; >>>>> hide >>>>>> >>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might be >>>>>> >>>>>> implemented behind the new API in ConnectionRegistry. On option 1., >>>>>> >>>>>> replicating master-local Region to RegionServers, options considered >>>>>> >>>>>> included >>>>>> >>>>>> * Listener on master-local Region WAL to replicate. >>>>>> >>>>>> * Perhaps Read-Replica but master-local is not an actual Region >>>>>> >>>>>> * Needs to be incremental edits because ROOT could get too big to >>>> ship >>>>>> >>>>>> in a lump; need to visit how... >>>>>> >>>>>> * Possibly in-memory-only Regions on RS replicated from master-local >>>>>> >>>>>> Region via WAL tailing <= [email protected] >>>>>> >>>>>> * Which RegionServers? Those hosting ROOT replicas? >>>>>> >>>>>> * How to bootstrap? Failure scenarios. >>>>>> >>>>>> * This would be a new replication system alongside current; could >>>>>> >>>>>> evolve to replace/improve old? >>>>>> >>>>>> Duo offered to look into means of replicating the master-local Region >>>>>> >>>>>> out to RegionServers. >>>>>> >>>>>> Next up was discussion constrasting ROOT as a standalone table vs >>>>>> >>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e. >>>>>> >>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came up >>>>>> >>>>>> was whether a need to specify one replica count for a ROOT table vs >>>>>> >>>>>> another for hbase:meta. If so, then it would be argument for ROOT as >>>>>> >>>>>> standalone table (Others of us argued it not a concern of >>>> consequence). >>>>>> >>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, how >>>>>> >>>>>> to stop clients reading hbase:meta table if not Master or fronted by >>>>>> >>>>>> a ConnectionRegistry request? (Should be able to switch on client >>>>>> >>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was >>>>>> >>>>>> NOT returning the first Region to the client post-meta split when >>>>>> >>>>>> accessing via the simple API. Some concern this would confuse old >>>>>> >>>>>> Clients (Francis was going to take a look). >>>>>> >>>>>> Moved to discussion how we'd move ConnectionRegistry from >>>>>> >>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a >>>>>> >>>>>> system came up? Where do clients go? How do they know which >>>>>> >>>>>> RegionServers (special regionserver group? Every RS fields >>>>>> >>>>>> ConnectionRegistry requests but only designated core serve the ROOT >>>>>> >>>>>> lookup APIs?). This was a TODO. >>>>>> >>>>>> This led naturally into 4.1.5 System RS group for client meta >>>> services >>>>>> >>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to look >>>>>> >>>>>> into feedback. >>>>>> >>>>>> On the end of the discussion, group expressed support for adding >>>>>> >>>>>> simple API to the ConnectionRegistry to hide ROOT implementation >>>>>> >>>>>> detail from client. Support was expressed for moving >>>> ConnectionRegistry >>>>>> >>>>>> from Master to RegionServers. Intent is to move forward on design of >>>>>> >>>>>> these pieces: e.g. how client bootstraps. >>>>>> >>>>>> Support was expressed for getting at least the bones of a split meta >>>>>> >>>>>> into an hbase3 before the RCs. >>>>>> >>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a >>>>>> >>>>>> "ROOT' would be implemented -- was for our next meeting informed by >>>>>> >>>>>> research of the various approaches noted mostly above. It was >>>>>> >>>>>> also thought that the new ConnectionRegistry should not preclude >>>>>> >>>>>> making progress on the "ROOT" implementation. >>>>>> >>>>>> Will post notice of next meeting (next Weds or the one >>>>>> >>>>>> following). >>>>>> >>>>>> 1. >>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n >>>>>> >>>>>> 2. >>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq >>>>>> >>>>>> 3. >>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153 >>>>>> >>>>>> 4. >>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle >>>>>> >>>>>> 5. >>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b >>>>>> >>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <[email protected]> wrote: >>>>>> >>>>>>> Lets meet tomorrow. Please review the design doc >>>> "Design/Brainstorming" >>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a refresh >>>> of >>>>> the >>>>>>> requirements section while you are at it). >>>>>>> >>>>>>> Topic: Split Meta Design Reset Status >>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada) >>>>>>> >>>>>>> Join Zoom Meeting >>>>>>> >>>>> >>>> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09 >>>>>>> >>>>>>> Meeting ID: 773 1892 0525 >>>>>>> Passcode: hbase >>>>>>> >>>>>>> Thanks, >>>>>>> S >>>>>>> >>>>>>> 1. 1. >>>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n >>>>>>> >>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <[email protected]> wrote: >>>>>>> >>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a >>>> 'waiting >>>>>>>> room' -- sorry Duo) >>>>>>>> >>>>>>>> Split Meta Status Zoom Meeting >>>>>>>> Wed Jul 7, 2021 @ 5pm for ~90minutes >>>>>>>> Attendees: Duo, Francis, Stack, and Clay >>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed in >>>> [2] >>>>>>>> and added to the >>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as >>>> ROOT' >>>>>>>> approach) >>>>>>>> >>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta >>>> table >>>>>>>> special; as a "root" >>>>>>>> and other comments (see remarks in [2]). >>>>>>>> >>>>>>>> Some pushback. When meta is NOT split, all works as it did before >>>> (big >>>>>>>> on backward-compatible). >>>>>>>> >>>>>>>> Duo suggested just intro a ROOT table altogether rather than treat >>>> the >>>>>>>> first Region >>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the first >>>>> meta >>>>>>>> region; if no >>>>>>>> split, then old clients should just work even though now hbase >>>> cluster >>>>>>>> has a ROOT table. >>>>>>>> Discussion. If no split, what's to do, etc.? >>>>>>>> >>>>>>>> Duo expressed concern that if split-meta is not on always -- >>>> enabled >>>>>>>> optionally -- then >>>>>>>> the code path will not see exercise and split-meta will likely >>>> fail and >>>>>>>> go the way of >>>>>>>> prefix tree and distributed log replay -- a feature that failed, >>>>>>>> cluttered up the >>>>>>>> code base, and only later was removed. Discussion. Was allowed this >>>>>>>> could happen. >>>>>>>> Counter examples: RegionServer Groups. A few of the attendees >>>>>>>> volunteered they need >>>>>>>> split meta for their production so would try to see it through. >>>>>>>> >>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 >>>> as >>>>>>>> ROOT' if we don't >>>>>>>> split; not much changes. >>>>>>>> >>>>>>>> Comment on the PoC that its all >>>>>>>> >>>>>>>> if 'first special meta region' do this >>>>>>>> else do something else... >>>>>>>> >>>>>>>> (all over the codebase) but it was suggested that this will be the >>>> case >>>>>>>> if ROOT table >>>>>>>> added also and argued any implementation will have this issue (if >>>> ROOT >>>>>>>> then....) and >>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the >>>>>>>> 'hbase:meta,,1 as ROOT' >>>>>>>> approach. >>>>>>>> >>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' and >>>>>>>> 'new-to-the-topic' >>>>>>>> roles). Some discussion around how indeed it is and why we want to >>>>> split >>>>>>>> meta at all. >>>>>>>> >>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted >>>>>>>> Registry).... >>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to >>>> be >>>>>>>> hosted on >>>>>>>> Master -- and Master will return to its background role -- soon. >>>>>>>> >>>>>>>> Clay brought up rollback after meta split. Discussion. Some >>>> agreement >>>>> it >>>>>>>> could be done for >>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root >>>> table >>>>> and >>>>>>>> client had >>>>>>>> started to use ROOT, it might be harder... >>>>>>>> >>>>>>>> Duo suggested that we not change the client at all; that client >>>> stays >>>>>>>> same however split >>>>>>>> meta is implemented (with addition of a root table or using >>>>>>>> hbase:meta,,1 as 'root'). >>>>>>>> This sounded attractive. We discussed how this could be done in a >>>>>>>> backward compatible way; >>>>>>>> add simple location lookup API to Registry...A write-up on how this >>>>>>>> might work will be posted >>>>>>>> in next day or so for others to review (Need to figure getting >>>> Registry >>>>>>>> off Master). >>>>>>>> >>>>>>>> Clay suggested, as an operator, how he thought the split meta >>>> should >>>>>>>> roll out. One of us >>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach. >>>>>>>> >>>>>>>> Duo had to go to work so we called it quits and said we'd meet >>>> again >>>>>>>> same time, next week. >>>>>>>> >>>>>>>> 1. >>>>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle >>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761 >>>>>>>> >>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) < >>>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Is it only me? I tried to join the meeting but it always tell me >>>> that >>>>> I >>>>>>>>> need to wait for the presenter to invite me to join... >>>>>>>>> >>>>>>>>> Stack <[email protected]> 于2021年7月8日周四 上午1:04写道： >>>>>>>>> >>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST >>>> (We >>>>>>>>> talked >>>>>>>>>> of doing it earlier but in the end lets just keep the original >>>> 5pm >>>>>>>>> time). >>>>>>>>>> Thanks, >>>>>>>>>> S >>>>>>>>>> >>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta >>>> project. >>>>>>>>>>> >>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761 >>>>> which >>>>>>>>>> would >>>>>>>>>>> be good to review before the meeting if you can. Lets discuss >>>> this >>>>>>>>> and >>>>>>>>>> any >>>>>>>>>>> other suggestions for moving the project forward. >>>>>>>>>>> >>>>>>>>>>> Meeting details below. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> S >>>>>>>>>>> >>>>>>>>>>> Topic Split Meta Design Reset Status >>>>>>>>>>> Description Review one-pager design attached to >>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761 >>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada) >>>>>>>>>>> >>>>>>>>>>> Meeting ID 736 3907 8852 >>>>>>>>>>> Security >>>>>>>>>>> Passcode *hbase* Hide >>>>>>>>>>> Waiting Room >>>>>>>>>>> Invite Link >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> >>>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 >>>>>>>>>> Copy >>>>>>>>>>> Invitation >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <[email protected]> >>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Now the requirements are in [1], we're going to move to the >>>> next >>>>>>>>> stage >>>>>>>>>> -- >>>>>>>>>>>> actual design for split-meta -- and have set up a chat for >>>> this >>>>>>>>> thursday >>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the >>>> ball >>>>>>>>>> rolling. >>>>>>>>>>>> Please come if interested. Zoom details are below. >>>>>>>>>>>> >>>>>>>>>>>> Yours, >>>>>>>>>>>> S >>>>>>>>>>>> 1. >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> >>>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Topic: hbase split-meta design warmup chat >>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada) >>>>>>>>>>>> >>>>>>>>>>>> Join Zoom Meeting >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>> >>>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09 >>>>>>>>>>>> >>>>>>>>>>>> Meeting ID: 759 8800 3798 >>>>>>>>>>>> Passcode: hbase >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <[email protected]> >>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the >>>>> split >>>>>>>>> meta >>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the >>>>> requirements. >>>>>>>>> Are >>>>>>>>>> there >>>>>>>>>>>>> any others to consider? >>>>>>>>>>>>> >>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next >>>> phase >>>>> -- >>>>>>>>>> design. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> S >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>> >>>> >>>
