Notes from yesterday's meeting (attendees, please amend if I misrepresent or if you have anything extra to add!)
Split Meta Design Reset Status Wed Jul 21 21:24:38 PDT 2021 Attendees: Bharath, Stack, Duo, and Francis We went over the new updates to the Brainstorming [1] section under Design in the Super Split Meta Design doc [2]. First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; hide ROOT from Client [3]. In particular, filling out how "ROOT" might be implemented behind the new API in ConnectionRegistry. On option 1., replicating master-local Region to RegionServers, options considered included * Listener on master-local Region WAL to replicate. * Perhaps Read-Replica but master-local is not an actual Region * Needs to be incremental edits because ROOT could get too big to ship in a lump; need to visit how... * Possibly in-memory-only Regions on RS replicated from master-local Region via WAL tailing <= [email protected] * Which RegionServers? Those hosting ROOT replicas? * How to bootstrap? Failure scenarios. * This would be a new replication system alongside current; could evolve to replace/improve old? Duo offered to look into means of replicating the master-local Region out to RegionServers. Next up was discussion constrasting ROOT as a standalone table vs First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e. options 2 and 3 for how we'd implement a ROOT. One item that came up was whether a need to specify one replica count for a ROOT table vs another for hbase:meta. If so, then it would be argument for ROOT as standalone table (Others of us argued it not a concern of consequence). If ROOT access is behind a new simple API in ConnectionRegistry, how to stop clients reading hbase:meta table if not Master or fronted by a ConnectionRegistry request? (Should be able to switch on client identity/source). One suggestion for First-meta-Region-as-ROOT was NOT returning the first Region to the client post-meta split when accessing via the simple API. Some concern this would confuse old Clients (Francis was going to take a look). Moved to discussion how we'd move ConnectionRegistry from hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a system came up? Where do clients go? How do they know which RegionServers (special regionserver group? Every RS fields ConnectionRegistry requests but only designated core serve the ROOT lookup APIs?). This was a TODO. This led naturally into 4.1.5 System RS group for client meta services [5], a new addition under Brainstorming. Discussion. Bharath to look into feedback. On the end of the discussion, group expressed support for adding simple API to the ConnectionRegistry to hide ROOT implementation detail from client. Support was expressed for moving ConnectionRegistry from Master to RegionServers. Intent is to move forward on design of these pieces: e.g. how client bootstraps. Support was expressed for getting at least the bones of a split meta into an hbase3 before the RCs. Where we'd actually store hbase:meta Region locations -- i.e. how a "ROOT' would be implemented -- was for our next meeting informed by research of the various approaches noted mostly above. It was also thought that the new ConnectionRegistry should not preclude making progress on the "ROOT" implementation. Will post notice of next meeting (next Weds or the one following). 1. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n 2. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq 3. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153 4. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle 5. https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b On Tue, Jul 20, 2021 at 11:00 AM Stack <[email protected]> wrote: > Lets meet tomorrow. Please review the design doc "Design/Brainstorming" > Section 4.1 [1] before the meeting if you can (No harm if a refresh of the > requirements section while you are at it). > > Topic: Split Meta Design Reset Status > Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada) > > Join Zoom Meeting > https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09 > > Meeting ID: 773 1892 0525 > Passcode: hbase > > Thanks, > S > > 1. 1. > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n > > On Thu, Jul 8, 2021 at 1:04 PM Stack <[email protected]> wrote: > >> Meeting notes (Meeting happend after I figured the zoom had a 'waiting >> room' -- sorry Duo) >> >> Split Meta Status Zoom Meeting >> Wed Jul 7, 2021 @ 5pm for ~90minutes >> Attendees: Duo, Francis, Stack, and Clay >> Agenda: Mainly talk about the one-pager design and PoC proposed in [2] >> and added to the >> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT' >> approach) >> >> Duo suggested no advantage treating the first meta of hbase:meta table >> special; as a "root" >> and other comments (see remarks in [2]). >> >> Some pushback. When meta is NOT split, all works as it did before (big >> on backward-compatible). >> >> Duo suggested just intro a ROOT table altogether rather than treat the >> first Region >> in the hbase:meta table as a 'root' and then mirror to zk the first meta >> region; if no >> split, then old clients should just work even though now hbase cluster >> has a ROOT table. >> Discussion. If no split, what's to do, etc.? >> >> Duo expressed concern that if split-meta is not on always -- enabled >> optionally -- then >> the code path will not see exercise and split-meta will likely fail and >> go the way of >> prefix tree and distributed log replay -- a feature that failed, >> cluttered up the >> code base, and only later was removed. Discussion. Was allowed this could >> happen. >> Counter examples: RegionServer Groups. A few of the attendees volunteered >> they need >> split meta for their production so would try to see it through. >> >> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as >> ROOT' if we don't >> split; not much changes. >> >> Comment on the PoC that its all >> >> if 'first special meta region' do this >> else do something else... >> >> (all over the codebase) but it was suggested that this will be the case >> if ROOT table >> added also and argued any implementation will have this issue (if ROOT >> then....) and >> THAT ugly 'root' comparator too whether a ROOT table or the >> 'hbase:meta,,1 as ROOT' >> approach. >> >> Clay asking if meta is a bottleneck (Clay played the 'operator' and >> 'new-to-the-topic' >> roles). Some discussion around how indeed it is and why we want to split >> meta at all. >> >> Clay mentioned how Master is inline for data now (Master-hosted >> Registry).... >> Discussion. Hopefully temporary state -- Registry doesn't need to be >> hosted on >> Master -- and Master will return to its background role -- soon. >> >> Clay brought up rollback after meta split. Discussion. Some agreement it >> could be done for >> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and >> client had >> started to use ROOT, it might be harder... >> >> Duo suggested that we not change the client at all; that client stays >> same however split >> meta is implemented (with addition of a root table or using hbase:meta,,1 >> as 'root'). >> This sounded attractive. We discussed how this could be done in a >> backward compatible way; >> add simple location lookup API to Registry...A write-up on how this might >> work will be posted >> in next day or so for others to review (Need to figure getting Registry >> off Master). >> >> Clay suggested, as an operator, how he thought the split meta should roll >> out. One of us >> claimed this described the 'hbase:meta,,1 as ROOT' approach. >> >> Duo had to go to work so we called it quits and said we'd meet again same >> time, next week. >> >> 1. >> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle >> 2. https://issues.apache.org/jira/browse/HBASE-25761 >> >> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) <[email protected]> >> wrote: >> >>> Is it only me? I tried to join the meeting but it always tell me that I >>> need to wait for the presenter to invite me to join... >>> >>> Stack <[email protected]> 于2021年7月8日周四 上午1:04写道： >>> >>> > Short notice but reminder that this meeting is today at 5pm PST (We >>> talked >>> > of doing it earlier but in the end lets just keep the original 5pm >>> time). >>> > Thanks, >>> > S >>> > >>> > On Tue, Jul 6, 2021 at 11:36 AM Stack <[email protected]> wrote: >>> > >>> > > Lets do a quick chat on current state of hbase split-meta project. >>> > > >>> > > Francis has posted a suggested one-pager design in HBASE-25761 which >>> > would >>> > > be good to review before the meeting if you can. Lets discuss this >>> and >>> > any >>> > > other suggestions for moving the project forward. >>> > > >>> > > Meeting details below. >>> > > >>> > > Thanks, >>> > > S >>> > > >>> > > Topic Split Meta Design Reset Status >>> > > Description Review one-pager design attached to >>> > > https://issues.apache.org/jira/browse/HBASE-25761 >>> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada) >>> > > >>> > > Meeting ID 736 3907 8852 >>> > > Security >>> > > Passcode *hbase* Hide >>> > > Waiting Room >>> > > Invite Link >>> > > >>> > >>> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 >>> > Copy >>> > > Invitation >>> > > >>> > > On Mon, Mar 22, 2021 at 4:28 PM Stack <[email protected]> wrote: >>> > > >>> > >> Now the requirements are in [1], we're going to move to the next >>> stage >>> > -- >>> > >> actual design for split-meta -- and have set up a chat for this >>> thursday >>> > >> afternoon (4PM California time/8AM Beijing time) to get the ball >>> > rolling. >>> > >> Please come if interested. Zoom details are below. >>> > >> >>> > >> Yours, >>> > >> S >>> > >> 1. >>> > >> >>> > >>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2 >>> > >> >>> > >> >>> > >> Topic: hbase split-meta design warmup chat >>> > >> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada) >>> > >> >>> > >> Join Zoom Meeting >>> > >> >>> > >>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09 >>> > >> >>> > >> Meeting ID: 759 8800 3798 >>> > >> Passcode: hbase >>> > >> >>> > >> >>> > >> On Tue, Jan 5, 2021 at 9:13 AM Stack <[email protected]> wrote: >>> > >> >>> > >>> FYI, a few of us have been working on the redo/reset of the split >>> meta >>> > >>> design (HBASE-25382). We (think we've) finished the requirements. >>> Are >>> > there >>> > >>> any others to consider? >>> > >>> >>> > >>> Feedback and contribs welcome. Otherwise, on to the next phase -- >>> > design. >>> > >>> >>> > >>> Thanks, >>> > >>> S >>> > >>> >>> > >> >>> > >>> >>
