On Fri, Aug 13, 2021 at 8:39 AM Andrew Purtell <[email protected]> wrote:
> Also let me say that your entreaty for more to join your calls is fine and > I hope that happens, but this is Apache, and discussions should take place > in suitably asynchronous fashion so all can participate - even those who > have competing responsibilities and commitments and cannot attend the > calls. This is not criticism of the calls! I’m so glad to see you lot are > continuing to communicate. But we need to allow and support asynchronous > communication and comment either by mailing list or on a document. Perhaps > a doc that lists notes from each of your session… Or otherwise randos like > me will be posting responses to your emailed notes here. > > Commenting on the notes here is great. Thanks. Otherwise, there is the (public) design doc which is what the calls are trying to move forward; all are free to comment/trash and improve it. (We're thinking of moving the doc off gdoc to dev-support/design-docs/split-meta to make it easier to see changes). Thanks Andrew, S > > On Aug 13, 2021, at 8:30 AM, Andrew Purtell <[email protected]> > wrote: > > > > Moving state into zookeeper is the wrong direction. Absolutely not. No, > no, no. We can discuss it but it is very unlikely I would even not veto > such a change. It doesn’t seem like a problem because you have other MUCH > better and more viable options. > > > >> On Aug 13, 2021, at 7:50 AM, Stack <[email protected]> wrote: > >> > >> Rough minutes from our meeting of Wed Aug 11 17:04:44 PDT 2021 > >> > >> Attendees: Francis, Duo, Francis. > >> > >> We moved to discuss the proposed agenda item, how to implement the > >> "ROOT" Region. > >> > >> Duo had a new proposal that we host meta Regions in zookeeper (either > >> all in a single znode or a znode per location). > >> > >> We talked about it. Mainly, it was noted that zookeeper doesn't scale, > >> that a metatable in zk would not be able to satisfy > >> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also > >> in violation of the old invariant that we keep no permanent state in > >> zookeeper [2]. > >> > >> Much back and forth on the fact that we violate the invariant > currently; i.e. > >> there is info in zk now that can't be scrubbed; that we need to fix > this before > >> we can go forward -- that the fact that meta location is currently > persistent > >> means we could make split meta Region locations also zk persistent. > Counter > >> arguments note that the original invariant script allowed that we had > >> violations, > >> that we have worked to undo, that addressing invariant violations was > off-topic, > >> as well as variations on two wrongs do not make a right, etc. > >> > >> We tried to reset to talk of previously suggested options of ROOT as a > >> new table to host hbase:meta Regions or just use first-Region in > >> hbase:meta as special case Region. A new ROOT table was generally > >> dismissed as an impediment when small installs in no need of split meta. > >> > >> The second approach was called 'ugly' -- I think 'ugly' was the term > >> used -- and too much if/else and possibly in the way of future > evolutions. > >> Counter suggestions thought any uglyness could be hidden from client and > >> isolated and that this approach seemed to meet requirements. > Counter-arguments > >> to the counter-argument where this approach can not hide everything > from client, > >> as opposed to separate ROOT table or master local region... see [1] for > detail. > >> > >> We ran out of time. Will try again next week (meantime on-going > >> developments on design doc [1] and in master). > >> > >> (Would be good if we could get others on the call, other PMCers > >> with in an interest in split meta. Could be of help when we hit > >> blockage) > >> Thanks, > >> S > >> > >> 1. > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75 > >> 2. http://hbase.apache.org/book.html#design.invariants.zk.data > >> > >> > >>>> On Tue, Aug 10, 2021 at 6:11 PM Stack <[email protected]> wrote: > >>> > >>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets > >>> continue our Split Meta Design Reset discussion. > >>> > >>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada) > >>> > >>> Join Zoom Meeting > >>> > https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09 > >>> > >>> Meeting ID: 739 0014 2989 > >>> Passcode: hbase > >>> > >>> Please review the design doc "Design/Brainstorming" Section 4.1 [1] > before > >>> the meeting. > >>> > >>> Topics for discussion: > >>> > >>> * Reports on progress since last meeting. > >>> * How will we actually implement the ROOT table (as a distinct ROOT > table, > >>> as the first region of hbase:meta, etc.) > >>> > >>> 1. > >>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle > >>> > >>> Thanks, > >>> S > >>> > >>> > >>>> On Tue, Aug 3, 2021 at 4:13 PM Stack <[email protected]> wrote: > >>>> > >>>> Any time Yu Li! > >>>> > >>>> No meeting tomorrow... Lets meet next week, the 10th. > >>>> > >>>> Thanks, > >>>> S > >>>> > >>>>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li <[email protected]> wrote: > >>>> > >>>>> Thanks for the notes and efforts Stack, it's pretty helpful to know > the > >>>>> progress and latest status of the work! > >>>>> > >>>>> Best Regards, > >>>>> Yu > >>>>> > >>>>> > >>>>> On Wed, 28 Jul 2021 at 12:13, Stack <[email protected]> wrote: > >>>>> > >>>>>> Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of > >>>>>> August. > >>>>>> Thanks, > >>>>>> S > >>>>>> > >>>>>> On Thu, Jul 22, 2021 at 8:53 PM Stack <[email protected]> wrote: > >>>>>> > >>>>>>> Notes from yesterday's meeting (attendees, please amend if I > >>>>> misrepresent > >>>>>>> or if you have anything extra to add!) > >>>>>>> > >>>>>>> Split Meta Design Reset Status > >>>>>>> > >>>>>>> Wed Jul 21 21:24:38 PDT 2021 > >>>>>>> > >>>>>>> Attendees: Bharath, Stack, Duo, and Francis > >>>>>>> > >>>>>>> We went over the new updates to the Brainstorming [1] section under > >>>>>>> > >>>>>>> Design in the Super Split Meta Design doc [2]. > >>>>>>> > >>>>>>> First was the new addition, 4.1.2 Extend (& Move) > ConnectionRegistry; > >>>>>> hide > >>>>>>> > >>>>>>> ROOT from Client [3]. In particular, filling out how "ROOT" might > be > >>>>>>> > >>>>>>> implemented behind the new API in ConnectionRegistry. On option 1., > >>>>>>> > >>>>>>> replicating master-local Region to RegionServers, options > considered > >>>>>>> > >>>>>>> included > >>>>>>> > >>>>>>> * Listener on master-local Region WAL to replicate. > >>>>>>> > >>>>>>> * Perhaps Read-Replica but master-local is not an actual Region > >>>>>>> > >>>>>>> * Needs to be incremental edits because ROOT could get too big to > >>>>> ship > >>>>>>> > >>>>>>> in a lump; need to visit how... > >>>>>>> > >>>>>>> * Possibly in-memory-only Regions on RS replicated from > master-local > >>>>>>> > >>>>>>> Region via WAL tailing <= [email protected] > >>>>>>> > >>>>>>> * Which RegionServers? Those hosting ROOT replicas? > >>>>>>> > >>>>>>> * How to bootstrap? Failure scenarios. > >>>>>>> > >>>>>>> * This would be a new replication system alongside current; could > >>>>>>> > >>>>>>> evolve to replace/improve old? > >>>>>>> > >>>>>>> Duo offered to look into means of replicating the master-local > Region > >>>>>>> > >>>>>>> out to RegionServers. > >>>>>>> > >>>>>>> Next up was discussion constrasting ROOT as a standalone table vs > >>>>>>> > >>>>>>> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); > i.e. > >>>>>>> > >>>>>>> options 2 and 3 for how we'd implement a ROOT. One item that came > up > >>>>>>> > >>>>>>> was whether a need to specify one replica count for a ROOT table vs > >>>>>>> > >>>>>>> another for hbase:meta. If so, then it would be argument for ROOT > as > >>>>>>> > >>>>>>> standalone table (Others of us argued it not a concern of > >>>>> consequence). > >>>>>>> > >>>>>>> If ROOT access is behind a new simple API in ConnectionRegistry, > how > >>>>>>> > >>>>>>> to stop clients reading hbase:meta table if not Master or fronted > by > >>>>>>> > >>>>>>> a ConnectionRegistry request? (Should be able to switch on client > >>>>>>> > >>>>>>> identity/source). One suggestion for First-meta-Region-as-ROOT was > >>>>>>> > >>>>>>> NOT returning the first Region to the client post-meta split when > >>>>>>> > >>>>>>> accessing via the simple API. Some concern this would confuse old > >>>>>>> > >>>>>>> Clients (Francis was going to take a look). > >>>>>>> > >>>>>>> Moved to discussion how we'd move ConnectionRegistry from > >>>>>>> > >>>>>>> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such > a > >>>>>>> > >>>>>>> system came up? Where do clients go? How do they know which > >>>>>>> > >>>>>>> RegionServers (special regionserver group? Every RS fields > >>>>>>> > >>>>>>> ConnectionRegistry requests but only designated core serve the ROOT > >>>>>>> > >>>>>>> lookup APIs?). This was a TODO. > >>>>>>> > >>>>>>> This led naturally into 4.1.5 System RS group for client meta > >>>>> services > >>>>>>> > >>>>>>> [5], a new addition under Brainstorming. Discussion. Bharath to > look > >>>>>>> > >>>>>>> into feedback. > >>>>>>> > >>>>>>> On the end of the discussion, group expressed support for adding > >>>>>>> > >>>>>>> simple API to the ConnectionRegistry to hide ROOT implementation > >>>>>>> > >>>>>>> detail from client. Support was expressed for moving > >>>>> ConnectionRegistry > >>>>>>> > >>>>>>> from Master to RegionServers. Intent is to move forward on design > of > >>>>>>> > >>>>>>> these pieces: e.g. how client bootstraps. > >>>>>>> > >>>>>>> Support was expressed for getting at least the bones of a split > meta > >>>>>>> > >>>>>>> into an hbase3 before the RCs. > >>>>>>> > >>>>>>> Where we'd actually store hbase:meta Region locations -- i.e. how a > >>>>>>> > >>>>>>> "ROOT' would be implemented -- was for our next meeting informed by > >>>>>>> > >>>>>>> research of the various approaches noted mostly above. It was > >>>>>>> > >>>>>>> also thought that the new ConnectionRegistry should not preclude > >>>>>>> > >>>>>>> making progress on the "ROOT" implementation. > >>>>>>> > >>>>>>> Will post notice of next meeting (next Weds or the one > >>>>>>> > >>>>>>> following). > >>>>>>> > >>>>>>> 1. > >>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n > >>>>>>> > >>>>>>> 2. > >>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq > >>>>>>> > >>>>>>> 3. > >>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153 > >>>>>>> > >>>>>>> 4. > >>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle > >>>>>>> > >>>>>>> 5. > >>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b > >>>>>>> > >>>>>>> On Tue, Jul 20, 2021 at 11:00 AM Stack <[email protected]> wrote: > >>>>>>> > >>>>>>>> Lets meet tomorrow. Please review the design doc > >>>>> "Design/Brainstorming" > >>>>>>>> Section 4.1 [1] before the meeting if you can (No harm if a > refresh > >>>>> of > >>>>>> the > >>>>>>>> requirements section while you are at it). > >>>>>>>> > >>>>>>>> Topic: Split Meta Design Reset Status > >>>>>>>> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada) > >>>>>>>> > >>>>>>>> Join Zoom Meeting > >>>>>>>> > >>>>>> > >>>>> > https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09 > >>>>>>>> > >>>>>>>> Meeting ID: 773 1892 0525 > >>>>>>>> Passcode: hbase > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> S > >>>>>>>> > >>>>>>>> 1. 1. > >>>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n > >>>>>>>> > >>>>>>>> On Thu, Jul 8, 2021 at 1:04 PM Stack <[email protected]> wrote: > >>>>>>>> > >>>>>>>>> Meeting notes (Meeting happend after I figured the zoom had a > >>>>> 'waiting > >>>>>>>>> room' -- sorry Duo) > >>>>>>>>> > >>>>>>>>> Split Meta Status Zoom Meeting > >>>>>>>>> Wed Jul 7, 2021 @ 5pm for ~90minutes > >>>>>>>>> Attendees: Duo, Francis, Stack, and Clay > >>>>>>>>> Agenda: Mainly talk about the one-pager design and PoC proposed > in > >>>>> [2] > >>>>>>>>> and added to the > >>>>>>>>> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as > >>>>> ROOT' > >>>>>>>>> approach) > >>>>>>>>> > >>>>>>>>> Duo suggested no advantage treating the first meta of hbase:meta > >>>>> table > >>>>>>>>> special; as a "root" > >>>>>>>>> and other comments (see remarks in [2]). > >>>>>>>>> > >>>>>>>>> Some pushback. When meta is NOT split, all works as it did before > >>>>> (big > >>>>>>>>> on backward-compatible). > >>>>>>>>> > >>>>>>>>> Duo suggested just intro a ROOT table altogether rather than > treat > >>>>> the > >>>>>>>>> first Region > >>>>>>>>> in the hbase:meta table as a 'root' and then mirror to zk the > first > >>>>>> meta > >>>>>>>>> region; if no > >>>>>>>>> split, then old clients should just work even though now hbase > >>>>> cluster > >>>>>>>>> has a ROOT table. > >>>>>>>>> Discussion. If no split, what's to do, etc.? > >>>>>>>>> > >>>>>>>>> Duo expressed concern that if split-meta is not on always -- > >>>>> enabled > >>>>>>>>> optionally -- then > >>>>>>>>> the code path will not see exercise and split-meta will likely > >>>>> fail and > >>>>>>>>> go the way of > >>>>>>>>> prefix tree and distributed log replay -- a feature that failed, > >>>>>>>>> cluttered up the > >>>>>>>>> code base, and only later was removed. Discussion. Was allowed > this > >>>>>>>>> could happen. > >>>>>>>>> Counter examples: RegionServer Groups. A few of the attendees > >>>>>>>>> volunteered they need > >>>>>>>>> split meta for their production so would try to see it through. > >>>>>>>>> > >>>>>>>>> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 > >>>>> as > >>>>>>>>> ROOT' if we don't > >>>>>>>>> split; not much changes. > >>>>>>>>> > >>>>>>>>> Comment on the PoC that its all > >>>>>>>>> > >>>>>>>>> if 'first special meta region' do this > >>>>>>>>> else do something else... > >>>>>>>>> > >>>>>>>>> (all over the codebase) but it was suggested that this will be > the > >>>>> case > >>>>>>>>> if ROOT table > >>>>>>>>> added also and argued any implementation will have this issue (if > >>>>> ROOT > >>>>>>>>> then....) and > >>>>>>>>> THAT ugly 'root' comparator too whether a ROOT table or the > >>>>>>>>> 'hbase:meta,,1 as ROOT' > >>>>>>>>> approach. > >>>>>>>>> > >>>>>>>>> Clay asking if meta is a bottleneck (Clay played the 'operator' > and > >>>>>>>>> 'new-to-the-topic' > >>>>>>>>> roles). Some discussion around how indeed it is and why we want > to > >>>>>> split > >>>>>>>>> meta at all. > >>>>>>>>> > >>>>>>>>> Clay mentioned how Master is inline for data now (Master-hosted > >>>>>>>>> Registry).... > >>>>>>>>> Discussion. Hopefully temporary state -- Registry doesn't need to > >>>>> be > >>>>>>>>> hosted on > >>>>>>>>> Master -- and Master will return to its background role -- soon. > >>>>>>>>> > >>>>>>>>> Clay brought up rollback after meta split. Discussion. Some > >>>>> agreement > >>>>>> it > >>>>>>>>> could be done for > >>>>>>>>> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root > >>>>> table > >>>>>> and > >>>>>>>>> client had > >>>>>>>>> started to use ROOT, it might be harder... > >>>>>>>>> > >>>>>>>>> Duo suggested that we not change the client at all; that client > >>>>> stays > >>>>>>>>> same however split > >>>>>>>>> meta is implemented (with addition of a root table or using > >>>>>>>>> hbase:meta,,1 as 'root'). > >>>>>>>>> This sounded attractive. We discussed how this could be done in a > >>>>>>>>> backward compatible way; > >>>>>>>>> add simple location lookup API to Registry...A write-up on how > this > >>>>>>>>> might work will be posted > >>>>>>>>> in next day or so for others to review (Need to figure getting > >>>>> Registry > >>>>>>>>> off Master). > >>>>>>>>> > >>>>>>>>> Clay suggested, as an operator, how he thought the split meta > >>>>> should > >>>>>>>>> roll out. One of us > >>>>>>>>> claimed this described the 'hbase:meta,,1 as ROOT' approach. > >>>>>>>>> > >>>>>>>>> Duo had to go to work so we called it quits and said we'd meet > >>>>> again > >>>>>>>>> same time, next week. > >>>>>>>>> > >>>>>>>>> 1. > >>>>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle > >>>>>>>>> 2. https://issues.apache.org/jira/browse/HBASE-25761 > >>>>>>>>> > >>>>>>>>> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) < > >>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Is it only me? I tried to join the meeting but it always tell me > >>>>> that > >>>>>> I > >>>>>>>>>> need to wait for the presenter to invite me to join... > >>>>>>>>>> > >>>>>>>>>> Stack <[email protected]> 于2021年7月8日周四 上午1:04写道： > >>>>>>>>>> > >>>>>>>>>>> Short notice but reminder that this meeting is today at 5pm PST > >>>>> (We > >>>>>>>>>> talked > >>>>>>>>>>> of doing it earlier but in the end lets just keep the original > >>>>> 5pm > >>>>>>>>>> time). > >>>>>>>>>>> Thanks, > >>>>>>>>>>> S > >>>>>>>>>>> > >>>>>>>>>>> On Tue, Jul 6, 2021 at 11:36 AM Stack <[email protected]> > wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Lets do a quick chat on current state of hbase split-meta > >>>>> project. > >>>>>>>>>>>> > >>>>>>>>>>>> Francis has posted a suggested one-pager design in HBASE-25761 > >>>>>> which > >>>>>>>>>>> would > >>>>>>>>>>>> be good to review before the meeting if you can. Lets discuss > >>>>> this > >>>>>>>>>> and > >>>>>>>>>>> any > >>>>>>>>>>>> other suggestions for moving the project forward. > >>>>>>>>>>>> > >>>>>>>>>>>> Meeting details below. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> S > >>>>>>>>>>>> > >>>>>>>>>>>> Topic Split Meta Design Reset Status > >>>>>>>>>>>> Description Review one-pager design attached to > >>>>>>>>>>>> https://issues.apache.org/jira/browse/HBASE-25761 > >>>>>>>>>>>> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada) > >>>>>>>>>>>> > >>>>>>>>>>>> Meeting ID 736 3907 8852 > >>>>>>>>>>>> Security > >>>>>>>>>>>> Passcode *hbase* Hide > >>>>>>>>>>>> Waiting Room > >>>>>>>>>>>> Invite Link > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>> > >>>>> > https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 > >>>>>>>>>>> Copy > >>>>>>>>>>>> Invitation > >>>>>>>>>>>> > >>>>>>>>>>>> On Mon, Mar 22, 2021 at 4:28 PM Stack <[email protected]> > >>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Now the requirements are in [1], we're going to move to the > >>>>> next > >>>>>>>>>> stage > >>>>>>>>>>> -- > >>>>>>>>>>>>> actual design for split-meta -- and have set up a chat for > >>>>> this > >>>>>>>>>> thursday > >>>>>>>>>>>>> afternoon (4PM California time/8AM Beijing time) to get the > >>>>> ball > >>>>>>>>>>> rolling. > >>>>>>>>>>>>> Please come if interested. Zoom details are below. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Yours, > >>>>>>>>>>>>> S > >>>>>>>>>>>>> 1. > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>> > >>>>> > https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Topic: hbase split-meta design warmup chat > >>>>>>>>>>>>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada) > >>>>>>>>>>>>> > >>>>>>>>>>>>> Join Zoom Meeting > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>> > >>>>> > https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09 > >>>>>>>>>>>>> > >>>>>>>>>>>>> Meeting ID: 759 8800 3798 > >>>>>>>>>>>>> Passcode: hbase > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Tue, Jan 5, 2021 at 9:13 AM Stack <[email protected]> > >>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> FYI, a few of us have been working on the redo/reset of the > >>>>>> split > >>>>>>>>>> meta > >>>>>>>>>>>>>> design (HBASE-25382). We (think we've) finished the > >>>>>> requirements. > >>>>>>>>>> Are > >>>>>>>>>>> there > >>>>>>>>>>>>>> any others to consider? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Feedback and contribs welcome. Otherwise, on to the next > >>>>> phase > >>>>>> -- > >>>>>>>>>>> design. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> S > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>>> > >>>> >
