We have closed the Google Doc for further comments, and all changes have been incorporated into the confluence document: https://cwiki.apache.org/confluence/display/SOLR/SIP-15+Node+roles Thanks for all the comments and feedback.
On Thu, Nov 25, 2021 at 12:09 PM Ishan Chattopadhyaya < [email protected]> wrote: > > Maybe one way to provide for the future is to place some requirements on > roles, and designate that role-specific coordination information > > be placed within the subtree for that role under /node-roles/<rolename>/ > To facilitate this we could > > 1. move the currently designed set of ephemeral nodes down one level > into am "/available" node that reflects which nodes are currently up with > the role (exactly what they do in the present design) > > We've introduced nesting in the ZK structure now. Ephemeral nodes for Solr > node names will now be placed at > /node-roles/<rolename>/nodes/<ephemeralnode>. > > > > > > > On Thu, Nov 25, 2021 at 12:04 PM Ishan Chattopadhyaya < > [email protected]> wrote: > >> >> >> On Thu, Nov 25, 2021 at 4:35 AM Gus Heck <[email protected]> wrote: >> >>> Things I think still need to be clarified to define "Role" explicitly. >>> Primarily to guide future implementations. Things I can think of include: >>> >>> - What the bar for something to be a role vs being more appropriate >>> as a property/tag type concept. Essentially what is and is not expected >>> (at >>> this time) to be a role. >>> >>> We don't wish to set a bar for anything at this point, since I can't >> imagine all future innovation that can happen. All we can do is present >> representative examples. Common sense will dictate what is useful to have >> as a role, and clearly we can't set a bar for that. >> >> >>> >>> - What's the process for this? Do we want a SIP per role to ensure >>> good discussion? Are there some simple things a role might do that are >>> "normal" and don't require a SIP? Or maybe no sip generally? >>> >>> The community can decide on a case by case basis using established >> community processes. I don't see the need to define a new process for this. >> >> >>> >>> - Where a role is defined (enum? class? which module? core? solrj?) >>> and what motivates that choice (specifics such as methods/properties can >>> be >>> left to impl of course) >>> >>> This can be discussed during the implementation phase. This SIP is for >> the general concept, public interface and other non-code concerns. I don't >> want to start another 20 page thread here debating enum vs class and other >> things that might tickle someone's fancy. Lets cross the bridge once we get >> to it. >> >> >>> >>> - Naming conventions for roles... camel case? snake case? all caps? >>> what characters are allowed, Possibly discourage any ideas with role >>> names >>> that embed config options such as overseer_priority_1 and any parsing of >>> role names. >>> >>> >> We can discuss that on a case by case basis when new roles are >> introduced. I've already proposed that the sysprops for roles should be in >> lowercase, which rules out camelCase. >> >> >>> >>> - >>> >>> If we answer those and bundle them with the 3 sections under the example >>> api calls into a "Roles" section or "Definition of Roles" or something like >>> that it seems good to me. >>> >>> Sure we could discuss any of this independently for each role in each >>> sip but if we put guidance here we probably can skip that discussion for N >>> out of M sips/tickets in the future (N being maximized if we choose well >>> and write with clarity). Thus we'd be saving time in the long run and >>> reduce the possibility that people propose something awkward and it slips >>> through on lazy consensus. >>> >>> And of course it's Apache software so it can all be changed later by a >>> vote or concensus. :) >>> >>> -Gus >>> >>> On Wed, Nov 24, 2021 at 3:21 PM Ishan Chattopadhyaya < >>> [email protected]> wrote: >>> >>>> > Moreover I think it's a very role specific notion, that is not a good >>>> > fit in the roles framework, so maybe should never be part of this SIP. >>>> >>>> Regarding "capable" vs "currently providing", I agree with Ilan here ^ >>>> >>>> >>>> >>>> On Wed, Nov 24, 2021 at 10:38 PM Gus Heck <[email protected]> wrote: >>>> >>>>> You do make a good point that the details can be very role specific. >>>>> Maybe one way to provide for the future is to place some requirements on >>>>> roles, and designate that role-specific coordination information be placed >>>>> within the subtree for that role under /node-roles/<rolename>/ To >>>>> facilitate this we could >>>>> >>>>> 1. move the currently designed set of ephemeral nodes down one >>>>> level into am "/available" node that reflects which nodes are >>>>> currently up >>>>> with the role (exactly what they do in the present design) >>>>> 2. Roles must store role related info in the data for the role >>>>> node, or in tree(s) that are peers to /available as they see fit. >>>>> 3. Roles must have documentation detailing what they store where >>>>> and how it's used. >>>>> 4. Roles (beyond those implemented in this SIP) should be >>>>> discussed in their own SIP (enforcing the non-trivial label concept we >>>>> all >>>>> seem to agree on) >>>>> >>>>> >>>> I am not in favour of complicating the currently proposed design by >>>> doing any of this now ^ If we need to, we can discuss this then. >>>> >>>> >>>>> The bit that's not clear is where that documentation would live. It's >>>>> not really material for the users, but more for the developers. >>>>> >>>>> On Wed, Nov 24, 2021 at 11:11 AM Ilan Ginzburg <[email protected]> >>>>> wrote: >>>>> >>>>>> I agree with Ishan that "currently providing" shouldn't be a current >>>>>> concern for the SIP. >>>>>> Moreover I think it's a very role specific notion, that is not a good >>>>>> fit in the roles framework, so maybe should never be part of this SIP. >>>>>> >>>>>> Even on the Overseer example. Suppose we changed the current design to >>>>>> have a "main" node being Overseer and a secondary to be somewhat >>>>>> active standby in case the main one goes down (imagine some form of >>>>>> replication between the two?). Both would have the "Overseer" role, >>>>>> but what they provide would be different. Overseer specific code >>>>>> should be able to know what is provided by which node, because a node >>>>>> trying to use the Overseer service would have to talk to a specific >>>>>> one of the two, not just one that currently provides. >>>>>> >>>>>> Same for data nodes. A data node can have replicas on it. But it may >>>>>> not. If it doesn't, shall we say it has the capability but does not >>>>>> provide? Of course not, nobody would suggest that. Because we know >>>>>> that tracking the actual data is not a role level responsibility but a >>>>>> data (collection/shard/replica etc) tracking responsibility, done >>>>>> using all the nice structures the code maintains in ZK and in caches >>>>>> all over the place. >>>>>> >>>>>> Ilan >>>>>> >>>>>> On Wed, Nov 24, 2021 at 5:10 PM Gus Heck <[email protected]> wrote: >>>>>> > >>>>>> > So we are potentially talking past each other. Let me re-clarify >>>>>> since you've interpreted "configuration" differently (I think). What I'm >>>>>> referring to in ZK is the "reflection of configuration" not the actual >>>>>> configuration. This is why I'm asking the question of how much info >>>>>> should >>>>>> be retained about a node after it goes down. Case in point zookeeper >>>>>> embedded: The code might take different actions if it can see that a node >>>>>> that had a running zookeeper did exist (and thus may return) vs a >>>>>> situation >>>>>> where there's only 2 nodes playing zookeeper roles right now and no >>>>>> evidence of a third ever having existed. >>>>>> > >>>>>> > Also you say you can't imagine anything beyond overseer... well I >>>>>> could imagine that a cluster would want to say these 10 machines (but not >>>>>> the other 40) are allowed to have zookeeper, and I want the cluster to to >>>>>> have 3 node level of redundancy in zookeeper, and a new zookeeper should >>>>>> be >>>>>> created and the data replicated to it if a zk node is lost for more than >>>>>> 30 >>>>>> minutes. >>>>>> > >>>>>> > On Wed, Nov 24, 2021 at 10:38 AM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Wed, Nov 24, 2021 at 9:05 PM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> On Wed, Nov 24, 2021 at 8:19 PM Gus Heck <[email protected]> >>>>>> wrote: >>>>>> >>>> >>>>>> >>>> Left some additional comments in the google doc, extracting the >>>>>> key point for others here so they can know if they want to read the >>>>>> google >>>>>> doc commentary: >>>>>> >>> >>>>>> >>> >>>>>> >>> I'm addressing them here, inline. >>>>>> >>> >>>>>> >>>> >>>>>> >>>> There is presently disagreement with my proposal that the sip >>>>>> specify both how we track configuration (which I have been referring to >>>>>> as >>>>>> "capability") and runtime state ("providing"). I see the primary value of >>>>>> having a role concept as one of organizing the code and configuration in >>>>>> a >>>>>> way that is easy to understand, which is why I'm in favor of this SiP >>>>>> overall. >>>>>> >>> >>>>>> >>> >>>>>> >>> I purposely don't want to complicate this design by allowing for >>>>>> "capability" vs "currently providing" to be represented as first class >>>>>> citizens. Besides the OVERSEER role (preferred overseer), I can't think >>>>>> of >>>>>> concrete scenarios where such a concept could be applicable, and hence I >>>>>> don't want to get ahead of myself in trying to address how to solve that. >>>>>> Since the current proposal for dealing with roles is flexible and generic >>>>>> enough, I don't think adding such extensions would be a problem. >>>>>> However, I >>>>>> don't want to go down that rabbithole at this point in time. >>>>>> >>> >>>>>> >>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> If you don't want to tackle implementing the runtime state bit >>>>>> I'm ok with that as long as we have a clear plan in the SIP of how this >>>>>> type of information should be organized in the future. >>>>>> >>>> The structure proposed in the SIP for information in zk seems to >>>>>> map more clearly to runtime state than configuration, and seems like it >>>>>> would inhibit a natural representation of such. >>>>>> >>> >>>>>> >>> >>>>>> >>> The structure proposed allows for role specific configurations >>>>>> (if needed). It doesn't provide for node specific configuration in ZK, >>>>>> since all node properties should come from either solr.xml or sysprops. >>>>>> At >>>>>> present, I don't think we ever have node specific info configuration in >>>>>> ZK >>>>>> in any part of Solr, and I think we shouldn't go that route here. >>>>>> >> >>>>>> >> >>>>>> >> Correction: "At present, I don't think we ever have node specific >>>>>> configuration in ZK in any part of Solr, and I think we shouldn't go that >>>>>> route here." >>>>>> >> >>>>>> >>> >>>>>> >>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> Finally the structure in the SIP for zk information doesn't >>>>>> specify if some of the nodes are ephemeral, and reliable for a "current" >>>>>> list or if they are persistent over time. I feel like we may not yet >>>>>> have a >>>>>> plan for how much information about nodes (n general) should be retained >>>>>> while a node is down, which maybe needs to underlie this design. >>>>>> >>> >>>>>> >>> >>>>>> >>> All the node names under the roles znodes are ephemeral. Added >>>>>> this to the document (confluence and gdoc both). >>>>>> >>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> -Gus >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> On Tue, Nov 23, 2021 at 10:34 PM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>> >>>>>> >>>>> I've synced back all changes to confluence. Both represent the >>>>>> same document as of now. >>>>>> >>>>> Please review and suggest if there are any outstanding >>>>>> concerns. Thanks for all the feedback. >>>>>> >>>>> >>>>>> >>>>> I wish to proceed with the implementation based on lazy >>>>>> consensus again. >>>>>> >>>>> >>>>>> >>>>> On Tue, Nov 23, 2021 at 9:38 PM Mike Drob <[email protected]> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> I should clarify, that my -1 was specifically about reaching >>>>>> lazy consensus and not about that proposal itself. >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>> >>>>>>> > -1, I would like to see a proposal on list where it will be >>>>>> permanently available >>>>>> >>>>>>> >>>>>> >>>>>>> It will be available in the confluence document (as it >>>>>> currently is). The google docs is just for collecting temporary feedback, >>>>>> thanks to easy inline commenting capability. We promised to consolidate >>>>>> feedback in the gdoc and sync back to the SIP document in confluence. >>>>>> This >>>>>> is reasonable enough expectations, and shouldn't be a worry enough for a >>>>>> -1. >>>>>> >>>>>>> >>>>>> >>>>>>> As of right now, while trying to address some concerns, some >>>>>> edits have been made in gdocs by Noble that we'll sync back into >>>>>> confluence; so that's the only divergence. >>>>>> >>>>>>> >>>>>> >>>>>>> On Tue, Nov 23, 2021 at 9:09 PM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>> >>>>>> >>>>>>>> > -1, I would like to see a proposal on list where it will >>>>>> be permanently available >>>>>> >>>>>>>> > in the archives instead of being directed to a Google doc >>>>>> which can be edited at any point in time. >>>>>> >>>>>>>> > This has been an incredibly difficult proposal to keep >>>>>> track of. >>>>>> >>>>>>>> >>>>>> >>>>>>>> This entire SIP process has been incredibly difficult to >>>>>> coordinate. The idea of doing this on a mail thread was a terrible one. >>>>>> There's no archive available publicly for this list that doesn't break >>>>>> the >>>>>> threading functionality (I guess someone among us is using some funky >>>>>> mail >>>>>> client that's messing with Pony Mail). JIRA for discussion would've been >>>>>> much better. I suppose JIRA used to have threaded discussions once upon a >>>>>> time, that would've been very handy here. >>>>>> >>>>>>>> >>>>>> >>>>>>>> On Tue, Nov 23, 2021 at 9:07 PM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> Okay, sure, I'll rescind my assertion about having reached >>>>>> a lazy consensus. I'll work through all the recent comments and feedback. >>>>>> >>>>>>>>> I think at the last moment, there was an edit by Noble to >>>>>> the Google Doc that hasn't made it into the SIP document, and I'll review >>>>>> that change, the recent comments and get back here tomorrow. Other than >>>>>> that last moment change, the SIP document is up to date. >>>>>> >>>>>>>>> >>>>>> >>>>>>>>> On Tue, Nov 23, 2021 at 8:24 PM Gus Heck < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> Also, I don't know that it's defined anywhere, but I feel >>>>>> that declaring lazy consensus on a SIP in less than a week is rushing >>>>>> things. Some folks won't have the flexibility to address things daily and >>>>>> I've been uncomfortably pulled away from paid work trying to keep up with >>>>>> this as is. Lazy consensus should include at least one weekend (IMHO) >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> On Tue, Nov 23, 2021 at 9:49 AM Gus Heck < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> IMHO two things must happen before lazy consensus wait >>>>>> period can begin: >>>>>> >>>>>>>>>>> 1) The wiki must be updated to reflect the results of >>>>>> discussion in the google doc. >>>>>> >>>>>>>>>>> 2) The fact that the wiki has been updated, and that >>>>>> you think the discussion is at an end must be posted here, preferably >>>>>> with >>>>>> a summary on the list. >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> If it didn't happen on the list it didn't happen. >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> Also the google doc has not been notifying me of >>>>>> changes/responses. So a note here when you've responded and some time to >>>>>> respond before the above steps are taken would be appreciated. >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> -Gus >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> On Tue, Nov 23, 2021 at 9:40 AM Mike Drob < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> -1, I would like to see a proposal on list where it will >>>>>> be permanently available in the archives instead of being directed to a >>>>>> Google doc which can be edited at any point in time. >>>>>> >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> This has been an incredibly difficult proposal to keep >>>>>> track of. >>>>>> >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> Mike >>>>>> >>>>>>>>>>>> >>>>>> >>>>>>>>>>>> On Tue, Nov 23, 2021 at 6:09 AM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>> This proposal has passed with lazy consensus. We can >>>>>> proceed to the implementation phase. >>>>>> >>>>>>>>>>>>> Thanks to everyone for feedback, esp. Gus for the >>>>>> patience. >>>>>> >>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>> On Sun, Nov 21, 2021 at 2:24 PM Gus Heck < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> Thanks for adding that :) and Thanks to Nobel for >>>>>> translating my ravings :). Sorry about the acronym. Amusingly I had to >>>>>> look >>>>>> up gish gallop to understand what you meant :) I can assure you that I >>>>>> will >>>>>> never gish gallop you or anyone on the list, and I find such techniques >>>>>> repugnant. If I'm not making sense, I'm likely covering ground too fast, >>>>>> skipping things that are in my head but not yours, expressing too many >>>>>> tangents (when I start nesting parentheses this is (usually) a sign I'm >>>>>> wandering too much), or I am simply mistaken about something. Always feel >>>>>> free to ask me to explain! >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> -Gus >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 12:17 AM Ishan Chattopadhyaya < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> Added both your suggestions to Google Docs for inline >>>>>> commenting and the SIP document at confluence. Thanks for the feedback. >>>>>> >>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 10:28 AM Ishan Chattopadhyaya >>>>>> <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> I'll add the lifecycle details to the document as >>>>>> well. >>>>>> >>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>> On Sun, 21 Nov, 2021, 9:08 am Ishan Chattopadhyaya, < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> Hi Gus, >>>>>> >>>>>>>>>>>>>>>>> Thanks for bringing it up again. Initially, I >>>>>> wasn't clear what you meant and mistook them for gish gallop. >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> These were the things I am/was confused about >>>>>> (Noble explained it to me what you mean): >>>>>> >>>>>>>>>>>>>>>>> - adherence to DRY via zk >>>>>> >>>>>>>>>>>>>>>>> - runtime config info should come from zk >>>>>> >>>>>>>>>>>>>>>>> - add dynamic features >>>>>> >>>>>>>>>>>>>>>>> - same interaction patterns for consulting config >>>>>> values. >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> Now, after Noble's help, I think I understand your >>>>>> motivations. >>>>>> >>>>>>>>>>>>>>>>> - DRY = Don't Repeat Yourself ;-) >>>>>> >>>>>>>>>>>>>>>>> - You're suggesting that we have a standard way to >>>>>> have role specific configurations (in ZK), so that a new role added later >>>>>> can access the configurations in a standard way >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> This is a very good suggestion, and makes complete >>>>>> sense now. This was definitely a missing piece! >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> Here's a proposal to address that (adding it to >>>>>> Google Docs: >>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit >>>>>> ): >>>>>> >>>>>>>>>>>>>>>>> - Per Role ZNode >>>>>> >>>>>>>>>>>>>>>>> - Every role's ZNode to hold configuration data >>>>>> (this is the key addition that you're asking for, IIUC) >>>>>> >>>>>>>>>>>>>>>>> - Children of role ZNodes to be list of ephemeral >>>>>> solr nodes >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> We can iterate over this on the Google Docs >>>>>> (internal representation ZK section) with inline commenting if we need >>>>>> to. >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> Please let us know how that sounds. >>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>> >>>>>>>>>>>>>>>>> Ishan >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>> On Sun, Nov 21, 2021 at 4:49 AM Gus Heck < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> It is difficult to track everything in this thread >>>>>> for sure. Specifically, I don't feel my email of Wed, Nov 17, 7:43 PM >>>>>> (EST) >>>>>> has been addressed or responded to except for one difference of opinion >>>>>> with Jan on whether or not we should prohibit the notion of any role ever >>>>>> changing at runtime. >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> Note that I made a specific proposal for an >>>>>> addition to the sip, regarding start up lifecycle and adherence to DRY >>>>>> via >>>>>> zk, in the (probably not clearly expressed) hope that we can move to a >>>>>> general overall application principle that at runtime config info should >>>>>> come from zk. If we follow such a principle, we will always be in a >>>>>> position to add dynamic features without significant rework, and all code >>>>>> could hopefully reuse the same interaction patterns for consulting config >>>>>> values. >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> Neither item is addressed in the SIP currently. >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> Also I had put that out there as a single item so >>>>>> it could be focused on (simplifying the discussion I hope), and depending >>>>>> on how discussion proceeds, I may have other follow on items. >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> -Gus >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> On Sat, Nov 20, 2021 at 8:45 AM Ishan >>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> Is the request here for everybody to express >>>>>> again the concerns >>>>>> >>>>>>>>>>>>>>>>>>>> already expressed in this email thread and not >>>>>> addressed? >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> I think the expectation now (after we've >>>>>> expressed our intention to reach a lazy consensus) is that if someone >>>>>> wants >>>>>> to block this SIP from adoption, then to put forward the objections. Sort >>>>>> of like a veto. >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> I suggest instead the authors review the thread, >>>>>> match expressed >>>>>> >>>>>>>>>>>>>>>>>>>> concerns with how the concern was addressed (or >>>>>> not addressed) and >>>>>> >>>>>>>>>>>>>>>>>>>> provide an exhaustive list. >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> I've summarized most of the discussion in the SIP >>>>>> document. If you feel I could do a better job with it, please help me >>>>>> with >>>>>> areas of improvement. >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> This proposal in its current form (data and >>>>>> overseer roles) doesn't >>>>>> >>>>>>>>>>>>>>>>>>>> offer much that can't be reasonably achieved by >>>>>> other means. I'd find >>>>>> >>>>>>>>>>>>>>>>>>>> much more value in making sure what is done now >>>>>> is a solid foundation >>>>>> >>>>>>>>>>>>>>>>>>>> for the future. >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>> Fair enough, I understand your perspective. >>>>>> Thanks for your feedback. >>>>>> >>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> Ilan >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Nov 18, 2021 at 11:24 PM Noble Paul < >>>>>> [email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>>>>>>> > After so many back and forth mails, I just >>>>>> can't say who has an outstanding concern and if they are already >>>>>> addressed >>>>>> or not. I think the Google doc would help us get clarity on that. Please >>>>>> take a moment to give your inputs >>>>>> >>>>>>>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>>>>>>> > On Fri, Nov 19, 2021, 9:18 AM Ishan >>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> >>>>>>>>>>>>>>>>>>>> >> Apologies, the vote hasn't passed formally >>>>>> and I was under some confusion on the process. >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> >>>>>>>>>>>>>>>>>>>> >> I'd like to proceed with a lazy consensus and >>>>>> proceed to the implementation phase now. >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> >>>>>>>>>>>>>>>>>>>> >> However, I would appreciate it if someone >>>>>> wants to bring out any outstanding concerns about the SIP document. >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> >>>>>>>>>>>>>>>>>>>> >> To facilitate in-line comments, here's a >>>>>> temporary Google Docs version of this document. >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> https://docs.google.com/document/d/1hijvM1WX9u2TOUdLEkFYVCofLeJlv2MRZqe-ncobJVw/edit?usp=sharing >>>>>> >>>>>>>>>>>>>>>>>>>> >> (I shall copy changes back to confluence >>>>>> eventually) >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> >>>>>>>>>>>>>>>>>>>> >> Thanks and apologies again regarding the >>>>>> confusion with the voting, >>>>>> >>>>>>>>>>>>>>>>>>>> >> Regards, >>>>>> >>>>>>>>>>>>>>>>>>>> >> Ishan >>>>>> >>>>>>>>>>>>>>>>>>>> >> >>>>>> >>>>>>>>>>>>>>>>>>>> >> On Thu, Nov 18, 2021 at 9:50 PM Ishan >>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>> The SIP passed the voting phase. Thanks for >>>>>> all for the feedback and insights. >>>>>> >>>>>>>>>>>>>>>>>>>> >>> Looking forward to your collaboration and >>>>>> reviews as we implement this. >>>>>> >>>>>>>>>>>>>>>>>>>> >>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>> On Thu, Nov 18, 2021 at 9:42 PM Ishan >>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>> > It's fine if we don't provide any ability >>>>>> for runtime modification of roles at this time but I'm leery of >>>>>> precluding >>>>>> it in the future. >>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>> In future, the necessity for such a >>>>>> facility can dictate our course of action. We cannot lay down rules cast >>>>>> in >>>>>> stone for functionality that we can't foresee yet. >>>>>> >>>>>>>>>>>>>>>>>>>> >>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>> On Thu, Nov 18, 2021 at 9:40 PM Ishan >>>>>> Chattopadhyaya <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> Thanks Jan, I added both those points to >>>>>> the SIP document in the Notes section. >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> On Thu, Nov 18, 2021 at 7:18 PM Jan >>>>>> Høydahl <[email protected]> wrote: >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> 18. nov. 2021 kl. 01:43 skrev Gus Heck < >>>>>> [email protected]>: >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> 2) Roles will not be checked by loading >>>>>> config from disk or caching disk config in memory. (zk ONLY source of >>>>>> truth) >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> It sounds a bit backward for a local node >>>>>> to first parse solr.node.roles, determine its local set of roles, then >>>>>> publish them to Zookeeper, and then read back its own roles from ZK. >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Code that only needs to determine "Do I >>>>>> have the XXX role?" or find out "What roles do I have" should be able to >>>>>> fetch the (static) roles from some roles utility class without consulting >>>>>> ZK. >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Code that needs to check what nodes have >>>>>> a certain role (such as placement) would obviously need ot consult ZK. >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Perhaps the SIP should also state some >>>>>> Non-goals or assertions such as >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> * Roles are static and immutable (also in >>>>>> zk) for the entire life cycle of a node >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> I also think we should state that the bar >>>>>> for adding new roles should be high so it is not abused as any other tag >>>>>> or >>>>>> label for any tiny feature. It should be reserved for functionality that >>>>>> may benefit from a dedicated set of nodes. That may be clear already, but >>>>>> you never know... >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> Jan >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> >>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>> [email protected] >>>>>> >>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>> [email protected] >>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>> >>>>>>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>> >>>>>>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> >>>>>> >>>>>>>>>>>>>> -- >>>>>> >>>>>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>> >>>>>>>>>>>>>> http://www.the111shift.com (play) >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> >>>>>> >>>>>>>>>>> -- >>>>>> >>>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>> >>>>>>>>>>> http://www.the111shift.com (play) >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> >>>>>> >>>>>>>>>> -- >>>>>> >>>>>>>>>> http://www.needhamsoftware.com (work) >>>>>> >>>>>>>>>> http://www.the111shift.com (play) >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> -- >>>>>> >>>> http://www.needhamsoftware.com (work) >>>>>> >>>> http://www.the111shift.com (play) >>>>>> > >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > http://www.needhamsoftware.com (work) >>>>>> > http://www.the111shift.com (play) >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: [email protected] >>>>>> For additional commands, e-mail: [email protected] >>>>>> >>>>>> >>>>> >>>>> -- >>>>> http://www.needhamsoftware.com (work) >>>>> http://www.the111shift.com (play) >>>>> >>>> >>> >>> -- >>> http://www.needhamsoftware.com (work) >>> http://www.the111shift.com (play) >>> >>
