Re: First class support for node roles

2021-12-08 Thread Noble Paul
Jan, We are actually saying the same things, but differently. Today we have only 2 roles - data - overseer We have set default values for them . A user who starts a 9.0 Solr will get the default behavior and it is the same exact behavior as we get in 8x. Imagine you introduce a new role

Re: First class support for node roles

2021-12-08 Thread Jan Høydahl
> And in some future version it may read an additional property > solr.overseer.priority=3 to give some priority to it, or it could read some > solr.overseer.strict=true to refuse to place overseer strictly on nodes with > the role This sysProp example was just an example, and maybe a bad

Re: First class support for node roles

2021-12-08 Thread Jan Høydahl
> > > My proposal is that ALL roles are always ALLOW if not specified explicitly. > > As explained several times before, this is a problem for new roles introduced > in future. > Those roles will get turned on on all nodes after an upgrade, whether a user > wants or not. But you're not

Re: First class support for node roles

2021-12-07 Thread Ishan Chattopadhyaya
> My proposal is that ALL roles are always ALLOW if not specified explicitly. As explained several times before, this is a problem for new roles introduced in future. Those roles will get turned on on all nodes after an upgrade, whether a user wants or not. A user explicitly mentions which roles

Re: First class support for node roles

2021-12-07 Thread Noble Paul
There is no complexity here. You just think something is complex because you haven't spent any time to understand this. A role is a tag for a node. Period. How complex is that? We just added standard mechanism to declare and discover the nodes. There is no extra cost or complexity On Wed,

Re: First class support for node roles

2021-12-07 Thread Jan Høydahl
> > Another user with a 100 node cluster who today have three overseer nodes that > they have shielded from having data by specifying createNodeSet manually or > by other means, can choose to adopt rhe role system, and define tree > dedicated nodes with the overseer role but without the data

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
Ilan, I missed addressing one aspect in your mail. On Mon, Dec 6, 2021 at 8:37 PM Ilan Ginzburg wrote: > Ishan, > > > > Using a string separate from the role definitions (Ishan) makes it too > easy to have roles for which the default configuration is unknown. > > > > Ilan, can you please

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
Hi all, I think we've captured all recent feedback in the updated proposal. Thanks to Mike and others for helping us streamline the proposal to better suit the current overseer role. Thanks to Ilan and others for the concept of defaultIfAbsent concept. I think the proposal is not only complete,

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
On Mon, Dec 6, 2021 at 9:18 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I've updated the SIP document with the recent changes. Also, added a new > section to provide guidance for adding new roles. > > On Mon, Dec 6, 2021 at 8:37 PM Ilan Ginzburg wrote: > >> Ishan, >> >> > >

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
I've updated the SIP document with the recent changes. Also, added a new section to provide guidance for adding new roles. On Mon, Dec 6, 2021 at 8:37 PM Ilan Ginzburg wrote: > Ishan, > > > > Using a string separate from the role definitions (Ishan) makes it too > easy to have roles for which

Re: First class support for node roles

2021-12-06 Thread Ilan Ginzburg
Ishan, > > Using a string separate from the role definitions (Ishan) makes it too easy > > to have roles for which the default configuration is unknown. > > Ilan, can you please elaborate (perhaps with an example) as to what you mean > here? If the default string for all roles for nodes with

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
On Mon, Dec 6, 2021 at 6:38 PM Jan Høydahl wrote: > > > Also, we should not put so much emhasis on "nodes without roles > defined" as if that should be a common way of starting nodes in a huge > cluster. > > > > Jan, the need to tackle "nodes without roles defined" separately is to > cater to

Re: First class support for node roles

2021-12-06 Thread Jan Høydahl
> > Also, we should not put so much emhasis on "nodes without roles defined" as > > if that should be a common way of starting nodes in a huge cluster. > > Jan, the need to tackle "nodes without roles defined" separately is to cater > to those users who do not use the roles functionality; we

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
> Using a string separate from the role definitions (Ishan) makes it too easy to have roles for which the default configuration is unknown. Ilan, can you please elaborate (perhaps with an example) as to what you mean here? As per my proposal, a node that was started with explicit roles, but

Re: First class support for node roles

2021-12-06 Thread Jan Høydahl
Are we making a non-issue into a configuration mess? The overseer's job is diminishing by every version, and we should not fool ourself into believing that a stray overseer will kill an upgrade, and therefore complicate the whole role system. Also, we should not put so much emhasis on "nodes

Re: First class support for node roles

2021-12-05 Thread Ishan Chattopadhyaya
Role specific configurations can go into /node_roles/${rolename} znode, and that is outside the scope of this SIP. The concept of role specific modes (eg allowed, preferred for overseer) is a welcome addition to original proposal to model the overseer functionality properly without any confusion

Re: First class support for node roles

2021-12-05 Thread Mark Miller
> I'm very strongly in favor of not letting users design a system in which the > cluster can be "live" without an overseer. > I understand that the overseer can be taxing to the cluster, That is really just an implementation choice. Bluntly, It doesn't do anything smartwatch could not handle on

Re: First class support for node roles

2021-12-05 Thread Gus Heck
So I think we're loosing sight of the original concept of "default" and conflating it with role configuration. When we started talking about "default roles" the idea was "default" was a flag that indicated if the role was active on a Solr Node where no roles had been specified. Plain and simple.

Re: First class support for node roles

2021-12-05 Thread Ishan Chattopadhyaya
I think I understand Ilan's motivation for two defaults. Here's a summary of what I understand Ilan's proposal, and a follow up proposal that achieves the similar effect with less perceived complexity to user. *Ilan's proposal (as I understand it):* 1. Every role to have two defaults. Example:

Re: First class support for node roles

2021-12-05 Thread Noble Paul
Ilan was asking how what should be the overseer role in the following situations a) role=overseer,data:on b) role=overseer: preferred,data:on c) role=data:on I'm saying a shouldn't be valid. Only b & c are valid On Mon, Dec 6, 2021, 12:44 PM Mike Drob wrote: > Ilan, > > Can you provide a

Re: First class support for node roles

2021-12-05 Thread Mike Drob
Ilan, Can you provide a more detailed concrete example? I’m having a lot of trouble understanding what you are proposing, beyond that it is somehow contraindicated with what Ishan/Noble suggest. Apologies for my failure to understand. Thanks, Mike On Sun, Dec 5, 2021 at 5:21 PM Ilan Ginzburg

Re: First class support for node roles

2021-12-05 Thread Noble Paul
On Mon, Dec 6, 2021, 10:21 AM Ilan Ginzburg wrote: > If we go with optional role params, we need two defaults: > 1. the param value to use when the role is specified without a parameter, > and > 2. the param value to use for the role on a node for which the role is > not specified at all. > > I

Re: First class support for node roles

2021-12-05 Thread Ilan Ginzburg
If we go with optional role params, we need two defaults: 1. the param value to use when the role is specified without a parameter, and 2. the param value to use for the role on a node for which the role is not specified at all. I don't know how to sensibly name these defaults, but the actual

Re: First class support for node roles

2021-12-04 Thread Ishan Chattopadhyaya
I'm +1 on this. It "looks" complicated at first, but simplifies all headaches going forward. On Sun, Dec 5, 2021 at 11:46 AM Noble Paul wrote: > I shall update the SIP proposal if we have a consensus on this > configuration > > On Sun, Dec 5, 2021 at 4:58 PM Noble Paul wrote: > >> >> >> On

Re: First class support for node roles

2021-12-04 Thread Noble Paul
I shall update the SIP proposal if we have a consensus on this configuration On Sun, Dec 5, 2021 at 4:58 PM Noble Paul wrote: > > > On Sun, Dec 5, 2021 at 4:47 PM Gus Heck wrote: > >> I like this in that it's an example of how the overseer might be extended >> without creating a new role :) >>

Re: First class support for node roles

2021-12-04 Thread Noble Paul
On Sun, Dec 5, 2021 at 4:47 PM Gus Heck wrote: > I like this in that it's an example of how the overseer might be extended > without creating a new role :) > > Not entirely sure if I'm for or against an enum implementation here, but > it makes me a bit nervous. Enums with complexity can quickly

Re: First class support for node roles

2021-12-04 Thread Noble Paul
typo - On Sun, Dec 5, 2021 at 2:37 PM Noble Paul wrote: > I recommend the following format for the role spec > > roles=: > > each role will have an enum of allowed values and a default value > > >- role name: *data* > - values: [*on*, *off]* > - default: *allowed* > >

Re: First class support for node roles

2021-12-04 Thread Gus Heck
On Sat, Dec 4, 2021 at 7:01 PM Ilan Ginzburg wrote: > If we go with no negative node roles and overseer node role is not strict > (i.e. it’s a "preferred overseer"), then one would need to define a second > node role "no_overseer" to explicitly exclude a node from ever becoming > overseer (which

Re: First class support for node roles

2021-12-04 Thread Noble Paul
I recommend the following format for the role spec roles=: each role will have an enum of allowed values and a default value - role name: *data* - values: [*on*, *off]* - default: *allowed* - role name: *overseer* - values: [*allowed*, *disallowed*, *preferred]* -

Re: First class support for node roles

2021-12-04 Thread Ilan Ginzburg
If we go with no negative node roles and overseer node role is not strict (i.e. it’s a "preferred overseer"), then one would need to define a second node role "no_overseer" to explicitly exclude a node from ever becoming overseer (which I think is a useful feature until we switch the cluster

Re: First class support for node roles

2021-12-03 Thread Gus Heck
I really don't think we should have types of roles. Not negative/positive and not strict/non-strict. You have a role or you don't. What that means is up to the code implementing the role. Roles should be free to configure a preference order (binary, or n-ary or whatever, strict or loose),

Re: First class support for node roles

2021-12-02 Thread Noble Paul
Negative roles have a place Example is overseer There are 3 possible choices for that role a) preferred: always be in front of the election queue b) on: not preferred, but can be an overseer if no preferred overseer nodes are available c) off: never become an overseer Today we only have

Re: First class support for node roles

2021-12-02 Thread Mike Drob
Negative roles add a lot of complexity, I would really want to stay away from them. That’s why I want strict roles up front. It’s maybe ok to push this decision out, but it also seems like the sort of thing we should consider at the start. On Thu, Dec 2, 2021 at 5:52 PM Noble Paul wrote: > Yes.

Re: First class support for node roles

2021-12-02 Thread Ilan Ginzburg
If we have non strict roles (like overseer), then it does make sense to have negative roles. That way I can define which are the two nodes that I'd prefer the overseer to run on, and a few other nodes on which it should definitely never run for various reasons. And in case these "!overseer" are

Re: First class support for node roles

2021-12-02 Thread Houston Putman
> > With the Strict/Loose option and sensible defaults, users cannot trip >> themselves up by default, but the option is there for people to tinker and >> have an iron grip over their cluster. >> > > +1 to sensible defaults so users don't trip themselves. The option to > tinker for tighter grip

Re: First class support for node roles

2021-12-01 Thread Ishan Chattopadhyaya
On Thu, Dec 2, 2021 at 1:31 AM Gus Heck wrote: > I think the key is to let the roles have full control of the implications > of having/not having that role. No need for even a strict/loose > designation. The question of do you have the role is yes/no with no logic > to guess if the role is

Re: First class support for node roles

2021-12-01 Thread Ishan Chattopadhyaya
On Thu, Dec 2, 2021 at 1:20 AM Houston Putman wrote: > This doesn't really address my concern around what happens if all of our >> existing OVERSEER candidates are down. When at least one of them is up, the >> overseer will go there, and that is good and expected. But what happens if >> all of

Re: First class support for node roles

2021-12-01 Thread Ishan Chattopadhyaya
On Thu, Dec 2, 2021 at 12:54 AM Mike Drob wrote: > Noble wrote: > > We are not modifying the way the "overseer role" works today. We are > just changing the definition and standardizing the configuration & > discoverability > Ishan wrote: > > As of this SIP, we're not planning to modify the

Re: First class support for node roles

2021-12-01 Thread Gus Heck
argh bad edit... The question of will it come up with the role is "have_explicit ? use_explicit : use_defaults. On Wed, Dec 1, 2021 at 3:01 PM Gus Heck wrote: > I think the key is to let the roles have full control of the implications > of having/not having that role. No need for even a

Re: First class support for node roles

2021-12-01 Thread Gus Heck
I think the key is to let the roles have full control of the implications of having/not having that role. No need for even a strict/loose designation. The question of do you have the role is yes/no with no logic to guess if the role is implied or not, The question of will it come up with the role

Re: First class support for node roles

2021-12-01 Thread Houston Putman
> > This doesn't really address my concern around what happens if all of our > existing OVERSEER candidates are down. When at least one of them is up, the > overseer will go there, and that is good and expected. But what happens if > all of the overseer eligible nodes are down. Your comment, and

Re: First class support for node roles

2021-12-01 Thread Mike Drob
Noble wrote: > We are not modifying the way the "overseer role" works today. We are just changing the definition and standardizing the configuration & discoverability Ishan wrote: > As of this SIP, we're not planning to modify the OVERSEER role (which currently stands for preferred overseer). We

Re: First class support for node roles

2021-11-30 Thread Ishan Chattopadhyaya
On Tue, Nov 30, 2021 at 12:53 AM Mike Drob wrote: > Replying to the top post in this thread because there has been a lot of > discussion and I don't want to look like I'm continuing any of those > particular threads. > > I finally had time to sit down and think about this with the attention it >

Re: First class support for node roles

2021-11-30 Thread Ishan Chattopadhyaya
On Tue, Nov 30, 2021 at 12:53 AM Mike Drob wrote: > Replying to the top post in this thread because there has been a lot of > discussion and I don't want to look like I'm continuing any of those > particular threads. > > I finally had time to sit down and think about this with the attention it >

Re: First class support for node roles

2021-11-30 Thread Ishan Chattopadhyaya
Hi Mike, Thanks for your thoughtful responses and the structured manner of your composing this mail (with clear labels as to your intent) is a refreshing change. I think we all should learn from this in future communication. On Tue, Nov 30, 2021 at 12:53 AM Mike Drob wrote: > Replying to the

Re: First class support for node roles

2021-11-29 Thread Gus Heck
> > This brings up another point, I’d like to see if we can somehow collapse > overseer election with overseer role advertising I would be strongly in > favor of that. > It would seem natural to move it into this structure, I agree, though that might be done in a follow on ticket. > > Or Zk

Re: First class support for node roles

2021-11-29 Thread Mike Drob
On Mon, Nov 29, 2021 at 2:40 PM Gus Heck wrote: > > >> CLARIFICATION: I do not like that we are storing node liveness in two >> different places now. We have the live nodes and we have the node roles >> stored in two different places in zookeeper and it feels like this would >> lead to race

Re: First class support for node roles

2021-11-29 Thread Mike Drob
Replying to the top post in this thread because there has been a lot of discussion and I don't want to look like I'm continuing any of those particular threads. I finally had time to sit down and think about this with the attention it deserves and am generally happy with how the conversation has

Re: First class support for node roles

2021-11-25 Thread Noble Paul
+1 On Thu, Nov 25, 2021, 6:02 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I've started a formal vote thread for the proposal here. Thanks all. > > On Thu, Nov 25, 2021 at 12:10 PM Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >> We have closed the Google Doc for

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
I've started a formal vote thread for the proposal here. Thanks all. On Thu, Nov 25, 2021 at 12:10 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > We have closed the Google Doc for further comments, and all changes have > been incorporated into the confluence document: >

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
We have closed the Google Doc for further comments, and all changes have been incorporated into the confluence document: https://cwiki.apache.org/confluence/display/SOLR/SIP-15+Node+roles Thanks for all the comments and feedback. On Thu, Nov 25, 2021 at 12:09 PM Ishan Chattopadhyaya <

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
> Maybe one way to provide for the future is to place some requirements on roles, and designate that role-specific coordination information > be placed within the subtree for that role under /node-roles// To facilitate this we could > 1. move the currently designed set of ephemeral nodes down one

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Thu, Nov 25, 2021 at 4:35 AM Gus Heck wrote: > Things I think still need to be clarified to define "Role" explicitly. > Primarily to guide future implementations. Things I can think of include: > >- What the bar for something to be a role vs being more appropriate as >a property/tag

Re: First class support for node roles

2021-11-24 Thread Noble Paul
On Thu, Nov 25, 2021 at 10:05 AM Gus Heck wrote: > Things I think still need to be clarified to define "Role" explicitly. > Primarily to guide future implementations. Things I can think of include: > >- What the bar for something to be a role vs being more appropriate as >a property/tag

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Thu, Nov 25, 2021 at 5:25 AM Jan Høydahl wrote: > Very good points to clarify Gus. > > Also, as mentioned in Google Doc, I think the role framework should > clarify the expectations for a node having a role. > Does it mean that the node is eligible for - "MAY" - a certain > feature/task, or

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
> Moreover I think it's a very role specific notion, that is not a good > fit in the roles framework, so maybe should never be part of this SIP. Regarding "capable" vs "currently providing", I agree with Ilan here ^ On Wed, Nov 24, 2021 at 10:38 PM Gus Heck wrote: > You do make a good point

Re: First class support for node roles

2021-11-24 Thread Gus Heck
You do make a good point that the details can be very role specific. Maybe one way to provide for the future is to place some requirements on roles, and designate that role-specific coordination information be placed within the subtree for that role under /node-roles// To facilitate this we could

Re: First class support for node roles

2021-11-24 Thread Gus Heck
So we are potentially talking past each other. Let me re-clarify since you've interpreted "configuration" differently (I think). What I'm referring to in ZK is the "reflection of configuration" not the actual configuration. This is why I'm asking the question of how much info should be retained

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Wed, Nov 24, 2021 at 9:05 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > > On Wed, Nov 24, 2021 at 8:19 PM Gus Heck wrote: > >> Left some additional comments in the google doc, extracting the key point >> for others here so they can know if they want to read the google doc >>

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Wed, Nov 24, 2021 at 8:19 PM Gus Heck wrote: > Left some additional comments in the google doc, extracting the key point > for others here so they can know if they want to read the google doc > commentary: > I'm addressing them here, inline. > There is presently disagreement with my

Re: First class support for node roles

2021-11-24 Thread Gus Heck
Left some additional comments in the google doc, extracting the key point for others here so they can know if they want to read the google doc commentary: There is presently disagreement with my proposal that the sip specify both how we track configuration (which I have been referring to as

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
I've synced back all changes to confluence. Both represent the same document as of now. Please review and suggest if there are any outstanding concerns. Thanks for all the feedback. I wish to proceed with the implementation based on lazy consensus again. On Tue, Nov 23, 2021 at 9:38 PM Mike Drob

Re: First class support for node roles

2021-11-23 Thread Mike Drob
I should clarify, that my -1 was specifically about reaching lazy consensus and not about that proposal itself. On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > -1, I would like to see a proposal on list where it will be permanently > available > > It

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
> -1, I would like to see a proposal on list where it will be permanently available > in the archives instead of being directed to a Google doc which can be edited at any point in time. > This has been an incredibly difficult proposal to keep track of. This entire SIP process has been incredibly

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
Okay, sure, I'll rescind my assertion about having reached a lazy consensus. I'll work through all the recent comments and feedback. I think at the last moment, there was an edit by Noble to the Google Doc that hasn't made it into the SIP document, and I'll review that change, the recent comments

Re: First class support for node roles

2021-11-23 Thread Gus Heck
Also, I don't know that it's defined anywhere, but I feel that declaring lazy consensus on a SIP in less than a week is rushing things. Some folks won't have the flexibility to address things daily and I've been uncomfortably pulled away from paid work trying to keep up with this as is. Lazy

Re: First class support for node roles

2021-11-23 Thread Gus Heck
IMHO two things must happen before lazy consensus wait period can begin: 1) The wiki must be updated to reflect the results of discussion in the google doc. 2) The fact that the wiki has been updated, and that you think the discussion is at an end must be posted here, preferably with a summary

Re: First class support for node roles

2021-11-23 Thread Mike Drob
-1, I would like to see a proposal on list where it will be permanently available in the archives instead of being directed to a Google doc which can be edited at any point in time. This has been an incredibly difficult proposal to keep track of. Mike On Tue, Nov 23, 2021 at 6:09 AM Ishan

Re: First class support for node roles

2021-11-23 Thread Jan Høydahl
There is still discussion and disagreement in the Google doc. The Google doc proposes that zk role structure is 1:1 for all roles -- except the "data" role. I assume the reason for that choice, even if not spelled out in the doc, is to optimize for large clusters with 1000 nodes, and thus

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
This proposal has passed with lazy consensus. We can proceed to the implementation phase. Thanks to everyone for feedback, esp. Gus for the patience. On Sun, Nov 21, 2021 at 2:24 PM Gus Heck wrote: > Thanks for adding that :) and Thanks to Nobel for translating my ravings > :). Sorry about the

Re: First class support for node roles

2021-11-21 Thread Gus Heck
Thanks for adding that :) and Thanks to Nobel for translating my ravings :). Sorry about the acronym. Amusingly I had to look up gish gallop to understand what you meant :) I can assure you that I will never gish gallop you or anyone on the list, and I find such techniques repugnant. If I'm not

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
Added both your suggestions to Google Docs for inline commenting and the SIP document at confluence. Thanks for the feedback. On Sun, Nov 21, 2021 at 10:28 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I'll add the lifecycle details to the document as well. > > On Sun, 21 Nov,

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
I'll add the lifecycle details to the document as well. On Sun, 21 Nov, 2021, 9:08 am Ishan Chattopadhyaya, < ichattopadhy...@gmail.com> wrote: > Hi Gus, > Thanks for bringing it up again. Initially, I wasn't clear what you meant > and mistook them for gish gallop. > > These were the things I

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
Hi Gus, Thanks for bringing it up again. Initially, I wasn't clear what you meant and mistook them for gish gallop. These were the things I am/was confused about (Noble explained it to me what you mean): - adherence to DRY via zk - runtime config info should come from zk - add dynamic features -

Re: First class support for node roles

2021-11-20 Thread Gus Heck
It is difficult to track everything in this thread for sure. Specifically, I don't feel my email of Wed, Nov 17, 7:43 PM (EST) has been addressed or responded to except for one difference of opinion with Jan on whether or not we should prohibit the notion of any role ever changing at runtime.

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, wrote: > Is the request here for everybody to express again the concerns > already expressed in this email thread and not addressed? > I think the expectation now (after we've expressed our intention to reach a lazy consensus) is that if someone

Re: First class support for node roles

2021-11-19 Thread Ilan Ginzburg
Is the request here for everybody to express again the concerns already expressed in this email thread and not addressed? I suggest instead the authors review the thread, match expressed concerns with how the concern was addressed (or not addressed) and provide an exhaustive list. This proposal

Re: First class support for node roles

2021-11-18 Thread Noble Paul
After so many back and forth mails, I just can't say who has an outstanding concern and if they are already addressed or not. I think the Google doc would help us get clarity on that. Please take a moment to give your inputs On Fri, Nov 19, 2021, 9:18 AM Ishan Chattopadhyaya <

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
Apologies, the vote hasn't passed formally and I was under some confusion on the process. I'd like to proceed with a lazy consensus and proceed to the implementation phase now. However, I would appreciate it if someone wants to bring out any outstanding concerns about the SIP document. To

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
The SIP passed the voting phase. Thanks for all for the feedback and insights. Looking forward to your collaboration and reviews as we implement this. On Thu, Nov 18, 2021 at 9:42 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > It's fine if we don't provide any ability for

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
> It's fine if we don't provide any ability for runtime modification of roles at this time but I'm leery of precluding it in the future. In future, the necessity for such a facility can dictate our course of action. We cannot lay down rules cast in stone for functionality that we can't foresee

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
Thanks Jan, I added both those points to the SIP document in the Notes section. On Thu, Nov 18, 2021 at 7:18 PM Jan Høydahl wrote: > 18. nov. 2021 kl. 01:43 skrev Gus Heck : > > > 2) Roles will not be checked by loading config from disk or caching disk > config in memory. (zk ONLY source of

Re: First class support for node roles

2021-11-18 Thread Gus Heck
Code may want to know the role of other nodes (i.e. Someone just sent me a request to create a collection, where can I put it?). DRY... one source of truth is all I'm after. Someone changes the file on disk before restarting, we shouldn't see the change locally but not see it globally, etc. It's

Re: First class support for node roles

2021-11-18 Thread Jan Høydahl
18. nov. 2021 kl. 01:43 skrev Gus Heck : > > 2) Roles will not be checked by loading config from disk or caching disk > config in memory. (zk ONLY source of truth) > It sounds a bit backward for a local node to first parse solr.node.roles, determine its local set of roles, then publish them

Re: First class support for node roles

2021-11-17 Thread Noble Paul
On Thu, Nov 18, 2021 at 6:15 AM Ilan Ginzburg wrote: > Current Overseer role defaults to all nodes can, and that changes only > when some nodes get the Overseer role. > The current "overseer" role means "preferred-oveerseer". It was implemented as such because if preferred overseers are not

Re: First class support for node roles

2021-11-17 Thread Gus Heck
ack meant to write "default +zookeep4r -overseer" any config should be explicit. On Wed, Nov 17, 2021 at 7:43 PM Gus Heck wrote: > I think we should establish a known set of default roles. Other roles are > non-default roles. New features can be added as default or non-default as > required.

Re: First class support for node roles

2021-11-17 Thread Gus Heck
I think we should establish a known set of default roles. Other roles are non-default roles. New features can be added as default or non-default as required. Default roles are explicitly loaded at startup if no other information is provided. I'd even be fine with a config that said +zookeeper

Re: First class support for node roles

2021-11-17 Thread Ilan Ginzburg
Current Overseer role defaults to all nodes can, and that changes only when some nodes get the Overseer role. This is a default value per node based on the presence of the role on any node of the cluster. But the Overseer role is "weak" in that nodes not having the role can become overseer even

Re: First class support for node roles

2021-11-17 Thread Ishan Chattopadhyaya
On Wed, Nov 17, 2021 at 8:12 PM Jan Høydahl wrote: > I think your VOTE is premature as several design decisions are obviously > not landed. That may be the reason there are no votes yet, and I'm not > going to vote either. > > >1. If "-Dsolr.node.roles" parameter is not passed, it is

Re: First class support for node roles

2021-11-17 Thread Jan Høydahl
I think your VOTE is premature as several design decisions are obviously not landed. That may be the reason there are no votes yet, and I'm not going to vote either. > If "-Dsolr.node.roles" parameter is not passed, it is implicitly assumed to > be "-Dsolr.nodes.role=data" (due to backcompat

Re: First class support for node roles

2021-11-15 Thread Ishan Chattopadhyaya
Thanks Jan, I've updated the SIP document with all the applicable changes with a link to this thread (which contains the summary at the end). I'll initiate the vote thread now. Thanks to everyone for contributing. On Mon, Nov 15, 2021 at 6:53 PM Jan Høydahl wrote: > Thanks for trying to

Re: First class support for node roles

2021-11-15 Thread Jan Høydahl
Thanks for trying to summarize and drive the work Ishan. I'd like to add Scope of SIP Ishan: Role API and config Jan: Role API, config, and impact of one real role e.g. the "data" role, to examplify and justify the role infrastructure According to SIP process the next step is not

Re: First class support for node roles

2021-11-14 Thread Ishan Chattopadhyaya
Thanks to everyone for the feedback. Here's an attempt to summarize broad topics discussed. *No negative roles* Everyone agree *Roles on/off by default?* Jason+(Ilan,Houston?): All roles to be on by default Gus,Ishan,Noble: Only those roles to be on by default that are needed for backcompat

Re: First class support for node roles

2021-11-12 Thread Gus Heck
Yeah we should only be looking for and only be reporting (if we choose to report to the user) a specific set of env variables. Anything else should be ignored.Should be an enum or constants somewhere listing what solr cares about, and we should ignore or be blind to anything else. Perhaps we'd

Re: First class support for node roles

2021-11-11 Thread Ilan Ginzburg
Houston made a very valid comment back then on the placement plugin support of environment variables (dropped as a consequence). https://issues.apache.org/jira/browse/SOLR-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286680#comment-17286680 It could be possible

Re: First class support for node roles

2021-11-11 Thread Eric Pugh
Agreed! I’ve noticed that in the Play Framework, you can configure everything via a property based configuration file, however it makes it easy to override the property file via another one, or via an ENV variables: db.default.username="smui" db.default.username=${?SMUI_DB_USER} Which

Re: First class support for node roles

2021-11-11 Thread Jan Høydahl
+1 to a roundup of env and props across the board. I think SIP 11 is on the track of something. But can be done independent of this. Jan Høydahl > 11. nov. 2021 kl. 17:44 skrev Gus Heck : >  > I guess all I mean is that it shouldn't be only sysprops. Enabling sysprops, > Env vars etc seems

Re: First class support for node roles

2021-11-11 Thread Gus Heck
I guess all I mean is that it shouldn't be only sysprops. Enabling sysprops, Env vars etc seems fine but we need to clearly document precedence among any/all options. What is convenient varies from case to case and in a perfect world what I'd like to see is full support across each style (files,

Re: First class support for node roles

2021-11-11 Thread Houston Putman
I agree with Jan, when thinking about making Solr as cloud friendly as possible EnvVars and (to a lesser extent) sysProps are much preferable than having a setting in the solr.xml. This is because it's easier to customize EnvVars per-node, while customizing a config file is much harder, as those

Re: First class support for node roles

2021-11-05 Thread Jan Høydahl
Thinking of these roles as labels, I think sysProps and envVars are the two universal methods, and nothing wrong with that. I keep trying to think cloud native and container, so having excellent 1st class support for env.vars for such configs is a priority to me. Most tools, CI-environments etc

  1   2   >