Re: Approach for a new Autoscaling framework

Good 'ol software engineering practices is certainly the core of what I intend, though I am also raising the question of whether or not we want to consign some defined set of things that plug-in to the far side of those API's, and whether or not that entails a more explicit notion of what

Re: Approach for a new Autoscaling framework

To everyone and especially Gus: I think the "plugin" word in this thread is basically a stop-word to the intent/scope of the thread. A plugin to Solr both has been and will be nothing more than a class that's loaded *dynamically* by a configurable name -- as opposed to a class within Solr that

Re: Approach for a new Autoscaling framework

On Sun, Jul 26, 2020 at 1:05 AM Ilan Ginzburg wrote: > Varun, you're correct. > This PR was built based on what's needed for creation (easiest starting > point for me and likely most urgent need). It's still totally WIP and > following steps include building the API required for move and other >

Re: Approach for a new Autoscaling framework

"There's value in starting simple, understanding the tradeoffs and generalizing later" Yes, this is what I am alluding to when I said the facade should "grow and expand such that more plugins can rely on it". One plugin that might re-use some of the same informational API's is the

Re: Approach for a new Autoscaling framework

Varun, you're correct. This PR was built based on what's needed for creation (easiest starting point for me and likely most urgent need). It's still totally WIP and following steps include building the API required for move and other placement based needs, then also everything related to triggers

Re: Approach for a new Autoscaling framework

Hi Ilan, I like where we're going with https://github.com/apache/lucene-solr/pull/1684 . Correct me if I am wrong, but my understanding of this PR is we're defining the interfaces for creating policies What's not clear to me is how will existing collection APIs like

Re: Approach for a new Autoscaling framework

Thanks Gus! This makes a lot of sense but significantly increases IMO the scope and effort to define an "Autoscaling" framework interface. I'd be happy to try to see what concepts could be shared and how a generic plugin facade could be defined. What are the other types of plugins that would

Re: Approach for a new Autoscaling framework

Scanned through the PR and read some of this thread. I likely have missed much other discussion, so forgive me if I'm dredging up somethings that are already discussed elsewhere. The idea of designing the interfaces defining what information is available seems good here, but I worry that it's too

Re: Approach for a new Autoscaling framework

> Not clear to me what type of "alternative proposal" you're thinking of Jan That would be the responsibility of Noble and others who have concerns to detail - and try convince other peers. It’s hard for me as a spectator to know whether to agree with Noble without a clear picture of what the

Re: Approach for a new Autoscaling framework

In my opinion we have to (and therefore will) ship at least a basic prod ready implementation on top of the API that does simple things (not sure about rack, but for example balance cores and disk size without co locating replicas of same shard on same node). Without such an implementation, I

Re: Approach for a new Autoscaling framework

Important discussion indeed. I don’t have time to dive deep into the PR or make up my mind whether there is a simpler and more future proof way of designing these APIs. But I understand that autoscaling is a complex beast and it is important we get it right. One question regarding having to

Re: Approach for a new Autoscaling framework

I think this is a valid thing to discuss on the dev list, since this isn't just about code comments. It seems to me that Ilan wants to discuss the philosophy around how to design plugins and the interfaces in Solr which the plugins will talk to. This is broad and affects much more than just the

Re: Approach for a new Autoscaling framework

I think we should move the discussion back to the PR because it has more context and inline comments are possible. Having this discussion in 4 places (jira, pr, slack and dev list is very hard to keep track of). On Thu, 23 Jul, 2020, 5:57 pm Ilan Ginzburg, wrote: > [I’m moving a discussion from

Approach for a new Autoscaling framework

[I’m moving a discussion from the PR for SOLR-14613 to the dev list for a wider audience. This is about replacing the now (in master) gone Autoscaling framework with a way for clients to write