Thanks a lot for the elaborated reply. It helped a lot in getting familiar with Drools by running samples as you've pointed. And I've built the code base.
After going through scaling.drl (products/autoscaler/modules/distribution/src/main/conf/scaling.drl) it was clear that currently stratos uses RuleTasksDelegator.getPredictedValueForNextMinute() method to compare, stat values against the thresholds. *Approach on deciding the number of instances that might need to handle the load:* Using existing method on predicting next minute Requests inflight, Load average and Memory Consumption. - Assumption: current thresholds of those metrics are the optimal values for an instance. - Based on that implementing a simple algorithm to decide, how many number of instances that might need for the next minute using predicted values for those metrics. - That algorithm will be implemented in such a way that it always will keep the instances under thresholds (or near thresholds ) of one or more metrics , with out exceeding them. - Assumption : metrics act inverse or direct proportionally when instances are spawned. (for an ex. load is equally distributed among all the instances + newly spawned instances. ) *Predict the load according to a schedule defined by end user * *Does this mean providing a functionality in web UI to define a schedule and make it active? *It's not clear to me. *Can this be achieved by generating an auto scale policy xml with user defined thresholds similar to how it's done currently and making it possible to override the *auto-scaling* algorithm in use when it's needed (like in a specific time *which is already defined) ? . Thanks Asiri On Wed, Mar 12, 2014 at 8:05 AM, Lahiru Sandaruwan <lahi...@wso2.com> wrote: > Hi Asiri, > > It is a pleasure to see your interest. Sorry for the late reply. I missed > the mail. > > Get the code base and build as a starting point for Stratos. > > You will not find Drools hard, after running some samples. [1] looks like > a good sample. You can just run those in WSO2 BRS. You can use your Java > knowledge as we can write Java code in "then" section. > > AMQP knowledge means you have to understand pub/sub model with topics. > Conceptually thats it. In addition, handling subs/pubs using java codes. > > Great research, find the comments inline. > > > On Tue, Mar 11, 2014 at 11:23 AM, Asiri Liyana Arachchi < > asiriw...@gmail.com> wrote: > >> 1. Improve Auto-scaling to predict the number of instances required in >> the next time interval. >> >> As far as I understood, this project aims at introducing a new auto >> scaling strategy apart from the threshold based auto scaling which is >> currently in use, to stratos making it more proactive on auto-scaling. >> > > Correct. So system should scale, understanding the load and hence the > number of instances that would require to handle that load. > > We have 3 types of information about load, and should consider all 3 for > our decision. > > - Requests inflight(Information about how many requests are waiting to > get the response) > - Load average of cartridge instances running > - Memory consumption of cartridge instances running > > > To do that there are several strategies suggested. >> >> 1. Kalman Filter >> 2. Control theory >> 3. Time Series Analysis. >> 4. FFT >> >> As I've gone through these techniques as for now I felt that Kalman >> Filter would be the most viable candidate and it can be used to address >> this issue effectively.There is an apache API for Kalman filter [1]. >> > > We should find an efficient, yet simplest way to get the job done. We > currently use S = u*t + 0.5 *a*t*t prediction(motion) equation. This is one > of the equations that Kalman filter used to do prediction. But with this, > we have to compare with a threshold to take the decision. > > We receive second derivative, gradient and average values at a given time. > Lets say we time interval we consider is minute. So we can predict the load > in the next minute using them. > Also we know the number of instances that are running at the moment. The > algorithm does not need to be complex. It should be just intelligent enough > to find the matching number of instances that should be there in the next > minute. > > [1] https://docs.wso2.org/display/BRS200/Sample+Rule+Definition > > Thanks. > >> >> But I think selecting an auto scaling algorithm would involve more of >> research and testing. Even selecting metrics to predict on will also be >> challenging because some of the metrics for an example *load average *depends >> on autos-scalling causing predictions to deviate from the actual values. >> > I would appreciate if you can comment on this. >> >> [1] : >> http://commons.apache.org/proper/commons-math/apidocs/org/apache/commons/math3/filter/KalmanFilter.html >> >> >> >> Thanks >> Asiri >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Mar 6, 2014 at 7:38 AM, Udara Liyanage <ud...@wso2.com> wrote: >> >>> Hi Asiri, >>> >>> Glad to hear your interest on Stratos. I don't think it will take more >>> than few days to learn drools and amqp. You will be able to do it within >>> given time period. >>> Happy to see your project proposal soon. >>> >>> Touched, not typed. Erroneous words are a feature, not a typo. >>> On Mar 6, 2014 7:13 AM, "Asiri Liyana Arachchi" <asiriw...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> I'm Asiri Liyana Arachchi , third year student studying Computer >>>> Science and Engineering in University of Moratuwa , Sri Lanka. >>>> I would like to start contributing towards the project $subject .I've >>>> gone through the resources about this project including stratos >>>> documentation and the code-base. >>>> >>>> As expected I'm familiur with java , json and SOA . I would like to >>>> know how well and in what cases Drools and APQM skills are required. Also >>>> would it be feasible to complete the project in the projects limited time, >>>> considered that the Drools and APQM are to be learnt along with the total >>>> work of the project. >>>> >>>> >>>> Thanks. >>>> >>>> Asiri >>>> >>>> >> > > > -- > -- > Lahiru Sandaruwan > Software Engineer, > Platform Technologies, > WSO2 Inc., http://wso2.com > lean.enterprise.middleware > > email: lahi...@wso2.com cell: (+94) 773 325 954 > blog: http://lahiruwrites.blogspot.com/ > twitter: http://twitter.com/lahirus > linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 > >
