Hi all,
Since number of executors are dependent on resources[1] we have to come up
with a new strategy. If we are going with a priority based method, It seems
like we could use a LB plugin("Scoring Load Balancer plugin"[2][3] or
"Least Load Plugin"[4]) to select the best slave node and the Priority
Sorter Plugin[5] to prioritize the jobs.[1] http://stackoverflow.com/questions/9626899/jenkins-maximum-number-of-concurrent-jobs [2] https://wiki.jenkins-ci.org/display/JENKINS/Scoring+Load+Balancer+plugin [3] http://stackoverflow.com/questions/25900538/can-jenkins-nodes-be-given-priority-for-build-jobs [4] https://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin [5] https://wiki.jenkins-ci.org/display/JENKINS/Priority+Sorter+Plugin Thanks, Samith On Mon, May 4, 2015 at 5:02 PM, Dimuthu Leelarathne <[email protected]> wrote: > Hi all, > > This model will have a problem at app creation. Imagine the number of > executors we give are 10. At a time where all 10 executors are engaged what > will happen a person clicks on "CreateApplication". > > Then AF has to wait until at least one job completes and the person who > creates the application waits. This will be a major UX problem and not > acceptable. We need to work around it. > > Something that comes into my head is to have a priority method for jobs > that are relevant to app creation. Ok. The easiest would be to over > allocate the number of executors :D, but that is too dangerous and > impractical. > > thanks, > dimuthu > > On Wed, Apr 22, 2015 at 3:40 PM, Amila Maha Arachchi <[email protected]> > wrote: > >> +1 >> >> This looks clean and easy to scale. >> >> On Wed, Apr 22, 2015 at 1:26 PM, Samith Dassanayake <[email protected]> >> wrote: >> >>> Hi all, >>> I have started to working on $subject [1] problem definition as proposed >>> solution as below. >>> >>> Problem >>> Jenkins remains one of App Cloud weakest links: it is not highly >>> available, does not scale well, and increasingly fails with OOM. The >>> current deployment of Jenkins is not scalable. We give one Jenkins per >>> tenant. Therefore we have come with a more scalable model for jenkins in >>> App Factory. >>> >>> Proposed solution >>> Instead of having one jenkins per tenant, we will have jenkins as >>> an underlying service. According to the [2] and [3] distributing the >>> jobs among multiple jenkins(clusters) is the more scalable way for jenkins. >>> When >>> we create a job, the job will be created in the format of >>> "tenantDomain_appkey_version" and each tenant will have a shared maven repo. >>> But with growth of number tenants, number of jobs will be increased. >>> Therefore we have to come up with a job distributing mechanism with >>> Jenkins clusters. If we use a simple job distributing method, based on >>> the number of jenkins clusters, when number of clustered has to be >>> increased, we have to migrate all jobs according to their new cluster >>> values. As a solution for that, after discussing with Srinath, we came up >>> with a bucketing mechanism to do the job distribution. Solution would be >>> like below. >>> >>> >>> [image: Inline image 1] >>> >>> >>> We'll have pre-defined number of buckets(large value eg: 256). >>> Each bucket will deployed on one of the jenkins clusters and one jenkins >>> cluster will have multiple buckets. >>> During the job creation, we decide bucket id for the job, based on the >>> tenantDomain and number of buckets.(Therefore jobs with same tenantDomain >>> will always be stored in same bucket) >>> >>> ex: hash(tenantDomain) % number of buckets = bukcetId >>> >>> Therefore once the cluster size is increased we only have move the >>> relevant buckets, instead of redistributing all the jobs. >>> >>> Please let me know any suggestions/feedback. >>> >>> [1] https://redmine.wso2.com/issues/3757 >>> [1] >>> https://wiki.jenkins-ci.org/display/JENKINS/Consideration+for+Large+Scale+Jenkins+Deployment >>> [2] >>> http://www.slideshare.net/andrewbayer/7-habits-of-highly-effective-jenkins-users >>> [4] >>> http://soldering-iron.blogspot.com/2014/01/jenkins-performance-hints.html >>> >>> Thanks, >>> Samith >>> >>> -- >>> Best Regards >>> >>> Samith Dassanayake >>> Software Engineer | Cloud TG >>> WSO2, Inc. | http://wso2.com >>> lean. enterprise. middleware >>> >>> Mobile : +947 76207351 >>> >> >> >> >> -- >> *Amila Maharachchi* >> Senior Technical Lead >> WSO2, Inc.; http://wso2.com >> >> Blog: http://maharachchi.blogspot.com >> Mobile: +94719371446 >> >> > > > -- > Dimuthu Leelarathne > Architect & Product Lead of App Factory > > WSO2, Inc. (http://wso2.com) > email: [email protected] > Mobile : 0773661935 > > Lean . Enterprise . Middleware > -- Best Regards Samith Dassanayake Software Engineer | Cloud TG WSO2, Inc. | http://wso2.com lean. enterprise. middleware Mobile : +947 76207351
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
