Awesome! Thanks, Kohsuke! I will test it as soon as possible.
Christian On Fri, Sep 21, 2012 at 1:38 AM, Kohsuke Kawaguchi <[email protected]> wrote: > > http://kohsuke.org/private/20120920/jenkins.war > > I've deployed this exact binary on https://ci.jenkins-ci.org/ > > I just set up https://ci.jenkins-ci.org/job/jenkins_lazy_load/ so this > should provide up-to-date build. > > > On 09/20/2012 12:28 PM, Emanuele Zattin wrote: >> >> Great job Kohsuke! I'll try and get somebody in Nokia to test this if >> possible. >> >> Emanuele Zattin >> --------------------------------------------------- >> -I don't have to know an answer. I don't feel frightened by not knowing >> things; >> by being lost in a mysterious universe without any purpose — which is the >> way it >> really is, as far as I can tell, possibly. It doesn't frighten me.- >> Richard Feynman >> >> >> On Thu, Sep 20, 2012 at 9:22 PM, Arnaud Héritier <[email protected] >> <mailto:[email protected]>> wrote: >> >> Thx a lot KK. >> >> It's a great improvement to come. >> >> As I reported this issue also a long long time ago I'll be happy to >> test it. >> My new infrastructure is using the .deb of jenkins. >> How can I build it from your branch to give it a try ? >> >> Arnaud >> >> >> On Thu, Sep 20, 2012 at 9:16 PM, Kohsuke Kawaguchi >> <[email protected] >> <mailto:[email protected]>> wrote: >> >> >> As any serious Jenkins users would know, a big Jenkins instance >> takes a >> long time to start up. One of the problems here is that it loads >> every >> build record before it starts accepting HTTP requests, and >> therefore a >> natural solution to this problem is to try to lazy load build >> records >> on-demand. >> >> I've been working on this in and out, but I finally got to a >> milestone. >> And incidentally I got contacted by Christian Wolfgang about >> this, as >> this very topic was discussed in the Copenhagen hackathon [3]. >> >> My changes are in the lazy-load branch [1]. You can also see the >> commits >> made in this branch [2] that aren't in master. >> >> >> >> What's already done >> =================== >> >> RunMap is conceptually a map from build number to Run, and Runs >> form a >> bi-directional linked list between their adjacent neighbors. So >> the >> first step was to turn RunMap into a virtual map. That is, it >> loads each >> build records on-demand as requested. Ditto to the bi-directional >> linking between Runs. >> >> What makes this interesting is that build records are keyed by >> their >> timestamps on disk, not by numbers (and numbers are available as >> symlinks but not in all the platforms.) So RunMap uses a binary >> search >> to locate the build that's currently needed. For this to work, I >> made an >> assumption that for build #M and #N. M>N iff >> M.timestamp>N.timestamp. >> This process also incorporates symlinks as a hint to speed up the >> search >> if they are available. >> >> Much of this logic is in AbstractLazyLoadRunMap, which has no >> dependencies to the rest of Jenkins core. This was so that I can >> test >> this class quickly and efficiently without using HudsonTestCase >> (and >> it's because historically I developed this code outside core.) >> See its >> search method that is the heart of this logic. >> >> RunMap was then modified to inherit this class to provide this >> necessary >> behaviour. >> >> >> The next step was to make Job/AbstractProject take advantages of >> this. >> The Job class expects subtypes to provide the _getRuns method >> that >> returns SortedMap of the builds, and all the other methods on Job >> that >> deal with obtaining build records work on this map. Some of these >> methods are overriden in AbstractProject to take advantages of >> RunMap. >> >> >> I then proceeded to update RunList, which is a class that's used >> to list >> up builds that satisfy a certain criteria. In the master, this >> class >> extends from ArrayList and every time a new filter is applied, >> all >> builds that satisfy the criteria gets copied into this array. >> This >> doesn't work very well with lazy loading, so I modified this >> class to >> only lazily walk through the builds and pick up ones that satisfy >> the >> criteria. >> >> For example, a typical use case of this class is "take all the >> builds of >> a job, narrow it down those that have failed, then list up first >> 10, >> render it to RSS". The new lazy implementation works very well >> with this. >> >> Unfortunately, to make this work, I needed to make a signature >> breaking >> change --- the class now extends from AbstractList and not from >> ArrayList. I did scan the source code of all the plugins to see >> their >> use of RunList, and I didn't spot anywhere it's casted to >> ArrayList, so >> I think the impact of this would be small. >> >> >> And at this point, it passes all the unit tests. >> >> >> >> What needs to be done >> ===================== >> Clearly more testing needs to be done. Code coverage of >> AbstractLazyLoadRunMap is actually pretty good, but the logic is >> complex. >> >> I also need to try this with a real Jenkins instance, mainly to >> see if >> there's some code in Jenkins or plugins that tries to eagerly >> load all >> the build records of all the jobs. >> >> To help us find this, we probably need some logging that tells us >> who's >> loading build records when. >> >> I'll try this with some real Jenkins deployments, and when that's >> ready, >> I'd like to merge this to the master. If anyone is willing to >> give this >> a shot before it hits the master, I'd highly appreciate that. >> >> >> >> The next goal after that is to deal with places where someone >> tries to >> load all the build records of one job. Unfortunately, this >> happens today >> in numerous places. Just in the job top page, test report trend >> or code >> coverage trend will cover the entire build history. So this will >> be a >> slow process. More about this later. >> >> >> >> WDYT? >> >> >> [1] http://github.com/jenkinsci/__jenkins/tree/lazy-load >> <http://github.com/jenkinsci/jenkins/tree/lazy-load> >> [2] >> https://github.com/jenkinsci/__jenkins/compare/master...lazy-__load >> <https://github.com/jenkinsci/jenkins/compare/master...lazy-load> >> [3] >> >> http://wiki.praqma.net/__jcicodecamp12/sessions/__improve-start-up-time >> >> >> <http://wiki.praqma.net/jcicodecamp12/sessions/improve-start-up-time> >> -- >> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ >> Try Nectar, our professional version of Jenkins >> >> >> >> >> -- >> ----- >> Arnaud Héritier >> 06-89-76-64-24 >> http://aheritier.net >> Mail/GTalk: [email protected] <mailto:[email protected]> >> Twitter/Skype : aheritier >> >> > > > -- > Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ > Try Nectar, our professional version of Jenkins
