Thx a lot KK. It's a great improvement to come.
As I reported this issue also a long long time ago I'll be happy to test it. My new infrastructure is using the .deb of jenkins. How can I build it from your branch to give it a try ? Arnaud On Thu, Sep 20, 2012 at 9:16 PM, Kohsuke Kawaguchi <[email protected] > wrote: > > As any serious Jenkins users would know, a big Jenkins instance takes a > long time to start up. One of the problems here is that it loads every > build record before it starts accepting HTTP requests, and therefore a > natural solution to this problem is to try to lazy load build records > on-demand. > > I've been working on this in and out, but I finally got to a milestone. > And incidentally I got contacted by Christian Wolfgang about this, as this > very topic was discussed in the Copenhagen hackathon [3]. > > My changes are in the lazy-load branch [1]. You can also see the commits > made in this branch [2] that aren't in master. > > > > What's already done > =================== > > RunMap is conceptually a map from build number to Run, and Runs form a > bi-directional linked list between their adjacent neighbors. So the first > step was to turn RunMap into a virtual map. That is, it loads each build > records on-demand as requested. Ditto to the bi-directional linking between > Runs. > > What makes this interesting is that build records are keyed by their > timestamps on disk, not by numbers (and numbers are available as symlinks > but not in all the platforms.) So RunMap uses a binary search to locate the > build that's currently needed. For this to work, I made an assumption that > for build #M and #N. M>N iff M.timestamp>N.timestamp. This process also > incorporates symlinks as a hint to speed up the search if they are > available. > > Much of this logic is in AbstractLazyLoadRunMap, which has no dependencies > to the rest of Jenkins core. This was so that I can test this class quickly > and efficiently without using HudsonTestCase (and it's because historically > I developed this code outside core.) See its search method that is the > heart of this logic. > > RunMap was then modified to inherit this class to provide this necessary > behaviour. > > > The next step was to make Job/AbstractProject take advantages of this. The > Job class expects subtypes to provide the _getRuns method that returns > SortedMap of the builds, and all the other methods on Job that deal with > obtaining build records work on this map. Some of these methods are > overriden in AbstractProject to take advantages of RunMap. > > > I then proceeded to update RunList, which is a class that's used to list > up builds that satisfy a certain criteria. In the master, this class > extends from ArrayList and every time a new filter is applied, all builds > that satisfy the criteria gets copied into this array. This doesn't work > very well with lazy loading, so I modified this class to only lazily walk > through the builds and pick up ones that satisfy the criteria. > > For example, a typical use case of this class is "take all the builds of a > job, narrow it down those that have failed, then list up first 10, render > it to RSS". The new lazy implementation works very well with this. > > Unfortunately, to make this work, I needed to make a signature breaking > change --- the class now extends from AbstractList and not from ArrayList. > I did scan the source code of all the plugins to see their use of RunList, > and I didn't spot anywhere it's casted to ArrayList, so I think the impact > of this would be small. > > > And at this point, it passes all the unit tests. > > > > What needs to be done > ===================== > Clearly more testing needs to be done. Code coverage of > AbstractLazyLoadRunMap is actually pretty good, but the logic is complex. > > I also need to try this with a real Jenkins instance, mainly to see if > there's some code in Jenkins or plugins that tries to eagerly load all the > build records of all the jobs. > > To help us find this, we probably need some logging that tells us who's > loading build records when. > > I'll try this with some real Jenkins deployments, and when that's ready, > I'd like to merge this to the master. If anyone is willing to give this a > shot before it hits the master, I'd highly appreciate that. > > > > The next goal after that is to deal with places where someone tries to > load all the build records of one job. Unfortunately, this happens today in > numerous places. Just in the job top page, test report trend or code > coverage trend will cover the entire build history. So this will be a slow > process. More about this later. > > > > WDYT? > > > [1] > http://github.com/jenkinsci/**jenkins/tree/lazy-load<http://github.com/jenkinsci/jenkins/tree/lazy-load> > [2] > https://github.com/jenkinsci/**jenkins/compare/master...lazy-**load<https://github.com/jenkinsci/jenkins/compare/master...lazy-load> > [3] http://wiki.praqma.net/**jcicodecamp12/sessions/** > improve-start-up-time<http://wiki.praqma.net/jcicodecamp12/sessions/improve-start-up-time> > -- > Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ > Try Nectar, our professional version of Jenkins > -- ----- Arnaud Héritier 06-89-76-64-24 http://aheritier.net Mail/GTalk: [email protected] Twitter/Skype : aheritier
