Done.
On 09/21/2012 02:09 AM, Baptiste MATHUS wrote:
That's an absolutely great news! Congrats again Kohsuke. Seeing this makes me think about a side-feature. Lazy-loading build records is great when it comes to start the server, but might we consider its unloading somehow? For example, on our instance (~10 nodes, 250 jobs), we've encountered some memory issues. We didn't investigate that very far, but couldn't we consider using some kind of WeakHashMap to handle build results? This way, the GC might be able to regain memory by freeing some build results accessed not so often. And it would eventually get naturally reloaded with that great new lazy-loading feature when needed. What do you think? Cheers PS : Please excuse me if this sounds stupid according to things I would have missed about the current behaviour. Just let know if so. 2012/9/21 Kohsuke Kawaguchi <[email protected] <mailto:[email protected]>> http://kohsuke.org/private/__20120920/jenkins.war <http://kohsuke.org/private/20120920/jenkins.war> I've deployed this exact binary on https://ci.jenkins-ci.org/ I just set up https://ci.jenkins-ci.org/job/__jenkins_lazy_load/ <https://ci.jenkins-ci.org/job/jenkins_lazy_load/> so this should provide up-to-date build. On 09/20/2012 12:28 PM, Emanuele Zattin wrote: Great job Kohsuke! I'll try and get somebody in Nokia to test this if possible. Emanuele Zattin ------------------------------__--------------------- -I don't have to know an answer. I don't feel frightened by not knowing things; by being lost in a mysterious universe without any purpose — which is the way it really is, as far as I can tell, possibly. It doesn't frighten me.- Richard Feynman On Thu, Sep 20, 2012 at 9:22 PM, Arnaud Héritier <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: Thx a lot KK. It's a great improvement to come. As I reported this issue also a long long time ago I'll be happy to test it. My new infrastructure is using the .deb of jenkins. How can I build it from your branch to give it a try ? Arnaud On Thu, Sep 20, 2012 at 9:16 PM, Kohsuke Kawaguchi <[email protected] <mailto:[email protected]> <mailto:kkawaguchi@cloudbees.__com <mailto:[email protected]>>> wrote: As any serious Jenkins users would know, a big Jenkins instance takes a long time to start up. One of the problems here is that it loads every build record before it starts accepting HTTP requests, and therefore a natural solution to this problem is to try to lazy load build records on-demand. I've been working on this in and out, but I finally got to a milestone. And incidentally I got contacted by Christian Wolfgang about this, as this very topic was discussed in the Copenhagen hackathon [3]. My changes are in the lazy-load branch [1]. You can also see the commits made in this branch [2] that aren't in master. What's already done =================== RunMap is conceptually a map from build number to Run, and Runs form a bi-directional linked list between their adjacent neighbors. So the first step was to turn RunMap into a virtual map. That is, it loads each build records on-demand as requested. Ditto to the bi-directional linking between Runs. What makes this interesting is that build records are keyed by their timestamps on disk, not by numbers (and numbers are available as symlinks but not in all the platforms.) So RunMap uses a binary search to locate the build that's currently needed. For this to work, I made an assumption that for build #M and #N. M>N iff M.timestamp>N.timestamp. This process also incorporates symlinks as a hint to speed up the search if they are available. Much of this logic is in AbstractLazyLoadRunMap, which has no dependencies to the rest of Jenkins core. This was so that I can test this class quickly and efficiently without using HudsonTestCase (and it's because historically I developed this code outside core.) See its search method that is the heart of this logic. RunMap was then modified to inherit this class to provide this necessary behaviour. The next step was to make Job/AbstractProject take advantages of this. The Job class expects subtypes to provide the _getRuns method that returns SortedMap of the builds, and all the other methods on Job that deal with obtaining build records work on this map. Some of these methods are overriden in AbstractProject to take advantages of RunMap. I then proceeded to update RunList, which is a class that's used to list up builds that satisfy a certain criteria. In the master, this class extends from ArrayList and every time a new filter is applied, all builds that satisfy the criteria gets copied into this array. This doesn't work very well with lazy loading, so I modified this class to only lazily walk through the builds and pick up ones that satisfy the criteria. For example, a typical use case of this class is "take all the builds of a job, narrow it down those that have failed, then list up first 10, render it to RSS". The new lazy implementation works very well with this. Unfortunately, to make this work, I needed to make a signature breaking change --- the class now extends from AbstractList and not from ArrayList. I did scan the source code of all the plugins to see their use of RunList, and I didn't spot anywhere it's casted to ArrayList, so I think the impact of this would be small. And at this point, it passes all the unit tests. What needs to be done ===================== Clearly more testing needs to be done. Code coverage of AbstractLazyLoadRunMap is actually pretty good, but the logic is complex. I also need to try this with a real Jenkins instance, mainly to see if there's some code in Jenkins or plugins that tries to eagerly load all the build records of all the jobs. To help us find this, we probably need some logging that tells us who's loading build records when. I'll try this with some real Jenkins deployments, and when that's ready, I'd like to merge this to the master. If anyone is willing to give this a shot before it hits the master, I'd highly appreciate that. The next goal after that is to deal with places where someone tries to load all the build records of one job. Unfortunately, this happens today in numerous places. Just in the job top page, test report trend or code coverage trend will cover the entire build history. So this will be a slow process. More about this later. WDYT? [1] http://github.com/jenkinsci/____jenkins/tree/lazy-load <http://github.com/jenkinsci/__jenkins/tree/lazy-load> <http://github.com/jenkinsci/__jenkins/tree/lazy-load <http://github.com/jenkinsci/jenkins/tree/lazy-load>> [2] https://github.com/jenkinsci/____jenkins/compare/master...__lazy-__load <https://github.com/jenkinsci/__jenkins/compare/master...lazy-__load> <https://github.com/jenkinsci/__jenkins/compare/master...lazy-__load <https://github.com/jenkinsci/jenkins/compare/master...lazy-load>> [3] http://wiki.praqma.net/____jcicodecamp12/sessions/____improve-start-up-time <http://wiki.praqma.net/__jcicodecamp12/sessions/__improve-start-up-time> <http://wiki.praqma.net/__jcicodecamp12/sessions/__improve-start-up-time <http://wiki.praqma.net/jcicodecamp12/sessions/improve-start-up-time>> -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Nectar, our professional version of Jenkins -- ----- Arnaud Héritier 06-89-76-64-24 http://aheritier.net Mail/GTalk: [email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> Twitter/Skype : aheritier -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Nectar, our professional version of Jenkins -- Baptiste <Batmat> MATHUS - http://batmat.net Sauvez un arbre, Mangez un castor ! nbsp;!
-- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Nectar, our professional version of Jenkins
