We recently dared and upgraded all our production masters from 1.480 to 1.532 after quite a long time of testing, so I realize that we might be a bit late to the party. We started two weeks ago with one of our smaller masters (about 700 jobs) and let that run for two weeks to test the waters so to say, some "interesting" things happened during that time, but those are for another thread ;) It should have been an alarm bell when we noticed a performance hit on a ListView we measure in graphite that basically includes all jobs. A curl operation on that view went from taking 0.5 seconds to 5 seconds after the upgrade. In retrospect I shouldn't have ignored that.
Now on our other servers that were upgraded yesterday we could see for the same
curl on the other servers
>From average 2 sec to 50 sec for a master with 2.5k jobs
And from average 2.5 sec to never going below 200 sec for the master with 6k
jobs
It does not seem to be lazy loading related as most thread dumps that I do
reveal the following stack trace:
Handling GET /view/All/ : http-8080-85
"Handling GET /view/All/ : http-8080-85" Id=4552 Group=main RUNNABLE
at java.lang.String$CaseInsensitiveComparator.compare(String.java:1170)
at java.lang.String.compareToIgnoreCase(String.java:1220)
at
hudson.util.CaseInsensitiveComparator.compare(CaseInsensitiveComparator.java:40)
at
hudson.util.CaseInsensitiveComparator.compare(CaseInsensitiveComparator.java:34)
at java.util.TreeMap.put(TreeMap.java:545)
at java.util.TreeSet.add(TreeSet.java:255)
at hudson.model.ListView.includeItems(ListView.java:214)
at hudson.model.ListView.getItems(ListView.java:164)
at hudson.model.ListView.getItems(ListView.java:60)
at hudson.Functions.getRelativeLinkTo(Functions.java:1014)
at sun.reflect.GeneratedMethodAccessor425.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258)
at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104)
at
org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
at
org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
at
org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
at
org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
at
hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
As you can see no loading of builds, we disabled the weather column to relieve
some of those problems.
I've been digging through the code, and not only is ListView.getItems quite
complex (I lost count after 3n and 2 n log n) but when the index view has
gotten all the items and renders the table it calls Functions.getRelativeLinkTo
for every item in the list which in turn again calls ListView.getItems and
viewItems.contains, so no wonder the view is rendering slow.
Now, one quick fix I could do would be to pass along the item list as a
parameter to Functions.getRelativeLinkTo since the view already has the list
and is iterating through it, but does anyone see anything else I could do as
well, or any problems with that approach or my analysis?
Robert Sandell
Staff Engineer
Development Environment
Software Environment and Product Configuration
Sony Mobile Communications
Tel: +46 10 80 12721
[email protected]<mailto:[email protected]>
sonymobile.com<http://sonymobile.com/>
[cid:[email protected]]
--
You received this message because you are subscribed to the Google Groups
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
<<inline: image002.png>>
