Interesting - we use the disk usage plugin too, though had disabled it (thought not removed it).
On Mon, Dec 9, 2013 at 5:13 PM, Robert CHAVANNE <[email protected]>wrote: > Tim Drury <tdrury <at> gmail.com> writes: > > > > > > > I'm doing a heap-dump analysis now and I think I might know what the > issue > was. The start of this whole problem was the disk-usage plugin hanging our > attempts to view a job in Jenkins > (see https://issues.jenkins-ci.org/browse/JENKINS-20876) so we disabled > that > plugin. After disabling, Jenkins complained about data in an > older/unreadable format: > > > > You have data stored in an older format and/or unreadable data.If I click > the "Manage" button to delete it, it takes a _long_ time for it to display > all the disk-usage plugin data - there must be thousands of rows, but it > does display it all eventually. The error shown in each row is: > > > > > > CannotResolveClassException: > hudson.plugins.disk_usage.BuildDiskUsageAction > > > > > > If I click "Discard Unreadable Data" at the bottom of the page, I quickly > get a stack trace: > > > > > > javax.servlet.ServletException: java.util.ConcurrentModificationException > > > > at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:735) > > > > at org.kohsuke.stapler.Stapler.invoke(Stapler.java:799) > > > > at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:239) > > > > at > > org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) > > > > at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:685) > > > > at org.kohsuke.stapler.Stapler.invoke(Stapler.java:799) > > > > at org.kohsuke.stapler.Stapler.invoke(Stapler.java:587) > > > > at org.kohsuke.stapler.Stapler.service(Stapler.java:218) > > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) > > > > at winstone.ServletConfiguration.execute(ServletConfiguration.java: > > > > at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) > > > > at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) > > > > at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter. > > > > at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter. > > > > at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter. > > > > at > net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter. > > > > and it fails to discard the data. Older data isn't usually a problem so > I > brushed off this error. However, here is dominator_tree of the heap dump: > > > > > > Class Name > > | Shallow Heap | Retained Heap > | Percentage > > > -------------------------------------------------------------------- > > hudson.diagnosis.OldDataMonitor <at> 0x6f9f2c4a0 > > | 24 | > 3,278,466,984 | 88.69% > > com.thoughtworks.xstream.converters.SingleValueConverterWrapper <at> > 0x6f9da8780 | 16 | > 13,825,616 | 0.37% > > hudson.model.Hudson <at> 0x6f9b8b8e8 > > | 272 | > 3,572,400 | 0.10% > > org.eclipse.jetty.webapp.WebAppClassLoader <at> 0x6f9a73598 > | 88 | > 2,308,760 | 0.06% > > org.apache.commons.jexl.util.introspection.Introspector <at> > 0x6fbb74710 > | 32 | > 1,842,392 | 0.05% > > org.kohsuke.stapler.WebApp <at> 0x6f9c0ff10 > | 64 | > 1,127,480 | 0.03% > > java.lang.Thread <at> 0x7d5c2d138 Handling GET > /view/Alle/job/common-translation-main/ : RequestHandlerThread[#105] > Thread| > 112 | 971,336 | 0.03% > > > > ------------------------------------------------------------------------------ > > > > > > What is hudson.diagnosis.OldDataMonitor? Could the disk-usage plugin > data > be the cause of all my recent OOM errors? If so, how do I get rid of it? > > > > -tim > > > > On Monday, December 9, 2013 9:41:25 AM UTC-5, Tim Drury wrote:I intended > to install 1.532 on Friday, but mistakenly installed 1.539. It gave us the > same OOM exceptions. I'm installing 1.532 now and will - hopefully - know > tomorrow whether it's stable or not. I'm not exactly sure what's going to > happen with our plugins though. Hopefully Jenkins will tell me if they > must > be downgraded too. > > -timOn Monday, December 9, 2013 7:45:28 AM UTC-5, Stephen Connolly wrote: > > How does the current LTS (1.532.1) hold up? > > > > On 6 December 2013 13:33, Tim Drury <tdr... <at> gmail.com> wrote: > > We updated Jenkins to 1.542 two days ago (from 1.514) and we're getting a > lot of OOM errors. (info: Windows server 2008 R2, Jenkins JVM > is jdk-x64-1.6.0_26) > > > > > > At first I did the simplest thing and increased the heap from 3G to 4.2G > (and bumped up permgen). This didn't help so I started looking at threads > via the Jenkins monitoring tool. It indicated the disk-usage plugin was > hung. When you tried to view a page for a particularly large job, the page > would "hang" and the stack trace showed the disk-usage plugin was to blame > (or so I thought). Jira report with thread dump > here: https://issues.jenkins-ci.org/browse/JENKINS-20876 > > > > > > We disabled the disk-usage plugin and restarted and now we can visit that > job page. However, we still get OOM and lots of GCs in the logs at least > once a day. The stack trace looks frighteningly similar to that from the > disk-usage plugin. Here is an edited stack trace showing the methods > common > between the two OOM incidents: one during the disk-usage plugin and one > after it was disabled: > > > > > > [lots of xstream methods snipped] > > hudson.XmlFile.unmarshal(XmlFile.java:165) > > hudson.model.Run.reload(Run.java:323) > > hudson.model.Run.<init>(Run.java:312) > > > > hudson.model.AbstractBuild.<init>(AbstractBuild.java:185) > > hudson.maven.AbstractMavenBuild.<init>(AbstractMavenBuild.java:54) > > hudson.maven.MavenModuleSetBuild.<init>(MavenModuleSetBuild.java:146) > > ... [JVM methods snipped] > > hudson.model.AbstractProject.loadBuild(AbstractProject.java:1155) > > hudson.model.AbstractProject$1.create(AbstractProject.java:342) > > hudson.model.AbstractProject$1.create(AbstractProject.java:340) > > hudson.model.RunMap.retrieve(RunMap.java:225) > > hudson.model.RunMap.retrieve(RunMap.java:59) > > > > jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:677) > > > > jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:660) > > > > jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:502) > > > > jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.ja > > hudson.model.AbstractProject.getBuildByNumber(AbstractProject.java:1077) > > hudson.maven.MavenBuild.getParentBuild(MavenBuild.java:165) > > hudson.maven.MavenBuild.getWhyKeepLog(MavenBuild.java:273) > > hudson.model.Run.isKeepLog(Run.java:572) > > ... > > > > > > It seems something in "core" Jenkins has changed and not for the better. > Anyone seeing these issues? > > > > > > -tim > > > > > > > > > > > > Hello, > I don't know if that can help but I had an issue with both disk usage > plugin and Jenkins Job Configuration History Plugin. > Configuration History Plugin was badly set and kept history of many things, > for example each time a configuration page was visited. > It seems that recent update of disk usage plugin increased that effect. > I didn't fully understand why (perhaps the usage for each build and each > configuration was logged). > > I have multi-configuration projects with thousands of configurations. > So after some time, Jenkins got slower and finally we ran out of inodes (I > am working on Linux). > To fix that, I had to clean config-history and change settings of > Job Config History in Configure System tab to disable some of the logs. > > Regards, > Robert > > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
