|
I've had similar situation and have thread dump with information of acquired locks (attached).
disk-usage-plugin thread so some "computation" in synchronized section
(...)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156)
at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133)
at hudson.model.RunMap.retrieve(RunMap.java:223)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:367)
- locked <0x00000005f57899b0> (a hudson.model.RunMap)
at hudson.model.RunMap.getById(RunMap.java:204)
at hudson.model.RunMap.getById(RunMap.java:56)
(...)
while other requests are blocked on synchronized section:
"Handling GET / from 192.168.72.4 : RequestHandlerThread[#1110] View/index.jelly Dashboard/main.jelly LatestBuilds/portlet.jelly LatestBuilds/latestbuilds.jelly" #1979423 daemon prio=5 os_prio=0 tid=0x00007f84b0008000 nid=0x147de waiting for monitor entry [0x00007f85728f1000]
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:356)
I don't see a classic deadlock there. My suspicion is rather that the reason (in additional to newly added synchronized block to fix
JENKINS-22767
) was large number of historical builds (7K+). It could just take some time for disk-usage-plugin to calculate required data (which the other several request threads for that project just was waiting to the first one to finish).
However I'm not sure if it is related only to disk-usage-plugin. After its removal (and restart) I observed the same situation with the thread handling side panel request (below). I'm not sure why sidepanel.jelly should also take much time in that place.
"Handling GET /job/somejob-pr/ from 9.9.9.9 : RequestHandlerThread[#15] Job/index.jelly AbstractProject/sidepanel.jelly HistoryWidget/index.jelly" #97 daemon prio=5 os_prio=0 tid=0x00007f8104008800 nid=0x15ad2 runnable [0x00007f81e28e0000]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.hashCode(Native Method)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.get(HashMap.java:556)
at com.thoughtworks.xstream.mapper.DefaultImplementationsMapper.defaultImplementationOf(DefaultImplementationsMapper.java:69)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:143)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.mapper.AnnotationMapper.defaultImplementationOf(AnnotationMapper.java:143)
at hudson.util.xstream.MapperDelegate.defaultImplementationOf(MapperDelegate.java:59)
at com.thoughtworks.xstream.mapper.MapperWrapper.defaultImplementationOf(MapperWrapper.java:46)
at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:54)
at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:352)
(...)
at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
at hudson.util.XStream2.unmarshal(XStream2.java:114)
at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
at hudson.XmlFile.unmarshal(XmlFile.java:163)
at hudson.model.Run.reload(Run.java:326)
at hudson.model.Run.<init>(Run.java:314)
at hudson.model.AbstractBuild.<init>(AbstractBuild.java:175)
at hudson.model.Build.<init>(Build.java:103)
at hudson.model.FreeStyleBuild.<init>(FreeStyleBuild.java:38)
at sun.reflect.GeneratedConstructorAccessor160.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:156)
at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:133)
at hudson.model.RunMap.retrieve(RunMap.java:223)
at hudson.model.RunMap.retrieve(RunMap.java:56)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:479)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:461)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:367)
- locked <0x00000005eda16310> (a hudson.model.RunMap)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:332)
at jenkins.model.lazy.LazyBuildMixIn$RunMixIn.getPreviousBuild(LazyBuildMixIn.java:355)
at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:196)
at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:104)
at hudson.model.RunMap$1.next(RunMap.java:109)
at hudson.model.RunMap$1.next(RunMap.java:98)
at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:849)
at org.apache.commons.collections.IteratorUtils.toList(IteratorUtils.java:825)
at hudson.widgets.BuildHistoryWidget.getHistoryPageFilter(BuildHistoryWidget.java:81)
(...)
|
