[JIRA] [core] (JENKINS-25742) Concurrent "job rename" and "edit view" results in deadlock that puts Jenkins in a non responsive state

Fri, 21 Nov 2014 20:31:15 -0800

Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 22/Nov/14 4:29 AM
Description:

When there are multiple users using Jenkins:

One user has clicked on "Yes" confirmation to rename a job.
Another user has clicked on "OK" on the "Edit View" screen at the same time.

Results in a race condition that results in a deadlock. Immediately after this deadlock occurs, Jenkins "create new job" and other operations are unresponsive. And after some more time Jenkins goes down.

Name: Handling GET /jenkins/view/ABCDEF/api/json : qtp817406040-243035
State: BLOCKED on hudson.model.ListView@57affa54 owned by: Handling POST /jenkins/view/ABCDEF/configSubmit : qtp817406040-42
Total blocked: 9 Total waited: 189

Stack trace:
hudson.model.ListView.getItems(ListView.java:164)
hudson.model.ListView.getItems(ListView.java:66)
sun.reflect.GeneratedMethodAccessor1929.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
org.kohsuke.stapler.export.Property.writeTo(Property.java:116)
org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:190)
org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185)
org.kohsuke.stapler.export.Model.writeTo(Model.java:157)
org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267)
hudson.model.Api.doJson(Api.java:204)
sun.reflect.GeneratedMethodAccessor771.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
org.kohsuke.stapler.Stapler.service(Stapler.java:225)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:91)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)

Name: Handling POST /jenkins/view/ABCDEF/configSubmit : qtp817406040-42
State: BLOCKED on hudson.model.Hudson@715617bb owned by: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
Total blocked: 2,007 Total waited: 12,613

Stack trace:
jenkins.model.Jenkins.save(Jenkins.java:2675)
hudson.model.View.save(View.java:341)
hudson.util.PersistedList.onModified(PersistedList.java:173)
hudson.util.PersistedList.replaceBy(PersistedList.java:85)
hudson.util.DescribableList.rebuildHetero(DescribableList.java:203)
hudson.model.ListView.submit(ListView.java:348)
hudson.model.View.doConfigSubmit(View.java:956)

  • locked hudson.model.ListView@57affa54
    sun.reflect.GeneratedMethodAccessor1966.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
    org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
    org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
    org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)

Name: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
State: BLOCKED on hudson.model.ListView@7e01a6a2 owned by: Handling POST /jenkins/view/XYZXYZ/configSubmit : qtp817406040-117
Total blocked: 54,155 Total waited: 11,725

Stack trace:
hudson.model.ListView$Listener.renameViewItem(ListView.java:421)
hudson.model.ListView$Listener.onLocationChanged(ListView.java:404)
hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:199)
hudson.model.AbstractItem.renameTo(AbstractItem.java:317)

  • locked hudson.model.FreeStyleProject@5dafef50
  • locked hudson.model.Hudson@715617bb
    hudson.model.Job.renameTo(Job.java:617)
    hudson.model.Job.doDoRename(Job.java:1403)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
    org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
    org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
    org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)

Name: Handling POST /jenkins/view/XYZXYZ/configSubmit : qtp817406040-117
State: BLOCKED on hudson.model.Hudson@715617bb owned by: Handling POST /jenkins/view/XYZXYZ/job/turkeytimeXYZ/doRename : qtp817406040-109
Total blocked: 2,045 Total waited: 12,350

Stack trace:
jenkins.model.Jenkins.save(Jenkins.java:2675)
hudson.model.View.save(View.java:341)
hudson.model.View.doConfigSubmit(View.java:967)

  • locked hudson.model.ListView@7e01a6a2
    sun.reflect.GeneratedMethodAccessor1966.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:483)
    org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)
    org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
    org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
    org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
    org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
    org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248)
    org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
    org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:733)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:863)
    org.kohsuke.stapler.Stapler.invoke(Stapler.java:636)
    org.kohsuke.stapler.Stapler.service(Stapler.java:225)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1667)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96)
    hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
    hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99)
    hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:85)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:114)
    hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:46)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
    hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
Environment: Summary: HP DL160 G6, 2 x Xeon E5645 2.40GHz, 93.8GB / 96GB 1333MHz DDR3
System: HP ProLiant DL160 G6, ySPEC 34.5
Processors: 2 x Xeon E5645 2.40GHz (HT enabled, 12 cores, 24 threads) - Gulftown B1, 64-bit, six-core, 32nm, L3: 12MB
Memory: 93.8GB / 96GB 1333MHz DDR3 == 6 x 8GB - 8GB PC3-10600 Samsung DDR3-1333 ECC Registered CL9 2Rx4
6 x 8GB - 8GB PC3-10600 Hynix DDR3-1333 ECC Registered CL9 2Rx4
6 x empty
Disk: sda (hpsa0): 1.2TB (0%) RAID-10 == 1 x 1.2TB HP-LOGICAL-VOLUME
Disk-Control: ahci0: HP/Intel ICH10 82801J 6 Port SATA/300 AHCI
Disk-Control: hpsa0: HP Smart Array P410
Chipset: Intel 5520 IOH-36D B3 (Tylersburg), 82801JIR A0 (ICH10R)
Network: eth0 (igb): HP NC362i/Intel 82576 Gigabit, 1Gb/s <full-duplex>
Network: eth1 (igb): HP NC362i/Intel 82576 Gigabit, no carrier
OS: RHEL Server 6.4, Linux 2.6.32-358, 64-bit
BIOS: HP O33 08/16/2010, rev 8.15
Hostname: xxx


Jenkins ver. 1.565.3
JDK 8u25 x.64
Jetty-9.2.2.v20140723


Project: Jenkins
Labels: jenkins
Priority: Critical Critical
Reporter: Manav Chimakurthi
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.

Reply via email to