Ed Palazzo created JENKINS-12672:
------------------------------------
Summary: Jobs take a long time to complete - get stuck sending
success/failure email
Key: JENKINS-12672
URL: https://issues.jenkins-ci.org/browse/JENKINS-12672
Project: Jenkins
Issue Type: Bug
Components: perforce
Affects Versions: current
Environment: Jenkins 1.450
Perforce plugin 1.3.7
Red Hat Enterprise Linux 4.5
Reporter: Ed Palazzo
We've noticed that sometimes our jobs get in a state where they're taking a
long time to complete. Investigation reveals that the maven build has
completing successfully, and it's "stuck" trying to send a success email.
Left alone, the build will eventually complete but a 10 minute build may take
several hours to succeed and blocks other jobs since the executors are busy.
Manual monitoring and intervention is usually required to free up the jobs.
Here's an example thread taken from a Jenkins thread dump that shows the
"stuck" job. It appears to be processing the change list to retrieve the
checkin user(s) that checked in code. Why is this taking so long?
The exact stack changes with each thread dump, but it is consistent up to
PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111).
"Executor #2 for master : executing markview-7.0.x_document-server #103"
prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000]
java.lang.Thread.State: RUNNABLE
at java.util.ArrayList.size(ArrayList.java:177)
at org.dom4j.tree.BackedList.<init>(BackedList.java:36)
at
org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373)
at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392)
at
org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428)
at
org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233)
at
org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98)
at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180)
at
org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140)
at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102)
at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674)
at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213)
at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234)
at org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159)
at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185)
at
hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111)
at
hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18)
at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808)
at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782)
at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356)
at
hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366)
at hudson.model.User.getProjects(User.java:402)
at
hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38)
at
hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514)
at
hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331)
at
hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251)
at
hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243)
at
hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at
hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700)
at
hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675)
at
hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022)
at hudson.model.Run.run(Run.java:1453)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
