On the other hand, is it really necessary to call all associated Listeners (like in [1].update() - L222, L236, L242) when the Queue is still locked (is it guaranteed and documented for a ProvisioningListener contract)? It can be a quite time spent operation/call...
[1] https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/slaves/NodeProvisioner.java -- Pavel Janousek Senior Jenkins QA Engineer ----- Original Message ----- > From: "Stephen Connolly" <[email protected]> > To: [email protected] > Sent: Monday, October 17, 2016 11:51:10 PM > Subject: Re: Adding a node with Queue.withLock in Jenkins > > On 17 October 2016 at 22:42, Surya Gaddipati <[email protected]> > wrote: > > > Hi Stephen, > > > > Thank you for your quick response. I don't want to belabor this more than > > it needs to be . > > > > >If you change the list of nodes during that time, the queue thread will > > get a concurrent modification exception (best case) and die > > > > I am guessing you are referring to this line > > <https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L1388> > > > > > > for (Computer c : Jenkins.getInstance().getComputers()) > > > > This is not same collection that addnode modifies, its a copy . So it > > seems unlikely that would cause a concurrent modification exception. > > > > > It's when you then end up down at > https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L1525-L1526 > that the issues start cropping up. > > In theory, add should be safe and only remove requiring the lock... in > practice we found that a lot of the cloud plugins blow up if we remove the > add lock. If you want to remove the add lock for your Jenkins and have > complete control over the plugins installed, you may get lucky > > > > > > > it will assign work to a node that no longer exists, except for a > > phantom object reference that it held onto... or worse > > > > This won't happen in this specific case I am describing here since a > > computer is tied to *one* particular build , it cannot be assigned to a > > phantom node. > > > > > > My main motivation is to remove queue locking as much as possible as our > > Jenkins instance has major scalability issues, almost *all* of it stemming > > from overzealous queue locking. > > > > > > > It is a long term goal to remove the queue lock > > > > I would love for queue.lock to go away but do you think it realistic to > > expect that to go away anytime soon given how deeply embedded it is into > > the core structure of the code. eg: Here it looks like we are acquiring a > > queue lock > > <https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Executor.java#L337>within > > a queue lock > > <https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Executor.java#L351> > > ? > > > > > > > > > > > > On Monday, October 17, 2016 at 1:54:11 PM UTC-5, Stephen Connolly wrote: > > > >> > >> > >> On Monday 17 October 2016, Surya Gaddipati <[email protected]> wrote: > >> > >>> Thanks Stephen for your quick response. > >>> > >>> > as otherwise the scheduling will blow up in your face. > >>> > >>> Curious, What do you mean by this ? > >>> > >> > >> When the queue starts scheduling it has to iterate the list of nodes to > >> build up the candidate nodes for the load balancing algorithm. > >> > >> If you change the list of nodes during that time, the queue thread will > >> get a concurrent modification exception (best case) and die... or it will > >> assign work to a node that no longer exists, except for a phantom object > >> reference that it held onto... or worse > >> > >> It is a long term goal to remove the queue lock as it impacts scalability > >> when you have 1000's of executors that can match a job... but even then > >> the > >> impact is not so large that removing the lock would be a priority. > >> > >> For now, use the queue lock methods, when we remove the need for a lock > >> they will become no-ops that the JVM will inline away for plugins compiled > >> against current cores > >> > >> > >>> thanks again. > >>> > >>> On Monday, October 17, 2016 at 9:53:23 AM UTC-5, Surya Gaddipati wrote: > >>>> > >>>> Hi all, > >>>> > >>>> I am working on plugin > >>>> <https://github.com/suryagaddipati/jenkins-docker-swarm-plugin> that > >>>> creates a single use computer/node whose lifecycle is tied to a single > >>>> build . I am currently adding the node like this > >>>> > >>>> Jenkins.getInstance().addNode(node); > >>>> > >>>> > >>>> However that method requires multiple Queue locks while doing so. > >>>> > >>>> > >>>> I believe in my particular case there is no need for queue locking > >>>> since only a single build can ever be scheduled on the computer via > >>>> LabelAssignmentAction. > >>>> > >>>> I wanted to check , > >>>> > >>>> 1. if that assumption is correct > >>>> > >>>> 2. If the team is open to accepting a patch to jenkins.instance for a > >>>> new method which adds a node without the Queue lock. > >>>> > >>>> > >>>> Thank you. > >>>> > >>>> > >>>> --Surya > >>>> > >>> -- > >>> You received this message because you are subscribed to the Google > >>> Groups "Jenkins Developers" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > >>> an email to [email protected]. > >>> To view this discussion on the web visit https://groups.google.com/d/ms > >>> gid/jenkinsci-dev/6a9cb43d-8eab-4413-89e9-f85775f5bc03%40goo > >>> glegroups.com > >>> <https://groups.google.com/d/msgid/jenkinsci-dev/6a9cb43d-8eab-4413-89e9-f85775f5bc03%40googlegroups.com?utm_medium=email&utm_source=footer> > >>> . > >>> For more options, visit https://groups.google.com/d/optout. > >>> > >> > >> > >> -- > >> Sent from my phone > >> > > -- > > You received this message because you are subscribed to the Google Groups > > "Jenkins Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To view this discussion on the web visit https://groups.google.com/d/ > > msgid/jenkinsci-dev/8daeff83-4cbb-4654-8886-c78586655d91% > > 40googlegroups.com > > <https://groups.google.com/d/msgid/jenkinsci-dev/8daeff83-4cbb-4654-8886-c78586655d91%40googlegroups.com?utm_medium=email&utm_source=footer> > > . > > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMwMv85%3D9nTLOYfiJUzAiiassAEhLDocD%3DOMncMmd9yJ-Q%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/382396873.181768.1476774525919.JavaMail.zimbra%40redhat.com. For more options, visit https://groups.google.com/d/optout.
