Hello Keith, I will try to list some pointers using which you should be able to trace the material update flow both on server and agent side,
Server material update, 1. MaterialUpdateService class [1] responsible to trigger the material update, there is timer task which is responsible to creates MaterialUpdateMessages and puts in on a queue. 2. MaterialUpdateListenerFactory [2] responsible to create MaterialUpdateListeners which process the MaterialUpdateMessage. This class uses the *material.check.threads *property to configure the number of listeners. 3. MaterialUpdateListener [3] responsible to update the materials. 4. material.check.threads are defined in SystemEnvironment[4] with a default value of 10. Agent material update, 1. AgentController [5] which retrieves work from GoCD server and runs the job. 2. BuildWork#prepareJob [6] where materials are updated sequential as part of executing a Job. Material update involves lot more classes, hope these pointers will help you in tracing the flow. Also, I usually do not come across a pipeline configured with 20 materials. Would you be able to share the details around what the pipeline does and why its configured with that many materials. [1] https://github.com/gocd/gocd/blob/master/server/src/com/thoughtworks/go/server/materials/MaterialUpdateService.java#L106 [2] https://github.com/gocd/gocd/blob/master/server/src/com/thoughtworks/go/server/materials/MaterialUpdateListenerFactory.java#L88 [3] https://github.com/gocd/gocd/blob/master/server/src/com/thoughtworks/go/server/materials/MaterialUpdateListener.java#L45 [4] https://github.com/gocd/gocd/blob/master/base/src/com/thoughtworks/go/util/SystemEnvironment.java#L454 [5] https://github.com/gocd/gocd/blob/master/agent/src/com/thoughtworks/go/agent/AgentHTTPClientController.java#L97 [6] https://github.com/gocd/gocd/blob/master/common/src/com/thoughtworks/go/remote/work/BuildWork.java#L192 Thanks, Mahesh On Tuesday, February 28, 2017 at 10:36:54 PM UTC+5:30, > > Hi Mahesh, > > Could you point me the source code where the logic resides on the GoCD > Server and also where the current logic for this in the Agent side. I want > to explore the possibility of enhancing this code. If the feature is not > possible, we have to drastically alter our pipelines to achieve desired > performance. Thanks for all your help and insight. > Also, the location of where I can find this setting on the GoCD Server. > > > Thanks, > Keith > > On Tuesday, February 28, 2017 at 4:40:46 AM UTC-8, Mahesh Panchaksharaiah > wrote: >> >> Hello Keith, >> >> The property *material.check.threads* to control the number of threads >> for material updates is used only on GoCD server. You seem to be looking >> for concurrent updates on the agent side which is not possible as of now, >> updates on the agent is sequential. >> >> Thanks, >> Mahesh >> >> >> On Tuesday, February 28, 2017 at 5:46:36 AM UTC+5:30, >>> >>> Hey Mahesh, >>> >>> Could you share with me where this property can be found I would like >>> see if changing it would solve my issue, I tried looking for any >>> documentation on it, but to no avail. >>> >>> Thanks, >>> Keith >>> >>> >>> On Sunday, February 26, 2017 at 8:17:11 PM UTC-8, Mahesh Panchakasahriah >>> wrote: >>>> >>>> Hello, >>>> >>>> Material updates are not sequential, rather there are 10 threads to >>>> handle updates at a given point of time. The number of threads can be >>>> configured through the System Property *material.check.threads *which >>>> defaults to 10. >>>> >>>> Why do you think the updates are sequential, are you facing any issues >>>> with the material update? >>>> >>>> Thanks, >>>> Mahesh >>>> >>>> On Saturday, February 25, 2017 at 12:18:17 AM UTC+5:30, >>>>> >>>>> My pipeline has a list of greater than 20 materials, I am seeing that >>>>> Go I fetching these materials one after another. Is there a way that I >>>>> can >>>>> make these fetches concurrent? >>>>> What other solutions you might have to shorten the fetch process. >>>>> Thanks, >>>>> Keith >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
