Hi Mahesh, Thanks for the insight provided on the source code. At this time, I have decided to go with the route of having a local TFS enlistment on build agents and using the Go Material solely for triggering without fetching. I will use this info you provided in case our team decides to go with Go Agent fetch of the materials. (Out team currently has on some pipelines about 40 materials, to fix it to fewer material used per pipeline requires lots of restructuring and revamping of the pipelines).
Thanks for your help, Keith On Wednesday, March 1, 2017 at 8:57:14 AM UTC-8, Mahesh Panchaksharaiah wrote: > > Hello Keith, > > I will try to list some pointers using which you should be able to trace > the material update flow both on server and agent side, > > Server material update, > 1. MaterialUpdateService class [1] responsible to trigger the material > update, there is timer task which is responsible to creates > MaterialUpdateMessages and puts in on a queue. > 2. MaterialUpdateListenerFactory [2] responsible to create > MaterialUpdateListeners which process the MaterialUpdateMessage. This class > uses the *material.check.threads *property to configure the number of > listeners. > 3. MaterialUpdateListener [3] responsible to update the materials. > 4. material.check.threads are defined in SystemEnvironment[4] with a > default value of 10. > > Agent material update, > 1. AgentController [5] which retrieves work from GoCD server and runs the > job. > 2. BuildWork#prepareJob [6] where materials are updated sequential as part > of executing a Job. > > Material update involves lot more classes, hope these pointers will help > you in tracing the flow. > > Also, I usually do not come across a pipeline configured with 20 > materials. Would you be able to share the details around what the pipeline > does and why its configured with that many materials. > > [1] > https://github.com/gocd/gocd/blob/master/server/src/com/thoughtworks/go/server/materials/MaterialUpdateService.java#L106 > [2] > https://github.com/gocd/gocd/blob/master/server/src/com/thoughtworks/go/server/materials/MaterialUpdateListenerFactory.java#L88 > [3] > https://github.com/gocd/gocd/blob/master/server/src/com/thoughtworks/go/server/materials/MaterialUpdateListener.java#L45 > [4] > https://github.com/gocd/gocd/blob/master/base/src/com/thoughtworks/go/util/SystemEnvironment.java#L454 > [5] > https://github.com/gocd/gocd/blob/master/agent/src/com/thoughtworks/go/agent/AgentHTTPClientController.java#L97 > [6] > https://github.com/gocd/gocd/blob/master/common/src/com/thoughtworks/go/remote/work/BuildWork.java#L192 > > Thanks, > Mahesh > > On Tuesday, February 28, 2017 at 10:36:54 PM UTC+5:30, >> >> Hi Mahesh, >> >> Could you point me the source code where the logic resides on the GoCD >> Server and also where the current logic for this in the Agent side. I want >> to explore the possibility of enhancing this code. If the feature is not >> possible, we have to drastically alter our pipelines to achieve desired >> performance. Thanks for all your help and insight. >> Also, the location of where I can find this setting on the GoCD Server. >> >> >> Thanks, >> Keith >> >> On Tuesday, February 28, 2017 at 4:40:46 AM UTC-8, Mahesh Panchaksharaiah >> wrote: >>> >>> Hello Keith, >>> >>> The property *material.check.threads* to control the number of threads >>> for material updates is used only on GoCD server. You seem to be looking >>> for concurrent updates on the agent side which is not possible as of now, >>> updates on the agent is sequential. >>> >>> Thanks, >>> Mahesh >>> >>> >>> On Tuesday, February 28, 2017 at 5:46:36 AM UTC+5:30, >>>> >>>> Hey Mahesh, >>>> >>>> Could you share with me where this property can be found I would like >>>> see if changing it would solve my issue, I tried looking for any >>>> documentation on it, but to no avail. >>>> >>>> Thanks, >>>> Keith >>>> >>>> >>>> On Sunday, February 26, 2017 at 8:17:11 PM UTC-8, Mahesh >>>> Panchakasahriah wrote: >>>>> >>>>> Hello, >>>>> >>>>> Material updates are not sequential, rather there are 10 threads to >>>>> handle updates at a given point of time. The number of threads can be >>>>> configured through the System Property *material.check.threads *which >>>>> defaults to 10. >>>>> >>>>> Why do you think the updates are sequential, are you facing any issues >>>>> with the material update? >>>>> >>>>> Thanks, >>>>> Mahesh >>>>> >>>>> On Saturday, February 25, 2017 at 12:18:17 AM UTC+5:30, >>>>>> >>>>>> My pipeline has a list of greater than 20 materials, I am seeing that >>>>>> Go I fetching these materials one after another. Is there a way that I >>>>>> can >>>>>> make these fetches concurrent? >>>>>> What other solutions you might have to shorten the fetch process. >>>>>> Thanks, >>>>>> Keith >>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "go-cd" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
