Hi Luke and David, We'll be happy to take a deeper look at this for you - can you please file a support ticket with Enterprise Support so that we can discuss and investigate in more detail?
Thanks! Stafford On Thursday, May 17, 2012 12:15:25 PM UTC-7, Luke Tracy wrote: > > Hi Claudio, I'm working with David and lead up our provisioning team. We > did eventually finish our migration of the 84,000 groups, but the rate > dropped off again on day 4 to 1 transaction per 20 seconds per thread. We > had to increase from 5 drivers/threads to 10 in order to finish in the nick > of time. A week later we attempted to use the provisioning API's to move > 4500 users from one sub-org to another to support their move to Google mail > and calendar and migration of their existing data. The response times from > the Google API servers were once again abissmal. We were processing 1 > transaction per 5 seconds per thread. We see delays of 1.5-2 seconds in > response times to simple API calls like retrieveCustomerId, and 4 or more > seconds to updateOrganizationUser. Some sanitized log file examples below > to show the gaps in our timestamps as our process sits idle waiting for a > response from the Google API servers: > > [05/10/12 04:47:58.294]:GA1 ST:appsForYourDomain.retrieveCustomerId: > sDomain = umich.edu > [05/10/12 04:48:00.452]:GA1 ST:appsForYourDomain.updateOrganizationUser: > CustomerId = ********************* > > > [05/10/12 04:48:00.453]:GA1 ST:appsForYourDomain.updateOrganizationUser: > NewOrgUnitPath = All Services > [05/10/12 04:48:04.551]:GA1 ST:addHandler: attr-name == 'GivenName' > > > How can we verify that this was due to overloaded servers or being caused > by something else, perhaps something we can correct or compensate for > locally? If we do verify it was overloaded servers, for 5 days, what can > be done to shore up Google's provisioning API infrastructure and is that > being done? > > We have other large migrations to complete before we are fully deployed > and we need to find a way of predicting and protecting the throughput of > our provisioning processes or our deployments will continue to be plagued > with these issues and Google's image will suffer here at the University of > Michigan. > > Thank you for any help you can provide us. > > Luke Tracy > Technical Manager > Identity and Access Management > Universtiy of Michigan > > > > On Friday, May 4, 2012 2:36:43 PM UTC-4, Claudio Cherubino wrote: >> >> I'll need to double check, but in general Google Data API requests' >> limits are per user, so using different admins should allow to work around >> them. >> >> Claudio >> >> On Fri, May 4, 2012 at 11:31 AM, David Spangler <[email protected]>wrote: >> >>> Also, does it matter if the threads are using different admin IDs? >>> Should we be using different admin IDs for each thread? >>> >>> >>> On Friday, May 4, 2012 1:24:49 PM UTC-5, wrote: >>>> >>>> There were no errors reported in the driver. >>>> >>>> On Friday, May 4, 2012 1:21:31 PM UTC-5, Claudio Cherubino wrote: >>>>> >>>>> I don't think we were specifically throttling your app, but instead it >>>>> might have been that our servers were overloaded. >>>>> Did you get any specific error message? We'll send you 503s if it is >>>>> your app that needs to slow down the requests. >>>>> >>>>> Claudio >>>>> >>>>> On Fri, May 4, 2012 at 11:19 AM, wrote: >>>>> >>>>>> There was a period where the 5 drivers were working at the 1 >>>>>> transaction per second (for 4 hours), then there was a drop-off in speed >>>>>> (by half). Does this indicate that we reached some limit? If we add >>>>>> more >>>>>> threads will it also work properly for some time until another limit is >>>>>> reached? >>>>>> >>>>>> >>>>>> On Friday, May 4, 2012 1:06:43 PM UTC-5, Claudio Cherubino wrote: >>>>>> >>>>>>> Hi David, >>>>>>> >>>>>>> All limits are per thread so adding more threads should definitely >>>>>>> increase the throughput. >>>>>>> Just be prepared to slow down using exponential backoff in case you >>>>>>> get a 503 response code. >>>>>>> For more details, check our Usage Limits page in the docs: >>>>>>> >>>>>>> https://developers.google.com/****google-apps/provisioning/**limit** >>>>>>> s <https://developers.google.com/google-apps/provisioning/limits> >>>>>>> >>>>>>> Claudio >>>>>>> >>>>>>> >>>>>>> In addition, we are interested in whether or not adding another 5 >>>>>>>> threads will increase the overall rates, i.e. is the process for >>>>>>>> provisioning Groups using the same API actions as "regular" user >>>>>>>> provisioning, hitting the same services that might be affecting the >>>>>>>> limits? >>>>>>>> >>>>>>>> In addition to the existing 5 threads, there are 5 threads that are >>>>>>>> dedicated to user provisioning and 1 for password sync. Are these >>>>>>>> additive? In other words, we would be going from 11 to 16 threads >>>>>>>> overall, >>>>>>>> would this help? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Friday, May 4, 2012 10:30:14 AM UTC-5, David Spangler wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I have a situation where we are attempting to populate around >>>>>>>>> 90,000 groups using 5 different threads against the Provisioning API. >>>>>>>>> When >>>>>>>>> we started at 5pm last night, we had "expected" throughput of around >>>>>>>>> 200-300 users per minute processed. After 4 hours, the rates dropped >>>>>>>>> more >>>>>>>>> than in half. Did we reach an API limit? If so, is it possible to >>>>>>>>> request >>>>>>>>> a lift of this limit temporarily until all the Groups are provisioned? >>>>>>>>> >>>>>>>>> Attached is a graph of what last night looked like. >>>>>>>>> >>>>>>>>> Thanks for looking at this. >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Google Apps Domain Information and Management APIs" group. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/**ms**g/google-apps-mgmt-apis/-/**MSD9* >>>>>>>> *rGLQ6AcJ<https://groups.google.com/d/msg/google-apps-mgmt-apis/-/MSD9rGLQ6AcJ> >>>>>>>> . >>>>>>>> >>>>>>>> To post to this group, send email to google-apps-mgmt-apis@** >>>>>>>> googlegr**oups.com <[email protected]>. >>>>>>>> To unsubscribe from this group, send email to >>>>>>>> google-apps-mgmt-apis+**unsubscr**[email protected]<google-apps-mgmt-apis%[email protected]> >>>>>>>> . >>>>>>>> For more options, visit this group at http://groups.google.com/** >>>>>>>> group**/google-apps-mgmt-apis?**hl=en<http://groups.google.com/group/google-apps-mgmt-apis?hl=en> >>>>>>>> . >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Google Apps Domain Information and Management APIs" group. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> **msg/google-apps-mgmt-apis/-/**lYs7PleFp4EJ<https://groups.google.com/d/msg/google-apps-mgmt-apis/-/lYs7PleFp4EJ> >>>>>> . >>>>>> >>>>>> To post to this group, send email to google-apps-mgmt-apis@** >>>>>> googlegroups.com <[email protected]>. >>>>>> To unsubscribe from this group, send email to google-apps-mgmt-apis+* >>>>>> *[email protected]<google-apps-mgmt-apis%[email protected]> >>>>>> . >>>>>> For more options, visit this group at http://groups.google.com/** >>>>>> group/google-apps-mgmt-apis?**hl=en<http://groups.google.com/group/google-apps-mgmt-apis?hl=en> >>>>>> . >>>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google Apps Domain Information and Management APIs" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-apps-mgmt-apis/-/IU4gwh0GW_4J. >>> >>> To post to this group, send email to >>> [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-apps-mgmt-apis?hl=en. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Google Apps Domain Information and Management APIs" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-apps-mgmt-apis/-/phYdXqTMWQEJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-apps-mgmt-apis?hl=en.
