Hi Nadine, Thanks for the update, we canceled the jobs from this morning as well. While doing so, we observed that for some jobs, it was not possible to cancel them although they have been in state active. For these jobs, the progress showed 100% completed. Retrying is not really an option for us, if the jobs are not completed by early morning (around 07:00am german time), it's pointless or even harmful for us to retry. I suppose that, under normal conditions, we can expect a job to be finished in time if they have been submitted by 5:00am? At least that was not a problem in the past. Also, it feels like pushing more and more jobs to the "queue" might make things even worse.
I checked again some of the jobs we did cancel (including jobs from tuesday) and they all are still in status canceling, so no progress on that side. Best, Stefan Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine Sundquist (AdWords API Team): > > Hi Stefan, > > The good news is that we found where the hanging is happening, but the bad > news is that we still don't have a complete fix. If the cancel continues to > hang, please get back to me. Right now, it looks like you're doing the > right thing by cancelling and retrying. > > Best, > Nadine, AdWords API Team > > On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine > Stegemann wrote: >> >> >> Hi Nadine, >> >> To update you on this issue, the situation has not improved but rather >> got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) >> with eight adwords accounts being affected. Yesterday, it was 30 batch jobs >> with three accounts being affected and on tuesday it was seven jobs, also >> with three accounts. So we see a slightly "negative" tendency here. I also >> checked some of the batch jobs we canceled yesterday and they are still in >> status "CANCELING". Even the jobs which got stuck on tuesday and which we >> also have canceled yesterday remain in "CANCELING". It seems that these >> jobs are not doing anything anymore and that there is something broken >> fundamentally. >> >> Do you have any update on this issue? Can we expect batch jobs to come >> back to "normal"? We'd like to know that because if not, we need to take >> action to deal with such scenarios. >> >> Best, >> Stefan >> >> Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine Sundquist >> (AdWords API Team): >>> >>> Hi Stefan, >>> >>> Thanks for reporting this. I personally experienced some longer than >>> usual running batch jobs last night as well. We're looking into what caused >>> this last spike in execution time. >>> >>> Cheers, >>> Nadine, AdWords API Team >>> >>> On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine >>> Stegemann wrote: >>>> >>>> Hi, >>>> >>>> sorry for hijacking this thread but since yesterday morning, we also >>>> started to have significant troubles with batch job service. Jobs remained >>>> active for a very long time. The following jobs have been started >>>> yesterday >>>> morning, german time (see start time is noted for each job) and were still >>>> active this morning: >>>> >>>> Job 445516765 | 2017-02-14T03:34 >>>> Job 445553000 | 2017-02-14T03:20 >>>> Job 445532407 | 2017-02-14T03:58 >>>> Job 445532515 | 2017-02-14T03:58 >>>> Job 445693959 | 2017-02-14T03:58 >>>> Job 445694061 | 2017-02-14T03:59 >>>> Job 445694595 | 2017-02-14T03:59 >>>> >>>> We've canceled the jobs this morning (after they are still reported to >>>> be active, how can that be?), but now they are in canceling state for >>>> about >>>> an hour (at the time of this writing). Furthermore, the situation got >>>> worse >>>> this morning, with 30 batch jobs being active for a couple of hours now. >>>> The ids of these jobs are attached to the end of this post. We will cancel >>>> these jobs as well because we do not expect them to finish, based on >>>> yesterday's experience. >>>> >>>> As an additional information, the problems seem to affect accounts with >>>> campaigns with a large to huge number or product partitions / keywords. >>>> The >>>> problematic jobs were "spreaded" over three adwords accounts yesterday and >>>> three accounts today. Two of the accounts affected yesterday are also >>>> affected today. We do use these batch jobs to set cpc bids for keywords >>>> and >>>> product partitions and use a maximum of 40.000 operations per job. We >>>> experimented with this number in the past and found that to be a >>>> "suitable" >>>> balance between performance and reliability of batch jobs. All jobs >>>> contain >>>> only one type of operation. While multiple jobs might affect the same >>>> campaign / adgroups, they do not affect the same keywords / product >>>> partitions. I was told on the past API workshop that this is ok and will >>>> not result in concurrency issues. In addition to concrete information on >>>> the issue, we would be happy to know if it makes sense to "play" with the >>>> limit of 40.000 operations per job again. >>>> >>>> As a general note, batch job continously tend to keep us busy since we >>>> switched to them last year. The runtime performance seems to be rather >>>> unpredictable. Especially on test accounts, batch jobs can run for a very >>>> long time (up to 30 minutes) which is painful if you want to run automated >>>> tests. Combined with the unclear advise on the number of operations to put >>>> in a batch job and best practices in general, that leaves us with a very >>>> uncertain feeling. Given the reliabilty issues we experienced in the past >>>> half year, we're currently thinking about ditching batch jobs entirely and >>>> implement batching on our site which would give us at least more control. >>>> We understand that the intend of batch jobs is like "fire and forget" >>>> (yeah, not really forget but check results etc.), we offload operations to >>>> AdWords and it takes care of it. Currently its more like "fire and pray". >>>> While we do not want to bring back the past, but we never had such >>>> troubles >>>> with mutate job service. I apologize for this bit of ranting, but it's >>>> constantly keeping us busy for a while now. >>>> >>>> Finally, here are the job ids of the still-active (soon to be canceled) >>>> jobs from this morning. >>>> >>>> 445954525 >>>> 445954540 >>>> 445989185 >>>> 446102085 >>>> 446102088 >>>> 445952002 >>>> 445986527 >>>> 445986596 >>>> 445986662 >>>> 445986857 >>>> 445987025 >>>> 445987106 >>>> 445987562 >>>> 445987577 >>>> 445987715 >>>> 446107815 >>>> 446107821 >>>> 446108097 >>>> 446108442 >>>> 446108472 >>>> 446108553 >>>> 446108598 >>>> 446108796 >>>> 446108910 >>>> 446108958 >>>> 45948708 >>>> 445983389 >>>> 445983824 >>>> 446105442 >>>> 446105460 >>>> >>>> Thanks for taking care of this and regards, >>>> Stefan >>>> >>>> >>>> Am Montag, 13. Februar 2017 22:08:05 UTC+1 schrieb AdWordsApiUser: >>>>> >>>>> Thanks Nadine - appreciate your quick reply. We'll keep an eye out and >>>>> let you know if things get out of hand. >>>>> >>>>> On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine Sundquist >>>>> (AdWords API Team) wrote: >>>>>> >>>>>> Hi MG, >>>>>> >>>>>> Thanks for sending over the batch job IDs. I took a look at the time >>>>>> it took for the jobs to run for the slow running jobs. I didn't see any >>>>>> that exceeded 15 minutes and most were completing within 5 minutes. >>>>>> There >>>>>> is a bit of load balancing going on with a lot of job requests, so it's >>>>>> expected that while some jobs may only take 2 minutes, there will be a >>>>>> few >>>>>> that may exceed 10 minutes. Luckily, the jobs don't appear to be getting >>>>>> stuck indefinitely, and they're not hanging for hours on end. I'll keep >>>>>> an >>>>>> eye on things as I did see a spike in processing time this morning, but >>>>>> it >>>>>> looks like things are returning to normal again. Please reach out if you >>>>>> see the times increasing even more. >>>>>> >>>>>> Thanks again, >>>>>> Nadine, AdWords API Team >>>>>> >>>>>> On Monday, February 13, 2017 at 9:39:03 AM UTC-5, AdWordsApiUser >>>>>> wrote: >>>>>>> >>>>>>> Hello team, >>>>>>> >>>>>>> The issue seems to happen again, though not as bad as last time, so >>>>>>> I thought I'd let you know before it becomes worse. Some jobs are >>>>>>> taking a >>>>>>> long time to process, while others process very fast. Last time it >>>>>>> happened, over 2-3 days, the jobs became slower, and a higher >>>>>>> proportion of >>>>>>> jobs started getting stuck till you guys did some magic to make >>>>>>> everything >>>>>>> fast again: >>>>>>> >>>>>>> stuck/took very long: >>>>>>> 445491084 >>>>>>> 445522515 >>>>>>> 445523004 >>>>>>> >>>>>>> not stuck/ran fast: >>>>>>> 445491588 >>>>>>> 445352074 >>>>>>> >>>>>>> Cheers, >>>>>>> MG >>>>>>> >>>>>>> On Thursday, February 2, 2017 at 6:51:38 PM UTC+1, AdWordsApiUser >>>>>>> wrote: >>>>>>>> >>>>>>>> Thanks Nadine! You and your team saved us a costly rewrite that we >>>>>>>> had started planning for (without using the bulk job service). We'll >>>>>>>> keep >>>>>>>> an eye out and report back if we see issues. Thanks team - you're the >>>>>>>> most >>>>>>>> responsive team at Google I've interacted with ;) >>>>>>>> >>>>>>>> On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine Sundquist >>>>>>>> (AdWords API Team) wrote: >>>>>>>>> >>>>>>>>> Hello Everyone, >>>>>>>>> >>>>>>>>> I have good news! We found the root cause, and the issue has been >>>>>>>>> fixed. Your batch job processing times should be back to normal now. >>>>>>>>> Thanks >>>>>>>>> again for reporting this! >>>>>>>>> >>>>>>>>> Take care, >>>>>>>>> Nadine, AdWords API Team >>>>>>>>> >>>>>>>>> On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine >>>>>>>>> Sundquist (AdWords API Team) wrote: >>>>>>>>>> >>>>>>>>>> Greetings Fellow AdWordsians, >>>>>>>>>> >>>>>>>>>> Thanks for the batch job IDs. They are a BIG help. I've taken a >>>>>>>>>> look at the history of those batch jobs in our internal systems, and >>>>>>>>>> I've >>>>>>>>>> found a pattern. I need to run it by a few teammates here to see if >>>>>>>>>> my >>>>>>>>>> observations result in finding a root cause. I'll update you when >>>>>>>>>> I've made >>>>>>>>>> a bit more progress. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Nadine, AdWords API Team >>>>>>>>>> >>>>>>>>>> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, >>>>>>>>>> AdWordsApiUser wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Vishal, >>>>>>>>>>> >>>>>>>>>>> Thanks for your reply. However, we have had 100% of the jobs >>>>>>>>>>> we've submitted in the last 6 hours take over 15 mintues to >>>>>>>>>>> complete. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> MG >>>>>>>>>>> >>>>>>>>>>> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal >>>>>>>>>>> Vinayak (Adwords API Team) wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi MG, >>>>>>>>>>>> >>>>>>>>>>>> Issues related to batch job performance tend to be intermittent >>>>>>>>>>>> some of the times. In such cases, cancelling the job and sending a >>>>>>>>>>>> new >>>>>>>>>>>> request solves the issue. Could you please try cancelling >>>>>>>>>>>> <https://developers.google.com/adwords/api/docs/guides/batch-jobs#canceling_a_batch_job> >>>>>>>>>>>> all >>>>>>>>>>>> current jobs with the status >>>>>>>>>>>> <https://developers.google.com/adwords/api/docs/reference/v201609/BatchJobService.BatchJob#status> >>>>>>>>>>>> AWAITING_FILE >>>>>>>>>>>> or ACTIVE and retry? If that doesn't help, please feel free to >>>>>>>>>>>> revert and >>>>>>>>>>>> I'll check with my team to see if we can dig into this issue >>>>>>>>>>>> further. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Vishal, AdWords API Team >>>>>>>>>>>> >>>>>>>>>>> -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog and Google+: https://googleadsdeveloper.blogspot.com/ https://plus.google.com/+GoogleAdsDevelopers/posts =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. Visit this group at https://groups.google.com/group/adwords-api. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/43f73996-e4ef-43f3-bc9d-c61992b89565%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
