Hi Nadine, Can you check this Batch Job Id:575510402
My first batch job, its been 4 hrs and status did not change. On Tuesday, February 21, 2017 at 11:20:12 PM UTC+8, Nadine Sundquist (AdWords API Team) wrote: > > Hi Stefan, > > Thanks for the update! I'm keeping an eye on this as well. > > Take care, > Nadine, AdWords API Team > > On Saturday, February 18, 2017 at 1:43:38 AM UTC-5, Stefan Kleine > Stegemann wrote: >> >> >> Hi Nadine, >> >> good news on our side as well. Today, we did not have stuck jobs any more >> and all jobs have been processed in time as they have before this issue. We >> did decrease the batch job size to 20k though, I don't know whether this >> relevant or not. Furthermore, I've checked a couple of jobs from the past >> days and they all are canceled now. >> >> At the moment, it looks like the issue is solved. We'll keep an I on it >> and let you know if it happens again. Thanks for taking care of this and >> your constant updates on this issue. >> >> Best, >> Stefan >> >> Am Freitag, 17. Februar 2017 23:28:11 UTC+1 schrieb Nadine Sundquist >> (AdWords API Team): >>> >>> Hello Stefan, >>> >>> I have good news! We've been working through processing through the >>> queue of hanging jobs. You should be seeing an improvement in that these >>> hanging jobs should be reaching a final state e.g. canceling should now be >>> canceled. You may still see a few hanging, but the majority should have >>> reached a final state. We're still working on fixing the root cause. >>> >>> Take care, >>> Nadine, AdWords API Team >>> >>> On Friday, February 17, 2017 at 1:47:04 PM UTC-5, Nadine Sundquist >>> (AdWords API Team) wrote: >>>> >>>> Hi Stefan, >>>> >>>> We're actively working on clearing the queue, which includes those jobs >>>> in the canceling state, as we're searching for a fix for this. >>>> Unfortunately, it won't make a difference whether or not it's a small job >>>> versus a larger job. One of my teammates put it quite well when he said >>>> that a larger job may have a higher chance of getting stuck, but on the >>>> other hand when it does go through, it gets through more operations. Just >>>> so you know, when a job is in a canceling state, it will not execute any >>>> more operations, so you can submit the rest of the operations that did not >>>> execute in that canceling job. It might take awhile for the job to reach a >>>> canceled state, but at least it won't execute any more operations. I >>>> realize this isn't optimal, but I do want you to know that is of higher >>>> priority to us. It's just really tricky trying to find a solution for what >>>> you're seeing. I'll get back to you as we make more progress. Thanks again >>>> for providing so much helpful information! >>>> >>>> Cheers, >>>> Nadine, AdWords API Team >>>> >>>> On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine >>>> Stegemann wrote: >>>>> >>>>> >>>>> Hi Nadine, >>>>> >>>>> still no improvement, today we have 28 stuck batch jobs across seven >>>>> adwords accounts. I also checked a sample of 10 batch jobs out of the >>>>> ones >>>>> we have canceled over the past days and 2 jobs are actually canceled but >>>>> 8 >>>>> are still in state canceling. As this issue is present for four days now, >>>>> it becomes more critical for us because we are unable to deliver the full >>>>> service to some of our customers. Thus, any advise for how the deal with >>>>> the "broken" batch jobs is appreciated. Does it, for example, make sense >>>>> to >>>>> try smaller or larger jobs? >>>>> >>>>> Best, >>>>> Stefan >>>>> >>>>> Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1 >>>>> schrieb Nadine Sundquist (AdWords API Team): >>>>>> >>>>>> Greetings Stefan, >>>>>> >>>>>> This is useful information when we're trying to figure out how to >>>>>> find a solution for this one. Thanks for passing it along. If jobs have >>>>>> been finishing for you normally by a certain time, then I would expect >>>>>> it >>>>>> to continue to do so. If it doesn't, then it's probably fallen into the >>>>>> issue that we're working on. I'll continue to track this so we can find >>>>>> a >>>>>> solution. >>>>>> >>>>>> Regards, >>>>>> Nadine, AdWords API Team >>>>>> >>>>>> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine >>>>>> Stegemann wrote: >>>>>>> >>>>>>> >>>>>>> Hi Nadine, >>>>>>> >>>>>>> Thanks for the update, we canceled the jobs from this morning as >>>>>>> well. While doing so, we observed that for some jobs, it was not >>>>>>> possible >>>>>>> to cancel them although they have been in state active. For these jobs, >>>>>>> the >>>>>>> progress showed 100% completed. Retrying is not really an option for >>>>>>> us, if >>>>>>> the jobs are not completed by early morning (around 07:00am german >>>>>>> time), >>>>>>> it's pointless or even harmful for us to retry. I suppose that, under >>>>>>> normal conditions, we can expect a job to be finished in time if they >>>>>>> have >>>>>>> been submitted by 5:00am? At least that was not a problem in the past. >>>>>>> Also, it feels like pushing more and more jobs to the "queue" might >>>>>>> make >>>>>>> things even worse. >>>>>>> >>>>>>> I checked again some of the jobs we did cancel (including jobs from >>>>>>> tuesday) and they all are still in status canceling, so no progress on >>>>>>> that >>>>>>> side. >>>>>>> >>>>>>> Best, >>>>>>> Stefan >>>>>>> >>>>>>> Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine >>>>>>> Sundquist (AdWords API Team): >>>>>>>> >>>>>>>> Hi Stefan, >>>>>>>> >>>>>>>> The good news is that we found where the hanging is happening, but >>>>>>>> the bad news is that we still don't have a complete fix. If the cancel >>>>>>>> continues to hang, please get back to me. Right now, it looks like >>>>>>>> you're >>>>>>>> doing the right thing by cancelling and retrying. >>>>>>>> >>>>>>>> Best, >>>>>>>> Nadine, AdWords API Team >>>>>>>> >>>>>>>> On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine >>>>>>>> Stegemann wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Hi Nadine, >>>>>>>>> >>>>>>>>> To update you on this issue, the situation has not improved but >>>>>>>>> rather got slightly worse. Today we have 33 stuck batch jobs (in >>>>>>>>> status >>>>>>>>> ACTIVE) with eight adwords accounts being affected. Yesterday, it was >>>>>>>>> 30 >>>>>>>>> batch jobs with three accounts being affected and on tuesday it was >>>>>>>>> seven >>>>>>>>> jobs, also with three accounts. So we see a slightly "negative" >>>>>>>>> tendency >>>>>>>>> here. I also checked some of the batch jobs we canceled yesterday and >>>>>>>>> they >>>>>>>>> are still in status "CANCELING". Even the jobs which got stuck on >>>>>>>>> tuesday >>>>>>>>> and which we also have canceled yesterday remain in "CANCELING". It >>>>>>>>> seems >>>>>>>>> that these jobs are not doing anything anymore and that there is >>>>>>>>> something >>>>>>>>> broken fundamentally. >>>>>>>>> >>>>>>>>> Do you have any update on this issue? Can we expect batch jobs to >>>>>>>>> come back to "normal"? We'd like to know that because if not, we need >>>>>>>>> to >>>>>>>>> take action to deal with such scenarios. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Stefan >>>>>>>>> >>>>>>>>> Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine >>>>>>>>> Sundquist (AdWords API Team): >>>>>>>>>> >>>>>>>>>> Hi Stefan, >>>>>>>>>> >>>>>>>>>> Thanks for reporting this. I personally experienced some longer >>>>>>>>>> than usual running batch jobs last night as well. We're looking into >>>>>>>>>> what >>>>>>>>>> caused this last spike in execution time. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Nadine, AdWords API Team >>>>>>>>>> >>>>>>>>>> On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan >>>>>>>>>> Kleine Stegemann wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> sorry for hijacking this thread but since yesterday morning, we >>>>>>>>>>> also started to have significant troubles with batch job service. >>>>>>>>>>> Jobs >>>>>>>>>>> remained active for a very long time. The following jobs have been >>>>>>>>>>> started >>>>>>>>>>> yesterday morning, german time (see start time is noted for each >>>>>>>>>>> job) and >>>>>>>>>>> were still active this morning: >>>>>>>>>>> >>>>>>>>>>> Job 445516765 | 2017-02-14T03:34 >>>>>>>>>>> Job 445553000 | 2017-02-14T03:20 >>>>>>>>>>> Job 445532407 | 2017-02-14T03:58 >>>>>>>>>>> Job 445532515 | 2017-02-14T03:58 >>>>>>>>>>> Job 445693959 | 2017-02-14T03:58 >>>>>>>>>>> Job 445694061 | 2017-02-14T03:59 >>>>>>>>>>> Job 445694595 | 2017-02-14T03:59 >>>>>>>>>>> >>>>>>>>>>> We've canceled the jobs this morning (after they are still >>>>>>>>>>> reported to be active, how can that be?), but now they are in >>>>>>>>>>> canceling >>>>>>>>>>> state for about an hour (at the time of this writing). Furthermore, >>>>>>>>>>> the >>>>>>>>>>> situation got worse this morning, with 30 batch jobs being active >>>>>>>>>>> for a >>>>>>>>>>> couple of hours now. The ids of these jobs are attached to the end >>>>>>>>>>> of this >>>>>>>>>>> post. We will cancel these jobs as well because we do not expect >>>>>>>>>>> them to >>>>>>>>>>> finish, based on yesterday's experience. >>>>>>>>>>> >>>>>>>>>>> As an additional information, the problems seem to affect >>>>>>>>>>> accounts with campaigns with a large to huge number or product >>>>>>>>>>> partitions / >>>>>>>>>>> keywords. The problematic jobs were "spreaded" over three adwords >>>>>>>>>>> accounts >>>>>>>>>>> yesterday and three accounts today. Two of the accounts affected >>>>>>>>>>> yesterday >>>>>>>>>>> are also affected today. We do use these batch jobs to set cpc bids >>>>>>>>>>> for >>>>>>>>>>> keywords and product partitions and use a maximum of 40.000 >>>>>>>>>>> operations per >>>>>>>>>>> job. We experimented with this number in the past and found that to >>>>>>>>>>> be a >>>>>>>>>>> "suitable" balance between performance and reliability of batch >>>>>>>>>>> jobs. All >>>>>>>>>>> jobs contain only one type of operation. While multiple jobs might >>>>>>>>>>> affect >>>>>>>>>>> the same campaign / adgroups, they do not affect the same keywords >>>>>>>>>>> / >>>>>>>>>>> product partitions. I was told on the past API workshop that this >>>>>>>>>>> is ok and >>>>>>>>>>> will not result in concurrency issues. In addition to concrete >>>>>>>>>>> information >>>>>>>>>>> on the issue, we would be happy to know if it makes sense to "play" >>>>>>>>>>> with >>>>>>>>>>> the limit of 40.000 operations per job again. >>>>>>>>>>> >>>>>>>>>>> As a general note, batch job continously tend to keep us busy >>>>>>>>>>> since we switched to them last year. The runtime performance seems >>>>>>>>>>> to be >>>>>>>>>>> rather unpredictable. Especially on test accounts, batch jobs can >>>>>>>>>>> run for a >>>>>>>>>>> very long time (up to 30 minutes) which is painful if you want to >>>>>>>>>>> run >>>>>>>>>>> automated tests. Combined with the unclear advise on the number of >>>>>>>>>>> operations to put in a batch job and best practices in general, >>>>>>>>>>> that leaves >>>>>>>>>>> us with a very uncertain feeling. Given the reliabilty issues we >>>>>>>>>>> experienced in the past half year, we're currently thinking about >>>>>>>>>>> ditching >>>>>>>>>>> batch jobs entirely and implement batching on our site which would >>>>>>>>>>> give us >>>>>>>>>>> at least more control. We understand that the intend of batch jobs >>>>>>>>>>> is like >>>>>>>>>>> "fire and forget" (yeah, not really forget but check results etc.), >>>>>>>>>>> we >>>>>>>>>>> offload operations to AdWords and it takes care of it. Currently >>>>>>>>>>> its more >>>>>>>>>>> like "fire and pray". While we do not want to bring back the past, >>>>>>>>>>> but we >>>>>>>>>>> never had such troubles with mutate job service. I apologize for >>>>>>>>>>> this bit >>>>>>>>>>> of ranting, but it's constantly keeping us busy for a while now. >>>>>>>>>>> >>>>>>>>>>> Finally, here are the job ids of the still-active (soon to be >>>>>>>>>>> canceled) jobs from this morning. >>>>>>>>>>> >>>>>>>>>>> 445954525 >>>>>>>>>>> 445954540 >>>>>>>>>>> 445989185 >>>>>>>>>>> 446102085 >>>>>>>>>>> 446102088 >>>>>>>>>>> 445952002 >>>>>>>>>>> 445986527 >>>>>>>>>>> 445986596 >>>>>>>>>>> 445986662 >>>>>>>>>>> 445986857 >>>>>>>>>>> 445987025 >>>>>>>>>>> 445987106 >>>>>>>>>>> 445987562 >>>>>>>>>>> 445987577 >>>>>>>>>>> 445987715 >>>>>>>>>>> 446107815 >>>>>>>>>>> 446107821 >>>>>>>>>>> 446108097 >>>>>>>>>>> 446108442 >>>>>>>>>>> 446108472 >>>>>>>>>>> 446108553 >>>>>>>>>>> 446108598 >>>>>>>>>>> 446108796 >>>>>>>>>>> 446108910 >>>>>>>>>>> 446108958 >>>>>>>>>>> 45948708 >>>>>>>>>>> 445983389 >>>>>>>>>>> 445983824 >>>>>>>>>>> 446105442 >>>>>>>>>>> 446105460 >>>>>>>>>>> >>>>>>>>>>> Thanks for taking care of this and regards, >>>>>>>>>>> Stefan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Am Montag, 13. Februar 2017 22:08:05 UTC+1 schrieb >>>>>>>>>>> AdWordsApiUser: >>>>>>>>>>>> >>>>>>>>>>>> Thanks Nadine - appreciate your quick reply. We'll keep an eye >>>>>>>>>>>> out and let you know if things get out of hand. >>>>>>>>>>>> >>>>>>>>>>>> On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine >>>>>>>>>>>> Sundquist (AdWords API Team) wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi MG, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for sending over the batch job IDs. I took a look at >>>>>>>>>>>>> the time it took for the jobs to run for the slow running jobs. I >>>>>>>>>>>>> didn't >>>>>>>>>>>>> see any that exceeded 15 minutes and most were completing within >>>>>>>>>>>>> 5 minutes. >>>>>>>>>>>>> There is a bit of load balancing going on with a lot of job >>>>>>>>>>>>> requests, so >>>>>>>>>>>>> it's expected that while some jobs may only take 2 minutes, there >>>>>>>>>>>>> will be a >>>>>>>>>>>>> few that may exceed 10 minutes. Luckily, the jobs don't appear to >>>>>>>>>>>>> be >>>>>>>>>>>>> getting stuck indefinitely, and they're not hanging for hours on >>>>>>>>>>>>> end. I'll >>>>>>>>>>>>> keep an eye on things as I did see a spike in processing time >>>>>>>>>>>>> this morning, >>>>>>>>>>>>> but it looks like things are returning to normal again. Please >>>>>>>>>>>>> reach out if >>>>>>>>>>>>> you see the times increasing even more. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks again, >>>>>>>>>>>>> Nadine, AdWords API Team >>>>>>>>>>>>> >>>>>>>>>>>>> On Monday, February 13, 2017 at 9:39:03 AM UTC-5, >>>>>>>>>>>>> AdWordsApiUser wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hello team, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The issue seems to happen again, though not as bad as last >>>>>>>>>>>>>> time, so I thought I'd let you know before it becomes worse. >>>>>>>>>>>>>> Some jobs are >>>>>>>>>>>>>> taking a long time to process, while others process very fast. >>>>>>>>>>>>>> Last time it >>>>>>>>>>>>>> happened, over 2-3 days, the jobs became slower, and a higher >>>>>>>>>>>>>> proportion of >>>>>>>>>>>>>> jobs started getting stuck till you guys did some magic to make >>>>>>>>>>>>>> everything >>>>>>>>>>>>>> fast again: >>>>>>>>>>>>>> >>>>>>>>>>>>>> stuck/took very long: >>>>>>>>>>>>>> 445491084 >>>>>>>>>>>>>> 445522515 >>>>>>>>>>>>>> 445523004 >>>>>>>>>>>>>> >>>>>>>>>>>>>> not stuck/ran fast: >>>>>>>>>>>>>> 445491588 >>>>>>>>>>>>>> 445352074 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> MG >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thursday, February 2, 2017 at 6:51:38 PM UTC+1, >>>>>>>>>>>>>> AdWordsApiUser wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks Nadine! You and your team saved us a costly rewrite >>>>>>>>>>>>>>> that we had started planning for (without using the bulk job >>>>>>>>>>>>>>> service). >>>>>>>>>>>>>>> We'll keep an eye out and report back if we see issues. Thanks >>>>>>>>>>>>>>> team - >>>>>>>>>>>>>>> you're the most responsive team at Google I've interacted with >>>>>>>>>>>>>>> ;) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine >>>>>>>>>>>>>>> Sundquist (AdWords API Team) wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello Everyone, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have good news! We found the root cause, and the issue >>>>>>>>>>>>>>>> has been fixed. Your batch job processing times should be back >>>>>>>>>>>>>>>> to normal >>>>>>>>>>>>>>>> now. Thanks again for reporting this! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Take care, >>>>>>>>>>>>>>>> Nadine, AdWords API Team >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine >>>>>>>>>>>>>>>> Sundquist (AdWords API Team) wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Greetings Fellow AdWordsians, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the batch job IDs. They are a BIG help. I've >>>>>>>>>>>>>>>>> taken a look at the history of those batch jobs in our >>>>>>>>>>>>>>>>> internal systems, >>>>>>>>>>>>>>>>> and I've found a pattern. I need to run it by a few teammates >>>>>>>>>>>>>>>>> here to see >>>>>>>>>>>>>>>>> if my observations result in finding a root cause. I'll >>>>>>>>>>>>>>>>> update you when >>>>>>>>>>>>>>>>> I've made a bit more progress. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>> Nadine, AdWords API Team >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, >>>>>>>>>>>>>>>>> AdWordsApiUser wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Vishal, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for your reply. However, we have had 100% of the >>>>>>>>>>>>>>>>>> jobs we've submitted in the last 6 hours take over 15 >>>>>>>>>>>>>>>>>> mintues to complete. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> MG >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, >>>>>>>>>>>>>>>>>> Vishal Vinayak (Adwords API Team) wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi MG, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Issues related to batch job performance tend to be >>>>>>>>>>>>>>>>>>> intermittent some of the times. In such cases, cancelling >>>>>>>>>>>>>>>>>>> the job and >>>>>>>>>>>>>>>>>>> sending a new request solves the issue. Could you please >>>>>>>>>>>>>>>>>>> try >>>>>>>>>>>>>>>>>>> cancelling >>>>>>>>>>>>>>>>>>> <https://developers.google.com/adwords/api/docs/guides/batch-jobs#canceling_a_batch_job> >>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>> current jobs with the status >>>>>>>>>>>>>>>>>>> <https://developers.google.com/adwords/api/docs/reference/v201609/BatchJobService.BatchJob#status> >>>>>>>>>>>>>>>>>>> AWAITING_FILE >>>>>>>>>>>>>>>>>>> or ACTIVE and retry? If that doesn't help, please feel free >>>>>>>>>>>>>>>>>>> to revert and >>>>>>>>>>>>>>>>>>> I'll check with my team to see if we can dig into this >>>>>>>>>>>>>>>>>>> issue further. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Vishal, AdWords API Team >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog and Google+: https://googleadsdeveloper.blogspot.com/ https://plus.google.com/+GoogleAdsDevelopers/posts =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. Visit this group at https://groups.google.com/group/adwords-api. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/ecd761a9-864d-47de-9414-d17fb86a37cd%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
