Re: Issues with BatchJobService - jobs remain stuck

Hi Thorsten, You are correct. I do see an anomaly in our batch job processing where the time to process has spiked for a number of people. Thank you for giving us a heads-up! It started May 21st. I can see that it's getting better, so the processing should be speeding up. I'm working with a

Re: Issues with BatchJobService - jobs remain stuck

Hello Vlad, I took a look at your job. It looks like it took 4 hours to complete, but it did finally complete. There was a bit of a load balancing glitch on our batch jobs that is currently clearing up, so your batch jobs should be getting faster again. Best, Nadine, AdWords API Team On

Re: Issues with BatchJobService - jobs remain stuck

Hey there, it seems that starting today we are experiencing performance issues with the batch job service (API v201710). Usually our batchjobs take only seconds to finish up to 1-2 minutes. Since today our batch jobs run within 10, 20, 30 or even 60 minutes with similar amount of operations

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine! I the same problem with my batch job. The ID is 743497970. It hangs a 3 hours. Thanks for the help in advance! Regards, Vlad, Independent Developer -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/

Re: Issues with BatchJobService - jobs remain stuck

Hello Roger, Could you please open a new forum thread for this? This one is pretty old, and it really helps to have a new forum thread for a new issue. Thanks, Nadine, AdWords API Team On Tuesday, October 3, 2017 at 9:29:57 AM UTC-4, ro...@growth-rocket.com wrote: > > Hi Nadine, > > Can you

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine, Can you check this Batch Job Id:575510402 My first batch job, its been 4 hrs and status did not change. On Tuesday, February 21, 2017 at 11:20:12 PM UTC+8, Nadine Sundquist (AdWords API Team) wrote: > > Hi Stefan, > > Thanks for the update! I'm keeping an eye on this as well. > >

Re: Issues with BatchJobService - jobs remain stuck

Hi Josh, i turned out that the jobs eventually went through, it just took very long (around 3 hours). Here are some job ids: 458099621 458108326 458358726 458161757 This morning, everything went back to normal and jobs go through as the weeks before. So no issue any more at the moment.

Re: Issues with BatchJobService - jobs remain stuck

Hi Stefan, Could you share the job IDs for some of the 22 jobs you mentioned? Thanks, Josh, AdWords API Team On Thursday, March 9, 2017 at 12:37:27 AM UTC-5, Stefan Kleine Stegemann wrote: > > > Hi, > > the problem remains and "increases" as far as we can see. This morning > (german time), we

Re: Issues with BatchJobService - jobs remain stuck

Hi, the problem remains and "increases" as far as we can see. This morning (german time), we have 22 jobs stuck in ACTIVE for around two to three hours now. The "old" jobs from yesterday and tuesday are in state CANCELED meanwhile. However, as mentioned before in this thread, it makes no

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine, over the past weeks, batch jobs have been stable and also much faster than before. However, starting yesterday, we have some stuck batch jobs again. This is annoying, to be honest. It creates a lot of effort on our side. Do we have to expect that to happen on a regular base? Here

Re: Issues with BatchJobService - jobs remain stuck

Hi Mike, It looks like the job is still waiting for a file to upload with operations. If you haven't cancelled yet, you can try uploading a file with operations to the upload URL. Cheers, Nadine, AdWords API Team On Friday, February 24, 2017 at 9:41:07 AM UTC-5, Nadine Sundquist (AdWords API

Re: Issues with BatchJobService - jobs remain stuck

Hi Mike, It looks like that job was submitted over 10 hours ago, but it wasn't picked up to be processed. I'm taking a look into it. In the meantime, please try cancelling the job and resubmitting. Thanks, Nadine, AdWords API Team On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine Now in 24 Feb it's stuck again with job ID 449587116. Can you check what happen please? -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog and Google+: https://googleadsdeveloper.blogspot.com/ https://plus.google.com/+GoogleAdsDevelopers/posts

Re: Issues with BatchJobService - jobs remain stuck

Hi Stefan, Thanks for the update! I'm keeping an eye on this as well. Take care, Nadine, AdWords API Team On Saturday, February 18, 2017 at 1:43:38 AM UTC-5, Stefan Kleine Stegemann wrote: > > > Hi Nadine, > > good news on our side as well. Today, we did not have stuck jobs any more > and all

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine, good news on our side as well. Today, we did not have stuck jobs any more and all jobs have been processed in time as they have before this issue. We did decrease the batch job size to 20k though, I don't know whether this relevant or not. Furthermore, I've checked a couple of jobs

Re: Issues with BatchJobService - jobs remain stuck

Hello Stefan, I have good news! We've been working through processing through the queue of hanging jobs. You should be seeing an improvement in that these hanging jobs should be reaching a final state e.g. canceling should now be canceled. You may still see a few hanging, but the majority

Re: Issues with BatchJobService - jobs remain stuck

Hi Stefan, We're actively working on clearing the queue, which includes those jobs in the canceling state, as we're searching for a fix for this. Unfortunately, it won't make a difference whether or not it's a small job versus a larger job. One of my teammates put it quite well when he said

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine, still no improvement, today we have 28 stuck batch jobs across seven adwords accounts. I also checked a sample of 10 batch jobs out of the ones we have canceled over the past days and 2 jobs are actually canceled but 8 are still in state canceling. As this issue is present for four

Re: Issues with BatchJobService - jobs remain stuck

Greetings Stefan, This is useful information when we're trying to figure out how to find a solution for this one. Thanks for passing it along. If jobs have been finishing for you normally by a certain time, then I would expect it to continue to do so. If it doesn't, then it's probably fallen

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine, Thanks for the update, we canceled the jobs from this morning as well. While doing so, we observed that for some jobs, it was not possible to cancel them although they have been in state active. For these jobs, the progress showed 100% completed. Retrying is not really an option

Re: Issues with BatchJobService - jobs remain stuck

Hi Stefan, The good news is that we found where the hanging is happening, but the bad news is that we still don't have a complete fix. If the cancel continues to hang, please get back to me. Right now, it looks like you're doing the right thing by cancelling and retrying. Best, Nadine,

Re: Issues with BatchJobService - jobs remain stuck

Hi Biniam, It looks like we found the culprit on our side, but we haven't found the solution, yet. The workaround for now is to cancel the job that is hanging and to retry the operations that have not executed. I'll keep you in the loop as we make progress. Cheers, Nadine, AdWords API Team

Re: Issues with BatchJobService - jobs remain stuck

We are also experiencing this issue since some time. The following batches are stuck on the same percentage for morethan 24 hours! Please fix. Here is some information from our log file. The batch-job 446153939 returned the status ACTIVE. Progress Stats: 46% Executed 1934

Re: Issues with BatchJobService - jobs remain stuck

Hi Nadine, To update you on this issue, the situation has not improved but rather got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) with eight adwords accounts being affected. Yesterday, it was 30 batch jobs with three accounts being affected and on tuesday it was seven

Re: Issues with BatchJobService - jobs remain stuck

Hi Stefan, Thanks for reporting this. I personally experienced some longer than usual running batch jobs last night as well. We're looking into what caused this last spike in execution time. Cheers, Nadine, AdWords API Team On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine

Re: Issues with BatchJobService - jobs remain stuck

Hi, sorry for hijacking this thread but since yesterday morning, we also started to have significant troubles with batch job service. Jobs remained active for a very long time. The following jobs have been started yesterday morning, german time (see start time is noted for each job) and were

Re: Issues with BatchJobService - jobs remain stuck

Thanks Nadine - appreciate your quick reply. We'll keep an eye out and let you know if things get out of hand. On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine Sundquist (AdWords API Team) wrote: > > Hi MG, > > Thanks for sending over the batch job IDs. I took a look at the time it >

Re: Issues with BatchJobService - jobs remain stuck

Hi MG, Thanks for sending over the batch job IDs. I took a look at the time it took for the jobs to run for the slow running jobs. I didn't see any that exceeded 15 minutes and most were completing within 5 minutes. There is a bit of load balancing going on with a lot of job requests, so it's

Re: Issues with BatchJobService - jobs remain stuck

Hello team, The issue seems to happen again, though not as bad as last time, so I thought I'd let you know before it becomes worse. Some jobs are taking a long time to process, while others process very fast. Last time it happened, over 2-3 days, the jobs became slower, and a higher proportion

Re: Issues with BatchJobService - jobs remain stuck

Thanks Nadine! You and your team saved us a costly rewrite that we had started planning for (without using the bulk job service). We'll keep an eye out and report back if we see issues. Thanks team - you're the most responsive team at Google I've interacted with ;) On Thursday, February 2,

Re: Issues with BatchJobService - jobs remain stuck

Hello Everyone, I have good news! We found the root cause, and the issue has been fixed. Your batch job processing times should be back to normal now. Thanks again for reporting this! Take care, Nadine, AdWords API Team On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist

Re: Issues with BatchJobService - jobs remain stuck

Greetings Fellow AdWordsians, Thanks for the batch job IDs. They are a BIG help. I've taken a look at the history of those batch jobs in our internal systems, and I've found a pattern. I need to run it by a few teammates here to see if my observations result in finding a root cause. I'll

Re: Issues with BatchJobService - jobs remain stuck

Hi Vishal, Thanks for your reply. However, we have had 100% of the jobs we've submitted in the last 6 hours take over 15 mintues to complete. Regards, MG On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak (Adwords API Team) wrote: > > Hi MG, > > Issues related to batch job

Re: Issues with BatchJobService - jobs remain stuck

We are also seeing an issue with the BatchJobService as well. Jobs are stuck in an "ACTIVE" state and when we check the Progress Stats everything says 0 even though we definitely uploaded a file with operations. It appears to be intermittent. Job IDs 441119824 441270816 441145043 On

Re: Issues with BatchJobService - jobs remain stuck

Hi MG, Issues related to batch job performance tend to be intermittent some of the times. In such cases, cancelling the job and sending a new request solves the issue. Could you please try cancelling all