Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API and Google Ads API Forum]

Hi Thorsten,

You are correct. I do see an anomaly in our batch job processing where the 
time to process has spiked for a number of people. Thank you for giving us 
a heads-up! It started May 21st. I can see that it's getting better, so the 
processing should be speeding up. I'm working with a couple people here to 
make sure that the processing time stays down. Please get back to me if the 
processing time continues to be troublesome tomorrow.

Thanks,
Nadine, AdWords API Team

On Wednesday, May 23, 2018 at 6:47:09 AM UTC-4, Thorsten Merz wrote:
>
> Hey there,
>
> it seems that starting today we are experiencing performance issues with 
> the batch job service (API v201710).
>
> Usually our batchjobs take only seconds to finish up to 1-2 minutes. Since 
> today our batch jobs run within 10, 20, 30 or even 60 minutes with similar 
> amount of operations and operation types.
>
> Are there any specific issues currently ongoing with the batch  job queue 
> / service / API?
>
> BatchID  example: 743564948
>
> Some help on how to resolve these issues would be appreciated.
>
> Regards
>
> Thorsten
>
> --
>
> Seems i am not able to create a new discussion :/
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/f8b2e7d5-0bc2-4a84-a00c-84aedb4a6a72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API and Google Ads API Forum]

Hello Vlad,

I took a look at your job. It looks like it took 4 hours to complete, but 
it did finally complete. There was a bit of a load balancing glitch on our 
batch jobs that is currently clearing up, so your batch jobs should be 
getting faster again.

Best,
Nadine, AdWords API Team

On Wednesday, May 23, 2018 at 4:55:34 AM UTC-4, Vlad Gukasov wrote:
>
> Hi Nadine!
>
> I the same problem with my batch job. The ID is 743497970. It hangs a 3 
> hours. Thanks for the help in advance!
>
> Regards,
> Vlad, Independent Developer
>
>
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/fd70f428-5631-4f6e-a96f-799d9741e0fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Thorsten Merz]

Hey there,

it seems that starting today we are experiencing performance issues with 
the batch job service (API v201710).

Usually our batchjobs take only seconds to finish up to 1-2 minutes. Since 
today our batch jobs run within 10, 20, 30 or even 60 minutes with similar 
amount of operations and operation types.

Are there any specific issues currently ongoing with the batch  job queue / 
service / API?

BatchID  example: 743564948

Some help on how to resolve these issues would be appreciated.

Regards

Thorsten

--

Seems i am not able to create a new discussion :/

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/20b64b2d-83e2-406a-889f-d32cf204a141%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Vlad Gukasov]

Hi Nadine!

I the same problem with my batch job. The ID is 743497970. It hangs a 3 
hours. Thanks for the help in advance!

Regards,
Vlad, Independent Developer


-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/3d33a440-4e27-40f8-b373-2c34392d03c7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hello Roger,

Could you please open a new forum thread for this? This one is pretty old, 
and it really helps to have a new forum thread for a new issue.

Thanks,
Nadine, AdWords API Team

On Tuesday, October 3, 2017 at 9:29:57 AM UTC-4, ro...@growth-rocket.com 
wrote:
>
> Hi Nadine,
>
> Can you check this Batch Job Id:575510402
>
> My first batch job, its been 4 hrs and status did not change. 
>
> On Tuesday, February 21, 2017 at 11:20:12 PM UTC+8, Nadine Sundquist 
> (AdWords API Team) wrote:
>>
>> Hi Stefan,
>>
>> Thanks for the update! I'm keeping an eye on this as well.
>>
>> Take care,
>> Nadine, AdWords API Team
>>
>> On Saturday, February 18, 2017 at 1:43:38 AM UTC-5, Stefan Kleine 
>> Stegemann wrote:
>>>
>>>
>>> Hi Nadine,
>>>
>>> good news on our side as well. Today, we did not have stuck jobs any 
>>> more and all jobs have been processed in time as they have before this 
>>> issue. We did decrease the batch job size to 20k though, I don't know 
>>> whether this relevant or not. Furthermore, I've checked a couple of jobs 
>>> from the past days and they all are canceled now.
>>>
>>> At the moment, it looks like the issue is solved. We'll keep an I on it 
>>> and let you know if it happens again. Thanks for taking care of this and 
>>> your constant updates on this issue.
>>>
>>> Best,
>>> Stefan
>>>
>>> Am Freitag, 17. Februar 2017 23:28:11 UTC+1 schrieb Nadine Sundquist 
>>> (AdWords API Team):

 Hello Stefan,

 I have good news! We've been working through processing through the 
 queue of hanging jobs. You should be seeing an improvement in that these 
 hanging jobs should be reaching a final state e.g. canceling should now be 
 canceled. You may still see a few hanging, but the majority should have 
 reached a final state. We're still working on fixing the root cause.

 Take care,
 Nadine, AdWords API Team

 On Friday, February 17, 2017 at 1:47:04 PM UTC-5, Nadine Sundquist 
 (AdWords API Team) wrote:
>
> Hi Stefan,
>
> We're actively working on clearing the queue, which includes those 
> jobs in the canceling state, as we're searching for a fix for this. 
> Unfortunately, it won't make a difference whether or not it's a small job 
> versus a larger job. One of my teammates put it quite well when he said 
> that a larger job may have a higher chance of getting stuck, but on the 
> other hand when it does go through, it gets through more operations. Just 
> so you know, when a job is in a canceling state, it will not execute any 
> more operations, so you can submit the rest of the operations that did 
> not 
> execute in that canceling job. It might take awhile for the job to reach 
> a 
> canceled state, but at least it won't execute any more operations. I 
> realize this isn't optimal, but I do want you to know that is of higher 
> priority to us. It's just really tricky trying to find a solution for 
> what 
> you're seeing. I'll get back to you as we make more progress. Thanks 
> again 
> for providing so much helpful information!
>
> Cheers,
> Nadine, AdWords API Team
>
> On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>>
>> Hi Nadine,
>>
>> still no improvement, today we have 28 stuck batch jobs across seven 
>> adwords accounts. I also checked a sample of 10 batch jobs out of the 
>> ones 
>> we have canceled over the past days and 2 jobs are actually canceled but 
>> 8 
>> are still in state canceling. As this issue is present for four days 
>> now, 
>> it becomes more critical for us because we are unable to deliver the 
>> full 
>> service to some of our customers. Thus, any advise for how the deal with 
>> the "broken" batch jobs is appreciated. Does it, for example, make sense 
>> to 
>> try smaller or larger jobs?
>>
>> Best,
>> Stefan
>>
>> Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
>>  schrieb Nadine Sundquist (AdWords API Team):
>>>
>>> Greetings Stefan,
>>>
>>> This is useful information when we're trying to figure out how to 
>>> find a solution for this one. Thanks for passing it along. If jobs have 
>>> been finishing for you normally by a certain time, then I would expect 
>>> it 
>>> to continue to do so. If it doesn't, then it's probably fallen into the 
>>> issue that we're working on. I'll continue to track this so we can find 
>>> a 
>>> solution.
>>>
>>> Regards,
>>> Nadine, AdWords API Team
>>>
>>> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
>>> Stegemann wrote:


 Hi Nadine,

 Thanks for the update, we canceled the jobs from this morning as 
 well. While doing so, we observed that for some jobs, it was not 
 possible 

Re: Issues with BatchJobService - jobs remain stuck

[roger]

Hi Nadine,

Can you check this Batch Job Id:575510402

My first batch job, its been 4 hrs and status did not change. 

On Tuesday, February 21, 2017 at 11:20:12 PM UTC+8, Nadine Sundquist 
(AdWords API Team) wrote:
>
> Hi Stefan,
>
> Thanks for the update! I'm keeping an eye on this as well.
>
> Take care,
> Nadine, AdWords API Team
>
> On Saturday, February 18, 2017 at 1:43:38 AM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>>
>> Hi Nadine,
>>
>> good news on our side as well. Today, we did not have stuck jobs any more 
>> and all jobs have been processed in time as they have before this issue. We 
>> did decrease the batch job size to 20k though, I don't know whether this 
>> relevant or not. Furthermore, I've checked a couple of jobs from the past 
>> days and they all are canceled now.
>>
>> At the moment, it looks like the issue is solved. We'll keep an I on it 
>> and let you know if it happens again. Thanks for taking care of this and 
>> your constant updates on this issue.
>>
>> Best,
>> Stefan
>>
>> Am Freitag, 17. Februar 2017 23:28:11 UTC+1 schrieb Nadine Sundquist 
>> (AdWords API Team):
>>>
>>> Hello Stefan,
>>>
>>> I have good news! We've been working through processing through the 
>>> queue of hanging jobs. You should be seeing an improvement in that these 
>>> hanging jobs should be reaching a final state e.g. canceling should now be 
>>> canceled. You may still see a few hanging, but the majority should have 
>>> reached a final state. We're still working on fixing the root cause.
>>>
>>> Take care,
>>> Nadine, AdWords API Team
>>>
>>> On Friday, February 17, 2017 at 1:47:04 PM UTC-5, Nadine Sundquist 
>>> (AdWords API Team) wrote:

 Hi Stefan,

 We're actively working on clearing the queue, which includes those jobs 
 in the canceling state, as we're searching for a fix for this. 
 Unfortunately, it won't make a difference whether or not it's a small job 
 versus a larger job. One of my teammates put it quite well when he said 
 that a larger job may have a higher chance of getting stuck, but on the 
 other hand when it does go through, it gets through more operations. Just 
 so you know, when a job is in a canceling state, it will not execute any 
 more operations, so you can submit the rest of the operations that did not 
 execute in that canceling job. It might take awhile for the job to reach a 
 canceled state, but at least it won't execute any more operations. I 
 realize this isn't optimal, but I do want you to know that is of higher 
 priority to us. It's just really tricky trying to find a solution for what 
 you're seeing. I'll get back to you as we make more progress. Thanks again 
 for providing so much helpful information!

 Cheers,
 Nadine, AdWords API Team

 On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine 
 Stegemann wrote:
>
>
> Hi Nadine,
>
> still no improvement, today we have 28 stuck batch jobs across seven 
> adwords accounts. I also checked a sample of 10 batch jobs out of the 
> ones 
> we have canceled over the past days and 2 jobs are actually canceled but 
> 8 
> are still in state canceling. As this issue is present for four days now, 
> it becomes more critical for us because we are unable to deliver the full 
> service to some of our customers. Thus, any advise for how the deal with 
> the "broken" batch jobs is appreciated. Does it, for example, make sense 
> to 
> try smaller or larger jobs?
>
> Best,
> Stefan
>
> Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
>  schrieb Nadine Sundquist (AdWords API Team):
>>
>> Greetings Stefan,
>>
>> This is useful information when we're trying to figure out how to 
>> find a solution for this one. Thanks for passing it along. If jobs have 
>> been finishing for you normally by a certain time, then I would expect 
>> it 
>> to continue to do so. If it doesn't, then it's probably fallen into the 
>> issue that we're working on. I'll continue to track this so we can find 
>> a 
>> solution.
>>
>> Regards,
>> Nadine, AdWords API Team
>>
>> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
>> Stegemann wrote:
>>>
>>>
>>> Hi Nadine,
>>>
>>> Thanks for the update, we canceled the jobs from this morning as 
>>> well. While doing so, we observed that for some jobs, it was not 
>>> possible 
>>> to cancel them although they have been in state active. For these jobs, 
>>> the 
>>> progress showed 100% completed. Retrying is not really an option for 
>>> us, if 
>>> the jobs are not completed by early morning (around 07:00am german 
>>> time), 
>>> it's pointless or even harmful for us to retry. I suppose that, under 
>>> normal conditions, we can expect a job to be finished in time if they 
>>> 

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi Josh,

i turned out that the jobs eventually went through, it just took very long 
(around 3 hours). Here are some job ids:

458099621
458108326
458358726
458161757

This morning, everything went back to normal and jobs go through as the 
weeks before. So no issue any more at the moment.

Thanks for taking care of this.

Best,
Stefan


Am Donnerstag, 9. März 2017 18:45:49 UTC+1 schrieb Josh Radcliff (AdWords 
API Team):
>
> Hi Stefan,
>
> Could you share the job IDs for some of the 22 jobs you mentioned?
>
> Thanks,
> Josh, AdWords API Team
>
> On Thursday, March 9, 2017 at 12:37:27 AM UTC-5, Stefan Kleine Stegemann 
> wrote:
>>
>>
>> Hi,
>>
>> the problem remains and "increases" as far as we can see. This morning 
>> (german time), we have 22 jobs stuck in ACTIVE for around two to three 
>> hours now. The "old" jobs from yesterday and tuesday are in state CANCELED 
>> meanwhile. However, as mentioned before in this thread, it makes no sense 
>> to re-submit a job if it does not get through in the early morning.
>>
>> Regards,
>> Stefan
>>
>>
>> Am Mittwoch, 8. März 2017 20:18:19 UTC+1 schrieb Stefan Kleine Stegemann:
>>>
>>> Hi Nadine,
>>>
>>> over the past weeks, batch jobs have been stable and also much faster 
>>> than before. However, starting yesterday, we have some
>>> stuck batch jobs again. This is annoying, to be honest. It creates a lot 
>>> of effort on our side. Do we have to expect that to
>>> happen on a regular base?
>>>
>>> Here are the job ids of jobs stuck today:
>>> - 456510200
>>> - 456526321
>>> - 456728007
>>> - 456485951
>>> - 456703203
>>> - 456703755
>>>
>>> I did cancel the jobs around 8pm german time after they have remained 
>>> active for around 14 hours. Could you check whether this
>>> is the "old issue" appearing again?
>>>
>>> Best,
>>> Stefan
>>>
>>>
>>> Am Freitag, 24. Februar 2017 16:53:40 UTC+1 schrieb Nadine Sundquist 
>>> (AdWords API Team):

 Hi Mike,

 It looks like the job is still waiting for a file to upload with 
 operations. If you haven't cancelled yet, you can try uploading a file 
 with 
 operations to the upload URL.

 Cheers,
 Nadine, AdWords API Team

 On Friday, February 24, 2017 at 9:41:07 AM UTC-5, Nadine Sundquist 
 (AdWords API Team) wrote:
>
> Hi Mike,
>
> It looks like that job was submitted over 10 hours ago, but it wasn't 
> picked up to be processed. I'm taking a look into it. In the meantime, 
> please try cancelling the job and resubmitting.
>
> Thanks,
> Nadine, AdWords API Team
>
> On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike wrote:
>>
>> Hi Nadine   
>>Now in 24 Feb it's stuck again with job ID 449587116. Can you 
>> check what happen please? 
>>
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/b01a4d90-71d2-449c-9794-158d3f726408%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Josh Radcliff (AdWords API Team)' via AdWords API Forum]

Hi Stefan,

Could you share the job IDs for some of the 22 jobs you mentioned?

Thanks,
Josh, AdWords API Team

On Thursday, March 9, 2017 at 12:37:27 AM UTC-5, Stefan Kleine Stegemann 
wrote:
>
>
> Hi,
>
> the problem remains and "increases" as far as we can see. This morning 
> (german time), we have 22 jobs stuck in ACTIVE for around two to three 
> hours now. The "old" jobs from yesterday and tuesday are in state CANCELED 
> meanwhile. However, as mentioned before in this thread, it makes no sense 
> to re-submit a job if it does not get through in the early morning.
>
> Regards,
> Stefan
>
>
> Am Mittwoch, 8. März 2017 20:18:19 UTC+1 schrieb Stefan Kleine Stegemann:
>>
>> Hi Nadine,
>>
>> over the past weeks, batch jobs have been stable and also much faster 
>> than before. However, starting yesterday, we have some
>> stuck batch jobs again. This is annoying, to be honest. It creates a lot 
>> of effort on our side. Do we have to expect that to
>> happen on a regular base?
>>
>> Here are the job ids of jobs stuck today:
>> - 456510200
>> - 456526321
>> - 456728007
>> - 456485951
>> - 456703203
>> - 456703755
>>
>> I did cancel the jobs around 8pm german time after they have remained 
>> active for around 14 hours. Could you check whether this
>> is the "old issue" appearing again?
>>
>> Best,
>> Stefan
>>
>>
>> Am Freitag, 24. Februar 2017 16:53:40 UTC+1 schrieb Nadine Sundquist 
>> (AdWords API Team):
>>>
>>> Hi Mike,
>>>
>>> It looks like the job is still waiting for a file to upload with 
>>> operations. If you haven't cancelled yet, you can try uploading a file with 
>>> operations to the upload URL.
>>>
>>> Cheers,
>>> Nadine, AdWords API Team
>>>
>>> On Friday, February 24, 2017 at 9:41:07 AM UTC-5, Nadine Sundquist 
>>> (AdWords API Team) wrote:

 Hi Mike,

 It looks like that job was submitted over 10 hours ago, but it wasn't 
 picked up to be processed. I'm taking a look into it. In the meantime, 
 please try cancelling the job and resubmitting.

 Thanks,
 Nadine, AdWords API Team

 On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike wrote:
>
> Hi Nadine   
>Now in 24 Feb it's stuck again with job ID 449587116. Can you 
> check what happen please? 
>


-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/ef8d721f-51d9-46e3-9dec-3a28ca192658%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi,

the problem remains and "increases" as far as we can see. This morning 
(german time), we have 22 jobs stuck in ACTIVE for around two to three 
hours now. The "old" jobs from yesterday and tuesday are in state CANCELED 
meanwhile. However, as mentioned before in this thread, it makes no sense 
to re-submit a job if it does not get through in the early morning.

Regards,
Stefan


Am Mittwoch, 8. März 2017 20:18:19 UTC+1 schrieb Stefan Kleine Stegemann:
>
> Hi Nadine,
>
> over the past weeks, batch jobs have been stable and also much faster than 
> before. However, starting yesterday, we have some
> stuck batch jobs again. This is annoying, to be honest. It creates a lot 
> of effort on our side. Do we have to expect that to
> happen on a regular base?
>
> Here are the job ids of jobs stuck today:
> - 456510200
> - 456526321
> - 456728007
> - 456485951
> - 456703203
> - 456703755
>
> I did cancel the jobs around 8pm german time after they have remained 
> active for around 14 hours. Could you check whether this
> is the "old issue" appearing again?
>
> Best,
> Stefan
>
>
> Am Freitag, 24. Februar 2017 16:53:40 UTC+1 schrieb Nadine Sundquist 
> (AdWords API Team):
>>
>> Hi Mike,
>>
>> It looks like the job is still waiting for a file to upload with 
>> operations. If you haven't cancelled yet, you can try uploading a file with 
>> operations to the upload URL.
>>
>> Cheers,
>> Nadine, AdWords API Team
>>
>> On Friday, February 24, 2017 at 9:41:07 AM UTC-5, Nadine Sundquist 
>> (AdWords API Team) wrote:
>>>
>>> Hi Mike,
>>>
>>> It looks like that job was submitted over 10 hours ago, but it wasn't 
>>> picked up to be processed. I'm taking a look into it. In the meantime, 
>>> please try cancelling the job and resubmitting.
>>>
>>> Thanks,
>>> Nadine, AdWords API Team
>>>
>>> On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike wrote:

 Hi Nadine   
Now in 24 Feb it's stuck again with job ID 449587116. Can you check 
 what happen please? 

>>>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/dc64d60c-c72f-456a-83c7-078477540e16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi Nadine,

over the past weeks, batch jobs have been stable and also much faster than 
before. However, starting yesterday, we have some
stuck batch jobs again. This is annoying, to be honest. It creates a lot of 
effort on our side. Do we have to expect that to
happen on a regular base?

Here are the job ids of jobs stuck today:
- 456510200
- 456526321
- 456728007
- 456485951
- 456703203
- 456703755

I did cancel the jobs around 8pm german time after they have remained 
active for around 14 hours. Could you check whether this
is the "old issue" appearing again?

Best,
Stefan


Am Freitag, 24. Februar 2017 16:53:40 UTC+1 schrieb Nadine Sundquist 
(AdWords API Team):
>
> Hi Mike,
>
> It looks like the job is still waiting for a file to upload with 
> operations. If you haven't cancelled yet, you can try uploading a file with 
> operations to the upload URL.
>
> Cheers,
> Nadine, AdWords API Team
>
> On Friday, February 24, 2017 at 9:41:07 AM UTC-5, Nadine Sundquist 
> (AdWords API Team) wrote:
>>
>> Hi Mike,
>>
>> It looks like that job was submitted over 10 hours ago, but it wasn't 
>> picked up to be processed. I'm taking a look into it. In the meantime, 
>> please try cancelling the job and resubmitting.
>>
>> Thanks,
>> Nadine, AdWords API Team
>>
>> On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike wrote:
>>>
>>> Hi Nadine   
>>>Now in 24 Feb it's stuck again with job ID 449587116. Can you check 
>>> what happen please? 
>>>
>>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/79abcf26-98cc-401d-93a9-85e27108e960%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Mike,

It looks like the job is still waiting for a file to upload with 
operations. If you haven't cancelled yet, you can try uploading a file with 
operations to the upload URL.

Cheers,
Nadine, AdWords API Team

On Friday, February 24, 2017 at 9:41:07 AM UTC-5, Nadine Sundquist (AdWords 
API Team) wrote:
>
> Hi Mike,
>
> It looks like that job was submitted over 10 hours ago, but it wasn't 
> picked up to be processed. I'm taking a look into it. In the meantime, 
> please try cancelling the job and resubmitting.
>
> Thanks,
> Nadine, AdWords API Team
>
> On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike wrote:
>>
>> Hi Nadine   
>>Now in 24 Feb it's stuck again with job ID 449587116. Can you check 
>> what happen please? 
>>
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/60035194-2fbb-4a1b-91a7-75d0d329c47b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Mike,

It looks like that job was submitted over 10 hours ago, but it wasn't 
picked up to be processed. I'm taking a look into it. In the meantime, 
please try cancelling the job and resubmitting.

Thanks,
Nadine, AdWords API Team

On Thursday, February 23, 2017 at 9:33:05 PM UTC-5, Mike wrote:
>
> Hi Nadine   
>Now in 24 Feb it's stuck again with job ID 449587116. Can you check 
> what happen please? 
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/c15d4ef0-b127-4b63-b23b-b774d94aabba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Mike]

Hi Nadine   
   Now in 24 Feb it's stuck again with job ID 449587116. Can you check what 
happen please? 

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/f77fde09-162f-4413-bc7b-95380f788d6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Stefan,

Thanks for the update! I'm keeping an eye on this as well.

Take care,
Nadine, AdWords API Team

On Saturday, February 18, 2017 at 1:43:38 AM UTC-5, Stefan Kleine Stegemann 
wrote:
>
>
> Hi Nadine,
>
> good news on our side as well. Today, we did not have stuck jobs any more 
> and all jobs have been processed in time as they have before this issue. We 
> did decrease the batch job size to 20k though, I don't know whether this 
> relevant or not. Furthermore, I've checked a couple of jobs from the past 
> days and they all are canceled now.
>
> At the moment, it looks like the issue is solved. We'll keep an I on it 
> and let you know if it happens again. Thanks for taking care of this and 
> your constant updates on this issue.
>
> Best,
> Stefan
>
> Am Freitag, 17. Februar 2017 23:28:11 UTC+1 schrieb Nadine Sundquist 
> (AdWords API Team):
>>
>> Hello Stefan,
>>
>> I have good news! We've been working through processing through the queue 
>> of hanging jobs. You should be seeing an improvement in that these hanging 
>> jobs should be reaching a final state e.g. canceling should now be 
>> canceled. You may still see a few hanging, but the majority should have 
>> reached a final state. We're still working on fixing the root cause.
>>
>> Take care,
>> Nadine, AdWords API Team
>>
>> On Friday, February 17, 2017 at 1:47:04 PM UTC-5, Nadine Sundquist 
>> (AdWords API Team) wrote:
>>>
>>> Hi Stefan,
>>>
>>> We're actively working on clearing the queue, which includes those jobs 
>>> in the canceling state, as we're searching for a fix for this. 
>>> Unfortunately, it won't make a difference whether or not it's a small job 
>>> versus a larger job. One of my teammates put it quite well when he said 
>>> that a larger job may have a higher chance of getting stuck, but on the 
>>> other hand when it does go through, it gets through more operations. Just 
>>> so you know, when a job is in a canceling state, it will not execute any 
>>> more operations, so you can submit the rest of the operations that did not 
>>> execute in that canceling job. It might take awhile for the job to reach a 
>>> canceled state, but at least it won't execute any more operations. I 
>>> realize this isn't optimal, but I do want you to know that is of higher 
>>> priority to us. It's just really tricky trying to find a solution for what 
>>> you're seeing. I'll get back to you as we make more progress. Thanks again 
>>> for providing so much helpful information!
>>>
>>> Cheers,
>>> Nadine, AdWords API Team
>>>
>>> On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine 
>>> Stegemann wrote:


 Hi Nadine,

 still no improvement, today we have 28 stuck batch jobs across seven 
 adwords accounts. I also checked a sample of 10 batch jobs out of the ones 
 we have canceled over the past days and 2 jobs are actually canceled but 8 
 are still in state canceling. As this issue is present for four days now, 
 it becomes more critical for us because we are unable to deliver the full 
 service to some of our customers. Thus, any advise for how the deal with 
 the "broken" batch jobs is appreciated. Does it, for example, make sense 
 to 
 try smaller or larger jobs?

 Best,
 Stefan

 Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
  schrieb Nadine Sundquist (AdWords API Team):
>
> Greetings Stefan,
>
> This is useful information when we're trying to figure out how to find 
> a solution for this one. Thanks for passing it along. If jobs have been 
> finishing for you normally by a certain time, then I would expect it to 
> continue to do so. If it doesn't, then it's probably fallen into the 
> issue 
> that we're working on. I'll continue to track this so we can find a 
> solution.
>
> Regards,
> Nadine, AdWords API Team
>
> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>>
>> Hi Nadine,
>>
>> Thanks for the update, we canceled the jobs from this morning as 
>> well. While doing so, we observed that for some jobs, it was not 
>> possible 
>> to cancel them although they have been in state active. For these jobs, 
>> the 
>> progress showed 100% completed. Retrying is not really an option for us, 
>> if 
>> the jobs are not completed by early morning (around 07:00am german 
>> time), 
>> it's pointless or even harmful for us to retry. I suppose that, under 
>> normal conditions, we can expect a job to be finished in time if they 
>> have 
>> been submitted by 5:00am? At least that was not a problem in the past. 
>> Also, it feels like pushing more and more jobs to the "queue" might make 
>> things even worse.
>>
>> I checked again some of the jobs we did cancel (including jobs from 
>> tuesday) and they all are still in status canceling, so no progress on 
>> 

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi Nadine,

good news on our side as well. Today, we did not have stuck jobs any more 
and all jobs have been processed in time as they have before this issue. We 
did decrease the batch job size to 20k though, I don't know whether this 
relevant or not. Furthermore, I've checked a couple of jobs from the past 
days and they all are canceled now.

At the moment, it looks like the issue is solved. We'll keep an I on it and 
let you know if it happens again. Thanks for taking care of this and your 
constant updates on this issue.

Best,
Stefan

Am Freitag, 17. Februar 2017 23:28:11 UTC+1 schrieb Nadine Sundquist 
(AdWords API Team):
>
> Hello Stefan,
>
> I have good news! We've been working through processing through the queue 
> of hanging jobs. You should be seeing an improvement in that these hanging 
> jobs should be reaching a final state e.g. canceling should now be 
> canceled. You may still see a few hanging, but the majority should have 
> reached a final state. We're still working on fixing the root cause.
>
> Take care,
> Nadine, AdWords API Team
>
> On Friday, February 17, 2017 at 1:47:04 PM UTC-5, Nadine Sundquist 
> (AdWords API Team) wrote:
>>
>> Hi Stefan,
>>
>> We're actively working on clearing the queue, which includes those jobs 
>> in the canceling state, as we're searching for a fix for this. 
>> Unfortunately, it won't make a difference whether or not it's a small job 
>> versus a larger job. One of my teammates put it quite well when he said 
>> that a larger job may have a higher chance of getting stuck, but on the 
>> other hand when it does go through, it gets through more operations. Just 
>> so you know, when a job is in a canceling state, it will not execute any 
>> more operations, so you can submit the rest of the operations that did not 
>> execute in that canceling job. It might take awhile for the job to reach a 
>> canceled state, but at least it won't execute any more operations. I 
>> realize this isn't optimal, but I do want you to know that is of higher 
>> priority to us. It's just really tricky trying to find a solution for what 
>> you're seeing. I'll get back to you as we make more progress. Thanks again 
>> for providing so much helpful information!
>>
>> Cheers,
>> Nadine, AdWords API Team
>>
>> On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine Stegemann 
>> wrote:
>>>
>>>
>>> Hi Nadine,
>>>
>>> still no improvement, today we have 28 stuck batch jobs across seven 
>>> adwords accounts. I also checked a sample of 10 batch jobs out of the ones 
>>> we have canceled over the past days and 2 jobs are actually canceled but 8 
>>> are still in state canceling. As this issue is present for four days now, 
>>> it becomes more critical for us because we are unable to deliver the full 
>>> service to some of our customers. Thus, any advise for how the deal with 
>>> the "broken" batch jobs is appreciated. Does it, for example, make sense to 
>>> try smaller or larger jobs?
>>>
>>> Best,
>>> Stefan
>>>
>>> Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
>>>  schrieb Nadine Sundquist (AdWords API Team):

 Greetings Stefan,

 This is useful information when we're trying to figure out how to find 
 a solution for this one. Thanks for passing it along. If jobs have been 
 finishing for you normally by a certain time, then I would expect it to 
 continue to do so. If it doesn't, then it's probably fallen into the issue 
 that we're working on. I'll continue to track this so we can find a 
 solution.

 Regards,
 Nadine, AdWords API Team

 On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
 Stegemann wrote:
>
>
> Hi Nadine,
>
> Thanks for the update, we canceled the jobs from this morning as well. 
> While doing so, we observed that for some jobs, it was not possible to 
> cancel them although they have been in state active. For these jobs, the 
> progress showed 100% completed. Retrying is not really an option for us, 
> if 
> the jobs are not completed by early morning (around 07:00am german time), 
> it's pointless or even harmful for us to retry. I suppose that, under 
> normal conditions, we can expect a job to be finished in time if they 
> have 
> been submitted by 5:00am? At least that was not a problem in the past. 
> Also, it feels like pushing more and more jobs to the "queue" might make 
> things even worse.
>
> I checked again some of the jobs we did cancel (including jobs from 
> tuesday) and they all are still in status canceling, so no progress on 
> that 
> side.
>
> Best,
> Stefan
>
> Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine 
> Sundquist (AdWords API Team):
>>
>> Hi Stefan,
>>
>> The good news is that we found where the hanging is happening, but 
>> the bad news is that we still don't have a complete fix. If the cancel 
>> 

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hello Stefan,

I have good news! We've been working through processing through the queue 
of hanging jobs. You should be seeing an improvement in that these hanging 
jobs should be reaching a final state e.g. canceling should now be 
canceled. You may still see a few hanging, but the majority should have 
reached a final state. We're still working on fixing the root cause.

Take care,
Nadine, AdWords API Team

On Friday, February 17, 2017 at 1:47:04 PM UTC-5, Nadine Sundquist (AdWords 
API Team) wrote:
>
> Hi Stefan,
>
> We're actively working on clearing the queue, which includes those jobs in 
> the canceling state, as we're searching for a fix for this. Unfortunately, 
> it won't make a difference whether or not it's a small job versus a larger 
> job. One of my teammates put it quite well when he said that a larger job 
> may have a higher chance of getting stuck, but on the other hand when it 
> does go through, it gets through more operations. Just so you know, when a 
> job is in a canceling state, it will not execute any more operations, so 
> you can submit the rest of the operations that did not execute in that 
> canceling job. It might take awhile for the job to reach a canceled state, 
> but at least it won't execute any more operations. I realize this isn't 
> optimal, but I do want you to know that is of higher priority to us. It's 
> just really tricky trying to find a solution for what you're seeing. I'll 
> get back to you as we make more progress. Thanks again for providing so 
> much helpful information!
>
> Cheers,
> Nadine, AdWords API Team
>
> On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine Stegemann 
> wrote:
>>
>>
>> Hi Nadine,
>>
>> still no improvement, today we have 28 stuck batch jobs across seven 
>> adwords accounts. I also checked a sample of 10 batch jobs out of the ones 
>> we have canceled over the past days and 2 jobs are actually canceled but 8 
>> are still in state canceling. As this issue is present for four days now, 
>> it becomes more critical for us because we are unable to deliver the full 
>> service to some of our customers. Thus, any advise for how the deal with 
>> the "broken" batch jobs is appreciated. Does it, for example, make sense to 
>> try smaller or larger jobs?
>>
>> Best,
>> Stefan
>>
>> Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
>>  schrieb Nadine Sundquist (AdWords API Team):
>>>
>>> Greetings Stefan,
>>>
>>> This is useful information when we're trying to figure out how to find a 
>>> solution for this one. Thanks for passing it along. If jobs have been 
>>> finishing for you normally by a certain time, then I would expect it to 
>>> continue to do so. If it doesn't, then it's probably fallen into the issue 
>>> that we're working on. I'll continue to track this so we can find a 
>>> solution.
>>>
>>> Regards,
>>> Nadine, AdWords API Team
>>>
>>> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
>>> Stegemann wrote:


 Hi Nadine,

 Thanks for the update, we canceled the jobs from this morning as well. 
 While doing so, we observed that for some jobs, it was not possible to 
 cancel them although they have been in state active. For these jobs, the 
 progress showed 100% completed. Retrying is not really an option for us, 
 if 
 the jobs are not completed by early morning (around 07:00am german time), 
 it's pointless or even harmful for us to retry. I suppose that, under 
 normal conditions, we can expect a job to be finished in time if they have 
 been submitted by 5:00am? At least that was not a problem in the past. 
 Also, it feels like pushing more and more jobs to the "queue" might make 
 things even worse.

 I checked again some of the jobs we did cancel (including jobs from 
 tuesday) and they all are still in status canceling, so no progress on 
 that 
 side.

 Best,
 Stefan

 Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine Sundquist 
 (AdWords API Team):
>
> Hi Stefan,
>
> The good news is that we found where the hanging is happening, but the 
> bad news is that we still don't have a complete fix. If the cancel 
> continues to hang, please get back to me. Right now, it looks like you're 
> doing the right thing by cancelling and retrying. 
>
> Best,
> Nadine, AdWords API Team
>
> On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>>
>> Hi Nadine,
>>
>> To update you on this issue, the situation has not improved but 
>> rather got slightly worse. Today we have 33 stuck batch jobs (in status 
>> ACTIVE) with eight adwords accounts being affected. Yesterday, it was 30 
>> batch jobs with three accounts being affected and on tuesday it was 
>> seven 
>> jobs, also with three accounts. So we see a slightly "negative" tendency 
>> here. I also checked some 

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Stefan,

We're actively working on clearing the queue, which includes those jobs in 
the canceling state, as we're searching for a fix for this. Unfortunately, 
it won't make a difference whether or not it's a small job versus a larger 
job. One of my teammates put it quite well when he said that a larger job 
may have a higher chance of getting stuck, but on the other hand when it 
does go through, it gets through more operations. Just so you know, when a 
job is in a canceling state, it will not execute any more operations, so 
you can submit the rest of the operations that did not execute in that 
canceling job. It might take awhile for the job to reach a canceled state, 
but at least it won't execute any more operations. I realize this isn't 
optimal, but I do want you to know that is of higher priority to us. It's 
just really tricky trying to find a solution for what you're seeing. I'll 
get back to you as we make more progress. Thanks again for providing so 
much helpful information!

Cheers,
Nadine, AdWords API Team

On Friday, February 17, 2017 at 3:29:54 AM UTC-5, Stefan Kleine Stegemann 
wrote:
>
>
> Hi Nadine,
>
> still no improvement, today we have 28 stuck batch jobs across seven 
> adwords accounts. I also checked a sample of 10 batch jobs out of the ones 
> we have canceled over the past days and 2 jobs are actually canceled but 8 
> are still in state canceling. As this issue is present for four days now, 
> it becomes more critical for us because we are unable to deliver the full 
> service to some of our customers. Thus, any advise for how the deal with 
> the "broken" batch jobs is appreciated. Does it, for example, make sense to 
> try smaller or larger jobs?
>
> Best,
> Stefan
>
> Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
>  schrieb Nadine Sundquist (AdWords API Team):
>>
>> Greetings Stefan,
>>
>> This is useful information when we're trying to figure out how to find a 
>> solution for this one. Thanks for passing it along. If jobs have been 
>> finishing for you normally by a certain time, then I would expect it to 
>> continue to do so. If it doesn't, then it's probably fallen into the issue 
>> that we're working on. I'll continue to track this so we can find a 
>> solution.
>>
>> Regards,
>> Nadine, AdWords API Team
>>
>> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
>> Stegemann wrote:
>>>
>>>
>>> Hi Nadine,
>>>
>>> Thanks for the update, we canceled the jobs from this morning as well. 
>>> While doing so, we observed that for some jobs, it was not possible to 
>>> cancel them although they have been in state active. For these jobs, the 
>>> progress showed 100% completed. Retrying is not really an option for us, if 
>>> the jobs are not completed by early morning (around 07:00am german time), 
>>> it's pointless or even harmful for us to retry. I suppose that, under 
>>> normal conditions, we can expect a job to be finished in time if they have 
>>> been submitted by 5:00am? At least that was not a problem in the past. 
>>> Also, it feels like pushing more and more jobs to the "queue" might make 
>>> things even worse.
>>>
>>> I checked again some of the jobs we did cancel (including jobs from 
>>> tuesday) and they all are still in status canceling, so no progress on that 
>>> side.
>>>
>>> Best,
>>> Stefan
>>>
>>> Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine Sundquist 
>>> (AdWords API Team):

 Hi Stefan,

 The good news is that we found where the hanging is happening, but the 
 bad news is that we still don't have a complete fix. If the cancel 
 continues to hang, please get back to me. Right now, it looks like you're 
 doing the right thing by cancelling and retrying. 

 Best,
 Nadine, AdWords API Team

 On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine 
 Stegemann wrote:
>
>
> Hi Nadine,
>
> To update you on this issue, the situation has not improved but rather 
> got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) 
> with eight adwords accounts being affected. Yesterday, it was 30 batch 
> jobs 
> with three accounts being affected and on tuesday it was seven jobs, also 
> with three accounts. So we see a slightly "negative" tendency here. I 
> also 
> checked some of the batch jobs we canceled yesterday and they are still 
> in 
> status "CANCELING". Even the jobs which got stuck on tuesday and which we 
> also have canceled yesterday remain in "CANCELING". It seems that these 
> jobs are not doing anything anymore and that there is something broken 
> fundamentally.
>
> Do you have any update on this issue? Can we expect batch jobs to come 
> back to "normal"? We'd like to know that because if not, we need to take 
> action to deal with such scenarios.
>
> Best,
> Stefan
>
> Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine 

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi Nadine,

still no improvement, today we have 28 stuck batch jobs across seven 
adwords accounts. I also checked a sample of 10 batch jobs out of the ones 
we have canceled over the past days and 2 jobs are actually canceled but 8 
are still in state canceling. As this issue is present for four days now, 
it becomes more critical for us because we are unable to deliver the full 
service to some of our customers. Thus, any advise for how the deal with 
the "broken" batch jobs is appreciated. Does it, for example, make sense to 
try smaller or larger jobs?

Best,
Stefan

Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1
 schrieb Nadine Sundquist (AdWords API Team):
>
> Greetings Stefan,
>
> This is useful information when we're trying to figure out how to find a 
> solution for this one. Thanks for passing it along. If jobs have been 
> finishing for you normally by a certain time, then I would expect it to 
> continue to do so. If it doesn't, then it's probably fallen into the issue 
> that we're working on. I'll continue to track this so we can find a 
> solution.
>
> Regards,
> Nadine, AdWords API Team
>
> On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>>
>> Hi Nadine,
>>
>> Thanks for the update, we canceled the jobs from this morning as well. 
>> While doing so, we observed that for some jobs, it was not possible to 
>> cancel them although they have been in state active. For these jobs, the 
>> progress showed 100% completed. Retrying is not really an option for us, if 
>> the jobs are not completed by early morning (around 07:00am german time), 
>> it's pointless or even harmful for us to retry. I suppose that, under 
>> normal conditions, we can expect a job to be finished in time if they have 
>> been submitted by 5:00am? At least that was not a problem in the past. 
>> Also, it feels like pushing more and more jobs to the "queue" might make 
>> things even worse.
>>
>> I checked again some of the jobs we did cancel (including jobs from 
>> tuesday) and they all are still in status canceling, so no progress on that 
>> side.
>>
>> Best,
>> Stefan
>>
>> Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine Sundquist 
>> (AdWords API Team):
>>>
>>> Hi Stefan,
>>>
>>> The good news is that we found where the hanging is happening, but the 
>>> bad news is that we still don't have a complete fix. If the cancel 
>>> continues to hang, please get back to me. Right now, it looks like you're 
>>> doing the right thing by cancelling and retrying. 
>>>
>>> Best,
>>> Nadine, AdWords API Team
>>>
>>> On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine 
>>> Stegemann wrote:


 Hi Nadine,

 To update you on this issue, the situation has not improved but rather 
 got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) 
 with eight adwords accounts being affected. Yesterday, it was 30 batch 
 jobs 
 with three accounts being affected and on tuesday it was seven jobs, also 
 with three accounts. So we see a slightly "negative" tendency here. I also 
 checked some of the batch jobs we canceled yesterday and they are still in 
 status "CANCELING". Even the jobs which got stuck on tuesday and which we 
 also have canceled yesterday remain in "CANCELING". It seems that these 
 jobs are not doing anything anymore and that there is something broken 
 fundamentally.

 Do you have any update on this issue? Can we expect batch jobs to come 
 back to "normal"? We'd like to know that because if not, we need to take 
 action to deal with such scenarios.

 Best,
 Stefan

 Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine Sundquist 
 (AdWords API Team):
>
> Hi Stefan,
>
> Thanks for reporting this. I personally experienced some longer than 
> usual running batch jobs last night as well. We're looking into what 
> caused 
> this last spike in execution time.
>
> Cheers,
> Nadine, AdWords API Team
>
> On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>> Hi,
>>
>> sorry for hijacking this thread but since yesterday morning, we also 
>> started to have significant troubles with batch job service. Jobs 
>> remained 
>> active for a very long time. The following jobs have been started 
>> yesterday 
>> morning, german time (see start time is noted for each job) and were 
>> still 
>> active this morning:
>>
>> Job 445516765 | 2017-02-14T03:34
>> Job 445553000 | 2017-02-14T03:20
>> Job 445532407 | 2017-02-14T03:58
>> Job 445532515 | 2017-02-14T03:58
>> Job 445693959 | 2017-02-14T03:58
>> Job 445694061 | 2017-02-14T03:59
>> Job 445694595 | 2017-02-14T03:59
>>
>> We've canceled the jobs this morning (after they are still reported 
>> to be active, how can that be?), but now 

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Greetings Stefan,

This is useful information when we're trying to figure out how to find a 
solution for this one. Thanks for passing it along. If jobs have been 
finishing for you normally by a certain time, then I would expect it to 
continue to do so. If it doesn't, then it's probably fallen into the issue 
that we're working on. I'll continue to track this so we can find a 
solution.

Regards,
Nadine, AdWords API Team

On Thursday, February 16, 2017 at 4:52:15 PM UTC-5, Stefan Kleine Stegemann 
wrote:
>
>
> Hi Nadine,
>
> Thanks for the update, we canceled the jobs from this morning as well. 
> While doing so, we observed that for some jobs, it was not possible to 
> cancel them although they have been in state active. For these jobs, the 
> progress showed 100% completed. Retrying is not really an option for us, if 
> the jobs are not completed by early morning (around 07:00am german time), 
> it's pointless or even harmful for us to retry. I suppose that, under 
> normal conditions, we can expect a job to be finished in time if they have 
> been submitted by 5:00am? At least that was not a problem in the past. 
> Also, it feels like pushing more and more jobs to the "queue" might make 
> things even worse.
>
> I checked again some of the jobs we did cancel (including jobs from 
> tuesday) and they all are still in status canceling, so no progress on that 
> side.
>
> Best,
> Stefan
>
> Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine Sundquist 
> (AdWords API Team):
>>
>> Hi Stefan,
>>
>> The good news is that we found where the hanging is happening, but the 
>> bad news is that we still don't have a complete fix. If the cancel 
>> continues to hang, please get back to me. Right now, it looks like you're 
>> doing the right thing by cancelling and retrying. 
>>
>> Best,
>> Nadine, AdWords API Team
>>
>> On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine 
>> Stegemann wrote:
>>>
>>>
>>> Hi Nadine,
>>>
>>> To update you on this issue, the situation has not improved but rather 
>>> got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) 
>>> with eight adwords accounts being affected. Yesterday, it was 30 batch jobs 
>>> with three accounts being affected and on tuesday it was seven jobs, also 
>>> with three accounts. So we see a slightly "negative" tendency here. I also 
>>> checked some of the batch jobs we canceled yesterday and they are still in 
>>> status "CANCELING". Even the jobs which got stuck on tuesday and which we 
>>> also have canceled yesterday remain in "CANCELING". It seems that these 
>>> jobs are not doing anything anymore and that there is something broken 
>>> fundamentally.
>>>
>>> Do you have any update on this issue? Can we expect batch jobs to come 
>>> back to "normal"? We'd like to know that because if not, we need to take 
>>> action to deal with such scenarios.
>>>
>>> Best,
>>> Stefan
>>>
>>> Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine Sundquist 
>>> (AdWords API Team):

 Hi Stefan,

 Thanks for reporting this. I personally experienced some longer than 
 usual running batch jobs last night as well. We're looking into what 
 caused 
 this last spike in execution time.

 Cheers,
 Nadine, AdWords API Team

 On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine 
 Stegemann wrote:
>
> Hi,
>
> sorry for hijacking this thread but since yesterday morning, we also 
> started to have significant troubles with batch job service. Jobs 
> remained 
> active for a very long time. The following jobs have been started 
> yesterday 
> morning, german time (see start time is noted for each job) and were 
> still 
> active this morning:
>
> Job 445516765 | 2017-02-14T03:34
> Job 445553000 | 2017-02-14T03:20
> Job 445532407 | 2017-02-14T03:58
> Job 445532515 | 2017-02-14T03:58
> Job 445693959 | 2017-02-14T03:58
> Job 445694061 | 2017-02-14T03:59
> Job 445694595 | 2017-02-14T03:59
>
> We've canceled the jobs this morning (after they are still reported to 
> be active, how can that be?), but now they are in canceling state for 
> about 
> an hour (at the time of this writing). Furthermore, the situation got 
> worse 
> this morning, with 30 batch jobs being active for a couple of hours now. 
> The ids of these jobs are attached to the end of this post. We will 
> cancel 
> these jobs as well because we do not expect them to finish, based on 
> yesterday's experience.
>
> As an additional information, the problems seem to affect accounts 
> with campaigns with a large to huge number or product partitions / 
> keywords. The problematic jobs were "spreaded" over three adwords 
> accounts 
> yesterday and three accounts today. Two of the accounts affected 
> yesterday 
> are also affected today. We do use these batch jobs 

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi Nadine,

Thanks for the update, we canceled the jobs from this morning as well. 
While doing so, we observed that for some jobs, it was not possible to 
cancel them although they have been in state active. For these jobs, the 
progress showed 100% completed. Retrying is not really an option for us, if 
the jobs are not completed by early morning (around 07:00am german time), 
it's pointless or even harmful for us to retry. I suppose that, under 
normal conditions, we can expect a job to be finished in time if they have 
been submitted by 5:00am? At least that was not a problem in the past. 
Also, it feels like pushing more and more jobs to the "queue" might make 
things even worse.

I checked again some of the jobs we did cancel (including jobs from 
tuesday) and they all are still in status canceling, so no progress on that 
side.

Best,
Stefan

Am Donnerstag, 16. Februar 2017 21:11:46 UTC+1 schrieb Nadine Sundquist 
(AdWords API Team):
>
> Hi Stefan,
>
> The good news is that we found where the hanging is happening, but the bad 
> news is that we still don't have a complete fix. If the cancel continues to 
> hang, please get back to me. Right now, it looks like you're doing the 
> right thing by cancelling and retrying. 
>
> Best,
> Nadine, AdWords API Team
>
> On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>>
>> Hi Nadine,
>>
>> To update you on this issue, the situation has not improved but rather 
>> got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) 
>> with eight adwords accounts being affected. Yesterday, it was 30 batch jobs 
>> with three accounts being affected and on tuesday it was seven jobs, also 
>> with three accounts. So we see a slightly "negative" tendency here. I also 
>> checked some of the batch jobs we canceled yesterday and they are still in 
>> status "CANCELING". Even the jobs which got stuck on tuesday and which we 
>> also have canceled yesterday remain in "CANCELING". It seems that these 
>> jobs are not doing anything anymore and that there is something broken 
>> fundamentally.
>>
>> Do you have any update on this issue? Can we expect batch jobs to come 
>> back to "normal"? We'd like to know that because if not, we need to take 
>> action to deal with such scenarios.
>>
>> Best,
>> Stefan
>>
>> Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine Sundquist 
>> (AdWords API Team):
>>>
>>> Hi Stefan,
>>>
>>> Thanks for reporting this. I personally experienced some longer than 
>>> usual running batch jobs last night as well. We're looking into what caused 
>>> this last spike in execution time.
>>>
>>> Cheers,
>>> Nadine, AdWords API Team
>>>
>>> On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine 
>>> Stegemann wrote:

 Hi,

 sorry for hijacking this thread but since yesterday morning, we also 
 started to have significant troubles with batch job service. Jobs remained 
 active for a very long time. The following jobs have been started 
 yesterday 
 morning, german time (see start time is noted for each job) and were still 
 active this morning:

 Job 445516765 | 2017-02-14T03:34
 Job 445553000 | 2017-02-14T03:20
 Job 445532407 | 2017-02-14T03:58
 Job 445532515 | 2017-02-14T03:58
 Job 445693959 | 2017-02-14T03:58
 Job 445694061 | 2017-02-14T03:59
 Job 445694595 | 2017-02-14T03:59

 We've canceled the jobs this morning (after they are still reported to 
 be active, how can that be?), but now they are in canceling state for 
 about 
 an hour (at the time of this writing). Furthermore, the situation got 
 worse 
 this morning, with 30 batch jobs being active for a couple of hours now. 
 The ids of these jobs are attached to the end of this post. We will cancel 
 these jobs as well because we do not expect them to finish, based on 
 yesterday's experience.

 As an additional information, the problems seem to affect accounts with 
 campaigns with a large to huge number or product partitions / keywords. 
 The 
 problematic jobs were "spreaded" over three adwords accounts yesterday and 
 three accounts today. Two of the accounts affected yesterday are also 
 affected today. We do use these batch jobs to set cpc bids for keywords 
 and 
 product partitions and use a maximum of 40.000 operations per job. We 
 experimented with this number in the past and found that to be a 
 "suitable" 
 balance between performance and reliability of batch jobs. All jobs 
 contain 
 only one type of operation. While multiple jobs might affect the same 
 campaign / adgroups, they do not affect the same keywords / product 
 partitions. I was told on the past API workshop that this is ok and will 
 not result in concurrency issues. In addition to concrete information on 
 the issue, we would be happy to know if it makes sense to "play" 

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Stefan,

The good news is that we found where the hanging is happening, but the bad 
news is that we still don't have a complete fix. If the cancel continues to 
hang, please get back to me. Right now, it looks like you're doing the 
right thing by cancelling and retrying. 

Best,
Nadine, AdWords API Team

On Thursday, February 16, 2017 at 1:54:41 AM UTC-5, Stefan Kleine Stegemann 
wrote:
>
>
> Hi Nadine,
>
> To update you on this issue, the situation has not improved but rather got 
> slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) with 
> eight adwords accounts being affected. Yesterday, it was 30 batch jobs with 
> three accounts being affected and on tuesday it was seven jobs, also with 
> three accounts. So we see a slightly "negative" tendency here. I also 
> checked some of the batch jobs we canceled yesterday and they are still in 
> status "CANCELING". Even the jobs which got stuck on tuesday and which we 
> also have canceled yesterday remain in "CANCELING". It seems that these 
> jobs are not doing anything anymore and that there is something broken 
> fundamentally.
>
> Do you have any update on this issue? Can we expect batch jobs to come 
> back to "normal"? We'd like to know that because if not, we need to take 
> action to deal with such scenarios.
>
> Best,
> Stefan
>
> Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine Sundquist 
> (AdWords API Team):
>>
>> Hi Stefan,
>>
>> Thanks for reporting this. I personally experienced some longer than 
>> usual running batch jobs last night as well. We're looking into what caused 
>> this last spike in execution time.
>>
>> Cheers,
>> Nadine, AdWords API Team
>>
>> On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine 
>> Stegemann wrote:
>>>
>>> Hi,
>>>
>>> sorry for hijacking this thread but since yesterday morning, we also 
>>> started to have significant troubles with batch job service. Jobs remained 
>>> active for a very long time. The following jobs have been started yesterday 
>>> morning, german time (see start time is noted for each job) and were still 
>>> active this morning:
>>>
>>> Job 445516765 | 2017-02-14T03:34
>>> Job 445553000 | 2017-02-14T03:20
>>> Job 445532407 | 2017-02-14T03:58
>>> Job 445532515 | 2017-02-14T03:58
>>> Job 445693959 | 2017-02-14T03:58
>>> Job 445694061 | 2017-02-14T03:59
>>> Job 445694595 | 2017-02-14T03:59
>>>
>>> We've canceled the jobs this morning (after they are still reported to 
>>> be active, how can that be?), but now they are in canceling state for about 
>>> an hour (at the time of this writing). Furthermore, the situation got worse 
>>> this morning, with 30 batch jobs being active for a couple of hours now. 
>>> The ids of these jobs are attached to the end of this post. We will cancel 
>>> these jobs as well because we do not expect them to finish, based on 
>>> yesterday's experience.
>>>
>>> As an additional information, the problems seem to affect accounts with 
>>> campaigns with a large to huge number or product partitions / keywords. The 
>>> problematic jobs were "spreaded" over three adwords accounts yesterday and 
>>> three accounts today. Two of the accounts affected yesterday are also 
>>> affected today. We do use these batch jobs to set cpc bids for keywords and 
>>> product partitions and use a maximum of 40.000 operations per job. We 
>>> experimented with this number in the past and found that to be a "suitable" 
>>> balance between performance and reliability of batch jobs. All jobs contain 
>>> only one type of operation. While multiple jobs might affect the same 
>>> campaign / adgroups, they do not affect the same keywords / product 
>>> partitions. I was told on the past API workshop that this is ok and will 
>>> not result in concurrency issues. In addition to concrete information on 
>>> the issue, we would be happy to know if it makes sense to "play" with the 
>>> limit of 40.000 operations per job again.
>>>
>>> As a general note, batch job continously tend to keep us busy since we 
>>> switched to them last year. The runtime performance seems to be rather 
>>> unpredictable. Especially on test accounts, batch jobs can run for a very 
>>> long time (up to 30 minutes) which is painful if you want to run automated 
>>> tests. Combined with the unclear advise on the number of operations to put 
>>> in a batch job and best practices in general, that leaves us with a very 
>>> uncertain feeling. Given the reliabilty issues we experienced in the past 
>>> half year, we're currently thinking about ditching batch jobs entirely and 
>>> implement batching on our site which would give us at least more control. 
>>> We understand that the intend of batch jobs is like "fire and forget" 
>>> (yeah, not really forget but check results etc.), we offload operations to 
>>> AdWords and it takes care of it. Currently its more like "fire and pray". 
>>> While we do not want to bring back the past, but we never had such troubles 
>>> with mutate 

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Biniam,

It looks like we found the culprit on our side, but we haven't found the 
solution, yet. The workaround for now is to cancel the job that is hanging 
and to retry the operations that have not executed. I'll keep you in the 
loop as we make progress.

Cheers,
Nadine, AdWords API Team

On Thursday, February 16, 2017 at 4:31:39 AM UTC-5, Biniam Asnake wrote:
>
> We are also experiencing this issue since some time.
>
> The following batches are stuck on the same percentage for morethan 24 
> hours! Please fix.
>
> Here is some information from our log file.
>
> The batch-job 446153939 returned the status ACTIVE.
> Progress Stats:
>   46% Executed
>   1934 Operations Executed
>   1932 Operations Succeeded
>   0 Results Written
>
>
> The batch-job 44643 returned the status ACTIVE.
> Progress Stats:
>   53% Executed
>   1733 Operations Executed
>   1725 Operations Succeeded
>   0 Results Written
>
>
> The batch-job 446270445 returned the status ACTIVE.
> Progress Stats:
>   61% Executed
>   2191 Operations Executed
>   2181 Operations Succeeded
>   0 Results Written
>
>
> Batch job status check for ID: 446153939 (Status: In-Progress) is 
> attempted 59 times while checking every 5 minutes
> Batch job status check for ID: 44643 (Status: In-Progress) is 
> attempted 59 times while checking every 5 minutes
> Batch job status check for ID: 446270445 (Status: In-Progress) is 
> attempted 59 times while checking every 5 minutes
>
>
> On Wednesday, February 1, 2017 at 10:50:12 PM UTC+1, AdWordsApiUser wrote:
>>
>> As mentioned by few other folks over the last few days, we're having 
>> issues with BatchJobService taking very long to process jobs. As requested, 
>> here are a few job IDs:
>>
>> Job Ids:
>> 441159032
>> 441282849
>>
>> These add AdGroupCriteria, but other jobs (e.g. that change bids) also 
>> get stuck. We would really appreciate some insight into this.
>>
>> Regards,
>> MG
>>
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/f42708b1-0475-4563-a787-63b23e4259b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Biniam Asnake]

We are also experiencing this issue since some time.

The following batches are stuck on the same percentage for morethan 24 
hours! Please fix.

Here is some information from our log file.

The batch-job 446153939 returned the status ACTIVE.
Progress Stats:
  46% Executed
  1934 Operations Executed
  1932 Operations Succeeded
  0 Results Written


The batch-job 44643 returned the status ACTIVE.
Progress Stats:
  53% Executed
  1733 Operations Executed
  1725 Operations Succeeded
  0 Results Written


The batch-job 446270445 returned the status ACTIVE.
Progress Stats:
  61% Executed
  2191 Operations Executed
  2181 Operations Succeeded
  0 Results Written


Batch job status check for ID: 446153939 (Status: In-Progress) is attempted 
59 times while checking every 5 minutes
Batch job status check for ID: 44643 (Status: In-Progress) is attempted 
59 times while checking every 5 minutes
Batch job status check for ID: 446270445 (Status: In-Progress) is attempted 
59 times while checking every 5 minutes


On Wednesday, February 1, 2017 at 10:50:12 PM UTC+1, AdWordsApiUser wrote:
>
> As mentioned by few other folks over the last few days, we're having 
> issues with BatchJobService taking very long to process jobs. As requested, 
> here are a few job IDs:
>
> Job Ids:
> 441159032
> 441282849
>
> These add AdGroupCriteria, but other jobs (e.g. that change bids) also get 
> stuck. We would really appreciate some insight into this.
>
> Regards,
> MG
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/1e054e54-2523-42c1-9184-6a91efc4f0d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi Nadine,

To update you on this issue, the situation has not improved but rather got 
slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) with 
eight adwords accounts being affected. Yesterday, it was 30 batch jobs with 
three accounts being affected and on tuesday it was seven jobs, also with 
three accounts. So we see a slightly "negative" tendency here. I also 
checked some of the batch jobs we canceled yesterday and they are still in 
status "CANCELING". Even the jobs which got stuck on tuesday and which we 
also have canceled yesterday remain in "CANCELING". It seems that these 
jobs are not doing anything anymore and that there is something broken 
fundamentally.

Do you have any update on this issue? Can we expect batch jobs to come back 
to "normal"? We'd like to know that because if not, we need to take action 
to deal with such scenarios.

Best,
Stefan

Am Mittwoch, 15. Februar 2017 15:16:22 UTC+1 schrieb Nadine Sundquist 
(AdWords API Team):
>
> Hi Stefan,
>
> Thanks for reporting this. I personally experienced some longer than usual 
> running batch jobs last night as well. We're looking into what caused this 
> last spike in execution time.
>
> Cheers,
> Nadine, AdWords API Team
>
> On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine 
> Stegemann wrote:
>>
>> Hi,
>>
>> sorry for hijacking this thread but since yesterday morning, we also 
>> started to have significant troubles with batch job service. Jobs remained 
>> active for a very long time. The following jobs have been started yesterday 
>> morning, german time (see start time is noted for each job) and were still 
>> active this morning:
>>
>> Job 445516765 | 2017-02-14T03:34
>> Job 445553000 | 2017-02-14T03:20
>> Job 445532407 | 2017-02-14T03:58
>> Job 445532515 | 2017-02-14T03:58
>> Job 445693959 | 2017-02-14T03:58
>> Job 445694061 | 2017-02-14T03:59
>> Job 445694595 | 2017-02-14T03:59
>>
>> We've canceled the jobs this morning (after they are still reported to be 
>> active, how can that be?), but now they are in canceling state for about an 
>> hour (at the time of this writing). Furthermore, the situation got worse 
>> this morning, with 30 batch jobs being active for a couple of hours now. 
>> The ids of these jobs are attached to the end of this post. We will cancel 
>> these jobs as well because we do not expect them to finish, based on 
>> yesterday's experience.
>>
>> As an additional information, the problems seem to affect accounts with 
>> campaigns with a large to huge number or product partitions / keywords. The 
>> problematic jobs were "spreaded" over three adwords accounts yesterday and 
>> three accounts today. Two of the accounts affected yesterday are also 
>> affected today. We do use these batch jobs to set cpc bids for keywords and 
>> product partitions and use a maximum of 40.000 operations per job. We 
>> experimented with this number in the past and found that to be a "suitable" 
>> balance between performance and reliability of batch jobs. All jobs contain 
>> only one type of operation. While multiple jobs might affect the same 
>> campaign / adgroups, they do not affect the same keywords / product 
>> partitions. I was told on the past API workshop that this is ok and will 
>> not result in concurrency issues. In addition to concrete information on 
>> the issue, we would be happy to know if it makes sense to "play" with the 
>> limit of 40.000 operations per job again.
>>
>> As a general note, batch job continously tend to keep us busy since we 
>> switched to them last year. The runtime performance seems to be rather 
>> unpredictable. Especially on test accounts, batch jobs can run for a very 
>> long time (up to 30 minutes) which is painful if you want to run automated 
>> tests. Combined with the unclear advise on the number of operations to put 
>> in a batch job and best practices in general, that leaves us with a very 
>> uncertain feeling. Given the reliabilty issues we experienced in the past 
>> half year, we're currently thinking about ditching batch jobs entirely and 
>> implement batching on our site which would give us at least more control. 
>> We understand that the intend of batch jobs is like "fire and forget" 
>> (yeah, not really forget but check results etc.), we offload operations to 
>> AdWords and it takes care of it. Currently its more like "fire and pray". 
>> While we do not want to bring back the past, but we never had such troubles 
>> with mutate job service. I apologize for this bit of ranting, but it's 
>> constantly keeping us busy for a while now.
>>
>> Finally, here are the job ids of the still-active (soon to be canceled) 
>> jobs from this morning.
>>
>> 445954525
>> 445954540
>> 445989185
>> 446102085
>> 446102088
>> 445952002
>> 445986527
>> 445986596
>> 445986662
>> 445986857
>> 445987025
>> 445987106
>> 445987562
>> 445987577
>> 445987715
>> 446107815
>> 446107821
>> 446108097
>> 446108442
>> 446108472
>> 446108553
>> 446108598

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi Stefan,

Thanks for reporting this. I personally experienced some longer than usual 
running batch jobs last night as well. We're looking into what caused this 
last spike in execution time.

Cheers,
Nadine, AdWords API Team

On Wednesday, February 15, 2017 at 5:04:20 AM UTC-5, Stefan Kleine 
Stegemann wrote:
>
> Hi,
>
> sorry for hijacking this thread but since yesterday morning, we also 
> started to have significant troubles with batch job service. Jobs remained 
> active for a very long time. The following jobs have been started yesterday 
> morning, german time (see start time is noted for each job) and were still 
> active this morning:
>
> Job 445516765 | 2017-02-14T03:34
> Job 445553000 | 2017-02-14T03:20
> Job 445532407 | 2017-02-14T03:58
> Job 445532515 | 2017-02-14T03:58
> Job 445693959 | 2017-02-14T03:58
> Job 445694061 | 2017-02-14T03:59
> Job 445694595 | 2017-02-14T03:59
>
> We've canceled the jobs this morning (after they are still reported to be 
> active, how can that be?), but now they are in canceling state for about an 
> hour (at the time of this writing). Furthermore, the situation got worse 
> this morning, with 30 batch jobs being active for a couple of hours now. 
> The ids of these jobs are attached to the end of this post. We will cancel 
> these jobs as well because we do not expect them to finish, based on 
> yesterday's experience.
>
> As an additional information, the problems seem to affect accounts with 
> campaigns with a large to huge number or product partitions / keywords. The 
> problematic jobs were "spreaded" over three adwords accounts yesterday and 
> three accounts today. Two of the accounts affected yesterday are also 
> affected today. We do use these batch jobs to set cpc bids for keywords and 
> product partitions and use a maximum of 40.000 operations per job. We 
> experimented with this number in the past and found that to be a "suitable" 
> balance between performance and reliability of batch jobs. All jobs contain 
> only one type of operation. While multiple jobs might affect the same 
> campaign / adgroups, they do not affect the same keywords / product 
> partitions. I was told on the past API workshop that this is ok and will 
> not result in concurrency issues. In addition to concrete information on 
> the issue, we would be happy to know if it makes sense to "play" with the 
> limit of 40.000 operations per job again.
>
> As a general note, batch job continously tend to keep us busy since we 
> switched to them last year. The runtime performance seems to be rather 
> unpredictable. Especially on test accounts, batch jobs can run for a very 
> long time (up to 30 minutes) which is painful if you want to run automated 
> tests. Combined with the unclear advise on the number of operations to put 
> in a batch job and best practices in general, that leaves us with a very 
> uncertain feeling. Given the reliabilty issues we experienced in the past 
> half year, we're currently thinking about ditching batch jobs entirely and 
> implement batching on our site which would give us at least more control. 
> We understand that the intend of batch jobs is like "fire and forget" 
> (yeah, not really forget but check results etc.), we offload operations to 
> AdWords and it takes care of it. Currently its more like "fire and pray". 
> While we do not want to bring back the past, but we never had such troubles 
> with mutate job service. I apologize for this bit of ranting, but it's 
> constantly keeping us busy for a while now.
>
> Finally, here are the job ids of the still-active (soon to be canceled) 
> jobs from this morning.
>
> 445954525
> 445954540
> 445989185
> 446102085
> 446102088
> 445952002
> 445986527
> 445986596
> 445986662
> 445986857
> 445987025
> 445987106
> 445987562
> 445987577
> 445987715
> 446107815
> 446107821
> 446108097
> 446108442
> 446108472
> 446108553
> 446108598
> 446108796
> 446108910
> 446108958
> 45948708
> 445983389
> 445983824
> 446105442
> 446105460
>
> Thanks for taking care of this and regards,
> Stefan
>
>
> Am Montag, 13. Februar 2017 22:08:05 UTC+1 schrieb AdWordsApiUser:
>>
>> Thanks Nadine - appreciate your quick reply. We'll keep an eye out and 
>> let you know if things get out of hand.
>>
>> On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine Sundquist 
>> (AdWords API Team) wrote:
>>>
>>> Hi MG,
>>>
>>> Thanks for sending over the batch job IDs. I took a look at the time it 
>>> took for the jobs to run for the slow running jobs. I didn't see any that 
>>> exceeded 15 minutes and most were completing within 5 minutes. There is a 
>>> bit of load balancing going on with a lot of job requests, so it's expected 
>>> that while some jobs may only take 2 minutes, there will be a few that may 
>>> exceed 10 minutes. Luckily, the jobs don't appear to be getting stuck 
>>> indefinitely, and they're not hanging for hours on end. I'll keep an eye on 
>>> things as I did see a spike in processing time this morning, but 

Re: Issues with BatchJobService - jobs remain stuck

[Stefan Kleine Stegemann]

Hi,

sorry for hijacking this thread but since yesterday morning, we also 
started to have significant troubles with batch job service. Jobs remained 
active for a very long time. The following jobs have been started yesterday 
morning, german time (see start time is noted for each job) and were still 
active this morning:

Job 445516765 | 2017-02-14T03:34
Job 445553000 | 2017-02-14T03:20
Job 445532407 | 2017-02-14T03:58
Job 445532515 | 2017-02-14T03:58
Job 445693959 | 2017-02-14T03:58
Job 445694061 | 2017-02-14T03:59
Job 445694595 | 2017-02-14T03:59

We've canceled the jobs this morning (after they are still reported to be 
active, how can that be?), but now they are in canceling state for about an 
hour (at the time of this writing). Furthermore, the situation got worse 
this morning, with 30 batch jobs being active for a couple of hours now. 
The ids of these jobs are attached to the end of this post. We will cancel 
these jobs as well because we do not expect them to finish, based on 
yesterday's experience.

As an additional information, the problems seem to affect accounts with 
campaigns with a large to huge number or product partitions / keywords. The 
problematic jobs were "spreaded" over three adwords accounts yesterday and 
three accounts today. Two of the accounts affected yesterday are also 
affected today. We do use these batch jobs to set cpc bids for keywords and 
product partitions and use a maximum of 40.000 operations per job. We 
experimented with this number in the past and found that to be a "suitable" 
balance between performance and reliability of batch jobs. All jobs contain 
only one type of operation. While multiple jobs might affect the same 
campaign / adgroups, they do not affect the same keywords / product 
partitions. I was told on the past API workshop that this is ok and will 
not result in concurrency issues. In addition to concrete information on 
the issue, we would be happy to know if it makes sense to "play" with the 
limit of 40.000 operations per job again.

As a general note, batch job continously tend to keep us busy since we 
switched to them last year. The runtime performance seems to be rather 
unpredictable. Especially on test accounts, batch jobs can run for a very 
long time (up to 30 minutes) which is painful if you want to run automated 
tests. Combined with the unclear advise on the number of operations to put 
in a batch job and best practices in general, that leaves us with a very 
uncertain feeling. Given the reliabilty issues we experienced in the past 
half year, we're currently thinking about ditching batch jobs entirely and 
implement batching on our site which would give us at least more control. 
We understand that the intend of batch jobs is like "fire and forget" 
(yeah, not really forget but check results etc.), we offload operations to 
AdWords and it takes care of it. Currently its more like "fire and pray". 
While we do not want to bring back the past, but we never had such troubles 
with mutate job service. I apologize for this bit of ranting, but it's 
constantly keeping us busy for a while now.

Finally, here are the job ids of the still-active (soon to be canceled) 
jobs from this morning.

445954525
445954540
445989185
446102085
446102088
445952002
445986527
445986596
445986662
445986857
445987025
445987106
445987562
445987577
445987715
446107815
446107821
446108097
446108442
446108472
446108553
446108598
446108796
446108910
446108958
45948708
445983389
445983824
446105442
446105460

Thanks for taking care of this and regards,
Stefan


Am Montag, 13. Februar 2017 22:08:05 UTC+1 schrieb AdWordsApiUser:
>
> Thanks Nadine - appreciate your quick reply. We'll keep an eye out and let 
> you know if things get out of hand.
>
> On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine Sundquist 
> (AdWords API Team) wrote:
>>
>> Hi MG,
>>
>> Thanks for sending over the batch job IDs. I took a look at the time it 
>> took for the jobs to run for the slow running jobs. I didn't see any that 
>> exceeded 15 minutes and most were completing within 5 minutes. There is a 
>> bit of load balancing going on with a lot of job requests, so it's expected 
>> that while some jobs may only take 2 minutes, there will be a few that may 
>> exceed 10 minutes. Luckily, the jobs don't appear to be getting stuck 
>> indefinitely, and they're not hanging for hours on end. I'll keep an eye on 
>> things as I did see a spike in processing time this morning, but it looks 
>> like things are returning to normal again. Please reach out if you see the 
>> times increasing even more.
>>
>> Thanks again,
>> Nadine, AdWords API Team
>>
>> On Monday, February 13, 2017 at 9:39:03 AM UTC-5, AdWordsApiUser wrote:
>>>
>>> Hello team,
>>>
>>> The issue seems to happen again, though not as bad as last time, so I 
>>> thought I'd let you know before it becomes worse. Some jobs are taking a 
>>> long time to process, while others process very fast. Last time it 
>>> happened, 

Re: Issues with BatchJobService - jobs remain stuck

[AdWordsApiUser]

Thanks Nadine - appreciate your quick reply. We'll keep an eye out and let 
you know if things get out of hand.

On Monday, February 13, 2017 at 6:19:50 PM UTC+1, Nadine Sundquist (AdWords 
API Team) wrote:
>
> Hi MG,
>
> Thanks for sending over the batch job IDs. I took a look at the time it 
> took for the jobs to run for the slow running jobs. I didn't see any that 
> exceeded 15 minutes and most were completing within 5 minutes. There is a 
> bit of load balancing going on with a lot of job requests, so it's expected 
> that while some jobs may only take 2 minutes, there will be a few that may 
> exceed 10 minutes. Luckily, the jobs don't appear to be getting stuck 
> indefinitely, and they're not hanging for hours on end. I'll keep an eye on 
> things as I did see a spike in processing time this morning, but it looks 
> like things are returning to normal again. Please reach out if you see the 
> times increasing even more.
>
> Thanks again,
> Nadine, AdWords API Team
>
> On Monday, February 13, 2017 at 9:39:03 AM UTC-5, AdWordsApiUser wrote:
>>
>> Hello team,
>>
>> The issue seems to happen again, though not as bad as last time, so I 
>> thought I'd let you know before it becomes worse. Some jobs are taking a 
>> long time to process, while others process very fast. Last time it 
>> happened, over 2-3 days, the jobs became slower, and a higher proportion of 
>> jobs started getting stuck till you guys did some magic to make everything 
>> fast again:
>>
>> stuck/took very long: 
>> 445491084
>> 445522515
>> 445523004 
>>
>> not stuck/ran fast: 
>> 445491588
>> 445352074
>>
>> Cheers,
>> MG
>>
>> On Thursday, February 2, 2017 at 6:51:38 PM UTC+1, AdWordsApiUser wrote:
>>>
>>> Thanks Nadine! You and your team saved us a costly rewrite that we had 
>>> started planning for (without using the bulk job service). We'll keep an 
>>> eye out and report back if we see issues. Thanks team - you're the most 
>>> responsive team at Google I've interacted with ;)
>>>
>>> On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine Sundquist 
>>> (AdWords API Team) wrote:

 Hello Everyone,

 I have good news! We found the root cause, and the issue has been 
 fixed. Your batch job processing times should be back to normal now. 
 Thanks 
 again for reporting this!

 Take care,
 Nadine, AdWords API Team

 On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist 
 (AdWords API Team) wrote:
>
> Greetings Fellow AdWordsians,
>
> Thanks for the batch job IDs. They are a BIG help. I've taken a look 
> at the history of those batch jobs in our internal systems, and I've 
> found 
> a pattern. I need to run it by a few teammates here to see if my 
> observations result in finding a root cause. I'll update you when I've 
> made 
> a bit more progress.
>
> Cheers,
> Nadine, AdWords API Team
>
> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser 
> wrote:
>>
>> Hi Vishal,
>>
>> Thanks for your reply. However, we have had 100% of the jobs we've 
>> submitted in the last 6 hours take over 15 mintues to complete. 
>>
>> Regards,
>> MG
>>
>> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
>> (Adwords API Team) wrote:
>>>
>>> Hi MG,
>>>
>>> Issues related to batch job performance tend to be intermittent some 
>>> of the times. In such cases, cancelling the job and sending a new 
>>> request 
>>> solves the issue. Could you please try cancelling 
>>> 
>>>  all 
>>> current jobs with the status 
>>> 
>>>  AWAITING_FILE 
>>> or ACTIVE and retry? If that doesn't help, please feel free to revert 
>>> and 
>>> I'll check with my team to see if we can dig into this issue further. 
>>>
>>> Regards,
>>> Vishal, AdWords API Team
>>>
>>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at 

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hi MG,

Thanks for sending over the batch job IDs. I took a look at the time it 
took for the jobs to run for the slow running jobs. I didn't see any that 
exceeded 15 minutes and most were completing within 5 minutes. There is a 
bit of load balancing going on with a lot of job requests, so it's expected 
that while some jobs may only take 2 minutes, there will be a few that may 
exceed 10 minutes. Luckily, the jobs don't appear to be getting stuck 
indefinitely, and they're not hanging for hours on end. I'll keep an eye on 
things as I did see a spike in processing time this morning, but it looks 
like things are returning to normal again. Please reach out if you see the 
times increasing even more.

Thanks again,
Nadine, AdWords API Team

On Monday, February 13, 2017 at 9:39:03 AM UTC-5, AdWordsApiUser wrote:
>
> Hello team,
>
> The issue seems to happen again, though not as bad as last time, so I 
> thought I'd let you know before it becomes worse. Some jobs are taking a 
> long time to process, while others process very fast. Last time it 
> happened, over 2-3 days, the jobs became slower, and a higher proportion of 
> jobs started getting stuck till you guys did some magic to make everything 
> fast again:
>
> stuck/took very long: 
> 445491084
> 445522515
> 445523004 
>
> not stuck/ran fast: 
> 445491588
> 445352074
>
> Cheers,
> MG
>
> On Thursday, February 2, 2017 at 6:51:38 PM UTC+1, AdWordsApiUser wrote:
>>
>> Thanks Nadine! You and your team saved us a costly rewrite that we had 
>> started planning for (without using the bulk job service). We'll keep an 
>> eye out and report back if we see issues. Thanks team - you're the most 
>> responsive team at Google I've interacted with ;)
>>
>> On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine Sundquist 
>> (AdWords API Team) wrote:
>>>
>>> Hello Everyone,
>>>
>>> I have good news! We found the root cause, and the issue has been fixed. 
>>> Your batch job processing times should be back to normal now. Thanks again 
>>> for reporting this!
>>>
>>> Take care,
>>> Nadine, AdWords API Team
>>>
>>> On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist 
>>> (AdWords API Team) wrote:

 Greetings Fellow AdWordsians,

 Thanks for the batch job IDs. They are a BIG help. I've taken a look at 
 the history of those batch jobs in our internal systems, and I've found a 
 pattern. I need to run it by a few teammates here to see if my 
 observations 
 result in finding a root cause. I'll update you when I've made a bit more 
 progress.

 Cheers,
 Nadine, AdWords API Team

 On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser 
 wrote:
>
> Hi Vishal,
>
> Thanks for your reply. However, we have had 100% of the jobs we've 
> submitted in the last 6 hours take over 15 mintues to complete. 
>
> Regards,
> MG
>
> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
> (Adwords API Team) wrote:
>>
>> Hi MG,
>>
>> Issues related to batch job performance tend to be intermittent some 
>> of the times. In such cases, cancelling the job and sending a new 
>> request 
>> solves the issue. Could you please try cancelling 
>> 
>>  all 
>> current jobs with the status 
>> 
>>  AWAITING_FILE 
>> or ACTIVE and retry? If that doesn't help, please feel free to revert 
>> and 
>> I'll check with my team to see if we can dig into this issue further. 
>>
>> Regards,
>> Vishal, AdWords API Team
>>
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/00a8ecac-4073-47cc-a57f-9661ee831985%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[AdWordsApiUser]

Hello team,

The issue seems to happen again, though not as bad as last time, so I 
thought I'd let you know before it becomes worse. Some jobs are taking a 
long time to process, while others process very fast. Last time it 
happened, over 2-3 days, the jobs became slower, and a higher proportion of 
jobs started getting stuck till you guys did some magic to make everything 
fast again:

stuck/took very long: 
445491084
445522515
445523004 

not stuck/ran fast: 
445491588
445352074

Cheers,
MG

On Thursday, February 2, 2017 at 6:51:38 PM UTC+1, AdWordsApiUser wrote:
>
> Thanks Nadine! You and your team saved us a costly rewrite that we had 
> started planning for (without using the bulk job service). We'll keep an 
> eye out and report back if we see issues. Thanks team - you're the most 
> responsive team at Google I've interacted with ;)
>
> On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine Sundquist 
> (AdWords API Team) wrote:
>>
>> Hello Everyone,
>>
>> I have good news! We found the root cause, and the issue has been fixed. 
>> Your batch job processing times should be back to normal now. Thanks again 
>> for reporting this!
>>
>> Take care,
>> Nadine, AdWords API Team
>>
>> On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist 
>> (AdWords API Team) wrote:
>>>
>>> Greetings Fellow AdWordsians,
>>>
>>> Thanks for the batch job IDs. They are a BIG help. I've taken a look at 
>>> the history of those batch jobs in our internal systems, and I've found a 
>>> pattern. I need to run it by a few teammates here to see if my observations 
>>> result in finding a root cause. I'll update you when I've made a bit more 
>>> progress.
>>>
>>> Cheers,
>>> Nadine, AdWords API Team
>>>
>>> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser wrote:

 Hi Vishal,

 Thanks for your reply. However, we have had 100% of the jobs we've 
 submitted in the last 6 hours take over 15 mintues to complete. 

 Regards,
 MG

 On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
 (Adwords API Team) wrote:
>
> Hi MG,
>
> Issues related to batch job performance tend to be intermittent some 
> of the times. In such cases, cancelling the job and sending a new request 
> solves the issue. Could you please try cancelling 
> 
>  all 
> current jobs with the status 
> 
>  AWAITING_FILE 
> or ACTIVE and retry? If that doesn't help, please feel free to revert and 
> I'll check with my team to see if we can dig into this issue further. 
>
> Regards,
> Vishal, AdWords API Team
>


-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/3948d3e7-b8f2-44fe-86ec-456de1694e06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[AdWordsApiUser]

Thanks Nadine! You and your team saved us a costly rewrite that we had 
started planning for (without using the bulk job service). We'll keep an 
eye out and report back if we see issues. Thanks team - you're the most 
responsive team at Google I've interacted with ;)

On Thursday, February 2, 2017 at 6:46:03 PM UTC+1, Nadine Sundquist 
(AdWords API Team) wrote:
>
> Hello Everyone,
>
> I have good news! We found the root cause, and the issue has been fixed. 
> Your batch job processing times should be back to normal now. Thanks again 
> for reporting this!
>
> Take care,
> Nadine, AdWords API Team
>
> On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist 
> (AdWords API Team) wrote:
>>
>> Greetings Fellow AdWordsians,
>>
>> Thanks for the batch job IDs. They are a BIG help. I've taken a look at 
>> the history of those batch jobs in our internal systems, and I've found a 
>> pattern. I need to run it by a few teammates here to see if my observations 
>> result in finding a root cause. I'll update you when I've made a bit more 
>> progress.
>>
>> Cheers,
>> Nadine, AdWords API Team
>>
>> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser wrote:
>>>
>>> Hi Vishal,
>>>
>>> Thanks for your reply. However, we have had 100% of the jobs we've 
>>> submitted in the last 6 hours take over 15 mintues to complete. 
>>>
>>> Regards,
>>> MG
>>>
>>> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
>>> (Adwords API Team) wrote:

 Hi MG,

 Issues related to batch job performance tend to be intermittent some of 
 the times. In such cases, cancelling the job and sending a new request 
 solves the issue. Could you please try cancelling 
 
  all 
 current jobs with the status 
 
  AWAITING_FILE 
 or ACTIVE and retry? If that doesn't help, please feel free to revert and 
 I'll check with my team to see if we can dig into this issue further. 

 Regards,
 Vishal, AdWords API Team

>>>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/edb8fb94-c007-467c-bbed-0875257dff81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Hello Everyone,

I have good news! We found the root cause, and the issue has been fixed. 
Your batch job processing times should be back to normal now. Thanks again 
for reporting this!

Take care,
Nadine, AdWords API Team

On Thursday, February 2, 2017 at 11:11:20 AM UTC-5, Nadine Sundquist 
(AdWords API Team) wrote:
>
> Greetings Fellow AdWordsians,
>
> Thanks for the batch job IDs. They are a BIG help. I've taken a look at 
> the history of those batch jobs in our internal systems, and I've found a 
> pattern. I need to run it by a few teammates here to see if my observations 
> result in finding a root cause. I'll update you when I've made a bit more 
> progress.
>
> Cheers,
> Nadine, AdWords API Team
>
> On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser wrote:
>>
>> Hi Vishal,
>>
>> Thanks for your reply. However, we have had 100% of the jobs we've 
>> submitted in the last 6 hours take over 15 mintues to complete. 
>>
>> Regards,
>> MG
>>
>> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
>> (Adwords API Team) wrote:
>>>
>>> Hi MG,
>>>
>>> Issues related to batch job performance tend to be intermittent some of 
>>> the times. In such cases, cancelling the job and sending a new request 
>>> solves the issue. Could you please try cancelling 
>>> 
>>>  all 
>>> current jobs with the status 
>>> 
>>>  AWAITING_FILE 
>>> or ACTIVE and retry? If that doesn't help, please feel free to revert and 
>>> I'll check with my team to see if we can dig into this issue further. 
>>>
>>> Regards,
>>> Vishal, AdWords API Team
>>>
>>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/068f41f2-3ca4-47ef-9cb0-da2d035a493b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Nadine Sundquist (AdWords API Team)' via AdWords API Forum]

Greetings Fellow AdWordsians,

Thanks for the batch job IDs. They are a BIG help. I've taken a look at the 
history of those batch jobs in our internal systems, and I've found a 
pattern. I need to run it by a few teammates here to see if my observations 
result in finding a root cause. I'll update you when I've made a bit more 
progress.

Cheers,
Nadine, AdWords API Team

On Wednesday, February 1, 2017 at 5:24:36 PM UTC-5, AdWordsApiUser wrote:
>
> Hi Vishal,
>
> Thanks for your reply. However, we have had 100% of the jobs we've 
> submitted in the last 6 hours take over 15 mintues to complete. 
>
> Regards,
> MG
>
> On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
> (Adwords API Team) wrote:
>>
>> Hi MG,
>>
>> Issues related to batch job performance tend to be intermittent some of 
>> the times. In such cases, cancelling the job and sending a new request 
>> solves the issue. Could you please try cancelling 
>> 
>>  all 
>> current jobs with the status 
>> 
>>  AWAITING_FILE 
>> or ACTIVE and retry? If that doesn't help, please feel free to revert and 
>> I'll check with my team to see if we can dig into this issue further. 
>>
>> Regards,
>> Vishal, AdWords API Team
>>
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/60df37eb-7be8-47eb-963d-428f1a54e4ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[AdWordsApiUser]

Hi Vishal,

Thanks for your reply. However, we have had 100% of the jobs we've 
submitted in the last 6 hours take over 15 mintues to complete. 

Regards,
MG

On Wednesday, February 1, 2017 at 11:16:53 PM UTC+1, Vishal Vinayak 
(Adwords API Team) wrote:
>
> Hi MG,
>
> Issues related to batch job performance tend to be intermittent some of 
> the times. In such cases, cancelling the job and sending a new request 
> solves the issue. Could you please try cancelling 
> 
>  all 
> current jobs with the status 
> 
>  AWAITING_FILE 
> or ACTIVE and retry? If that doesn't help, please feel free to revert and 
> I'll check with my team to see if we can dig into this issue further. 
>
> Regards,
> Vishal, AdWords API Team
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/acaae913-e841-4542-9314-3b1c557d1bc7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

[Brian Bachtel]

We are also seeing an issue with the BatchJobService as well. Jobs are 
stuck in an "ACTIVE" state and when we check the Progress Stats everything 
says 0 even though we definitely uploaded a file with operations. It 
appears to be intermittent.

Job IDs
441119824
441270816
441145043

On Wednesday, February 1, 2017 at 4:50:12 PM UTC-5, AdWordsApiUser wrote:
>
> As mentioned by few other folks over the last few days, we're having 
> issues with BatchJobService taking very long to process jobs. As requested, 
> here are a few job IDs:
>
> Job Ids:
> 441159032
> 441282849
>
> These add AdGroupCriteria, but other jobs (e.g. that change bids) also get 
> stuck. We would really appreciate some insight into this.
>
> Regards,
> MG
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/3df653a8-6e70-43e1-b789-96f73aeba736%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Issues with BatchJobService - jobs remain stuck

['Vishal Vinayak (Adwords API Team)' via AdWords API Forum]

Hi MG,

Issues related to batch job performance tend to be intermittent some of the 
times. In such cases, cancelling the job and sending a new request solves 
the issue. Could you please try cancelling 

 all 
current jobs with the status 

 AWAITING_FILE 
or ACTIVE and retry? If that doesn't help, please feel free to revert and 
I'll check with my team to see if we can dig into this issue further. 

Regards,
Vishal, AdWords API Team

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/2c9ae43e-678f-4610-9d52-fb168918de86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.