We're now seeing this on other calls as well. These are across multiple
customerIds.
2021-06-23 01:19:33,532 WARN RequestLogger - FAILURE REQUEST SUMMARY.
Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint:
googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null,
ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after
3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556,
remote_addr=googleads.googleapis.com/142.250.191.138:443].
2021-06-23 01:22:48,579 WARN RequestLogger - FAILURE REQUEST SUMMARY.
Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds,
Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID:
null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after
3599.985958315s. [buffered_nanos=65378596, buffered_nanos=67500777,
remote_addr=googleads.googleapis.com/172.217.8.202:443].
20212021-06-23 01:30:08,166 WARN RequestLogger - FAILURE REQUEST SUMMARY.
Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds,
Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID:
null, ResponseCode: *DEADLINE*_EXCEEDED, Fault: deadline exceeded after
3599.998821696s. [buffered_nanos=66057372, buffered_nanos=77619715,
remote_addr=googleads.googleapis.com/142.250.191.202:443].
On Tuesday, June 22, 2021 at 8:23:06 PM UTC-4 Pete Lavetsky (AdWords API
Guru) wrote:
> We put V8 into production earlier today. Everything was fine for about 6.5
> hours and then we started seeing an increased rate of DEADLINE_EXCEEDED
>
> Our GrpcCallContext is configured for a 5m timeout and that's been fine
> for months
>
> I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the last
> 20 minutes but I'm not sure if that proves anything
>
> Is it perhaps a transient issue on Google's end for maintenance work
> around increased capacity for V8?
>
> Here's the traceroute ... we've been asked to provide this before when we
> witnessed a much higher rate of DEADLINE_EXCEEDED
>
> traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60
> byte packets
>
> 1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 ms
> 0.096 ms 0.102 ms
>
> 2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 ms
> ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms
> ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms
>
> 3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16)
> 6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms
>
> 4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 (100.66.12.144)
> 3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms
>
> 5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202
> (100.66.14.202) 22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms
>
> 6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131)
> 50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms
>
> 7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117)
> 31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms
>
> 8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 (100.65.13.161)
> 0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms
>
> 9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161
> (15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms
>
> 10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10)
> 2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms
>
> 11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820
> ms 52.93.239.50 (52.93.239.50) 1.313 ms
>
> 12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 (100.92.53.76)
> 14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms
>
> 13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 (100.92.48.100)
> 16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms
>
> 14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 (100.92.48.139)
> 12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms
>
> 15 100.92.49.66 (100.92.49.66) 12.591 ms 100.92.44.90 (100.92.44.90)
> 18.647 ms 18.368 ms
>
> 16 100.92.49.7 (100.92.49.7) 12.986 ms 100.92.44.9 (100.92.44.9) 12.990
> ms 100.92.44.123 (100.92.44.123) 18.647 ms
>
> 17 52.93.132.62 (52.93.132.62) 12.399 ms 52.93.130.134 (52.93.130.134)
> 12.463 ms 52.93.133.56 (52.93.133.56) 12.744 ms
>
> 18 100.91.163.4 (100.91.163.4) 13.681 ms 100.91.167.8 (100.91.167.8)
> 18.976 ms 100.91.163.96 (100.91.163.96) 13.129 ms
>
> 19 100.91.163.89 (100.91.163.89) 12.558 ms 100.91.168.43
> (100.91.168.43) 13.010 ms 100.91.167.17 (100.91.167.17) 13.632 ms
>
> 20 100.91.164.108 (100.91.164.108) 12.366 ms 100.91.159.64
> (100.91.159.64) 17.061 ms 100.91.164.60 (100.91.164.60) 12.693 ms
>
> 21 100.91.159.73 (100.91.159.73) 17.008 ms 100.91.164.105
> (100.91.164.105) 13.024 ms 100.91.165.129 (100.91.165.129) 13.239 ms
>
> 22 100.91.177.41 (100.91.177.41) 11.699 ms 100.91.177.55
> (100.91.177.55) 11.945 ms 100.91.176.241 (100.91.176.241) 11.779 ms
>
> 23 100.100.6.21 (100.100.6.21) 12.162 ms 100.100.8.11 (100.100.8.11)
> 14.884 ms 100.100.8.117 (100.100.8.117) 12.359 ms
>
> 24 100.100.93.8 (100.100.93.8) 12.228 ms 100.100.77.198
> (100.100.77.198) 12.200 ms 100.100.76.134 (100.100.76.134) 12.463 ms
>
> 25 100.100.93.195 (100.100.93.195) 12.901 ms 100.100.73.221
> (100.100.73.221) 13.835 ms 100.100.85.221 (100.100.85.221) 12.385 ms
>
> 26 100.100.28.124 (100.100.28.124) 12.296 ms 100.100.34.92
> (100.100.34.92) 12.247 ms 100.100.2.36 (100.100.2.36) 12.940 ms
>
> 27 72.14.203.158 (72.14.203.158) 12.939 ms 99.82.181.23 (99.82.181.23)
> 12.316 ms 72.14.203.158 (72.14.203.158) 12.906 ms
>
> 28 108.170.246.66 (108.170.246.66) 12.874 ms * 12.818 ms
>
> 29 108.170.246.33 (108.170.246.33) 13.149 ms 72.14.203.158
> (72.14.203.158) 17.185 ms 99.82.181.23 (99.82.181.23) 15.395 ms
>
> 30 108.170.246.66 (108.170.246.66) 12.841 ms * 108.170.240.98
> (108.170.240.98) 12.831 ms
>
> 31 * * *
>
> 32 * * *
>
> 33 * * *
>
> 34 * * *
>
> 35 * * *
>
> 36 * * *
>
> 37 * * *
>
> 38 * * *
>
> 39 * * *
>
> 40 * * *
>
> 41 * * *
>
> 42 * * *
>
> 43 * * *
>
> 44 * * *
>
> 45 * * *
>
> 46 * * *
>
> 47 * * *
>
> 48 * * *
>
> 49 * * *
>
> 50 * * *
>
> Looking closer it looks like almost all of the DEADLINE_EXCEEDED were on
>
> google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
> calls
>
> Request
>
> -------
>
> MethodName:
> google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
>
> Endpoint: googleads.googleapis.com:443
>
> Headers: {developer-token=REDACTED, login-customer-id=5741853629,
> x-goog-api-client=gl-java/1.8.0_201 gapic/ gax/1.65.1 grpc/1.37.0}
>
> Body: customer_id: "REDACTED"
>
> operations {
>
> update {
>
> resource_name: "customers/REDACTED/campaignBudgets/7001096614"
>
> status: ENABLED
>
> delivery_method: STANDARD
>
> id: 7001096614
>
> amount_micros: 10000
>
> explicitly_shared: false
>
> }
>
> update_mask {
>
> paths: "resource_name"
>
> paths: "id"
>
> paths: "amount_micros"
>
> paths: "status"
>
> paths: "delivery_method"
>
> paths: "explicitly_shared"
>
> }
>
> }
>
> partial_failure: true
>
>
>
> Response
>
> --------
>
> Headers: null
>
> Body: null
>
> Failure message: null
>
> Status: Status{code=DEADLINE_EXCEEDED, description=deadline exceeded after
> 3599.999166595s. [buffered_nanos=65828859, buffered_nanos=107502844,
> remote_addr=googleads.googleapis.com/142.250.190.74:443], cause=null}.
>
> 2021-06-22 23:49:42,443 INFO RequestLogger - FAILURE REQUEST DETAIL.
>
> There's no request-id generated / given on a DEADLINE_EXCEEDED so I can't
> provide one.
>
> Pete
>
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/54cda46f-179e-42da-9424-8506adaf7f68n%40googlegroups.com.
