We put V8 into production earlier today. Everything was fine for about 6.5
hours and then we started seeing an increased rate of DEADLINE_EXCEEDED
Our GrpcCallContext is configured for a 5m timeout and that's been fine for
months
I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the last
20 minutes but I'm not sure if that proves anything
Is it perhaps a transient issue on Google's end for maintenance work around
increased capacity for V8?
Here's the traceroute ... we've been asked to provide this before when we
witnessed a much higher rate of DEADLINE_EXCEEDED
traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 byte
packets
1 ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206) 0.104 ms
0.096 ms 0.102 ms
2 ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37) 8.985 ms
ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57) 8.931 ms
ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43) 5.446 ms
3 100.65.27.16 (100.65.27.16) 3.778 ms 100.65.31.16 (100.65.31.16)
6.846 ms 100.65.28.0 (100.65.28.0) 3.651 ms
4 100.66.12.132 (100.66.12.132) 1.994 ms 100.66.12.144 (100.66.12.144)
3.632 ms 100.66.13.36 (100.66.13.36) 1.978 ms
5 100.66.14.196 (100.66.14.196) 16.689 ms 100.66.14.202 (100.66.14.202)
22.094 ms 100.66.14.130 (100.66.14.130) 12.970 ms
6 100.66.6.77 (100.66.6.77) 21.618 ms 100.66.6.131 (100.66.6.131)
50.483 ms 100.66.7.235 (100.66.7.235) 17.491 ms
7 100.66.4.101 (100.66.4.101) 32.121 ms 100.66.4.117 (100.66.4.117)
31.610 ms 100.66.4.125 (100.66.4.125) 42.423 ms
8 100.65.10.129 (100.65.10.129) 1.140 ms 100.65.13.161 (100.65.13.161)
0.937 ms 100.65.8.161 (100.65.8.161) 0.945 ms
9 15.230.134.209 (15.230.134.209) 2.346 ms 15.230.134.161
(15.230.134.161) 1.961 ms 15.230.134.203 (15.230.134.203) 2.729 ms
10 15.230.39.184 (15.230.39.184) 2.850 ms 15.230.39.10 (15.230.39.10)
2.794 ms 52.95.2.198 (52.95.2.198) 3.692 ms
11 52.95.2.205 (52.95.2.205) 2.088 ms 52.95.1.235 (52.95.1.235) 1.820 ms
52.93.239.50 (52.93.239.50) 1.313 ms
12 100.92.53.124 (100.92.53.124) 12.442 ms 100.92.53.76 (100.92.53.76)
14.483 ms 100.92.53.72 (100.92.53.72) 12.509 ms
13 100.92.43.34 (100.92.43.34) 12.711 ms 100.92.48.100 (100.92.48.100)
16.661 ms 100.92.48.10 (100.92.48.10) 13.001 ms
14 100.92.43.85 (100.92.43.85) 12.143 ms 100.92.48.139 (100.92.48.139)
12.547 ms 100.92.48.135 (100.92.48.135) 12.539 ms
15 100.92.49.66 (100.92.49.66) 12.591 ms 100.92.44.90 (100.92.44.90)
18.647 ms 18.368 ms
16 100.92.49.7 (100.92.49.7) 12.986 ms 100.92.44.9 (100.92.44.9) 12.990
ms 100.92.44.123 (100.92.44.123) 18.647 ms
17 52.93.132.62 (52.93.132.62) 12.399 ms 52.93.130.134 (52.93.130.134)
12.463 ms 52.93.133.56 (52.93.133.56) 12.744 ms
18 100.91.163.4 (100.91.163.4) 13.681 ms 100.91.167.8 (100.91.167.8)
18.976 ms 100.91.163.96 (100.91.163.96) 13.129 ms
19 100.91.163.89 (100.91.163.89) 12.558 ms 100.91.168.43 (100.91.168.43)
13.010 ms 100.91.167.17 (100.91.167.17) 13.632 ms
20 100.91.164.108 (100.91.164.108) 12.366 ms 100.91.159.64
(100.91.159.64) 17.061 ms 100.91.164.60 (100.91.164.60) 12.693 ms
21 100.91.159.73 (100.91.159.73) 17.008 ms 100.91.164.105
(100.91.164.105) 13.024 ms 100.91.165.129 (100.91.165.129) 13.239 ms
22 100.91.177.41 (100.91.177.41) 11.699 ms 100.91.177.55 (100.91.177.55)
11.945 ms 100.91.176.241 (100.91.176.241) 11.779 ms
23 100.100.6.21 (100.100.6.21) 12.162 ms 100.100.8.11 (100.100.8.11)
14.884 ms 100.100.8.117 (100.100.8.117) 12.359 ms
24 100.100.93.8 (100.100.93.8) 12.228 ms 100.100.77.198 (100.100.77.198)
12.200 ms 100.100.76.134 (100.100.76.134) 12.463 ms
25 100.100.93.195 (100.100.93.195) 12.901 ms 100.100.73.221
(100.100.73.221) 13.835 ms 100.100.85.221 (100.100.85.221) 12.385 ms
26 100.100.28.124 (100.100.28.124) 12.296 ms 100.100.34.92
(100.100.34.92) 12.247 ms 100.100.2.36 (100.100.2.36) 12.940 ms
27 72.14.203.158 (72.14.203.158) 12.939 ms 99.82.181.23 (99.82.181.23)
12.316 ms 72.14.203.158 (72.14.203.158) 12.906 ms
28 108.170.246.66 (108.170.246.66) 12.874 ms * 12.818 ms
29 108.170.246.33 (108.170.246.33) 13.149 ms 72.14.203.158
(72.14.203.158) 17.185 ms 99.82.181.23 (99.82.181.23) 15.395 ms
30 108.170.246.66 (108.170.246.66) 12.841 ms * 108.170.240.98
(108.170.240.98) 12.831 ms
31 * * *
32 * * *
33 * * *
34 * * *
35 * * *
36 * * *
37 * * *
38 * * *
39 * * *
40 * * *
41 * * *
42 * * *
43 * * *
44 * * *
45 * * *
46 * * *
47 * * *
48 * * *
49 * * *
50 * * *
Looking closer it looks like almost all of the DEADLINE_EXCEEDED were on
google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
calls
Request
-------
MethodName:
google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets
Endpoint: googleads.googleapis.com:443
Headers: {developer-token=REDACTED, login-customer-id=5741853629,
x-goog-api-client=gl-java/1.8.0_201 gapic/ gax/1.65.1 grpc/1.37.0}
Body: customer_id: "REDACTED"
operations {
update {
resource_name: "customers/REDACTED/campaignBudgets/7001096614"
status: ENABLED
delivery_method: STANDARD
id: 7001096614
amount_micros: 10000
explicitly_shared: false
}
update_mask {
paths: "resource_name"
paths: "id"
paths: "amount_micros"
paths: "status"
paths: "delivery_method"
paths: "explicitly_shared"
}
}
partial_failure: true
Response
--------
Headers: null
Body: null
Failure message: null
Status: Status{code=DEADLINE_EXCEEDED, description=deadline exceeded after
3599.999166595s. [buffered_nanos=65828859, buffered_nanos=107502844,
remote_addr=googleads.googleapis.com/142.250.190.74:443], cause=null}.
2021-06-22 23:49:42,443 INFO RequestLogger - FAILURE REQUEST DETAIL.
There's no request-id generated / given on a DEADLINE_EXCEEDED so I can't
provide one.
Pete
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"AdWords API and Google Ads API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/6fdb077e-35f9-4e0c-887d-9f2d37740d5bn%40googlegroups.com.
