Re: StreamingResponseIterator from search_stream gets stuck
Hello, The issue was resolved for a few hours, but then an additional issue popped up. We're working on resolving it. It's our top priority. Regards, Nadine Wang Google Ads API Team ref:_00D1U1174p._5004Q2egwqd:ref -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "Google Ads API and AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/TJJ550RIX8MW00AYWVAmoQRLuZjxkljGVUgg%40sfdc.net.
Re: StreamingResponseIterator from search_stream gets stuck
Hi,
Unfortunately the issue does not seem to have been resolved as of yet. The
query shown in the original post still gets stuck at exactly the same spot.
What could we do to investigate this further? And I think, as Robert
suggests, that a location where the status of this API issue can be tracked
would be great, as we are currently in a situation where we have to explain
this to our customers. An ETA would also be much appreciated.
Kind regards,
Op woensdag 28 september 2022 om 06:23:35 UTC+2 schreef Robert Heise:
> Hi,
>
> same here. Some reports succeed now. But some reports don't, most likely
> related to the data size. We didn't change anything on our side, all report
> fetches worked fine with their defined date ranges before Sept 27, when the
> issue started.
>
> The issue hasn't been fully resolved and we're still unable to fetch all
> reports via the Google Ads API. The behavior for the failing fetches is
> still the same. The iterator in the search_stream fetch gets stuck, is
> never finishing and at some point our wrapping timeout kills the task.
> Since there was no code change on our side and many developers are
> reporting the same issue here, using different programming languages, we
> assume the issue is on the Google Ads API side. My guess would be that the
> GRPC call for search_stream is not properly ended. To the best of our
> knowledge and our inspection so far, this is far too low level from the
> perspective of users of the Google Ads API (developers) and needs to be
> fixed on the Google side.
>
> Ideally please provide an ETA when the issue will be fixed on Google side.
> Also I wonder where we can best track the status of the API issue.
> https://ads.google.com/status/publisher/ continuously shows Google Ads
> operating fine, though the API clearly has severe issues for search_stream
> requests.
>
> Thanks,
>
> Robert
> On Wednesday, 28 September 2022 at 01:49:05 UTC+2 Andrew Repp wrote:
>
>> Hi,
>>
>> We are also continuing to see the same issue even after the all clear
>> message above. It appears to be related to the number of records involved.
>>
>> This query
>>
>> SELECT
>> segments.date,customer.id,campaign.name,campaign.id
>> ,campaign_criterion.bid_modifier,campaign_criterion.criterion_id,segments.device,metrics.impressions,metrics.clicks,metrics.cost_micros,metrics.gmail_forwards,metrics.gmail_saves,metrics.gmail_secondary_clicks
>> FROM
>> campaign_audience_view
>> WHERE metrics.impressions > 0 AND segments.date BETWEEN '2022-09-23'
>> AND '2022-09-27'
>>
>> fails with
>>
>> Response
>> ---
>> Headers: {}
>> Fault: {
>> "created": "@1664321827.318637000",
>> "description": "Error received from peer ipv4:142.250.217.74:443",
>> "file": "src/core/lib/surface/call.cc",
>> "file_line": 904,
>> "grpc_message": "Deadline Exceeded",
>> "grpc_status": 4
>> }
>>
>> But this query
>>
>> SELECT
>> segments.date,customer.id,campaign.name,campaign.id
>> ,campaign_criterion.bid_modifier,campaign_criterion.criterion_id,segments.device,metrics.impressions,metrics.clicks,metrics.cost_micros,metrics.gmail_forwards,metrics.gmail_saves,metrics.gmail_secondary_clicks
>> FROM
>> campaign_audience_view
>> WHERE metrics.impressions > 0 AND segments.date BETWEEN '2022-09-26'
>> AND '2022-09-27'
>>
>> succeeds. Notice that by only selecting 2 days of data instead of 5, the
>> query is able to complete. Before today, the query for 5 days completed
>> successfully.
>>
>> Any help from the Google folks on a resolution?
>>
>> Thanks,
>> Andrew
>>
>> On Tuesday, September 27, 2022 at 3:39:56 PM UTC-7 gel...@gmail.com
>> wrote:
>>
>>> We're still seeing this problem when streaming a large set of keywords
>>> from a customer account. It appears the request gets stuck and eventually
>>> the process times out. We have a query as follows:
>>>
>>> SELECT
>>> ad_group_criterion.criterion_id,
>>> ad_group_criterion.status,
>>> ad_group_criterion.keyword.match_type,
>>> ad_group_criterion.keyword.text,
>>>...
>>> FROM ad_group_criterion WHERE ad_group_criterion.type = KEYWORD
>>>
>>> and we use the searchStream API to process the results. The result set
>>> is very large, and could exceed 100,000 keywords but up until today we
>>> never had a problem with it.
>>>
>>> On Tuesday, September 27, 2022 at 3:28:02 PM UTC-5 adsapi wrote:
>>>
Hello Everyone,
Thank you so much for providing all this information! I believe the
issue that was causing this was cleared up a few hours ago. I see our
monitoring is showing that traffic is back to normal. If you continue to
have the same issue, please respond back to me.
Thanks,
[image: Google Logo]
Nadine Wang
Google Ads API Team
ref:_00D1U1174p._5004Q2egwqd:ref
>>>
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also
Re: StreamingResponseIterator from search_stream gets stuck
Hi,
same here. Some reports succeed now. But some reports don't, most likely
related to the data size. We didn't change anything on our side, all report
fetches worked fine with their defined date ranges before Sept 27, when the
issue started.
The issue hasn't been fully resolved and we're still unable to fetch all
reports via the Google Ads API. The behavior for the failing fetches is
still the same. The iterator in the search_stream fetch gets stuck, is
never finishing and at some point our wrapping timeout kills the task.
Since there was no code change on our side and many developers are
reporting the same issue here, using different programming languages, we
assume the issue is on the Google Ads API side. My guess would be that the
GRPC call for search_stream is not properly ended. To the best of our
knowledge and our inspection so far, this is far too low level from the
perspective of users of the Google Ads API (developers) and needs to be
fixed on the Google side.
Ideally please provide an ETA when the issue will be fixed on Google side.
Also I wonder where we can best track the status of the API issue.
https://ads.google.com/status/publisher/ continuously shows Google Ads
operating fine, though the API clearly has severe issues for search_stream
requests.
Thanks,
Robert
On Wednesday, 28 September 2022 at 01:49:05 UTC+2 Andrew Repp wrote:
> Hi,
>
> We are also continuing to see the same issue even after the all clear
> message above. It appears to be related to the number of records involved.
>
> This query
>
> SELECT
> segments.date,customer.id,campaign.name,campaign.id
> ,campaign_criterion.bid_modifier,campaign_criterion.criterion_id,segments.device,metrics.impressions,metrics.clicks,metrics.cost_micros,metrics.gmail_forwards,metrics.gmail_saves,metrics.gmail_secondary_clicks
> FROM
> campaign_audience_view
> WHERE metrics.impressions > 0 AND segments.date BETWEEN '2022-09-23'
> AND '2022-09-27'
>
> fails with
>
> Response
> ---
> Headers: {}
> Fault: {
> "created": "@1664321827.318637000",
> "description": "Error received from peer ipv4:142.250.217.74:443",
> "file": "src/core/lib/surface/call.cc",
> "file_line": 904,
> "grpc_message": "Deadline Exceeded",
> "grpc_status": 4
> }
>
> But this query
>
> SELECT
> segments.date,customer.id,campaign.name,campaign.id
> ,campaign_criterion.bid_modifier,campaign_criterion.criterion_id,segments.device,metrics.impressions,metrics.clicks,metrics.cost_micros,metrics.gmail_forwards,metrics.gmail_saves,metrics.gmail_secondary_clicks
> FROM
> campaign_audience_view
> WHERE metrics.impressions > 0 AND segments.date BETWEEN '2022-09-26'
> AND '2022-09-27'
>
> succeeds. Notice that by only selecting 2 days of data instead of 5, the
> query is able to complete. Before today, the query for 5 days completed
> successfully.
>
> Any help from the Google folks on a resolution?
>
> Thanks,
> Andrew
>
> On Tuesday, September 27, 2022 at 3:39:56 PM UTC-7 gel...@gmail.com wrote:
>
>> We're still seeing this problem when streaming a large set of keywords
>> from a customer account. It appears the request gets stuck and eventually
>> the process times out. We have a query as follows:
>>
>> SELECT
>> ad_group_criterion.criterion_id,
>> ad_group_criterion.status,
>> ad_group_criterion.keyword.match_type,
>> ad_group_criterion.keyword.text,
>>...
>> FROM ad_group_criterion WHERE ad_group_criterion.type = KEYWORD
>>
>> and we use the searchStream API to process the results. The result set is
>> very large, and could exceed 100,000 keywords but up until today we never
>> had a problem with it.
>>
>> On Tuesday, September 27, 2022 at 3:28:02 PM UTC-5 adsapi wrote:
>>
>>> Hello Everyone,
>>>
>>> Thank you so much for providing all this information! I believe the
>>> issue that was causing this was cleared up a few hours ago. I see our
>>> monitoring is showing that traffic is back to normal. If you continue to
>>> have the same issue, please respond back to me.
>>>
>>> Thanks,
>>> [image: Google Logo]
>>> Nadine Wang
>>> Google Ads API Team
>>>
>>>
>>> ref:_00D1U1174p._5004Q2egwqd:ref
>>>
>>
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"Google Ads API and AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an
Re: StreamingResponseIterator from search_stream gets stuck
We're still seeing this problem when streaming a large set of keywords from a customer account. It appears the request gets stuck and eventually the process times out. We have a query as follows: SELECT ad_group_criterion.criterion_id, ad_group_criterion.status, ad_group_criterion.keyword.match_type, ad_group_criterion.keyword.text, ... FROM ad_group_criterion WHERE ad_group_criterion.type = KEYWORD and we use the searchStream API to process the results. The result set is very large, and could exceed 100,000 keywords but up until today we never had a problem with it. On Tuesday, September 27, 2022 at 3:28:02 PM UTC-5 adsapi wrote: > Hello Everyone, > > Thank you so much for providing all this information! I believe the issue > that was causing this was cleared up a few hours ago. I see our monitoring > is showing that traffic is back to normal. If you continue to have the same > issue, please respond back to me. > > Thanks, > [image: Google Logo] > Nadine Wang > Google Ads API Team > > > ref:_00D1U1174p._5004Q2egwqd:ref > -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "Google Ads API and AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/387b2a4e-0d35-4034-8eb1-878035ecee0cn%40googlegroups.com.
Re: StreamingResponseIterator from search_stream gets stuck
Hi,
Not sure if I am doing anything wrong, but after running the following on
databricks (I run it before initialising client, as the documentation says
is needed):
import logging
logging.basicConfig(level=logging.INFO, format='[%(asctime)s -
%(levelname)s] %(message).5000s')
logging.getLogger("py4j").setLevel(logging.ERROR)
logging.getLogger('google.ads.googleads.client').setLevel(logging.DEBUG)
I don't get any logs before it gets stuck.
Kind regards,
Op dinsdag 27 september 2022 om 17:32:23 UTC+2 schreef Robert Heise:
> Hi,
>
> we're having the same problem, though we're using the official ruby gem
> from Google Ads. The exact same ruby code worked fine until today. Since
> today most jobs are getting stuck while iterating through the result. Since
> there are a few forum feeds reporting issues with search_stream behavior
> today (performance, timeout, getting stuck) it feels like an issue on the
> Google Ads API side. Can you confirm and give guidance how to overcome this
> behavior of the Google Ads API?
>
> Thanks,
>
> On Tuesday, 27 September 2022 at 11:42:45 UTC+2 tjadi@billygrace.com
> wrote:
>
>> Hi,
>>
>> I am wondering regarding what to do when a StreamingResponseIterator from
>> a ga_service.search_stream(search_request) seems to get stuck at a certain
>> 'row' and whether this is a bug or expected.
>>
>> Example (with printing to show that it gets stuck):
>> [image: Screenshot-stream.png]
>> [image: point where it gets stuck.png]
>> It doesn't matter how many times I redo the query, it gets stuck at some
>> point at a specific point (batch 17 after row 7000). Before yesterday
>> evening the exact same code/ data used to run fine within a minute. One of
>> our jobs continued in this state for 6 hours, before it got killed by its
>> time out (while yesterday it ran in 5 min). Not sure what I can do about
>> this?
>>
>> Kind regards,
>>
>>
--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog:
https://googleadsdeveloper.blogspot.com/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
You received this message because you are subscribed to the Google
Groups "AdWords API and Google Ads API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups
"Google Ads API and AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to adwords-api+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/adwords-api/7789cbb0-4cd9-497f-a405-93f647a7f5a8n%40googlegroups.com.
Re: StreamingResponseIterator from search_stream gets stuck
Hi, Please find the error log message I found before it gets stuck below: [2022-09-27 17:20:58,571 - INFO] Request made: ClientCustomerId: , Host: googleads.googleapis.com, Method: /google.ads.googleads.v11.services.GoogleAdsService/SearchStream, RequestId: Mzq9Od7rpC1j9df0ry-hLQ, IsFault: False, FaultMessage: None Is this sufficient? Kind regards, Op dinsdag 27 september 2022 om 17:32:23 UTC+2 schreef Robert Heise: > Hi, > > we're having the same problem, though we're using the official ruby gem > from Google Ads. The exact same ruby code worked fine until today. Since > today most jobs are getting stuck while iterating through the result. Since > there are a few forum feeds reporting issues with search_stream behavior > today (performance, timeout, getting stuck) it feels like an issue on the > Google Ads API side. Can you confirm and give guidance how to overcome this > behavior of the Google Ads API? > > Thanks, > > On Tuesday, 27 September 2022 at 11:42:45 UTC+2 tjadi@billygrace.com > wrote: > >> Hi, >> >> I am wondering regarding what to do when a StreamingResponseIterator from >> a ga_service.search_stream(search_request) seems to get stuck at a certain >> 'row' and whether this is a bug or expected. >> >> Example (with printing to show that it gets stuck): >> [image: Screenshot-stream.png] >> [image: point where it gets stuck.png] >> It doesn't matter how many times I redo the query, it gets stuck at some >> point at a specific point (batch 17 after row 7000). Before yesterday >> evening the exact same code/ data used to run fine within a minute. One of >> our jobs continued in this state for 6 hours, before it got killed by its >> time out (while yesterday it ran in 5 min). Not sure what I can do about >> this? >> >> Kind regards, >> >> -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "Google Ads API and AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/226fd310-e98c-4f6e-abde-0432ec5506b1n%40googlegroups.com.
Re: StreamingResponseIterator from search_stream gets stuck
Hi, we're having the same problem, though we're using the official ruby gem from Google Ads. The exact same ruby code worked fine until today. Since today most jobs are getting stuck while iterating through the result. Since there are a few forum feeds reporting issues with search_stream behavior today (performance, timeout, getting stuck) it feels like an issue on the Google Ads API side. Can you confirm and give guidance how to overcome this behavior of the Google Ads API? Thanks, On Tuesday, 27 September 2022 at 11:42:45 UTC+2 tjadi@billygrace.com wrote: > Hi, > > I am wondering regarding what to do when a StreamingResponseIterator from > a ga_service.search_stream(search_request) seems to get stuck at a certain > 'row' and whether this is a bug or expected. > > Example (with printing to show that it gets stuck): > [image: Screenshot-stream.png] > [image: point where it gets stuck.png] > It doesn't matter how many times I redo the query, it gets stuck at some > point at a specific point (batch 17 after row 7000). Before yesterday > evening the exact same code/ data used to run fine within a minute. One of > our jobs continued in this state for 6 hours, before it got killed by its > time out (while yesterday it ran in 5 min). Not sure what I can do about > this? > > Kind regards, > > -- -- =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ Also find us on our blog: https://googleadsdeveloper.blogspot.com/ =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~ You received this message because you are subscribed to the Google Groups "AdWords API and Google Ads API Forum" group. To post to this group, send email to adwords-api@googlegroups.com To unsubscribe from this group, send email to adwords-api+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/adwords-api?hl=en --- You received this message because you are subscribed to the Google Groups "Google Ads API and AdWords API Forum" group. To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/a3e74823-7fb8-421e-bfe5-8ea4c3455d46n%40googlegroups.com.
