[
https://issues.apache.org/jira/browse/BEAM-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17040971#comment-17040971
]
Elias Djurfeldt edited comment on BEAM-9145 at 2/20/20 1:40 PM:
----------------------------------------------------------------
Are there any other transforms in Beam that wait for an external API to finish
processing?
Reason I'm asking is that e.g. potentially annotating lots of longer videos,
there will be a lot of waiting time for the CPUs of the workers. How does Beam
deal with this? I get that we want the results back for potential processing in
a later step in the pipeline, but in the ideal case, we would send off an API
request, let the CPU move on and process the next item while awaiting the
response from the API.
For a e.g. Python web server, we would just do `async def` and then `await` the
response. But that's probably not feasible here.
What are your thoughts? Is this a non-issue?
[~kamilwu] [~mwalenia]
was (Author: eliasdjur):
Are there any other transforms in Beam that wait for an external API to finish
processing?
Reason I'm asking is that e.g. potentially annotating lots of longer videos,
there will be a lot of waiting time for the CPUs of the workers. How does Beam
deal with this? I get that we want the results back for potential processing in
a later step in the pipeline, but in the ideal case, we would send off an API
request, let the CPU move on and process the next item while awaiting the
response from the API.
For a Python web server, we would just do `async def` and then `await` the
response. But that's probably not feasible here.
What are your thoughts? Is this a non-issue?
[~kamilwu] [~mwalenia]
> Integrate Google Cloud AI functionalities
> -----------------------------------------
>
> Key: BEAM-9145
> URL: https://issues.apache.org/jira/browse/BEAM-9145
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp, io-py-gcp
> Reporter: Kamil Wasilewski
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
