liferoad commented on issue #29527:
URL: https://github.com/apache/beam/issues/29527#issuecomment-1824764211
I did my own test with this code:
```
import argparse
import logging
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import StandardOptions
from apache_beam.io import ReadFromPubSub
from google.cloud import translate_v3 as translate
class TranslateMessage(beam.DoFn):
def __init__(self, project_id):
self.project_id = project_id
def setup(self):
self.translate_client = translate.TranslationServiceClient()
def process(self, element):
text = element.decode(encoding="utf-8")
location = "global"
parent = f"projects/{self.project_id}/locations/{location}"
language_detected = self.translate_client.detect_language(
content=text, parent=parent
).languages[0]
logging.info("Translating message: " + text + " lang: " +
language_detected.language_code)
yield (text, language_detected)
def main(argv=None, save_main_session=True):
"""Main entry point; defines and runs the wordcount pipeline."""
parser = argparse.ArgumentParser()
known_args, pipeline_args = parser.parse_known_args(argv)
options = PipelineOptions(pipeline_args)
options.view_as(SetupOptions).save_main_session = True
options.view_as(StandardOptions).streaming = True
project = options.view_as(GoogleCloudOptions).project
topic = "projects/pubsub-public-data/topics/shakespeare-kinglear"
p = beam.Pipeline(options=options)
_ = (
p
| "Read PubSub topic" >> ReadFromPubSub(topic=topic)
| beam.ParDo(TranslateMessage(project))
)
p.run()
if __name__ == '__main__':
logging.getLogger().setLevel(logging.INFO)
main()
```
I do not see any problem:
<img width="628" alt="image"
src="https://github.com/apache/beam/assets/7833268/8c2f2763-4f80-4f1e-9df1-7a8073d16030";>
The code needs to be polished to handle throttling but it seems working well
with Runner V2 and google-cloud-translate-3.12.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
