Here is my launcher image's entry point: ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"]
Needs to be: /opt/apache/beam/boot On Sun, Nov 3, 2024 at 3:10 PM XQ Hu via user <user@beam.apache.org> wrote: > The default worker container image is built with > https://github.com/apache/beam/blob/master/sdks/python/container/py311/base_image_requirements.txt. > This does not install google-cloud-secret-manager. > > When your code needs more python packages, it is always recommended using > https://cloud.google.com/dataflow/docs/guides/using-custom-containers. > > Since you already use the container for your template launcher, it is > convenient to just use one container image for both launcher and workers. > So you do not need to maintain two images. > > On Sun, Nov 3, 2024 at 5:52 PM Henry Tremblay via user < > user@beam.apache.org> wrote: > >> You shouldn’t have to use an sdk_container_image. It is not in the docs, >> and I talked to Google, and they said a container image is not needed. >> Also, why does the requests library work, and the secret manager does not? >> >> >> >> *From:* XQ Hu via user <user@beam.apache.org> >> *Sent:* Sunday, November 3, 2024 1:53 PM >> *To:* user@beam.apache.org >> *Cc:* XQ Hu <x...@google.com> >> *Subject:* Re: Solution to import problem >> >> >> >> I think the problem is you do not specify sdk_container_image when >> running your template. >> >> >> >> >> https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#run-the-template >> has more details. >> >> >> >> Basically, you do not need to >> https://github.com/paulhtremblay/data-engineering/blob/main/dataflow_/flex_proj_with_secret_manager/Dockerfile >> for your template launcher. >> >> >> >> You can use the same image for both launcher and dataflow workers. You >> only need to copy python_template_launcher to your image like >> https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38. >> Then you can use this image for both launcher and dataflow workers. When >> running the template job, you need to add --parameters >> sdk_container_image=$SDK_CONTAINER_IMAGE. >> >> >> >> >> >> On Sun, Nov 3, 2024 at 4:18 PM Henry Tremblay <paulhtremb...@gmail.com> >> wrote: >> >> A few weeks ago I had posted a problem I had with importing the Google >> Cloud Secret Manager library in Python. >> >> >> >> Here is the problem and solution: >> >> >> >> >> https://github.com/paulhtremblay/data-engineering/tree/main/dataflow_/flex_proj_with_secret_manager >> >> >> >> -- >> >> Henry Tremblay >> >> Data Engineer, Best Buy >> >> -- Henry Tremblay Data Engineer, Best Buy
