The default worker container image is built with https://github.com/apache/beam/blob/master/sdks/python/container/py311/base_image_requirements.txt. This does not install google-cloud-secret-manager.
When your code needs more python packages, it is always recommended using https://cloud.google.com/dataflow/docs/guides/using-custom-containers. Since you already use the container for your template launcher, it is convenient to just use one container image for both launcher and workers. So you do not need to maintain two images. On Sun, Nov 3, 2024 at 5:52 PM Henry Tremblay via user <user@beam.apache.org> wrote: > You shouldn’t have to use an sdk_container_image. It is not in the docs, > and I talked to Google, and they said a container image is not needed. > Also, why does the requests library work, and the secret manager does not? > > > > *From:* XQ Hu via user <user@beam.apache.org> > *Sent:* Sunday, November 3, 2024 1:53 PM > *To:* user@beam.apache.org > *Cc:* XQ Hu <x...@google.com> > *Subject:* Re: Solution to import problem > > > > I think the problem is you do not specify sdk_container_image when running > your template. > > > > > https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#run-the-template > has more details. > > > > Basically, you do not need to > https://github.com/paulhtremblay/data-engineering/blob/main/dataflow_/flex_proj_with_secret_manager/Dockerfile > for your template launcher. > > > > You can use the same image for both launcher and dataflow workers. You > only need to copy python_template_launcher to your image like > https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38. > Then you can use this image for both launcher and dataflow workers. When > running the template job, you need to add --parameters > sdk_container_image=$SDK_CONTAINER_IMAGE. > > > > > > On Sun, Nov 3, 2024 at 4:18 PM Henry Tremblay <paulhtremb...@gmail.com> > wrote: > > A few weeks ago I had posted a problem I had with importing the Google > Cloud Secret Manager library in Python. > > > > Here is the problem and solution: > > > > > https://github.com/paulhtremblay/data-engineering/tree/main/dataflow_/flex_proj_with_secret_manager > > > > -- > > Henry Tremblay > > Data Engineer, Best Buy > >
