For ENTRYPOINT, as long as your image copies the launcher file (like https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38), you can just do `ENTRYPOINT ["/opt/apache/beam/boot"]`. Again, using one container image is more convenient if you start managing more Python package depecdencies.
On Mon, Nov 4, 2024 at 1:16 AM Henry Tremblay <paulhtremb...@gmail.com> wrote: > Sorry, yes, you are correct, though Google does not document this. > > 1. Formerly I can import pyscog and requests because they are in the > worker image you linked to. > 2. secretmanager cannot be imported because it is not in the worker image. > 3. passing the parameter --parameters > sdk_container_image=$IMAGE_URL_WORKER causes the worker to use the > pre-built image > 4. I cannot use the same Docker image for both launcher and worker because > of the ENTRYPOINT > > On Sun, Nov 3, 2024 at 1:53 PM XQ Hu via user <user@beam.apache.org> > wrote: > >> I think the problem is you do not specify sdk_container_image when >> running your template. >> >> >> https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/flex-templates/pipeline_with_dependencies#run-the-template >> has more details. >> >> Basically, you do not need to >> https://github.com/paulhtremblay/data-engineering/blob/main/dataflow_/flex_proj_with_secret_manager/Dockerfile >> for your template launcher. >> >> You can use the same image for both launcher and dataflow workers. You >> only need to copy python_template_launcher to your image like >> https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/dataflow/flex-templates/pipeline_with_dependencies/Dockerfile#L38. >> Then you can use this image for both launcher and dataflow workers. When >> running the template job, you need to add --parameters >> sdk_container_image=$SDK_CONTAINER_IMAGE. >> >> >> On Sun, Nov 3, 2024 at 4:18 PM Henry Tremblay <paulhtremb...@gmail.com> >> wrote: >> >>> A few weeks ago I had posted a problem I had with importing the Google >>> Cloud Secret Manager library in Python. >>> >>> Here is the problem and solution: >>> >>> >>> https://github.com/paulhtremblay/data-engineering/tree/main/dataflow_/flex_proj_with_secret_manager >>> >>> -- >>> Henry Tremblay >>> Data Engineer, Best Buy >>> >> > > -- > Henry Tremblay > Data Engineer, Best Buy >
