Ah, I think you need the DOCKER environment to use cross-language transforms not the EXTERNAL environment (agree that the terminology is confusing).
On Tue, May 4, 2021 at 11:04 AM Nir Gazit <nir....@gmail.com> wrote: > Yes that’s on purpose. I’m running in Kubernetes which makes it hard to > install docker on the pods so I don’t want to use the docker environment. > That’s why I specified EXTERNAL environment in PipelineOptions. However, it > seems that it doesn’t get propagated. > > On Tue, 4 May 2021 at 20:59 Chamikara Jayalath <chamik...@google.com> > wrote: > >> Is it possible that you don't have the "docker" command available in your >> system ? >> >> On Tue, May 4, 2021 at 10:28 AM Nir Gazit <nir....@gmail.com> wrote: >> >>> Hey, >>> I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL >>> environment. However, when the pipeline is run, the error below is thrown, >>> which implies that for some reason the external environment pipeline >>> options didn't get in. When replacing the Kafka Source with an S3 source >>> (for example) I don't get this error so it implies that it's >>> somewhere around the external transform / expansion service area. How can I >>> debug this? >>> >>> Caused by: java.io.IOException: Cannot run program "docker": error=2, No >>> such file or directory >>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:189) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:171) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runImage(DockerCommand.java:95) >>> at >>> org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory.createEnvironment(DockerEnvironmentFactory.java:131) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:252) >>> at >>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:231) >>> >>
