Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

On Thu, May 6, 2021 at 4:58 AM Nir Gazit wrote: > Hey, > Not sure I follow - from the code you sent me, it seems that the > environment is chosen according to the pipeline options > (JAVA_SDK_HARNESS_ENVIRONMENT is used by createOrGetDefaultEnvironment only > AFAIU). So if I'm passing

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Hey, Not sure I follow - from the code you sent me, it seems that the environment is chosen according to the pipeline options (JAVA_SDK_HARNESS_ENVIRONMENT is used by createOrGetDefaultEnvironment only AFAIU). So if I'm passing `--environment_type=EXTERNAL` why wouldn't I get to using an external

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

When you use cross-language Java transforms from Python we use the default environment for Java transforms which always gets set to Docker.

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

But looking at the code of the exception it seems that it tries to use docker only because it thinks it's in a docker environment, no?

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Ah, I think you need the DOCKER environment to use cross-language transforms not the EXTERNAL environment (agree that the terminology is confusing). On Tue, May 4, 2021 at 11:04 AM Nir Gazit wrote: > Yes that’s on purpose. I’m running in Kubernetes which makes it hard to > install docker on the

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Yes that’s on purpose. I’m running in Kubernetes which makes it hard to install docker on the pods so I don’t want to use the docker environment. That’s why I specified EXTERNAL environment in PipelineOptions. However, it seems that it doesn’t get propagated. On Tue, 4 May 2021 at 20:59 Chamikara

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Is it possible that you don't have the "docker" command available in your system ? On Tue, May 4, 2021 at 10:28 AM Nir Gazit wrote: > Hey, > I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL > environment. However, when the pipeline is run, the error below is thrown, > which

Potential bug with Kafka (or other external) IO in Python Portable Runner

Hey, I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL environment. However, when the pipeline is run, the error below is thrown, which implies that for some reason the external environment pipeline options didn't get in. When replacing the Kafka Source with an S3 source (for