Ross Rothenstine created BEAM-9525:
--------------------------------------
Summary: KafkaIO attempting to make connection when uploading via
Dataflow Runner
Key: BEAM-9525
URL: https://issues.apache.org/jira/browse/BEAM-9525
Project: Beam
Issue Type: Bug
Components: io-java-kafka
Reporter: Ross Rothenstine
Given our Kafka is running in GCP, within a VPC that is not accessible outside
of the VPC.
When uploading a pipeline to GCS using the Dataflow Runner via our CI/CD
worker, it attempts to make a connection to the broker. Our broker isn't
available outside the VPC, so this process fails. We can remedy this by running
a pod from within the VPC to upload, but it causes an awkward step within our
CI/CD process.
If there is a more elegant solution to this, such as KafkaIO taking in a
ValueProvider for bootstrap servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
