It does seem like the Dataflow will do some validation around PubSub params before actually creating the pipeline. That's fair for Dataflow because Dataflow will swap the PubSubIO from beam implementation into Dataflow native one.
I think if you really want to run your virtual PubSub with Dataflow, you need to try out --experiments=enable_custom_pubsub_sink to enforce Dataflow not to do the override. Would you like to share your job id thus we can verify the failure. Also I'm not sure about the motivation to test it against Dataflow. Would you like to elaborate more on that? On Mon, Jan 18, 2021 at 3:51 AM Ramazan Yapparov < [email protected]> wrote: > Hi Beam! > We've been writing E2E test for KafkaToPubsub example pipeline. Instead of > depending on some real > Cloud Pubsub and Kafka instances we decided to use Testcontainers. > We launch Kafka and PubSub Emulator containers and after that we pass > containers urls into pipeline options and run the pipeline. > During PR review we received a request for turning this test into IT so it > would run in Dataflow Runner > instead of Direct Runner. > Trying to do so, we've ran into some troubles with that: > 1. While running the test all Docker containers start at the machine where > the test is running, > so in order for this test to work properly dataflow job should be able > to reach test-runner machine by a public IP. > I certainly can't do it on my local machine, not sure how it will > behave when running in CI environment. > 2. When we pass our fake PubSub url into the dataflow job we receive > following error: > json > { > "code" : 400, > "errors" : [ { > "domain" : "global", > "message" : "(f214233f9dbe6968): The workflow could not be created. > Causes: (f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub > URL.", > "reason" : "badRequest" > } ], > "message" : "(f214233f9dbe6968): The workflow could not be created. > Causes: (f214233f9dbe6719): http://localhost:49169 is not a valid Pub/Sub > URL.", > "status" : "INVALID_ARGUMENT" > } > > Not sure how this can be avoided, looks like the job will only accept the > real Cloud PubSub url. > It would be great if you share some thoughts or any suggestions how it can > be solved! > >
