Re: Question about E2E tests for pipelines

KafkaIOIT can also use testcontainers to create a fake Kafka service [1]. In theory we could do something similar to test PubSubIO by using the provided emulator [2], but currently we only test against the production pubsub service. There's a useful TestPubsub [3] rule which can be used to create

Re: Question about E2E tests for pipelines

Thank you for the information and links, Alexey! We will try to follow this approach. On 25 Nov 2020, at 21:27, Alexey Romanenko mailto:aromanenko@gmail.com>> wrote: For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It leverages a real Kafka cluster that runs on k8s.

Re: Question about E2E tests for pipelines

For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It leverages a real Kafka cluster that runs on k8s. So, probably you can follow the similar approach. In the same time, we fake Kafka consumer and its output for KafkaIO unit tests. [1]

Question about E2E tests for pipelines

Hi Devs, We are finalizing this PR with a pipeline that reads from Kafka and writes to Pub/Sub without any transformations in between. We would like to implement e2e tests where we create and execute a pipeline, but we haven't found much information