Thank you for the information and links, Alexey! We will try to follow this approach.
On 25 Nov 2020, at 21:27, Alexey Romanenko <aromanenko....@gmail.com<mailto:aromanenko....@gmail.com>> wrote: For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It leverages a real Kafka cluster that runs on k8s. So, probably you can follow the similar approach. In the same time, we fake Kafka consumer and its output for KafkaIO unit tests. [1] https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java [2] https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy On 25 Nov 2020, at 13:05, Artur Khanin <artur.kha...@akvelon.com<mailto:artur.kha...@akvelon.com>> wrote: Hi Devs, We are finalizing this PR<https://github.com/apache/beam/pull/13112> with a pipeline that reads from Kafka and writes to Pub/Sub without any transformations in between. We would like to implement e2e tests where we create and execute a pipeline, but we haven't found much information and relevant examples about it.How exactly should we implement such kind of tests? Can we mock somehow Kafka and Pub/Sub or maybe can we set them up using some test environment? Any information and hints will be greatly appreciated! Thanks, Artur Khanin Akvelon, Inc
