For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It leverages a real Kafka cluster that runs on k8s. So, probably you can follow the similar approach.
In the same time, we fake Kafka consumer and its output for KafkaIO unit tests. [1] https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java [2] https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy > On 25 Nov 2020, at 13:05, Artur Khanin <[email protected]> wrote: > > Hi Devs, > > We are finalizing this PR <https://github.com/apache/beam/pull/13112> with a > pipeline that reads from Kafka and writes to Pub/Sub without any > transformations in between. We would like to implement e2e tests where we > create and execute a pipeline, but we haven't found much information and > relevant examples about it. > How exactly should we implement such kind of tests? Can we mock somehow Kafka > and Pub/Sub or maybe can we set them up using some test environment? > > Any information and hints will be greatly appreciated! > > Thanks, > Artur Khanin > Akvelon, Inc >
