KafkaIOIT can also use testcontainers to create a fake Kafka service [1]. In theory we could do something similar to test PubSubIO by using the provided emulator [2], but currently we only test against the production pubsub service. There's a useful TestPubsub [3] rule which can be used to create a test PubSub topic that you can read or write to and make assertions against. Jenkins has permissions to use the apache-beam-testing project to create these topics on GCP.
[1] https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java#L336 [2] https://cloud.google.com/pubsub/docs/emulator [3] https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java On Thu, Nov 26, 2020 at 9:37 AM Artur Khanin <[email protected]> wrote: > Thank you for the information and links, Alexey! We will try to follow > this approach. > > On 25 Nov 2020, at 21:27, Alexey Romanenko <[email protected]> > wrote: > > For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It > leverages a real Kafka cluster that runs on k8s. So, probably you can > follow the similar approach. > > In the same time, we fake Kafka consumer and its output for KafkaIO unit > tests. > > [1] > https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java > [2] > https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy > > > On 25 Nov 2020, at 13:05, Artur Khanin <[email protected]> wrote: > > Hi Devs, > > We are finalizing this PR <https://github.com/apache/beam/pull/13112> with > a pipeline that reads from Kafka and writes to Pub/Sub without any > transformations in between. We would like to implement e2e tests where we > create and execute a pipeline, but we haven't found much information and > relevant examples about it.How exactly should we implement such kind of > tests? Can we mock somehow Kafka and Pub/Sub or maybe can we set them up > using some test environment? > > Any information and hints will be greatly appreciated! > > Thanks, > Artur Khanin > Akvelon, Inc > > > >
