Re: Question about E2E tests for pipelines

[Brian Hulette]

KafkaIOIT can also use testcontainers to create a fake Kafka service [1].
In theory we could do something similar to test PubSubIO by using the
provided emulator [2], but currently we only test against the production
pubsub service. There's a useful TestPubsub [3] rule which can be used to
create a test PubSub topic that you can read or write to and make
assertions against. Jenkins has permissions to use the apache-beam-testing
project to create these topics on GCP.

[1]
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java#L336
[2] https://cloud.google.com/pubsub/docs/emulator
[3]
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java

On Thu, Nov 26, 2020 at 9:37 AM Artur Khanin 
wrote:

> Thank you for the information and links, Alexey! We will try to follow
> this approach.
>
> On 25 Nov 2020, at 21:27, Alexey Romanenko 
> wrote:
>
> For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It
> leverages a real Kafka cluster that runs on k8s. So, probably you can
> follow the similar approach.
>
> In the same time, we fake Kafka consumer and its output for KafkaIO unit
> tests.
>
> [1]
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
> [2]
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
>
>
> On 25 Nov 2020, at 13:05, Artur Khanin  wrote:
>
> Hi Devs,
>
> We are finalizing this PR  with
> a pipeline that reads from Kafka and writes to Pub/Sub without any
> transformations in between. We would like to implement e2e tests where we
> create and execute a pipeline, but we haven't found much information and
> relevant examples about it.How exactly should we implement such kind of
> tests? Can we mock somehow Kafka and Pub/Sub or maybe can we set them up
> using some test environment?
>
> Any information and hints will be greatly appreciated!
>
> Thanks,
> Artur Khanin
> Akvelon, Inc
>
>
>
>

Re: Question about E2E tests for pipelines

[Artur Khanin]

Thank you for the information and links, Alexey! We will try to follow this 
approach.

On 25 Nov 2020, at 21:27, Alexey Romanenko 
mailto:aromanenko@gmail.com>> wrote:

For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It 
leverages a real Kafka cluster that runs on k8s. So, probably you can follow 
the similar approach.

In the same time, we fake Kafka consumer and its output for KafkaIO unit tests.

[1] 
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
[2] 
https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy


On 25 Nov 2020, at 13:05, Artur Khanin 
mailto:artur.kha...@akvelon.com>> wrote:

Hi Devs,

We are finalizing this PR with a 
pipeline that reads from Kafka and writes to Pub/Sub without any 
transformations in between. We would like to implement e2e tests where we 
create and execute a pipeline, but we haven't found much information and 
relevant examples about it.How exactly should we implement such kind of tests? 
Can we mock somehow Kafka and Pub/Sub or maybe can we set them up using some 
test environment?

Any information and hints will be greatly appreciated!

Thanks,
Artur Khanin
Akvelon, Inc

Re: Question about E2E tests for pipelines

[Alexey Romanenko]

For Kafka testing, there is a Kafka IT [1] that runs on Jenkins [2]. It 
leverages a real Kafka cluster that runs on k8s. So, probably you can follow 
the similar approach. 

In the same time, we fake Kafka consumer and its output for KafkaIO unit tests.

[1] 
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
[2] 
https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
   

> On 25 Nov 2020, at 13:05, Artur Khanin  wrote:
> 
> Hi Devs,
> 
> We are finalizing this PR  with a 
> pipeline that reads from Kafka and writes to Pub/Sub without any 
> transformations in between. We would like to implement e2e tests where we 
> create and execute a pipeline, but we haven't found much information and 
> relevant examples about it.
> How exactly should we implement such kind of tests? Can we mock somehow Kafka 
> and Pub/Sub or maybe can we set them up using some test environment?
> 
> Any information and hints will be greatly appreciated!
> 
> Thanks,
> Artur Khanin
> Akvelon, Inc
> 

Question about E2E tests for pipelines

[Artur Khanin]

Hi Devs,

We are finalizing this PR with a 
pipeline that reads from Kafka and writes to Pub/Sub without any 
transformations in between. We would like to implement e2e tests where we 
create and execute a pipeline, but we haven't found much information and 
relevant examples about it.How exactly should we implement such kind of tests? 
Can we mock somehow Kafka and Pub/Sub or maybe can we set them up using some 
test environment?

Any information and hints will be greatly appreciated!

Thanks,
Artur Khanin
Akvelon, Inc