Welcome Evan! Note we do have some examples in Beam of running IO integration tests against testcontainers [1] that startup "fakes". We do this for Kafka [2], Kinesis [3], and there's a test that does this for both Kafka and Pubsub [4]. Is that the kind of thing you had in mind?
It looks like there is an Elasticsearch testcontainer you could use if appropriate [5]. Brian [1] https://www.testcontainers.org/ [2] https://github.com/apache/beam/blob/93ecc1d3a4b997b2490c4439972ffaf09125299f/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java#L67 [3] https://github.com/apache/beam/blob/93ecc1d3a4b997b2490c4439972ffaf09125299f/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java#L51 [4] https://github.com/apache/beam/blob/93ecc1d3a4b997b2490c4439972ffaf09125299f/examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java#L49-L50 [5] https://www.testcontainers.org/modules/elasticsearch/ On Thu, Apr 22, 2021 at 9:01 AM Alexey Romanenko <[email protected]> wrote: > Hi Evan, > > Great to hear that you are going to contribute to Beam. Welcome! > > For integration tests we mostly use k8s. Did you take a look on current > implementation of ITs for ElasticsearchIO (e.g. [1]) and how it runs on > Jenkins [2]? > Also, perhaps worth to mention our very good guide about writing ITs for > Beam [3] > > [1] > https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java > [2] > https://github.com/apache/beam/tree/master/.test-infra/kubernetes/elasticsearch > [3] > https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests > > --- > Alexey > > > > On 22 Apr 2021, at 15:54, Evan Galpin <[email protected]> wrote: > > Hi folks! > > I'm Evan, and I'm fairly new to developing the Beam SDK. I've been a user > for a number of years and have done some private SDK customizations along > the way for my day job, but have recently been given the green light to > contribute back to the OSS repo 🙌 In particular, I've worked with > ElasticsearchIO the most. > > I'm looking ahead at how to potentially revamp the ElasticsearchIO test > suite. I wanted to double-check if there are any WIP efforts underway so as > to not duplicate effort or step on toes. > > Barring that, I'm thinking that using docker containers to support > integration testing could be really beneficial. Do we have examples of > doing just that in the Java SDK at the moment? Do the Jenkins VMs support > docker? Looking around briefly outside of Beam it appears there are a few > options for running Docker containers from gradle [1][2]. Or perhaps > there's another alternative to Docker+gradle entirely. > > Thanks in advance for your advice, > Evan > > [1] https://github.com/bmuschko/gradle-docker-plugin > [2] https://github.com/palantir/gradle-docker > > >
