Thanks all for the warm welcome and documentation. I'll peruse the links sent and ask follow-ups once I've digested the info.
Thanks again, Evan On Thu, Apr 22, 2021 at 12:11 PM Brian Hulette <[email protected]> wrote: > Welcome Evan! > > Note we do have some examples in Beam of running IO integration tests > against testcontainers [1] that startup "fakes". We do this for Kafka [2], > Kinesis [3], and there's a test that does this for both Kafka and Pubsub > [4]. Is that the kind of thing you had in mind? > > It looks like there is an Elasticsearch testcontainer you could use if > appropriate [5]. > > Brian > > [1] https://www.testcontainers.org/ > [2] > https://github.com/apache/beam/blob/93ecc1d3a4b997b2490c4439972ffaf09125299f/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java#L67 > [3] > https://github.com/apache/beam/blob/93ecc1d3a4b997b2490c4439972ffaf09125299f/sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisIOIT.java#L51 > [4] > https://github.com/apache/beam/blob/93ecc1d3a4b997b2490c4439972ffaf09125299f/examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubE2ETest.java#L49-L50 > [5] https://www.testcontainers.org/modules/elasticsearch/ > > On Thu, Apr 22, 2021 at 9:01 AM Alexey Romanenko <[email protected]> > wrote: > >> Hi Evan, >> >> Great to hear that you are going to contribute to Beam. Welcome! >> >> For integration tests we mostly use k8s. Did you take a look on current >> implementation of ITs for ElasticsearchIO (e.g. [1]) and how it runs on >> Jenkins [2]? >> Also, perhaps worth to mention our very good guide about writing ITs for >> Beam [3] >> >> [1] >> https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java >> [2] >> https://github.com/apache/beam/tree/master/.test-infra/kubernetes/elasticsearch >> [3] >> https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests >> >> --- >> Alexey >> >> >> >> On 22 Apr 2021, at 15:54, Evan Galpin <[email protected]> wrote: >> >> Hi folks! >> >> I'm Evan, and I'm fairly new to developing the Beam SDK. I've been a user >> for a number of years and have done some private SDK customizations along >> the way for my day job, but have recently been given the green light to >> contribute back to the OSS repo 🙌 In particular, I've worked with >> ElasticsearchIO the most. >> >> I'm looking ahead at how to potentially revamp the ElasticsearchIO test >> suite. I wanted to double-check if there are any WIP efforts underway so as >> to not duplicate effort or step on toes. >> >> Barring that, I'm thinking that using docker containers to support >> integration testing could be really beneficial. Do we have examples of >> doing just that in the Java SDK at the moment? Do the Jenkins VMs support >> docker? Looking around briefly outside of Beam it appears there are a few >> options for running Docker containers from gradle [1][2]. Or perhaps >> there's another alternative to Docker+gradle entirely. >> >> Thanks in advance for your advice, >> Evan >> >> [1] https://github.com/bmuschko/gradle-docker-plugin >> [2] https://github.com/palantir/gradle-docker >> >> >>
