I created following JIRAs. Integration test for TextIO ReadAll and dynamic writes: https://issues.apache.org/jira/browse/BEAM-3211 Integration test for MongoDBIO: https://issues.apache.org/jira/browse/BEAM-3212 Large scale performance test for MongoDBIO: https://issues.apache.org/jira/browse/BEAM-3213 Integration test for HBaseIO: https://issues.apache.org/jira/browse/BEAM-3214 Large scale performance test for HBaseIO: https://issues.apache.org/jira/browse/BEAM-3215 Integration test for HadoopInputFormatIO: https://issues.apache.org/jira/browse/BEAM-3216 Large scale performance test for HadoopInputFormatIO: https://issues.apache.org/jira/browse/BEAM-3217
Thanks, Cham On Wed, Nov 15, 2017 at 2:09 AM Chamikara Jayalath <chamik...@google.com> wrote: > Sounds good. I'll create JIRAs for both small scale and large scale > versions. To be clear, large scale versions will use PerfKitBenchmarker > based performance testing framework which will allow us to (1) manage > Kubernetes based multi-node deployments of data stores (2) publish/track > benchmark results. > > Thanks, > Cham > > > On Tue, Nov 14, 2017 at 10:09 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Cham, >> >> Some are in progress on private branch (at least for MongoDbIO ;)). >> >> +1 to create the Jira, we will assign to the right persons. >> >> Regards >> JB >> >> On 11/15/2017 01:09 AM, Chamikara Jayalath wrote: >> > Hi All, >> > >> > I went through the current list of I/O transforms offered by Beam Java >> SDK >> > and it looks like we currently don't have enough integration test >> coverage >> > for some of the I/O transforms. For example, it might be good to improve >> > test coverage of following (not a complete list). >> > >> > MongoDbIO Read/Write >> > HBaseIO Read/Write >> > HadoopInputFormatIO Read >> > HCatalogIO Read/Write >> > SolrO Read/Write >> > File-based I/O (Text, Avro, etc) Read/ReadAll/Write//Write with dynamic >> > destinations >> > >> > I hope to work with folks cc'ed in this email to add some of these >> tests. >> > We'll create JIRAs for each of the tests and mention original >> > authors/reviewers in PRs. We extremely appreciate any feedback you can >> > provide to improve these tests. When applicable we also hope to develop >> > large scale versions of some of these tests using Beam performance >> testing >> > framework (https://beam.apache.org/documentation/io/testing/). >> > >> > Thanks, >> > Cham >> > >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> >
