[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378055#comment-16378055 ] Willy Lulciuc commented on BEAM-3664: - [~kenn] Great suggestions. I'll try a few approaches. Feel free to assign this ticket to me. [~iemejia] Happy to update ElasticsearchIO tests as well after I figure out an approach. > Port SolrIOTest off DoFnTester > -- > > Key: BEAM-3664 > URL: https://issues.apache.org/jira/browse/BEAM-3664 > Project: Beam > Issue Type: Sub-task > Components: io-java-solr >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372917#comment-16372917 ] Ismaël Mejía commented on BEAM-3664: Jump in just to say that the ElasticsearchIO tests use exactly the same pattern that Solr, so if you fix Solr you can easily solve the Elasticsearch ticket for free (BEAM-3674). Feel free to take it if you are interested. > Port SolrIOTest off DoFnTester > -- > > Key: BEAM-3664 > URL: https://issues.apache.org/jira/browse/BEAM-3664 > Project: Beam > Issue Type: Sub-task > Components: io-java-solr >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372869#comment-16372869 ] Kenneth Knowles commented on BEAM-3664: --- [~wslulciuc] I just read through the test. We have a few tests like this. I think they are a bit overspecified. I would suggest: 1. Eliminate the for loop and just pass the input docs to {{TestStream.addElements}}. 2. Verify that the output is as expected. 3. Determine a way to track insert requests to Solr. Could be via a fake {{solrClient}} (probably best) or by tweaking the data you insert with request ids. 4. Verify that no request exceeded the max size. > Port SolrIOTest off DoFnTester > -- > > Key: BEAM-3664 > URL: https://issues.apache.org/jira/browse/BEAM-3664 > Project: Beam > Issue Type: Sub-task > Components: io-java-solr >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester
[ https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372481#comment-16372481 ] Willy Lulciuc commented on BEAM-3664: - [~kenn]: Replacing DoFnTester with PipelineTest in testWriteWithMaxBatchSize() is not as straight forward here (at least it certainly would seem). Let me explain why. The test has the following comment: "write bundles size is the runner decision, we cannot force a bundle size, so we test the Writer as a DoFn outside of a runner." Meaning, DoFnTester.of() is used to invoke processElement() as the Solr documents are iterated over, with periodic calls to commit documents, then check insertion counters, etc. I've tried a couples ways. But none providing the convenience of DoFnTester.of() to compare the number inserted vs processed. Am I missing something? > Port SolrIOTest off DoFnTester > -- > > Key: BEAM-3664 > URL: https://issues.apache.org/jira/browse/BEAM-3664 > Project: Beam > Issue Type: Sub-task > Components: io-java-solr >Reporter: Kenneth Knowles >Priority: Major > Labels: beginner, newbie, starter > -- This message was sent by Atlassian JIRA (v7.6.3#76005)