[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

2018-02-26 Thread Willy Lulciuc (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16378055#comment-16378055
 ] 

Willy Lulciuc commented on BEAM-3664:
-

[~kenn] Great suggestions. I'll try a few approaches. Feel free to assign this 
ticket to me.

[~iemejia] Happy to update ElasticsearchIO tests as well after I figure out an 
approach.

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

2018-02-22 Thread JIRA

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372917#comment-16372917
 ] 

Ismaël Mejía commented on BEAM-3664:


Jump in just to say that the ElasticsearchIO tests use exactly the same pattern 
that Solr, so if you fix Solr you can easily solve the Elasticsearch ticket for 
free (BEAM-3674). Feel free to take it if you are interested.

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

2018-02-22 Thread Kenneth Knowles (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372869#comment-16372869
 ] 

Kenneth Knowles commented on BEAM-3664:
---

[~wslulciuc] I just read through the test. We have a few tests like this. I 
think they are a bit overspecified. I would suggest:

1. Eliminate the for loop and just pass the input docs to 
{{TestStream.addElements}}.
2. Verify that the output is as expected.
3. Determine a way to track insert requests to Solr. Could be via a fake 
{{solrClient}} (probably best) or by tweaking the data you insert with request 
ids.
4. Verify that no request exceeded the max size.

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

2018-02-21 Thread Willy Lulciuc (JIRA)

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372481#comment-16372481
 ] 

Willy Lulciuc commented on BEAM-3664:
-

[~kenn]: Replacing DoFnTester with PipelineTest in testWriteWithMaxBatchSize() 
is not as straight forward here (at least it certainly would seem). Let me 
explain why. The test has the following comment:

"write bundles size is the runner decision, we cannot force a bundle size, so 
we test the Writer as a DoFn outside of a runner."

Meaning, DoFnTester.of() is used to invoke processElement() as the Solr 
documents are iterated over, with periodic calls to commit documents, then 
check insertion counters, etc.

I've tried a couples ways. But none providing the convenience of 
DoFnTester.of() to compare the number inserted vs processed.

Am I missing something?

 

 

 

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)