[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754922#comment-15754922 ] ASF GitHub Bot commented on METRON-587: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-metron/pull/374 > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754357#comment-15754357 ] ASF GitHub Bot commented on METRON-587: --- Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/374 Done. Sorry about that > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[
https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15753000#comment-15753000
]
ASF GitHub Bot commented on METRON-587:
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/374#discussion_r92731270
--- Diff:
metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
---
@@ -153,9 +168,17 @@ public void test() throws Exception {
fluxComponent.submitTopology();
kafkaComponent.writeMessages(Constants.ENRICHMENT_TOPIC,
inputMessages);
- ProcessorResult>> result =
runner.process(getProcessor(inputMessages));
+ ProcessorResult>> result =
runner.process(getProcessor());
// We expect failures, so we don't care if result returned failure
or not
List> docs = result.getResult();
+ // OTTO>>>
--- End diff --
*might* not want the OTTO there, unless you want to be immortalized in
Metron code @ottobackwards ;) Lord knows the `StellarTest` has enough
mentions of my name, though, so I can't talk.
> Integration tests should use common processor implementations where possible
>
>
> Key: METRON-587
> URL: https://issues.apache.org/jira/browse/METRON-587
> Project: Metron
> Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> We have 7 or so integration tests that implement the Processor interface.
> Much of the code in these implementations is the same. It would be better
> for the system if we had some common implementations of the Processor
> interface.
> - This would make it easier to write correct tests
> - This would reduce errors from copy/paste test generation
> - This would clean up the codebase and make it easier to understand
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[
https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752999#comment-15752999
]
ASF GitHub Bot commented on METRON-587:
---
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/374#discussion_r92731374
--- Diff:
metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
---
@@ -153,9 +168,17 @@ public void test() throws Exception {
fluxComponent.submitTopology();
kafkaComponent.writeMessages(Constants.ENRICHMENT_TOPIC,
inputMessages);
- ProcessorResult>> result =
runner.process(getProcessor(inputMessages));
+ ProcessorResult>> result =
runner.process(getProcessor());
// We expect failures, so we don't care if result returned failure
or not
List> docs = result.getResult();
+ // OTTO>>>
--- End diff --
I also assume those printlns are debug code and should be removed.
> Integration tests should use common processor implementations where possible
>
>
> Key: METRON-587
> URL: https://issues.apache.org/jira/browse/METRON-587
> Project: Metron
> Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> We have 7 or so integration tests that implement the Processor interface.
> Much of the code in these implementations is the same. It would be better
> for the system if we had some common implementations of the Processor
> interface.
> - This would make it easier to write correct tests
> - This would reduce errors from copy/paste test generation
> - This would clean up the codebase and make it easier to understand
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735084#comment-15735084 ] ASF GitHub Bot commented on METRON-587: --- Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/374 Resolved the issue: The enrichment test is different in that it expect both errors and messages of the same amount. The code was not handling the circumstance where the errors arrived before the messages from kafka correctly. > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15735085#comment-15735085 ] ASF GitHub Bot commented on METRON-587: --- GitHub user ottobackwards reopened a pull request: https://github.com/apache/incubator-metron/pull/374 METRON-587 Integration tests should use common processor implementations where possible We have 7 or so integration tests that implement the Processor interface. Much of the code in these implementations is the same. It would be better for the system if we had some common implementations of the Processor interface. This would make it easier to write correct tests This would reduce errors from copy/paste test generation This would clean up the codebase and make it easier to understand This PR creates a KafkaProcessor class which implements the common processor code used for all the Parser Integration Test based classes, The WriterBolt Integration Test and the Enrichment Integration tests. The pcap and simple hbase enrichments are 'different' enough to continue to use their own interface implementations. The indexing tests are less likely to have new additions to them, and can be part of a subsequent effort if required. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-587 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/374.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #374 commit 0ae8378e21ad84a279cc41d829cb873fd1287e99 Author: Otto Fowler Date: 2016-11-23T19:09:53Z METRON-587 create Processor implementation for Kafka based tests Implment in EnrichmentIntegrationTest as poc commit 93a676d278b763ca6f92f6f8d8863ecd1ceabd96 Author: Otto Fowler Date: 2016-11-28T11:39:11Z merge tracking branch apache/master commit 1b725add240333f483de0f1922be597a3423f7b6 Author: Otto Fowler Date: 2016-11-28T15:01:43Z For new enrichment tests, get errors at the same time as getting messages commit 20ef76199fa5b67e289c96b584086f31a5799558 Author: Otto Fowler Date: 2016-11-28T19:45:19Z more api evolution handle more cases ASA integration failing with timeouts? commit 21a76c0f4023b1182e0463397689a5e54cc889e2 Author: Otto Fowler Date: 2016-11-28T20:36:42Z add in all the topics allow more retries commit 5e09cf57ab557037883e598fca4e48a28b968342 Author: Otto Fowler Date: 2016-11-28T20:37:56Z Merge remote-tracking branch 'apache/master' into METRON-587 commit 10e159d1054fa37b90d9353d8e14def8217dae00 Author: Otto Fowler Date: 2016-11-28T20:49:39Z fix up imports commit 905bf53d0269bb6a6eed027f3673f604af034288 Author: Otto Fowler Date: 2016-11-28T20:58:00Z fixed package for KafkaMessageSet commit 1f241b972728ed5ee22719a9439833ab4446b57f Author: Otto Fowler Date: 2016-11-29T16:58:29Z fix for message read counts failing asa tests commit e0221db83b498e78e87209e38f7ff107966d06dc Author: Otto Fowler Date: 2016-11-29T16:59:58Z Merge remote-tracking branch 'apache/master' into METRON-587 commit 946af47ebba6629bce35cfefebb203e0637479b4 Author: Otto Fowler Date: 2016-11-29T20:20:01Z fix for enrichment tests commit 82201d3130f0518bf2edd2e0950c66dbe67170eb Author: Otto Fowler Date: 2016-12-07T20:38:02Z merge tracking branch apache/master resolve conflicts commit 7062cd1bca5e43e37139f5af96065019fc51713c Author: Otto Fowler Date: 2016-12-07T20:57:18Z remove 2 vars that from merge commit f42b7094536758f05b1b6c8b5eeb76c2edf28507 Author: Otto Fowler Date: 2016-12-08T21:22:56Z refactored and simplified - will help for cases like the enrichment tests, where we want to wait for errors and messages > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732472#comment-15732472 ] ASF GitHub Bot commented on METRON-587: --- Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/374 I am going to close this until I figure out what in the latest sync is throwing of the timing in the tests > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732471#comment-15732471 ] ASF GitHub Bot commented on METRON-587: --- Github user ottobackwards closed the pull request at: https://github.com/apache/incubator-metron/pull/374 > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706313#comment-15706313 ] ASF GitHub Bot commented on METRON-587: --- GitHub user ottobackwards reopened a pull request: https://github.com/apache/incubator-metron/pull/374 METRON-587 Integration tests should use common processor implementations where possible We have 7 or so integration tests that implement the Processor interface. Much of the code in these implementations is the same. It would be better for the system if we had some common implementations of the Processor interface. This would make it easier to write correct tests This would reduce errors from copy/paste test generation This would clean up the codebase and make it easier to understand This PR creates a KafkaProcessor class which implements the common processor code used for all the Parser Integration Test based classes, The WriterBolt Integration Test and the Enrichment Integration tests. The pcap and simple hbase enrichments are 'different' enough to continue to use their own interface implementations. The indexing tests are less likely to have new additions to them, and can be part of a subsequent effort if required. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-587 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/374.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #374 commit 0ae8378e21ad84a279cc41d829cb873fd1287e99 Author: Otto Fowler Date: 2016-11-23T19:09:53Z METRON-587 create Processor implementation for Kafka based tests Implment in EnrichmentIntegrationTest as poc commit 93a676d278b763ca6f92f6f8d8863ecd1ceabd96 Author: Otto Fowler Date: 2016-11-28T11:39:11Z merge tracking branch apache/master commit 1b725add240333f483de0f1922be597a3423f7b6 Author: Otto Fowler Date: 2016-11-28T15:01:43Z For new enrichment tests, get errors at the same time as getting messages commit 20ef76199fa5b67e289c96b584086f31a5799558 Author: Otto Fowler Date: 2016-11-28T19:45:19Z more api evolution handle more cases ASA integration failing with timeouts? commit 21a76c0f4023b1182e0463397689a5e54cc889e2 Author: Otto Fowler Date: 2016-11-28T20:36:42Z add in all the topics allow more retries commit 5e09cf57ab557037883e598fca4e48a28b968342 Author: Otto Fowler Date: 2016-11-28T20:37:56Z Merge remote-tracking branch 'apache/master' into METRON-587 commit 10e159d1054fa37b90d9353d8e14def8217dae00 Author: Otto Fowler Date: 2016-11-28T20:49:39Z fix up imports commit 905bf53d0269bb6a6eed027f3673f604af034288 Author: Otto Fowler Date: 2016-11-28T20:58:00Z fixed package for KafkaMessageSet commit 1f241b972728ed5ee22719a9439833ab4446b57f Author: Otto Fowler Date: 2016-11-29T16:58:29Z fix for message read counts failing asa tests commit e0221db83b498e78e87209e38f7ff107966d06dc Author: Otto Fowler Date: 2016-11-29T16:59:58Z Merge remote-tracking branch 'apache/master' into METRON-587 > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706227#comment-15706227 ] ASF GitHub Bot commented on METRON-587: --- Github user ottobackwards closed the pull request at: https://github.com/apache/incubator-metron/pull/374 > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706230#comment-15706230 ] ASF GitHub Bot commented on METRON-587: --- Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/374 Re-Run travis - build working locally > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible
[ https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703083#comment-15703083 ] ASF GitHub Bot commented on METRON-587: --- GitHub user ottobackwards opened a pull request: https://github.com/apache/incubator-metron/pull/374 METRON-587 Integration tests should use common processor implementations where possible We have 7 or so integration tests that implement the Processor interface. Much of the code in these implementations is the same. It would be better for the system if we had some common implementations of the Processor interface. This would make it easier to write correct tests This would reduce errors from copy/paste test generation This would clean up the codebase and make it easier to understand This PR creates a KafkaProcessor class which implements the common processor code used for all the Parser Integration Test based classes, The WriterBolt Integration Test and the Enrichment Integration tests. The pcap and simple hbase enrichments are 'different' enough to continue to use their own interface implementations. The indexing tests are less likely to have new additions to them, and can be part of a subsequent effort if required. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/incubator-metron METRON-587 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/374.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #374 commit 0ae8378e21ad84a279cc41d829cb873fd1287e99 Author: Otto Fowler Date: 2016-11-23T19:09:53Z METRON-587 create Processor implementation for Kafka based tests Implment in EnrichmentIntegrationTest as poc commit 93a676d278b763ca6f92f6f8d8863ecd1ceabd96 Author: Otto Fowler Date: 2016-11-28T11:39:11Z merge tracking branch apache/master commit 1b725add240333f483de0f1922be597a3423f7b6 Author: Otto Fowler Date: 2016-11-28T15:01:43Z For new enrichment tests, get errors at the same time as getting messages commit 20ef76199fa5b67e289c96b584086f31a5799558 Author: Otto Fowler Date: 2016-11-28T19:45:19Z more api evolution handle more cases ASA integration failing with timeouts? commit 21a76c0f4023b1182e0463397689a5e54cc889e2 Author: Otto Fowler Date: 2016-11-28T20:36:42Z add in all the topics allow more retries commit 5e09cf57ab557037883e598fca4e48a28b968342 Author: Otto Fowler Date: 2016-11-28T20:37:56Z Merge remote-tracking branch 'apache/master' into METRON-587 commit 10e159d1054fa37b90d9353d8e14def8217dae00 Author: Otto Fowler Date: 2016-11-28T20:49:39Z fix up imports > Integration tests should use common processor implementations where possible > > > Key: METRON-587 > URL: https://issues.apache.org/jira/browse/METRON-587 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > We have 7 or so integration tests that implement the Processor interface. > Much of the code in these implementations is the same. It would be better > for the system if we had some common implementations of the Processor > interface. > - This would make it easier to write correct tests > - This would reduce errors from copy/paste test generation > - This would clean up the codebase and make it easier to understand -- This message was sent by Atlassian JIRA (v6.3.4#6332)
