[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605060#comment-16605060
]
ASF GitHub Bot commented on METRON-1748:
----------------------------------------
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215456000
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+ // retrieve the profile measurement using PROFILE_GET
+ String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+ List<Integer> actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
- while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+ while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
- // validate what was flushed
- List<Integer> actuals = read(
- profilerTable.getPutLog(),
- columnFamily,
- columnBuilder.getColumnQualifier("value"),
- Integer.class);
- assertEquals(1, actuals.size());
+ // the profile should count at least 3 messages
+ assertTrue(actuals.size() > 0);
--- End diff --
For my own understanding, is the List that's returned as `actuals` a List
of profile periods? I looked at the profiler.json in test resources and it
looks like it's counting the number of messages that have ip_src_addr. Is that
correct? Under what condition(s) would actuals.size() != 1 for the purposes of
this test? I think that having a clear understanding of the associated profile
for this test and why I would expect a value of 3 (as it's not just bc of the
flushing mechanism) would be useful for future maintainers.
Along those lines, as a small recommendation for readability (by no means a
firm requirement here), but you might consider using the multiline string
annotation that we've leveraged in other areas of code. It allows the config to
sit right along side the test code itself. Just easier to read imo when the
size of the multiline string is still manageable/small.
> Improve Storm Profiler Integration Test
> ---------------------------------------
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
> Reporter: Nick Allen
> Assignee: Nick Allen
> Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
