[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[ https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606418#comment-16606418 ] ASF GitHub Bot commented on METRON-1748: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1174 This looks great @nickwallen. Thanks for the contribution and for taking the time with my feedback. +1 > Improve Storm Profiler Integration Test > --- > > Key: METRON-1748 > URL: https://issues.apache.org/jira/browse/METRON-1748 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > We should use the Profiler Client, like PROFILE_GET, to validate the output > of the Storm Profiler Integration Test. This is better validation that > things are working end-to-end. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1174: METRON-1748 Improve Storm Profiler Integration Test
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1174 This looks great @nickwallen. Thanks for the contribution and for taking the time with my feedback. +1 ---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606415#comment-16606415
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215785684
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -127,94 +127,53 @@
@Multiline
private static String kryoSerializers;
- /**
- * The Profiler can generate profiles based on processing time. With
processing time,
- * the Profiler builds profiles based on when the telemetry is processed.
- *
- * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
- * to use processing time.
- *
- * There are two mechanisms that will cause a profile to flush.
- *
- * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
- * message (which can be either event or system time.) This advances
time and leads to profile
- * measurements being flushed.
- *
- * (2) If no messages arrive to advance time, then the "time-to-live"
mechanism will flush a profile
- * after a period of time.
- *
- * This test specifically tests the *first* mechanism where time is
advanced by incoming messages.
- */
@Test
public void testProcessingTime() throws Exception {
+uploadConfigToZookeeper(TEST_RESOURCES +
"/config/zookeeper/processing-time-test");
-// upload the config to zookeeper
-uploadConfig(TEST_RESOURCES +
"/config/zookeeper/processing-time-test");
-
-// start the topology and write test messages to kafka
+// start the topology and write 3 test messages to kafka
fluxComponent.submitTopology();
-
-// the messages that will be applied to the profile
kafkaComponent.writeMessages(inputTopic, message1);
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
// retrieve the profile measurement using PROFILE_GET
String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
-List actuals = execute(profileGetExpression, List.class);
+List measurements = execute(profileGetExpression, List.class);
-// storm needs at least one message to close its event window
+// need to keep checking for measurements until the profiler has
flushed one out
int attempt = 0;
-while(actuals.size() == 0 && attempt++ < 10) {
+while(measurements.size() == 0 && attempt++ < 10) {
// wait for the profiler to flush
long sleep = windowDurationMillis;
LOG.debug("Waiting {} millis for profiler to flush", sleep);
Thread.sleep(sleep);
- // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // write another message to advance time. this ensures we are
testing the 'normal' flush mechanism.
// if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
// will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
- // retrieve the profile measurement using PROFILE_GET
- actuals = execute(profileGetExpression, List.class);
+ // try again to retrieve the profile measurement using PROFILE_GET
+ measurements = execute(profileGetExpression, List.class);
}
-// the profile should count at least 3 messages
-assertTrue(actuals.size() > 0);
-assertTrue(actuals.get(0) >= 3);
+// expect to see only 1 measurement, but could be more (one for each
period) depending on
--- End diff --
Ok, that makes sense - that's more what I meant when I said "period" in my
original inquiry. It's a measurement that is associated with the k-eth period.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215785684
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -127,94 +127,53 @@
@Multiline
private static String kryoSerializers;
- /**
- * The Profiler can generate profiles based on processing time. With
processing time,
- * the Profiler builds profiles based on when the telemetry is processed.
- *
- * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
- * to use processing time.
- *
- * There are two mechanisms that will cause a profile to flush.
- *
- * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
- * message (which can be either event or system time.) This advances
time and leads to profile
- * measurements being flushed.
- *
- * (2) If no messages arrive to advance time, then the "time-to-live"
mechanism will flush a profile
- * after a period of time.
- *
- * This test specifically tests the *first* mechanism where time is
advanced by incoming messages.
- */
@Test
public void testProcessingTime() throws Exception {
+uploadConfigToZookeeper(TEST_RESOURCES +
"/config/zookeeper/processing-time-test");
-// upload the config to zookeeper
-uploadConfig(TEST_RESOURCES +
"/config/zookeeper/processing-time-test");
-
-// start the topology and write test messages to kafka
+// start the topology and write 3 test messages to kafka
fluxComponent.submitTopology();
-
-// the messages that will be applied to the profile
kafkaComponent.writeMessages(inputTopic, message1);
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
// retrieve the profile measurement using PROFILE_GET
String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
-List actuals = execute(profileGetExpression, List.class);
+List measurements = execute(profileGetExpression, List.class);
-// storm needs at least one message to close its event window
+// need to keep checking for measurements until the profiler has
flushed one out
int attempt = 0;
-while(actuals.size() == 0 && attempt++ < 10) {
+while(measurements.size() == 0 && attempt++ < 10) {
// wait for the profiler to flush
long sleep = windowDurationMillis;
LOG.debug("Waiting {} millis for profiler to flush", sleep);
Thread.sleep(sleep);
- // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // write another message to advance time. this ensures we are
testing the 'normal' flush mechanism.
// if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
// will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
- // retrieve the profile measurement using PROFILE_GET
- actuals = execute(profileGetExpression, List.class);
+ // try again to retrieve the profile measurement using PROFILE_GET
+ measurements = execute(profileGetExpression, List.class);
}
-// the profile should count at least 3 messages
-assertTrue(actuals.size() > 0);
-assertTrue(actuals.get(0) >= 3);
+// expect to see only 1 measurement, but could be more (one for each
period) depending on
--- End diff --
Ok, that makes sense - that's more what I meant when I said "period" in my
original inquiry. It's a measurement that is associated with the k-eth period.
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606345#comment-16606345
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215761683
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> I looked at the profiler.json in test resources and it looks like it's
counting the number of messages that have ip_src_addr. Is that correct? Under
what condition(s) would actuals.size() != 1 for the purposes of this test?
Hopefully the latest commit clears up your questions around this.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215761683
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> I looked at the profiler.json in test resources and it looks like it's
counting the number of messages that have ip_src_addr. Is that correct? Under
what condition(s) would actuals.size() != 1 for the purposes of this test?
Hopefully the latest commit clears up your questions around this.
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606344#comment-16606344
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215761476
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> I could certainly add that. This is just how the tests were originally
written.
So I looked at this. I'd rather not do this now, only because the
ConfigUploader component doesnt just let me pass it a String to upload to Zk.
I need to give it a path. But I would definitely like to come back to this and
embed the profile definitions in the test like you suggested.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215761476
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> I could certainly add that. This is just how the tests were originally
written.
So I looked at this. I'd rather not do this now, only because the
ConfigUploader component doesnt just let me pass it a String to upload to Zk.
I need to give it a path. But I would definitely like to come back to this and
embed the profile definitions in the test like you suggested.
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606342#comment-16606342
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215760609
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
+
+// upload the config to zookeeper
--- End diff --
Done
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606341#comment-16606341
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215760574
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
--- End diff --
I removed the javadocs from the integration test and put it in the bolt's
javadoc. It seemed to fit there nicely.
Hopefully this satisfies your valid point around DRY. But let me know if
there is something else I can do.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215760609
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
+
+// upload the config to zookeeper
--- End diff --
Done
---
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215760574
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
--- End diff --
I removed the javadocs from the integration test and put it in the bolt's
javadoc. It seemed to fit there nicely.
Hopefully this satisfies your valid point around DRY. But let me know if
there is something else I can do.
---
[jira] [Commented] (METRON-1770) Add Docs for Running the Profiler with Spark on YARN
[ https://issues.apache.org/jira/browse/METRON-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606258#comment-16606258 ] ASF GitHub Bot commented on METRON-1770: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1189 METRON-1770 Add Docs for Running the Profiler with Spark on YARN I updated the README with advice for running the Batch Profiler with Spark on YARN. I also added a link with instructions for running on a secure cluster. This is all transparent to the Batch Profiler, which is why this is only a doc change. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1770 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1189.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1189 commit eb419d8bdc6e60900bb08142d4d2bb50b2bd8c23 Author: Nick Allen Date: 2018-09-06T18:47:28Z METRON-1770 Add Docs for Running the Profiler with Spark on YARN > Add Docs for Running the Profiler with Spark on YARN > > > Key: METRON-1770 > URL: https://issues.apache.org/jira/browse/METRON-1770 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > Add some docs to get help a user get started running the Profiler using Spark > on YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1189: METRON-1770 Add Docs for Running the Profiler wit...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1189 METRON-1770 Add Docs for Running the Profiler with Spark on YARN I updated the README with advice for running the Batch Profiler with Spark on YARN. I also added a link with instructions for running on a secure cluster. This is all transparent to the Batch Profiler, which is why this is only a doc change. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1770 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1189.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1189 commit eb419d8bdc6e60900bb08142d4d2bb50b2bd8c23 Author: Nick Allen Date: 2018-09-06T18:47:28Z METRON-1770 Add Docs for Running the Profiler with Spark on YARN ---
[jira] [Created] (METRON-1770) Add Docs for Running the Profiler with Spark on YARN
Nick Allen created METRON-1770: -- Summary: Add Docs for Running the Profiler with Spark on YARN Key: METRON-1770 URL: https://issues.apache.org/jira/browse/METRON-1770 Project: Metron Issue Type: Sub-task Reporter: Nick Allen Assignee: Nick Allen Add some docs to get help a user get started running the Profiler using Spark on YARN. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Deleted] (METRON-1716) MPack Support for the Batch Profiler
[ https://issues.apache.org/jira/browse/METRON-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen deleted METRON-1716: --- > MPack Support for the Batch Profiler > > > Key: METRON-1716 > URL: https://issues.apache.org/jira/browse/METRON-1716 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1717) Relocate Storm Profiler Code
[ https://issues.apache.org/jira/browse/METRON-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1717: -- Assignee: Nick Allen > Relocate Storm Profiler Code > > > Key: METRON-1717 > URL: https://issues.apache.org/jira/browse/METRON-1717 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > The Storm port of the Profiler currently lives in > metron-analytics/metron-profiler. This should be moved to > metron-analytics/metron-profiler-storm. This would mirror the project names > for the Spark port (metron-profiler-spark) and the REPL port > (metron-profiler-repl). > The package name for the Storm port of the Profiler should be changed to > org.apache.metron.profiler.storm. This would mimic the package name used for > Spark; org.apache.metron.profiler.spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606149#comment-16606149 ] ASF GitHub Bot commented on METRON-1769: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1188 That's definitely an option. I know offhand there's a few more similar flags (e.g. -o pipefail), which may also be useful. It might be sufficient, at least as a first cut. > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1188 That's definitely an option. I know offhand there's a few more similar flags (e.g. -o pipefail), which may also be useful. It might be sufficient, at least as a first cut. ---
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606124#comment-16606124
]
ASF GitHub Bot commented on METRON-1761:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r215704742
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.commons.io.IOUtils;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class MultiLineGrokParserTest extends GrokParserTest {
--- End diff --
Comparing this test with the base class and `SampleGrokParserTest`, there
doesn't seem to be any value in having this extend the abstract
`GrokParserTest` class. The `test()` method is overridden, and the other nugget
of value, `compare(...)`, is not used. I think your test would be much cleaner
and succinct if it was simply separated from the 1-line-per-message test.
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1761) Allow a grok statement to be applied to each line in a file.
[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606123#comment-16606123
]
ASF GitHub Bot commented on METRON-1761:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r215710648
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.commons.io.IOUtils;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class MultiLineGrokParserTest extends GrokParserTest {
--- End diff --
Also, can you add some tests around parsing failure scenarios in the batch
case?
> Allow a grok statement to be applied to each line in a file.
>
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
>Reporter: Laurens Vets
>Assignee: Otto Fowler
>Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.73 0.001048 0.57 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.86 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.28 0.41 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.15 0.23 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606125#comment-16606125 ] ASF GitHub Bot commented on METRON-1769: Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1188 For error handling, we should look into `set -e` and the related downfalls. It has been a while since I looked into it. http://mywiki.wooledge.org/BashFAQ/105 > Script creation of a release candidate > -- > > Key: METRON-1769 > URL: https://issues.apache.org/jira/browse/METRON-1769 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Our process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly > error prone. There's a lot of typing out versions and other tedious busywork. > In particular, this JIRA is for steps 4/5, as they're the most egregious. > This script should be able to produce both the core artifacts as well as the > bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1188: METRON-1769: Script creation of a release candidate
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1188 For error handling, we should look into `set -e` and the related downfalls. It has been a while since I looked into it. http://mywiki.wooledge.org/BashFAQ/105 ---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r215704742
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.commons.io.IOUtils;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class MultiLineGrokParserTest extends GrokParserTest {
--- End diff --
Comparing this test with the base class and `SampleGrokParserTest`, there
doesn't seem to be any value in having this extend the abstract
`GrokParserTest` class. The `test()` method is overridden, and the other nugget
of value, `compare(...)`, is not used. I think your test would be much cleaner
and succinct if it was simply separated from the 1-line-per-message test.
---
[GitHub] metron pull request #1184: METRON-1761, allow application of grok statement ...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1184#discussion_r215710648
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
---
@@ -0,0 +1,112 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.commons.io.IOUtils;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class MultiLineGrokParserTest extends GrokParserTest {
--- End diff --
Also, can you add some tests around parsing failure scenarios in the batch
case?
---
[jira] [Commented] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
[ https://issues.apache.org/jira/browse/METRON-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605955#comment-16605955 ] ASF GitHub Bot commented on METRON-1768: Github user mmiklavc commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 +1 via inspection > Adjust versioning of metron-bro-plugin-kafka to be x.y.z > > > Key: METRON-1768 > URL: https://issues.apache.org/jira/browse/METRON-1768 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > > In order to comprehensively do this, it will require an upstream fix in bro. > I opened https://bro-tracker.atlassian.net/browse/BIT-1985 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron-bro-plugin-kafka issue #8: METRON-1768: Adjust versioning of metron-b...
Github user mmiklavc commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 +1 via inspection ---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605934#comment-16605934
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215673075
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
Really what I mean is its the latest/most recent timestamp. The records are
not expected to be in order.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215673075
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
Really what I mean is its the latest/most recent timestamp. The records are
not expected to be in order.
---
[jira] [Commented] (METRON-1769) Script creation of a release candidate
[ https://issues.apache.org/jira/browse/METRON-1769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605927#comment-16605927 ] ASF GitHub Bot commented on METRON-1769: GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1188 METRON-1769: Script creation of a release candidate ## Contributor Comments Preliminary script that I wanted to make available for feedback. It basically scripts out steps 4 and 5 of the RC creation process https://cwiki.apache.org/confluence/display/METRON/Release+Process Basically it will * Checkout the various repos needed during the release process * Setup branches/directories as needed * Create the release artifacts for both core metron as well as metron-bro-plugin-kafka * This includes the tar.gz * The signatures (SHA-256 and SHA-512. MD5 and SHA-1 files are no longer included in the releases) * The various supplementary files (LICENSE, CHANGES, etc). It also has a practice run mode that won't push anything to the servers. Things that need to be improved * Need to handle only releasing core and not the plugin. Probably the other way around, too? * Error handling when things go wrong (e.g. couldn't sign artifacts). Right now it's standard Bash "Keep going until the script crashes" * Input validation around version format and such? * Might be nice to tie this into maven Sidenote, the general process around the plugin is really rough and undocumented right now. We should clean that up (and preferably make it match formatting wise with the rest of Metron. This might be things like version numbers as discussed on release threads, but even the tag naming conventions differ, along with minor things like '-' vs '_' which made doing this extra fun) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron release_utils Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1188.patch To close this pull request, make a
[GitHub] metron pull request #1188: METRON-1769: Script creation of a release candida...
GitHub user justinleet opened a pull request: https://github.com/apache/metron/pull/1188 METRON-1769: Script creation of a release candidate ## Contributor Comments Preliminary script that I wanted to make available for feedback. It basically scripts out steps 4 and 5 of the RC creation process https://cwiki.apache.org/confluence/display/METRON/Release+Process Basically it will * Checkout the various repos needed during the release process * Setup branches/directories as needed * Create the release artifacts for both core metron as well as metron-bro-plugin-kafka * This includes the tar.gz * The signatures (SHA-256 and SHA-512. MD5 and SHA-1 files are no longer included in the releases) * The various supplementary files (LICENSE, CHANGES, etc). It also has a practice run mode that won't push anything to the servers. Things that need to be improved * Need to handle only releasing core and not the plugin. Probably the other way around, too? * Error handling when things go wrong (e.g. couldn't sign artifacts). Right now it's standard Bash "Keep going until the script crashes" * Input validation around version format and such? * Might be nice to tie this into maven Sidenote, the general process around the plugin is really rough and undocumented right now. We should clean that up (and preferably make it match formatting wise with the rest of Metron. This might be things like version numbers as discussed on release threads, but even the tag naming conventions differ, along with minor things like '-' vs '_' which made doing this extra fun) ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/justinleet/metron release_utils Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1188.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1188 commit a61af11f61bf85b34248eae2143b4c50df0d89af Author: Justin Leet Date: 2018-09-06T14:53:51Z Adding release script to make my li
[jira] [Created] (METRON-1769) Script creation of a release candidate
Justin Leet created METRON-1769: --- Summary: Script creation of a release candidate Key: METRON-1769 URL: https://issues.apache.org/jira/browse/METRON-1769 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet Our process at: https://cwiki.apache.org/confluence/display/METRON/Release+Process is fairly error prone. There's a lot of typing out versions and other tedious busywork. In particular, this JIRA is for steps 4/5, as they're the most egregious. This script should be able to produce both the core artifacts as well as the bro kafka plugin artifacts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605892#comment-16605892
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215660252
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
On looking at that data file more closely, it looks like the records are
not in time order. That TS shows up on line 92 in the telemetry file rather
than the last line. Is that expected? I think I'm misunderstanding something.
Can you elaborate on that a bit more?
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215660252
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
On looking at that data file more closely, it looks like the records are
not in time order. That TS shows up on line 92 in the telemetry file rather
than the last line. Is that expected? I think I'm misunderstanding something.
Can you elaborate on that a bit more?
---
[jira] [Commented] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
[ https://issues.apache.org/jira/browse/METRON-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605824#comment-16605824 ] ASF GitHub Bot commented on METRON-1768: Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 +1 > Adjust versioning of metron-bro-plugin-kafka to be x.y.z > > > Key: METRON-1768 > URL: https://issues.apache.org/jira/browse/METRON-1768 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > > In order to comprehensively do this, it will require an upstream fix in bro. > I opened https://bro-tracker.atlassian.net/browse/BIT-1985 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron-bro-plugin-kafka issue #8: METRON-1768: Adjust versioning of metron-b...
Github user ottobackwards commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/8 +1 ---
[jira] [Commented] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
[ https://issues.apache.org/jira/browse/METRON-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605820#comment-16605820 ] Jon Zeolla commented on METRON-1768: If BIT-1985 gets completed and included in a release, in addition to changing to MAJOR.MINOR.PATCH I will need to adjust bro-pkg.meta to require >= the correct version of bro that includes it. > Adjust versioning of metron-bro-plugin-kafka to be x.y.z > > > Key: METRON-1768 > URL: https://issues.apache.org/jira/browse/METRON-1768 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > > In order to comprehensively do this, it will require an upstream fix in bro. > I opened https://bro-tracker.atlassian.net/browse/BIT-1985 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
[ https://issues.apache.org/jira/browse/METRON-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Zeolla updated METRON-1768: --- Description: In order to comprehensively do this, it will require an upstream fix in bro. I opened https://bro-tracker.atlassian.net/browse/BIT-1985 > Adjust versioning of metron-bro-plugin-kafka to be x.y.z > > > Key: METRON-1768 > URL: https://issues.apache.org/jira/browse/METRON-1768 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > > In order to comprehensively do this, it will require an upstream fix in bro. > I opened https://bro-tracker.atlassian.net/browse/BIT-1985 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605763#comment-16605763
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215621040
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
--- End diff --
Ok, fair point. Let me see if I can't move this to the README and clean it
up.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215621040
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
--- End diff --
Ok, fair point. Let me see if I can't move this to the README and clean it
up.
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605761#comment-16605761
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215620641
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> Along those lines, as a small recommendation for readability (by no means
a firm requirement here), but you might consider using the multiline string
annotation that we've leveraged in other areas of code.
I could certainly add that. This is just how the tests were originally
written.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215620641
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> Along those lines, as a small recommendation for readability (by no means
a firm requirement here), but you might consider using the multiline string
annotation that we've leveraged in other areas of code.
I could certainly add that. This is just how the tests were originally
written.
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605714#comment-16605714
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215606430
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
yes, that is the last timestamp contained in the telemetry. the comment
was meant to make that point, but maybe its not clear enough?
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215606430
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
yes, that is the last timestamp contained in the telemetry. the comment
was meant to make that point, but maybe its not clear enough?
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605704#comment-16605704
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215603987
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> For my own understanding, is the List that's returned as actuals a List
of profile periods?
it is a list of profile measurements; retrieve the profile measurement
using PROFILE_GET
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215603987
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
> For my own understanding, is the List that's returned as actuals a List
of profile periods?
it is a list of profile measurements; retrieve the profile measurement
using PROFILE_GET
---
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605700#comment-16605700
]
ASF GitHub Bot commented on METRON-1748:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215603116
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
--- End diff --
The same.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215603116
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
--- End diff --
The same.
---
[jira] [Commented] (METRON-1760) Kill PCAP job should prompt for confirmation
[ https://issues.apache.org/jira/browse/METRON-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605645#comment-16605645 ] ASF GitHub Bot commented on METRON-1760: Github user ruffle1986 closed the pull request at: https://github.com/apache/metron/pull/1182 > Kill PCAP job should prompt for confirmation > > > Key: METRON-1760 > URL: https://issues.apache.org/jira/browse/METRON-1760 > Project: Metron > Issue Type: Improvement >Reporter: Tamas Fodor >Assignee: Tamas Fodor >Priority: Major > > Since killing a PCAP job is a disruptive operation, it would be good to > prompt for user confirmation before carrying out the operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1182: METRON-1760: show a confirmation popover before k...
Github user ruffle1986 closed the pull request at: https://github.com/apache/metron/pull/1182 ---
[jira] [Commented] (METRON-1760) Kill PCAP job should prompt for confirmation
[ https://issues.apache.org/jira/browse/METRON-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605644#comment-16605644 ] ASF GitHub Bot commented on METRON-1760: Github user ruffle1986 commented on the issue: https://github.com/apache/metron/pull/1182 Closing due to incorrect PR conventions. > Kill PCAP job should prompt for confirmation > > > Key: METRON-1760 > URL: https://issues.apache.org/jira/browse/METRON-1760 > Project: Metron > Issue Type: Improvement >Reporter: Tamas Fodor >Assignee: Tamas Fodor >Priority: Major > > Since killing a PCAP job is a disruptive operation, it would be good to > prompt for user confirmation before carrying out the operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1182: METRON-1760: show a confirmation popover before killing ...
Github user ruffle1986 commented on the issue: https://github.com/apache/metron/pull/1182 Closing due to incorrect PR conventions. ---
