[jira] [Commented] (METRON-1668) Remove login services and screens from UIs
[ https://issues.apache.org/jira/browse/METRON-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605124#comment-16605124 ] ASF GitHub Bot commented on METRON-1668: Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/1112 @mmiklavc the knoxsso is enabled with a toggle in the mpack, to follow the pattern for NiFi, Ranger and Atlas etc. There is a kind of failover of auth methods: knoxsso, ldap, nothing in our case. Many of the API calls require auth because they use username to load settings in the rest backend. As such I’m not sure no-auth makes sense, but maybe a default to anon would allow @rmerriman’s use case of dev runs, I’d still say that was a case for a mock backend though, in angular rather than rest. > Remove login services and screens from UIs > -- > > Key: METRON-1668 > URL: https://issues.apache.org/jira/browse/METRON-1668 > Project: Metron > Issue Type: Sub-task >Reporter: Simon Elliston Ball >Assignee: Simon Elliston Ball >Priority: Major > > Once we have integrated SSO, there is no need for the front-end apps to > present a login screen, we should remove these from the load footprint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user simonellistonball commented on the issue: https://github.com/apache/metron/pull/1112 @mmiklavc the knoxsso is enabled with a toggle in the mpack, to follow the pattern for NiFi, Ranger and Atlas etc. There is a kind of failover of auth methods: knoxsso, ldap, nothing in our case. Many of the API calls require auth because they use username to load settings in the rest backend. As such Iâm not sure no-auth makes sense, but maybe a default to anon would allow @rmerrimanâs use case of dev runs, Iâd still say that was a case for a mock backend though, in angular rather than rest. ---
[GitHub] metron-bro-plugin-kafka pull request #8: METRON-1768: Adjust versioning of m...
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/8 METRON-1768: Adjust versioning of metron-bro-plugin-kafka to be x.y.z You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1768 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/8.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8 commit f62b8b51195f804a81b5ac433b7b338d05d3d4e0 Author: Jon Zeolla Date: 2018-09-06T00:24:04Z METRON-1768: Adjust versioning of metron-bro-plugin-kafka to be x.y.z ---
[jira] [Commented] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
[ https://issues.apache.org/jira/browse/METRON-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605088#comment-16605088 ] ASF GitHub Bot commented on METRON-1768: GitHub user JonZeolla opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/8 METRON-1768: Adjust versioning of metron-bro-plugin-kafka to be x.y.z You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1768 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/8.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8 commit f62b8b51195f804a81b5ac433b7b338d05d3d4e0 Author: Jon Zeolla Date: 2018-09-06T00:24:04Z METRON-1768: Adjust versioning of metron-bro-plugin-kafka to be x.y.z > Adjust versioning of metron-bro-plugin-kafka to be x.y.z > > > Key: METRON-1768 > URL: https://issues.apache.org/jira/browse/METRON-1768 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1668) Remove login services and screens from UIs
[ https://issues.apache.org/jira/browse/METRON-1668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605084#comment-16605084 ] ASF GitHub Bot commented on METRON-1668: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1112 To @merrimanr 's point, can we provide a config flag to disable auth altogether, if desired? Maybe an Ambari config dropdown?, e.g. "NONE, KNOX SSO." Not sure what your plan is with the Ambari config in 1664, but I think that making SSO enabled/disabled as configurable like we do with Kerberos would be desirable. I realize there's very likely config and properties to deal with as supplied by Knox when setting it up in Ambari. I just don't know off the top of my head how much is automated (again, like Kerberos across the entire cluster) vs what's ad-hoc and per-component/service. Thoughts? > Remove login services and screens from UIs > -- > > Key: METRON-1668 > URL: https://issues.apache.org/jira/browse/METRON-1668 > Project: Metron > Issue Type: Sub-task >Reporter: Simon Elliston Ball >Assignee: Simon Elliston Ball >Priority: Major > > Once we have integrated SSO, there is no need for the front-end apps to > present a login screen, we should remove these from the load footprint. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1112: METRON-1668 Remove login services and screens from UIs
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1112 To @merrimanr 's point, can we provide a config flag to disable auth altogether, if desired? Maybe an Ambari config dropdown?, e.g. "NONE, KNOX SSO." Not sure what your plan is with the Ambari config in 1664, but I think that making SSO enabled/disabled as configurable like we do with Kerberos would be desirable. I realize there's very likely config and properties to deal with as supplied by Knox when setting it up in Ambari. I just don't know off the top of my head how much is automated (again, like Kerberos across the entire cluster) vs what's ad-hoc and per-component/service. Thoughts? ---
[jira] [Created] (METRON-1768) Adjust versioning of metron-bro-plugin-kafka to be x.y.z
Jon Zeolla created METRON-1768: -- Summary: Adjust versioning of metron-bro-plugin-kafka to be x.y.z Key: METRON-1768 URL: https://issues.apache.org/jira/browse/METRON-1768 Project: Metron Issue Type: Improvement Reporter: Jon Zeolla Assignee: Jon Zeolla -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605058#comment-16605058
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215452935
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
+
+// upload the config to zookeeper
--- End diff --
Can we remove the redundant javadoc? Your code speaks for itself here. I'd
probably just rename that method precisely as "uploadConfigToZookeeper". Very
clear and with 1 less non-executable line.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605057#comment-16605057
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215450093
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
--- End diff --
Is processing time == system time or are these actually 2 different types
of timestamp in addition to event/source time?
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605060#comment-16605060
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215456000
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
For my own understanding, is the List that's returned as `actuals` a List
of profile periods? I looked at the profiler.json in test resources and it
looks like it's counting the number of messages that have ip_src_addr. Is that
correct? Under what condition(s) would actuals.size() != 1 for the purposes of
this test? I think that having a clear understanding of the associated profile
for this test and why I would expect a value of 3 (as it's not just bc of the
flushing mechanism) would be useful for future maintainers.
Along those lines, as a small recommendation for readability (by no means a
firm requirement here), but you might consider using the multiline string
annotation that we've leveraged in other areas of code. It allows the config to
sit right along side the test code itself. Just easier to read imo when the
size of the multiline string is still manageable/small.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605061#comment-16605061
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215458574
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
What's the significance of this specific value `1530978728982L`? Is that
tied to the last record timestamp in the telemetry input data file?
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output
> of the Storm Profiler Integration Test. This is better validation that
> things are working end-to-end.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
[
https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605059#comment-16605059
]
ASF GitHub Bot commented on METRON-1748:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215452413
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
--- End diff --
Might it be better to have the core of this test's javadoc documentation in
the main README and simply reference the core terminology from the test?
Actually, your test name already has that info (test flushing via TTL), so even
that is probably redundant. My concern is that the info is encoded in all of:
* test code
* test javadoc
* source code
* source javadoc (maybe it's not, but I'm assuming there's also doc there)
* project README's
I think expressing the functionality in test/source code is of course good
because it's executable. And README's are great as a set of expectations about
how the project works. But my concern would be that if this functionality gets
updated by someone other than you, then they'll have to be sure it's updated in
5 places. If a test gets updated it might be easy to miss updating associated
javadoc of this detail. It's the kind of thing that Bob Martin, Martin Fowler,
and Kent Beck are constantly railing about wrt stale docs.
> Improve Storm Profiler Integration Test
> ---
>
> Key: METRON-1748
> URL: https://issues.apache.org/jira/browse/METRON-1748
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> We should
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215452413
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
--- End diff --
Might it be better to have the core of this test's javadoc documentation in
the main README and simply reference the core terminology from the test?
Actually, your test name already has that info (test flushing via TTL), so even
that is probably redundant. My concern is that the info is encoded in all of:
* test code
* test javadoc
* source code
* source javadoc (maybe it's not, but I'm assuming there's also doc there)
* project README's
I think expressing the functionality in test/source code is of course good
because it's executable. And README's are great as a set of expectations about
how the project works. But my concern would be that if this functionality gets
updated by someone other than you, then they'll have to be sure it's updated in
5 places. If a test gets updated it might be easy to miss updating associated
javadoc of this detail. It's the kind of thing that Bob Martin, Martin Fowler,
and Kent Beck are constantly railing about wrt stale docs.
---
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215450093
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
--- End diff --
Is processing time == system time or are these actually 2 different types
of timestamp in addition to event/source time?
---
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215452935
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
assertTrue(actuals.get(0) >= 3);
}
+ /**
+ * The Profiler can generate profiles based on processing time. With
processing time,
+ * the Profiler builds profiles based on when the telemetry is processed.
+ *
+ * Not defining a 'timestampField' within the Profiler configuration
tells the Profiler
+ * to use processing time.
+ *
+ * There are two mechanisms that will cause a profile to flush.
+ *
+ * (1) As new messages arrive, time is advanced. The splitter bolt
attaches a timestamp to each
+ * message (which can be either event or system time.) This advances
time and leads to profile
+ * measurements being flushed.
+ *
+ * (2) If no messages arrive to advance time, then the "time to live"
mechanism will flush a profile
+ * after a period of time.
+ *
+ * This test specifically tests the *second* mechanism when a profile
is flushed by the
+ * "time to live" mechanism.
+ */
+ @Test
+ public void testProcessingTimeWithTimeToLiveFlush() throws Exception {
+
+// upload the config to zookeeper
--- End diff --
Can we remove the redundant javadoc? Your code speaks for itself here. I'd
probably just rename that method precisely as "uploadConfigToZookeeper". Very
clear and with 1 less non-executable line.
---
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215458574
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -186,21 +266,40 @@ public void testEventTime() throws Exception {
// start the topology and write test messages to kafka
fluxComponent.submitTopology();
-kafkaComponent.writeMessages(inputTopic, message1);
-kafkaComponent.writeMessages(inputTopic, message2);
-kafkaComponent.writeMessages(inputTopic, message3);
-
-// wait until the profile is flushed
-waitOrTimeout(() -> profilerTable.getPutLog().size() > 0,
timeout(seconds(90)));
-
-List puts = profilerTable.getPutLog();
-assertEquals(1, puts.size());
-
-// inspect the row key to ensure the profiler used event time
correctly. the timestamp
-// embedded in the row key should match those in the source telemetry
-byte[] expectedRowKey = generateExpectedRowKey("event-time-test",
entity, startAt);
-byte[] actualRowKey = puts.get(0).getRow();
-assertArrayEquals(failMessage(expectedRowKey, actualRowKey),
expectedRowKey, actualRowKey);
+List messages = FileUtils.readLines(new
File("src/test/resources/telemetry.json"));
+kafkaComponent.writeMessages(inputTopic, messages);
+
+long timestamp = System.currentTimeMillis();
+LOG.debug("Attempting to close window period by sending message with
timestamp = {}", timestamp);
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.66.1",
timestamp));
+kafkaComponent.writeMessages(inputTopic, getMessage("192.168.138.158",
timestamp));
+
+// create the 'window' that looks up to 5 hours before the last
timestamp contained in the telemetry
+assign("lastTimestamp", "1530978728982L");
--- End diff --
What's the significance of this specific value `1530978728982L`? Is that
tied to the last record timestamp in the telemetry input data file?
---
[GitHub] metron pull request #1174: METRON-1748 Improve Storm Profiler Integration Te...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1174#discussion_r215456000
--- Diff:
metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
@@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
kafkaComponent.writeMessages(inputTopic, message2);
kafkaComponent.writeMessages(inputTopic, message3);
+// retrieve the profile measurement using PROFILE_GET
+String profileGetExpression = "PROFILE_GET('processing-time-test',
'10.0.0.1', PROFILE_FIXED('5', 'MINUTES'))";
+List actuals = execute(profileGetExpression, List.class);
+
// storm needs at least one message to close its event window
int attempt = 0;
-while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
+while(actuals.size() == 0 && attempt++ < 10) {
- // sleep, at least beyond the current window
- Thread.sleep(windowDurationMillis + windowLagMillis);
+ // wait for the profiler to flush
+ long sleep = windowDurationMillis;
+ LOG.debug("Waiting {} millis for profiler to flush", sleep);
+ Thread.sleep(sleep);
- // send another message to help close the current event window
+ // write another message to advance time. this ensures that we are
testing the 'normal' flush mechanism.
+ // if we do not send additional messages to advance time, then it is
the profile TTL mechanism which
+ // will ultimately flush the profile
kafkaComponent.writeMessages(inputTopic, message2);
+
+ // retrieve the profile measurement using PROFILE_GET
+ actuals = execute(profileGetExpression, List.class);
}
-// validate what was flushed
-List actuals = read(
-profilerTable.getPutLog(),
-columnFamily,
-columnBuilder.getColumnQualifier("value"),
-Integer.class);
-assertEquals(1, actuals.size());
+// the profile should count at least 3 messages
+assertTrue(actuals.size() > 0);
--- End diff --
For my own understanding, is the List that's returned as `actuals` a List
of profile periods? I looked at the profiler.json in test resources and it
looks like it's counting the number of messages that have ip_src_addr. Is that
correct? Under what condition(s) would actuals.size() != 1 for the purposes of
this test? I think that having a clear understanding of the associated profile
for this test and why I would expect a value of 3 (as it's not just bc of the
flushing mechanism) would be useful for future maintainers.
Along those lines, as a small recommendation for readability (by no means a
firm requirement here), but you might consider using the multiline string
annotation that we've leveraged in other areas of code. It allows the config to
sit right along side the test code itself. Just easier to read imo when the
size of the multiline string is still manageable/small.
---
[jira] [Commented] (METRON-1756) REST tests should use Embedded LDAP in metron-security
[ https://issues.apache.org/jira/browse/METRON-1756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604955#comment-16604955 ] ASF GitHub Bot commented on METRON-1756: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1186 +1 Looks good, thanks > REST tests should use Embedded LDAP in metron-security > -- > > Key: METRON-1756 > URL: https://issues.apache.org/jira/browse/METRON-1756 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Priority: Major > > Tests in metron-rest are still using JDBC-based authentication. These tests > (and all security-related tests) should use the testing infrastructure > introduced in METRON-1665. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1186: METRON-1756: REST tests should use Embedded LDAP in metr...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1186 +1 Looks good, thanks ---
[jira] [Commented] (METRON-1717) Relocate Storm Profiler Code
[ https://issues.apache.org/jira/browse/METRON-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604923#comment-16604923 ] ASF GitHub Bot commented on METRON-1717: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1187 METRON-1717 Relocate Storm Profiler Code - This change moves the Storm Profiler module to `metron-analytics/metron-profiler-storm`. - The core Storm Profiler package was renamed to `org.apache.metron.profiler.storm`. - All of the Profiler READMEs have been cleaned-up to contain only content relevant to each project. The main README is now in `metron-analytics/metron-profiler-common` which links to the others as needed. ## Testing 1. Stand-up a development environment. 1. Validate the development environment by ensuring alerts are visible within the Alerts UI and that the Metron Service Check in Ambari passes. 1. Launch the REPL and follow the instructions in the Profiler README to [create and execute a profile in the REPL](https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles). 1. Follow the instructions in the README to[ deploy the same profile in Storm](https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler-storm#getting-started). Ensure that you can retrieve values from HBase using `PROFILE_GET`. 1. Generate the site book and review the READMEs that have changed. ``` cd site-book mvn site ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1717 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1187.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1187 commit 290bc793a4cecb1c7c83ef4cfb77f67f5ef7dbbe Author: Nick Allen Date: 2018-09-05T16:12:51Z METRON-1717 Renamed Storm Profiler package commit b88c0e72974480750255d6e64faed24cf876527b Author: Nick Allen Date: 2018-09-05T17:15:46Z Rename package to org.apache.metron.profiler.storm commit 27e69d41c2e8a982dca23dfc6feca737b0e48c12 Author: Nick Allen Date: 2018-09-05T20:36:26Z Updated READMEs > Relocate Storm Profiler Code > > > Key: METRON-1717 > URL: https://issues.apache.org/jira/browse/METRON-1717 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Priority: Major > > The Storm
[GitHub] metron pull request #1187: METRON-1717 Relocate Storm Profiler Code
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1187 METRON-1717 Relocate Storm Profiler Code - This change moves the Storm Profiler module to `metron-analytics/metron-profiler-storm`. - The core Storm Profiler package was renamed to `org.apache.metron.profiler.storm`. - All of the Profiler READMEs have been cleaned-up to contain only content relevant to each project. The main README is now in `metron-analytics/metron-profiler-common` which links to the others as needed. ## Testing 1. Stand-up a development environment. 1. Validate the development environment by ensuring alerts are visible within the Alerts UI and that the Metron Service Check in Ambari passes. 1. Launch the REPL and follow the instructions in the Profiler README to [create and execute a profile in the REPL](https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler#creating-profiles). 1. Follow the instructions in the README to[ deploy the same profile in Storm](https://github.com/apache/metron/tree/master/metron-analytics/metron-profiler-storm#getting-started). Ensure that you can retrieve values from HBase using `PROFILE_GET`. 1. Generate the site book and review the READMEs that have changed. ``` cd site-book mvn site ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1717 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1187.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1187 commit 290bc793a4cecb1c7c83ef4cfb77f67f5ef7dbbe Author: Nick Allen Date: 2018-09-05T16:12:51Z METRON-1717 Renamed Storm Profiler package commit b88c0e72974480750255d6e64faed24cf876527b Author: Nick Allen Date: 2018-09-05T17:15:46Z Rename package to org.apache.metron.profiler.storm commit 27e69d41c2e8a982dca23dfc6feca737b0e48c12 Author: Nick Allen Date: 2018-09-05T20:36:26Z Updated READMEs ---
[jira] [Updated] (METRON-1607) update public web site to point at 0.5.0 new release
[ https://issues.apache.org/jira/browse/METRON-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1607: Fix Version/s: (was: 0.5.0) 0.6.0 > update public web site to point at 0.5.0 new release > > > Key: METRON-1607 > URL: https://issues.apache.org/jira/browse/METRON-1607 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Update the site for the 0.5.0 release per the process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1419) Create a SolrDao
[ https://issues.apache.org/jira/browse/METRON-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1419: Fix Version/s: 0.6.0 > Create a SolrDao > > > Key: METRON-1419 > URL: https://issues.apache.org/jira/browse/METRON-1419 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Create an implementation of the IndexDao for Solr. This will involve > implementing the various IndexDao methods using the SolrJ library and also > providing a SolrSearchIntegrationTest that extends SearchIntegrationTest > (similar to ElasticsearchSearchIntegrationTest). An integration test similar > to ElasticsearchUpdateIntegrationTest should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1436) Manually Install Solr Cloud in Full Dev
[ https://issues.apache.org/jira/browse/METRON-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1436: Fix Version/s: 0.6.0 > Manually Install Solr Cloud in Full Dev > --- > > Key: METRON-1436 > URL: https://issues.apache.org/jira/browse/METRON-1436 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.6.0 > > > Script and documentation for getting Solr setup in full dev -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1441) Create complementary Solr schemas for the main sensors
[ https://issues.apache.org/jira/browse/METRON-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1441: Fix Version/s: 0.6.0 > Create complementary Solr schemas for the main sensors > -- > > Key: METRON-1441 > URL: https://issues.apache.org/jira/browse/METRON-1441 > Project: Metron > Issue Type: Sub-task >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > We have ES templates for bro, snort, yaf, and error, we need corresponding > solr schemas for these collections. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1424) Kerberos: Solr
[ https://issues.apache.org/jira/browse/METRON-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1424: Fix Version/s: 0.6.0 > Kerberos: Solr > -- > > Key: METRON-1424 > URL: https://issues.apache.org/jira/browse/METRON-1424 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Verify that our Mpack passes the right credentials to Solr. Verify that Solr > kerberizes out of the box. This'll need to be spun up so we can find any > issues. > This should also be spun up for at least the ticket expiration time (over 24 > hours by default). We'd seen some issues with lowering the ticket expiration > time not seeming to be reflected in configs, so we should be careful to make > sure ticket expiration works as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1423) Ambari work to handle Solr configuration
[ https://issues.apache.org/jira/browse/METRON-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1423: Fix Version/s: 0.6.0 > Ambari work to handle Solr configuration > > > Key: METRON-1423 > URL: https://issues.apache.org/jira/browse/METRON-1423 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Ambari needs to be able to handle Solr configuration. This should be setup > similar to the existing ES config screen, and include anything necessary to > plug in a generic Solr connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1464) Convert schemas to be compatible with Solr 5.5.2
[ https://issues.apache.org/jira/browse/METRON-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1464: Fix Version/s: 0.6.0 > Convert schemas to be compatible with Solr 5.5.2 > > > Key: METRON-1464 > URL: https://issues.apache.org/jira/browse/METRON-1464 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > It would be ideal if our Solr schemas were compatible with Solr 5.5.2 or > 6.6.2. This would be users the option of using a manually installed Solr or > HDP Search. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1482) Update REST to work with Solr
[ https://issues.apache.org/jira/browse/METRON-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1482: Fix Version/s: 0.6.0 > Update REST to work with Solr > - > > Key: METRON-1482 > URL: https://issues.apache.org/jira/browse/METRON-1482 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Ambari should start REST with the correct Solr indexing jar on the classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1526) Location field types cause DocValuesField appear more than once error
[
https://issues.apache.org/jira/browse/METRON-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1526:
Fix Version/s: 0.6.0
> Location field types cause DocValuesField appear more than once error
> -
>
> Key: METRON-1526
> URL: https://issues.apache.org/jira/browse/METRON-1526
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> While testing [https://github.com/apache/metron/pull/970] I get this error
> when creating a meta alert:
> {code:java}
> Error from server at http://10.0.2.15:8983/solr/bro: Exception writing
> document id bbc150f5-92f8-485d-93cc-11730c1edf31 to the index; possible
> analysis error: DocValuesField
> \"enrichments.geo.ip_dst_addr.location_point_0_coordinate\" appears more than
> once in this document (only one value is allowed per field){code}
> I tracked it down to the fact that multiple fields are returned for a
> location field. For example when a field named
> "enrichments.geo.ip_dst_addr.location_point" is configured in a schema, these
> fields are returned in a query:
> {code:java}
> {
> "enrichments.geo.ip_dst_addr.location_point_0_coordinate": "33.4499",
> "enrichments.geo.ip_dst_addr.location_point_1_coordinate": "-112.0712",
> "enrichments.geo.ip_dst_addr.location_point": "33.4499,-112.0712"
> }
> {code}
> We need a way to either suppress these extra fields when querying or remove
> them before updating a document.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1503) Alerts are not getting populated in alerts UI when search engine is Solr
[
https://issues.apache.org/jira/browse/METRON-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1503:
Fix Version/s: 0.6.0
> Alerts are not getting populated in alerts UI when search engine is Solr
>
>
> Key: METRON-1503
> URL: https://issues.apache.org/jira/browse/METRON-1503
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> When navigating to the Alerts UI no alerts are displayed. REST is returning
> this error on a search call:
> {{Mar 27, 2018 6:51:50 AM org.apache.catalina.core.StandardWrapperValve
> invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in context
> with path [] threw exception [Request processing failed; nested exception is
> org.apache.solr.common.SolrException: Collection not found: websphere] with
> root cause org.apache.solr.common.SolrException: Collection not found:
> websphere at
> org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1401)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1094)
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) at
> org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) at
> org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957) at
> org.apache.metron.solr.dao.SolrSearchDao.search(SolrSearchDao.java:89) at
> org.apache.metron.solr.dao.SolrDao.search(SolrDao.java:90) at
> org.apache.metron.solr.dao.SolrMetaAlertDao.search(SolrMetaAlertDao.java:89)
> at
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:73)
> at
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
> at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1540) Solr Integration tests should use actual schemas
[ https://issues.apache.org/jira/browse/METRON-1540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1540: Fix Version/s: 0.6.0 > Solr Integration tests should use actual schemas > > > Key: METRON-1540 > URL: https://issues.apache.org/jira/browse/METRON-1540 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Right now some of the integration tests, e.g. SolrSearchIntegrationTest, > don't use the actual schemas for bro and snort. We've been bit a couple times > during the feature branch by these tests using stripped down test schemas, we > need to be using the actual ones and getting rid of the stripped down ones. > Given that the test schemas can vary a bit and things like the column > metadata varies, there's more work than just swapping them out. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1567) Large error message can't be written in Solr
[
https://issues.apache.org/jira/browse/METRON-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1567:
Fix Version/s: 0.6.0
> Large error message can't be written in Solr
>
>
> Key: METRON-1567
> URL: https://issues.apache.org/jira/browse/METRON-1567
> Project: Metron
> Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
> Fix For: 0.6.0
>
>
> Error message on the feature branch:
> {code:java}
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at
> http://ip-11-0-1-51.us-west-2.compute.internal:8983/solr/error: Exception
> writing document id cd6db5c1-f41b-4dcf-8f68-583c7fc08575 to the index;
> possible analysis error: Document contains at least one immense term in
> field="raw_message_1" (whose UTF8 encoding is longer than the max length
> 32766), all of which were skipped. Please correct the analyzer to not produce
> such terms. The prefix of the first immense term is: '[123, 34, 101, 120, 99,
> 101, 112, 116, 105, 111, 110, 34, 58, 34, 106, 97, 118, 97, 46, 105, 111, 46,
> 70, 105, 108, 101, 78, 111, 116, 70]...', original message: bytes can be at
> most 32766 in length; got 165866. Perhaps the document has an indexed string
> field (solr.StrField) which is too large
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
> ~[stormjar.jar:?]
> ...{code}
> This is a hard limit of string fields, per
> https://lucene.apache.org/solr/guide/6_6/field-types-included-with-solr.html
> It also mentions they aren't tokenized or analyzed, so it doesn't seem like
> we'd be able to turn this limit off.
> Text fields don't list any sort of limit (although they may still have one),
> so we may want to switch to that, but it would require testing.
> Additionally, it appears that raw_message is dynamic (since it's getting _1,
> but we don't define it in the schema).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1577) Solr searches don't include the index of the result
[
https://issues.apache.org/jira/browse/METRON-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1577:
Fix Version/s: 0.6.0
> Solr searches don't include the index of the result
> ---
>
> Key: METRON-1577
> URL: https://issues.apache.org/jira/browse/METRON-1577
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> For example
> {code:java}
> {
> "total": 370,
> "results": [
> {
> "id": "1dcf6e7e-9d16-477b-990e-e734bd400101",
> "source":
> {
> "guid": "1dcf6e7e-9d16-477b-990e-e734bd400101"
> },
> "score": 0,
> "index": null
> }
> ],
> "facetCounts": null
> }{code}
> We should also make sure that any other endpoints (if there are any) that
> return index, are populated properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1571) Correct KAFKA_TAIL Seek to End Logic
[ https://issues.apache.org/jira/browse/METRON-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1571: Fix Version/s: 0.6.0 > Correct KAFKA_TAIL Seek to End Logic > > > Key: METRON-1571 > URL: https://issues.apache.org/jira/browse/METRON-1571 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.6.0 > > > KAFKA_TAIL always needs to tail from the end of a topic. The current > implementation does not correctly seek to the end of the topic as the Kafka > API supports. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1421) Create a SolrMetaAlertDao
[ https://issues.apache.org/jira/browse/METRON-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1421: Fix Version/s: 0.6.0 > Create a SolrMetaAlertDao > - > > Key: METRON-1421 > URL: https://issues.apache.org/jira/browse/METRON-1421 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Create an implementation of the MetaAlertDao for Solr. This will involve > implementing the various MetaAlertDao methods using the SolrJ library and > also providing a SolrMetaAlertIntegrationTest (similar to > ElasticsearchMetaAlertIntegrationTest). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1579) Stellar should return the expression that failed in the exception
[ https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1579: Fix Version/s: 0.6.0 > Stellar should return the expression that failed in the exception > - > > Key: METRON-1579 > URL: https://issues.apache.org/jira/browse/METRON-1579 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > There are situations where we are not including the expression in the > exception. Also, in stellar enrichments, we should include the relevant > variable values in the exception to help with diagnosing issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1579) Stellar should return the expression that failed in the exception
[ https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1579: --- Assignee: Casey Stella > Stellar should return the expression that failed in the exception > - > > Key: METRON-1579 > URL: https://issues.apache.org/jira/browse/METRON-1579 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > There are situations where we are not including the expression in the > exception. Also, in stellar enrichments, we should include the relevant > variable values in the exception to help with diagnosing issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1593) Setting Metron rest additional classpath removes HBase and Hadoop configs from classpath
[
https://issues.apache.org/jira/browse/METRON-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1593:
Fix Version/s: 0.6.0
> Setting Metron rest additional classpath removes HBase and Hadoop configs
> from classpath
>
>
> Key: METRON-1593
> URL: https://issues.apache.org/jira/browse/METRON-1593
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.6.0
>
>
> There is a bug in the metron-rest.sh start script where HBase and Hadoop
> configuration directories get left off the classpath when the
> METRON_REST_CLASSPATH environment variable is populated initially. This
> happens when the "Metron rest additional classpath" setting in Ambari >
> Metron > Advanced > {color:#33}Advanced metron-rest-env is update with a
> value.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1547) Solr Comment Fields
[ https://issues.apache.org/jira/browse/METRON-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1547: Fix Version/s: 0.6.0 > Solr Comment Fields > --- > > Key: METRON-1547 > URL: https://issues.apache.org/jira/browse/METRON-1547 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Right now the Solr schemas don't have comment fields defined. It'll get > caught by the catch all with ignored type not multivalued. > ES just handles this correctly out of the box, but we'll need to take care of > it in Solr and document the schema restriction. > This actually is probably fairly problematic in comparison to ES. Solr > doesn't support an easy way of doing a complex structure without doing > something a bit weird (like parsing a string representation) or miserable > (nested document). > This will be incompatible with the current comment update system (just using > the patch() functionality). Preferably we need to add a new REST endpoint for > comments specifically so that we can handle it without the frontend knowing > the backend system. This also involves adjusting the UI to use the new REST > API, along with testing for both ES and Solr. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1589) '/api/v1/search/search' fails when 'Solr Zookeeper Urls' has comma separated multiple zookeeper urls
[
https://issues.apache.org/jira/browse/METRON-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1589:
Fix Version/s: 0.6.0
> '/api/v1/search/search' fails when 'Solr Zookeeper Urls' has comma separated
> multiple zookeeper urls
>
>
> Key: METRON-1589
> URL: https://issues.apache.org/jira/browse/METRON-1589
> Project: Metron
> Issue Type: Sub-task
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
> Fix For: 0.6.0
>
>
> http://metronnode:8082/api/v1/search/search with below payload fails with
> internal server error when 'Solr Zookeeper Urls' has comma separated multiple
> zookeeper urls
> {code:java}
> {
> "indices": [],
> "facetFields": [],
> "query": "*",
> "from": 0,
> "size": 25
> }
> {code}
> {code:java}
> {"responseCode":500,"message":"Cannot connect to cluster at
> ctr-e138-1518143905142-328005-01-06.hwx.site:2181/solr,ctr-e138-1518143905142-328005-01-05.hwx.site:2181/solr:
> cluster not found/not ready","fullMessage":"SolrException: Cannot connect to
> cluster at
> ctr-e138-1518143905142-328005-01-06.hwx.site:2181/solr,ctr-e138-1518143905142-328005-01-05.hwx.site:2181/solr:
> cluster not found/not ready"}
> {code}
>
> This appears to result from here
> [SolrDao.java#L137-L144|https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrDao.java#L137-L144]
> In `getSolrClient`, `withZkHost` should be called multiple times after
> splitting the comma delimited string, per
> [CloudSolrClient.Builder#withZkHost|https://lucene.apache.org/solr/6_5_0/solr-solrj/org/apache/solr/client/solrj/impl/CloudSolrClient.Builder.html#withZkHost-java.lang.String-].
> Overall, this could probably be done a couple ways, either to just split the
> String directly, or rearrange the methods to pass around Lists and have
> `getZkHost` be changed to `getZkHosts` and take care of it in a more
> contained manner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1594) KafkaWriter is asynchronous and may lose data on node failure
[ https://issues.apache.org/jira/browse/METRON-1594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1594: Fix Version/s: 0.6.0 > KafkaWriter is asynchronous and may lose data on node failure > - > > Key: METRON-1594 > URL: https://issues.apache.org/jira/browse/METRON-1594 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.6.0 > > > Currently, we do not block for data to be sent from kafka writer and we do > not batch. Unfortunately, the send call is asynchronous, so if the node dies > before the data is actually sent from kafka then it will drop data. It is > highly unlikely that we will be able to make kafkawriter synchronous in a > performant way, so we will likely need to batch in the parser and enrichment > topologies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1568) Stellar should have a _ special variable which returns the message in map form
[ https://issues.apache.org/jira/browse/METRON-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1568: --- Assignee: Casey Stella > Stellar should have a _ special variable which returns the message in map form > -- > > Key: METRON-1568 > URL: https://issues.apache.org/jira/browse/METRON-1568 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > In order to support functions which operate on the whole message, we should > have a special variable (_, keeping with the vaguely scala theme) which can > return the entire underlying message. This map should be immutable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1568) Stellar should have a _ special variable which returns the message in map form
[ https://issues.apache.org/jira/browse/METRON-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1568: Fix Version/s: 0.6.0 > Stellar should have a _ special variable which returns the message in map form > -- > > Key: METRON-1568 > URL: https://issues.apache.org/jira/browse/METRON-1568 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > In order to support functions which operate on the whole message, we should > have a special variable (_, keeping with the vaguely scala theme) which can > return the entire underlying message. This map should be immutable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1603) Fix multivalue field errors in Bro Solr schema
[
https://issues.apache.org/jira/browse/METRON-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1603:
Fix Version/s: 0.6.0
> Fix multivalue field errors in Bro Solr schema
> ---
>
> Key: METRON-1603
> URL: https://issues.apache.org/jira/browse/METRON-1603
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.6.0
>
>
> Running some additional test data through Bro with multiValued fields
> revealed that the Solr schema for Bro needs some attention. Exceptions are
> thrown like the following for fields that may have many values but aren't
> declared as such in the Solr schema.
> {code:java}
> 2018-06-05 07:34:11.903 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3]
> [ERROR]
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://10.0.2.15:7574/solr/bro: ERROR:
> [doc=26643986-b4ce-4ffe-b84e-6fe45143ac16] multiple values encountered for
> non multiValued field answers: [www.cisco.com.akadns.net,
> origin-www.cisco.com, 2001:420:1201:2::a]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:612)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:388)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1383)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1134)
> ~[stormjar.jar:?]
> at
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
> ~[stormjar.jar:?]
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
> ~[stormjar.jar:?]
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
> ~[stormjar.jar:?]
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
> ~[stormjar.jar:?]
> at org.apache.metron.solr.writer.SolrWriter.write(SolrWriter.java:208)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:258)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10252$tuple_action_fn__10254.invoke(executor.clj:735)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:466)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$fn__10252$fn__10265$fn__10320.invoke(executor.clj:855)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1601) Rename metaalert "alert" nested field to "metron_alert" to avoid collision
[ https://issues.apache.org/jira/browse/METRON-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1601: Fix Version/s: 0.6.0 > Rename metaalert "alert" nested field to "metron_alert" to avoid collision > -- > > Key: METRON-1601 > URL: https://issues.apache.org/jira/browse/METRON-1601 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently we add a nested field called "alert" to our ES templates to enable > meta alert querying. This field name is too common and should be something > like "metron_alert" instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1607) update public web site to point at 0.5.0 new release
[ https://issues.apache.org/jira/browse/METRON-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1607: Fix Version/s: 0.5.0 > update public web site to point at 0.5.0 new release > > > Key: METRON-1607 > URL: https://issues.apache.org/jira/browse/METRON-1607 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.5.0 > > > Update the site for the 0.5.0 release per the process at: > https://cwiki.apache.org/confluence/display/METRON/Release+Process -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1601) Rename metaalert "alert" nested field to "metron_alert" to avoid collision
[ https://issues.apache.org/jira/browse/METRON-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1601: --- Assignee: Casey Stella > Rename metaalert "alert" nested field to "metron_alert" to avoid collision > -- > > Key: METRON-1601 > URL: https://issues.apache.org/jira/browse/METRON-1601 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently we add a nested field called "alert" to our ES templates to enable > meta alert querying. This field name is too common and should be something > like "metron_alert" instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1608) Add configuration for threat.triage.field name
[ https://issues.apache.org/jira/browse/METRON-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1608: --- Assignee: Ryan Merriman > Add configuration for threat.triage.field name > -- > > Key: METRON-1608 > URL: https://issues.apache.org/jira/browse/METRON-1608 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Currently there is an option for replacing '.'s with ':'s in Elasticsearch > field names. This is the default behavior. However our current version of > Elasticsearch (5.6.2) now allows '.'s so it's possible for users to use '.'s > instead. In the DAO implementation (metaalerts specifically), the > threat.triage.field is hardcoded with ':'s and will not work properly if a > user switches to using '.'s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1585) SolrRetrieveLatestDao does not use the collection lookup
[ https://issues.apache.org/jira/browse/METRON-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1585: Fix Version/s: 0.6.0 > SolrRetrieveLatestDao does not use the collection lookup > > > Key: METRON-1585 > URL: https://issues.apache.org/jira/browse/METRON-1585 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > `getLatest` interface has the second arg as "sensorType", but the Solr DAO > makes the assumption that it's "collection" and renames the arg and uses it > without retrieving the actual collection. > https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrRetrieveLatestDao.java#L47 > `getAllLatest` at > https://github.com/apache/metron/blob/feature/METRON-1416-upgrade-solr/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrRetrieveLatestDao.java#L47 > This can affect other DAOs that defer to this DAO (e.g. update and metaalert) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1612) Fix website download links
[ https://issues.apache.org/jira/browse/METRON-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1612: Fix Version/s: 0.6.0 > Fix website download links > -- > > Key: METRON-1612 > URL: https://issues.apache.org/jira/browse/METRON-1612 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Couple issues to be fixed: > * Correct the https link for the keys (per > [https://www.apache.org/dev/release-distribution#download-links]) > * Remove MD5 > * Rename SHA to SHA1 > * Add SHA256 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1608) Add configuration for threat.triage.field name
[ https://issues.apache.org/jira/browse/METRON-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1608: Fix Version/s: 0.6.0 > Add configuration for threat.triage.field name > -- > > Key: METRON-1608 > URL: https://issues.apache.org/jira/browse/METRON-1608 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Currently there is an option for replacing '.'s with ':'s in Elasticsearch > field names. This is the default behavior. However our current version of > Elasticsearch (5.6.2) now allows '.'s so it's possible for users to use '.'s > instead. In the DAO implementation (metaalerts specifically), the > threat.triage.field is hardcoded with ':'s and will not work properly if a > user switches to using '.'s. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1611) Increment master version number to 0.5.1 for on-going development
[ https://issues.apache.org/jira/browse/METRON-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1611: Fix Version/s: 0.6.0 > Increment master version number to 0.5.1 for on-going development > - > > Key: METRON-1611 > URL: https://issues.apache.org/jira/browse/METRON-1611 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Now that 0.5.0 has been released, increment the working version -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1617) Make threat triage scores work for fields with colons as well as dots in the UI
[ https://issues.apache.org/jira/browse/METRON-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1617: --- Assignee: Casey Stella > Make threat triage scores work for fields with colons as well as dots in the > UI > --- > > Key: METRON-1617 > URL: https://issues.apache.org/jira/browse/METRON-1617 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > This is the UI portion of METRON-1608. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1617) Make threat triage scores work for fields with colons as well as dots in the UI
[ https://issues.apache.org/jira/browse/METRON-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1617: Fix Version/s: 0.6.0 > Make threat triage scores work for fields with colons as well as dots in the > UI > --- > > Key: METRON-1617 > URL: https://issues.apache.org/jira/browse/METRON-1617 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > This is the UI portion of METRON-1608. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1587) Make collection utility work for HDP search
[ https://issues.apache.org/jira/browse/METRON-1587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1587: Fix Version/s: 0.6.0 > Make collection utility work for HDP search > --- > > Key: METRON-1587 > URL: https://issues.apache.org/jira/browse/METRON-1587 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Collection scripts need to be improved so that they use the Solr REST api > instead for collection management. Kerberos support should also be included. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1616) Changing alert status fails if no metaalerts have been created yet
[ https://issues.apache.org/jira/browse/METRON-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1616: Fix Version/s: 0.6.0 > Changing alert status fails if no metaalerts have been created yet > -- > > Key: METRON-1616 > URL: https://issues.apache.org/jira/browse/METRON-1616 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Patching an alert will fail with a IndexNotFoundException if a metaalert > hasn't been created yet and the metaalert index doesn't exist. This is > because the ElasticsearchMetaAlertDao tries to lookup and update any > metaalerts that contain the alert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1616) Changing alert status fails if no metaalerts have been created yet
[ https://issues.apache.org/jira/browse/METRON-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1616: --- Assignee: Ryan Merriman > Changing alert status fails if no metaalerts have been created yet > -- > > Key: METRON-1616 > URL: https://issues.apache.org/jira/browse/METRON-1616 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > Patching an alert will fail with a IndexNotFoundException if a metaalert > hasn't been created yet and the metaalert index doesn't exist. This is > because the ElasticsearchMetaAlertDao tries to lookup and update any > metaalerts that contain the alert. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1626) Alerts UI: An empty result is returned when searching for a single alert contained in a metaalert
[ https://issues.apache.org/jira/browse/METRON-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1626: --- Assignee: Shane Ardell > Alerts UI: An empty result is returned when searching for a single alert > contained in a metaalert > - > > Key: METRON-1626 > URL: https://issues.apache.org/jira/browse/METRON-1626 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-19 at 3.18.43 PM.png, Screen Shot > 2018-06-19 at 3.19.09 PM.png > > > When a user tries to searc for a single alert contained within a metaalert, > an empty result set is returned from the API. Upon further inspection, it > looks like this is because the query string is not updated to use > metron_alert in place of alert. [This change happend in > METRON-1601|https://github.com/apache/metron/pull/1049]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1626) Alerts UI: An empty result is returned when searching for a single alert contained in a metaalert
[ https://issues.apache.org/jira/browse/METRON-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1626: Fix Version/s: 0.6.0 > Alerts UI: An empty result is returned when searching for a single alert > contained in a metaalert > - > > Key: METRON-1626 > URL: https://issues.apache.org/jira/browse/METRON-1626 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-19 at 3.18.43 PM.png, Screen Shot > 2018-06-19 at 3.19.09 PM.png > > > When a user tries to searc for a single alert contained within a metaalert, > an empty result set is returned from the API. Upon further inspection, it > looks like this is because the query string is not updated to use > metron_alert in place of alert. [This change happend in > METRON-1601|https://github.com/apache/metron/pull/1049]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1625) Merge master into Solr feature branch
[ https://issues.apache.org/jira/browse/METRON-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1625: Fix Version/s: 0.6.0 > Merge master into Solr feature branch > - > > Key: METRON-1625 > URL: https://issues.apache.org/jira/browse/METRON-1625 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > A PR was recently merged to master > ([https://github.com/apache/metron/pull/1061)] with changes to ES DAO > classes. Due to the significant refactor of the DAO layer in this feature > branch it was not a clean merge and a review is appropriate. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1627) Alerts UI: Metaalert details missing in details panel when trying to add alert to existing metaalert
[ https://issues.apache.org/jira/browse/METRON-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1627: --- Assignee: Shane Ardell > Alerts UI: Metaalert details missing in details panel when trying to add > alert to existing metaalert > > > Key: METRON-1627 > URL: https://issues.apache.org/jira/browse/METRON-1627 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-19 at 6.56.27 PM.png > > > When a user tries to add an alert to an already existing metaalert, the > details panel should show metaalerts to select from with details. Currently, > those details do not show. To reproduce, click on the checkbox on the > right-hand side to select an alert, click on the 'Actions' button dropdown > and select 'Add to Alert'. The details panel that opens should have details > for all the metaalerts, but the info is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1627) Alerts UI: Metaalert details missing in details panel when trying to add alert to existing metaalert
[ https://issues.apache.org/jira/browse/METRON-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1627: Fix Version/s: 0.6.0 > Alerts UI: Metaalert details missing in details panel when trying to add > alert to existing metaalert > > > Key: METRON-1627 > URL: https://issues.apache.org/jira/browse/METRON-1627 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-19 at 6.56.27 PM.png > > > When a user tries to add an alert to an already existing metaalert, the > details panel should show metaalerts to select from with details. Currently, > those details do not show. To reproduce, click on the checkbox on the > right-hand side to select an alert, click on the 'Actions' button dropdown > and select 'Add to Alert'. The details panel that opens should have details > for all the metaalerts, but the info is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1630) Add threat.triage.score.field to READMEs
[ https://issues.apache.org/jira/browse/METRON-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1630: Fix Version/s: 0.6.0 > Add threat.triage.score.field to READMEs > > > Key: METRON-1630 > URL: https://issues.apache.org/jira/browse/METRON-1630 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > This global config setting should be documented in the READMEs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1629) Update Solr documentation
[ https://issues.apache.org/jira/browse/METRON-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1629: Fix Version/s: 0.6.0 > Update Solr documentation > - > > Key: METRON-1629 > URL: https://issues.apache.org/jira/browse/METRON-1629 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > The documentation in various READMEs is likely out of date and needs to be > reviewed and updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1637) Wrong path to escalate alert REST endpoint
[ https://issues.apache.org/jira/browse/METRON-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1637: Fix Version/s: 0.6.0 > Wrong path to escalate alert REST endpoint > -- > > Key: METRON-1637 > URL: https://issues.apache.org/jira/browse/METRON-1637 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > A regression was introduced when the path to the REST escalation endpoint was > changed. This needs to be updated to match the current path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1630) Add threat.triage.score.field to READMEs
[ https://issues.apache.org/jira/browse/METRON-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1630: --- Assignee: Ryan Merriman > Add threat.triage.score.field to READMEs > > > Key: METRON-1630 > URL: https://issues.apache.org/jira/browse/METRON-1630 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > This global config setting should be documented in the READMEs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1624) Set Profiler and Enrichment batch parameters in Ambari
[
https://issues.apache.org/jira/browse/METRON-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet reassigned METRON-1624:
---
Assignee: Nick Allen
> Set Profiler and Enrichment batch parameters in Ambari
> --
>
> Key: METRON-1624
> URL: https://issues.apache.org/jira/browse/METRON-1624
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> METRON-1594 introduced a mechanism to control the batch size and timeout when
> writing records to Kafka by updating the global properties.
> Both Enrichment and the Profiler write records back into Kafka. The user can
> independently set the batch size and timeout for each of those.
> Currently, the user must set the following global properties using the CLI.
> {code}
> "profiler.writer.batchSize" : 15,
> "profiler.writer.batchTimeout" : 0,
> "enrichment.writer.batchSize" : 15,
> "enrichment.writer.batchTimeout" : 0
> {code}
> The user should be able to adjust these values in Ambari.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1624) Set Profiler and Enrichment batch parameters in Ambari
[
https://issues.apache.org/jira/browse/METRON-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1624:
Fix Version/s: 0.6.0
> Set Profiler and Enrichment batch parameters in Ambari
> --
>
> Key: METRON-1624
> URL: https://issues.apache.org/jira/browse/METRON-1624
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: 0.6.0
>
>
> METRON-1594 introduced a mechanism to control the batch size and timeout when
> writing records to Kafka by updating the global properties.
> Both Enrichment and the Profiler write records back into Kafka. The user can
> independently set the batch size and timeout for each of those.
> Currently, the user must set the following global properties using the CLI.
> {code}
> "profiler.writer.batchSize" : 15,
> "profiler.writer.batchTimeout" : 0,
> "enrichment.writer.batchSize" : 15,
> "enrichment.writer.batchTimeout" : 0
> {code}
> The user should be able to adjust these values in Ambari.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1637) Wrong path to escalate alert REST endpoint
[ https://issues.apache.org/jira/browse/METRON-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1637: --- Assignee: Ryan Merriman > Wrong path to escalate alert REST endpoint > -- > > Key: METRON-1637 > URL: https://issues.apache.org/jira/browse/METRON-1637 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > A regression was introduced when the path to the REST escalation endpoint was > changed. This needs to be updated to match the current path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs
[ https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1489: --- Assignee: Shane Ardell (was: Daniel Toth) > Retrofit UI tests to run reliably during nightly QE runs > > > Key: METRON-1489 > URL: https://issues.apache.org/jira/browse/METRON-1489 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1634) Alerts UI add comment doesn't immediately show up.
[ https://issues.apache.org/jira/browse/METRON-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1634: Fix Version/s: 0.6.0 > Alerts UI add comment doesn't immediately show up. > -- > > Key: METRON-1634 > URL: https://issues.apache.org/jira/browse/METRON-1634 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > For ES (and potentially Solr), when the Alerts UI adds a comment to an alert, > it calls the update, then immediately calls a findOne to retrieve it. This > comment might not immediately be available, so it doesn't show the new > comment. > Instead of running a findOne, assuming the update responds appropriately, we > should probably just add it directly in the UI. > Also applies to removing a comment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1489) Retrofit UI tests to run reliably during nightly QE runs
[ https://issues.apache.org/jira/browse/METRON-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1489: Fix Version/s: 0.6.0 > Retrofit UI tests to run reliably during nightly QE runs > > > Key: METRON-1489 > URL: https://issues.apache.org/jira/browse/METRON-1489 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1555) Update REST to run YARN and MR jobs
[ https://issues.apache.org/jira/browse/METRON-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1555: Fix Version/s: 0.6.0 > Update REST to run YARN and MR jobs > --- > > Key: METRON-1555 > URL: https://issues.apache.org/jira/browse/METRON-1555 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > This task involves enabling REST to submit YARN or MR jobs. We will likely > need to: > * update Maven dependencies to include YARN and MR libraries in the > classpath and resolve any version conflicts > * update REST start script to include properties required for YARN > * update the MPack for any additional setup work (create user HDFS directory > for example) and properties needed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1634) Alerts UI add comment doesn't immediately show up.
[ https://issues.apache.org/jira/browse/METRON-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1634: --- Assignee: Ryan Merriman > Alerts UI add comment doesn't immediately show up. > -- > > Key: METRON-1634 > URL: https://issues.apache.org/jira/browse/METRON-1634 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > For ES (and potentially Solr), when the Alerts UI adds a comment to an alert, > it calls the update, then immediately calls a findOne to retrieve it. This > comment might not immediately be available, so it doesn't show the new > comment. > Instead of running a findOne, assuming the update responds appropriately, we > should probably just add it directly in the UI. > Also applies to removing a comment -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1645) All the Metron Service stop/start fail after Kerberizing the cluster due to missing config parameters
[
https://issues.apache.org/jira/browse/METRON-1645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1645:
Fix Version/s: 0.6.0
> All the Metron Service stop/start fail after Kerberizing the cluster due to
> missing config parameters
> -
>
> Key: METRON-1645
> URL: https://issues.apache.org/jira/browse/METRON-1645
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
> Fix For: 0.6.0
>
>
> The installation of HCP 1.5.1.0-16 successful and all the services came up,
> When I kerberize the HCP and restart all services in the cluster, all the
> Metron services including the metron client fails with below error:
> {code:java}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.5.1.1.5.1.0/package/scripts/parser_master.py",
> line 97, in
> ParserMaster().execute()
> File
> "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py",
> line 375, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.5.1.1.5.1.0/package/scripts/parser_master.py",
> line 61, in start
> from params import params
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.5.1.1.5.1.0/package/scripts/params/params.py",
> line 27, in
> from params_linux import *
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.5.1.1.5.1.0/package/scripts/params/params_linux.py",
> line 262, in
> solr_principal_name = solr_principal_name.replace('_HOST',
> hostname_lowercase)
> File
> "/usr/lib/ambari-agent/lib/resource_management/libraries/script/config_dictionary.py",
> line 73, in __getattr__
> raise Fail("Configuration parameter '" + self.name + "' was not found in
> configurations dictionary!")
> resource_management.core.exceptions.Fail: Configuration parameter
> 'solr-config-env' was not found in configurations dictionary!
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1619) Stellar empty collections should be considered false in boolean expressions
[ https://issues.apache.org/jira/browse/METRON-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1619: Fix Version/s: 0.6.0 > Stellar empty collections should be considered false in boolean expressions > --- > > Key: METRON-1619 > URL: https://issues.apache.org/jira/browse/METRON-1619 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Similar to METRON-1618, we should follow the example of python and javascript > and make empty collections evaluate to false in boolean expressions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1619) Stellar empty collections should be considered false in boolean expressions
[ https://issues.apache.org/jira/browse/METRON-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1619: --- Assignee: Casey Stella > Stellar empty collections should be considered false in boolean expressions > --- > > Key: METRON-1619 > URL: https://issues.apache.org/jira/browse/METRON-1619 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Similar to METRON-1618, we should follow the example of python and javascript > and make empty collections evaluate to false in boolean expressions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1647) Change Runner Logging level error to info
[ https://issues.apache.org/jira/browse/METRON-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1647: Fix Version/s: (was: 0.5.0) 0.6.0 > Change Runner Logging level error to info > - > > Key: METRON-1647 > URL: https://issues.apache.org/jira/browse/METRON-1647 > Project: Metron > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: pravin rahangdale >Priority: Minor > Fix For: 0.6.0 > > > Change log level from error to info on line number "180" > org.apache.metron.maas.service.runner.Runner > It is quite misleading while debugging. I think it should be on info level > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1631) Alerts UI: Dash score does not show if only filtering by one group
[ https://issues.apache.org/jira/browse/METRON-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1631: Fix Version/s: 0.6.0 > Alerts UI: Dash score does not show if only filtering by one group > -- > > Key: METRON-1631 > URL: https://issues.apache.org/jira/browse/METRON-1631 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-21 at 2.44.28 PM.png, Screen Shot > 2018-06-21 at 2.44.34 PM.png > > > When a user clicks on a single Group By button, the dash score doesn't > display. If a user then clicks on another Group By button, the dash score > will then display. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1631) Alerts UI: Dash score does not show if only filtering by one group
[ https://issues.apache.org/jira/browse/METRON-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1631: --- Assignee: Shane Ardell > Alerts UI: Dash score does not show if only filtering by one group > -- > > Key: METRON-1631 > URL: https://issues.apache.org/jira/browse/METRON-1631 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.6.0 > > Attachments: Screen Shot 2018-06-21 at 2.44.28 PM.png, Screen Shot > 2018-06-21 at 2.44.34 PM.png > > > When a user clicks on a single Group By button, the dash score doesn't > display. If a user then clicks on another Group By button, the dash score > will then display. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1636) Fix broken unit test setup in metron-alerts
[ https://issues.apache.org/jira/browse/METRON-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1636: --- Assignee: Tibor Meller > Fix broken unit test setup in metron-alerts > --- > > Key: METRON-1636 > URL: https://issues.apache.org/jira/browse/METRON-1636 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > Fix For: 0.6.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > Unit tests are not runs in metron-alerts module because of some configuration > issue. > Please investigate and fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1636) Fix broken unit test setup in metron-alerts
[ https://issues.apache.org/jira/browse/METRON-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1636: Fix Version/s: 0.6.0 > Fix broken unit test setup in metron-alerts > --- > > Key: METRON-1636 > URL: https://issues.apache.org/jira/browse/METRON-1636 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Tibor Meller >Priority: Major > Fix For: 0.6.0 > > Original Estimate: 48h > Remaining Estimate: 48h > > Unit tests are not runs in metron-alerts module because of some configuration > issue. > Please investigate and fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1642) KafkaWriter should be able choose the topic from a field in addition to topology construction time
[ https://issues.apache.org/jira/browse/METRON-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1642: Fix Version/s: 0.6.0 > KafkaWriter should be able choose the topic from a field in addition to > topology construction time > -- > > Key: METRON-1642 > URL: https://issues.apache.org/jira/browse/METRON-1642 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently, we choose the kafka topic via the kafka.topic field. It would be > useful to allow people to specify the topic via a field. This would enable > multi-stage (or chain) parsing, among other use-cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1642) KafkaWriter should be able choose the topic from a field in addition to topology construction time
[ https://issues.apache.org/jira/browse/METRON-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1642: --- Assignee: Casey Stella > KafkaWriter should be able choose the topic from a field in addition to > topology construction time > -- > > Key: METRON-1642 > URL: https://issues.apache.org/jira/browse/METRON-1642 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently, we choose the kafka topic via the kafka.topic field. It would be > useful to allow people to specify the topic via a field. This would enable > multi-stage (or chain) parsing, among other use-cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1635) Alerts UI status update doesn't immediately show up
[ https://issues.apache.org/jira/browse/METRON-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1635: Fix Version/s: 0.6.0 > Alerts UI status update doesn't immediately show up > --- > > Key: METRON-1635 > URL: https://issues.apache.org/jira/browse/METRON-1635 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.6.0 > > > For ES (and potentially Solr), when the Alerts UI changes the status of an > alert, it calls the update, then immediately calls a findOne to retrieve it. > This comment might not immediately be available, so it doesn't show the new > comment. > Instead of running a findOne, we should probably just add it directly in the > UI when the patch call returns a 200. > Applies to all status values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1643) Create a REGEX_ROUTING field transformation
[ https://issues.apache.org/jira/browse/METRON-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1643: Fix Version/s: 0.6.0 > Create a REGEX_ROUTING field transformation > --- > > Key: METRON-1643 > URL: https://issues.apache.org/jira/browse/METRON-1643 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > A relatively common pattern is to choose the value of a field based on if > another field matches a set of regexes. This can be done via stellar with > match, but if the list of possible regexes are large, that can be unwieldy. > To ease that burden, we should have a REGEX_ROUTING field transformation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1643) Create a REGEX_ROUTING field transformation
[ https://issues.apache.org/jira/browse/METRON-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1643: --- Assignee: Casey Stella > Create a REGEX_ROUTING field transformation > --- > > Key: METRON-1643 > URL: https://issues.apache.org/jira/browse/METRON-1643 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > A relatively common pattern is to choose the value of a field based on if > another field matches a set of regexes. This can be done via stellar with > match, but if the list of possible regexes are large, that can be unwieldy. > To ease that burden, we should have a REGEX_ROUTING field transformation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1635) Alerts UI status update doesn't immediately show up
[ https://issues.apache.org/jira/browse/METRON-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1635: --- Assignee: Ryan Merriman > Alerts UI status update doesn't immediately show up > --- > > Key: METRON-1635 > URL: https://issues.apache.org/jira/browse/METRON-1635 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > For ES (and potentially Solr), when the Alerts UI changes the status of an > alert, it calls the update, then immediately calls a findOne to retrieve it. > This comment might not immediately be available, so it doesn't show the new > comment. > Instead of running a findOne, we should probably just add it directly in the > UI when the patch call returns a 200. > Applies to all status values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1644) Support parser chaining
[ https://issues.apache.org/jira/browse/METRON-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1644: --- Assignee: Casey Stella > Support parser chaining > --- > > Key: METRON-1644 > URL: https://issues.apache.org/jira/browse/METRON-1644 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently we have only one layer of parsing prior to enrichment, but often > real data is more complex. For instance, often data is wrapped in an > envelope (e.g. syslog data which contains a field which needs to be parsed). > This Jira allows us to support a DAG of parsers prior to enrichment by > allowing us to provide a strategy for interpreting what is data and what is > metadata in the parser bolt. This enables upstream parsers to pass in a JSON > Blob which contains the metadata and have the parser bolt choose which field > is the data to be parsed, the remaining fields would be considered metadata > (and merged into the resulting data or not depending on our existing rules > for handling metadata). > > To illustrate this better, I've provided a use-case with an example. Note, > this PR depends on METRON-1643 and METRON-1642, so those should be reviewed > prior to this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1655) Make REGEXP_MATCH take multiple regexs in the 2nd arg
[ https://issues.apache.org/jira/browse/METRON-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1655: Fix Version/s: 0.6.0 > Make REGEXP_MATCH take multiple regexs in the 2nd arg > - > > Key: METRON-1655 > URL: https://issues.apache.org/jira/browse/METRON-1655 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Otto Fowler >Priority: Major > Fix For: 0.6.0 > > > Right now, REGEXP_MATCH only takes 1 regex in the 2nd arg, if we took more > than one and treated them as an or, we would increase usability. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1644) Support parser chaining
[ https://issues.apache.org/jira/browse/METRON-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1644: Fix Version/s: 0.6.0 > Support parser chaining > --- > > Key: METRON-1644 > URL: https://issues.apache.org/jira/browse/METRON-1644 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.6.0 > > > Currently we have only one layer of parsing prior to enrichment, but often > real data is more complex. For instance, often data is wrapped in an > envelope (e.g. syslog data which contains a field which needs to be parsed). > This Jira allows us to support a DAG of parsers prior to enrichment by > allowing us to provide a strategy for interpreting what is data and what is > metadata in the parser bolt. This enables upstream parsers to pass in a JSON > Blob which contains the metadata and have the parser bolt choose which field > is the data to be parsed, the remaining fields would be considered metadata > (and merged into the resulting data or not depending on our existing rules > for handling metadata). > > To illustrate this better, I've provided a use-case with an example. Note, > this PR depends on METRON-1643 and METRON-1642, so those should be reviewed > prior to this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1641) Enable Pcap jobs to be submitted asynchronously
[ https://issues.apache.org/jira/browse/METRON-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1641: Fix Version/s: 0.6.0 > Enable Pcap jobs to be submitted asynchronously > --- > > Key: METRON-1641 > URL: https://issues.apache.org/jira/browse/METRON-1641 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1660) On Solr, sorting by threat score fails
[
https://issues.apache.org/jira/browse/METRON-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1660:
Fix Version/s: 0.6.0
> On Solr, sorting by threat score fails
> --
>
> Key: METRON-1660
> URL: https://issues.apache.org/jira/browse/METRON-1660
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Major
> Fix For: 0.6.0
>
>
> * HCP 1.5.1.0 bld 16
> * 12-node Ubuntu 14.04 cluster on Openstack
> * Solr indexing server + Kerberized
> *Steps to Reproduce*
> 1. Ingest snort events
> 2. Create a metaalert using snort alerts
> 3. Sort by 'Score'
> Following error is seen in REST API:
> {code}
> {"responseCode":500,"message":"No live SolrServers available to handle this
> request:[http://nat-u14-jaks-metron-1.openstacklocal:8983/solr/metaalert_shard1_replica1,
>
> http://nat-u14-jaks-metron-1.openstacklocal:8983/solr/yaf_shard1_replica1]","fullMessage":"HttpSolrClient.RemoteSolrException:
> Error from server at
> http://nat-u14-jaks-metron-1.openstacklocal:8983/solr/metaalert_shard1_replica1:
> java.lang.Float cannot be cast to java.lang.String"}
> {code}
> Following exceptions are seen in solr.log
> {code}
> 2018-07-09 09:41:25,817 [qtp705265961-18] ERROR [c:metaalert s:shard1
> r:core_node1 x:metaalert_shard1_replica1]
> org.apache.solr.common.SolrException (SolrException.java:159) -
> null:java.lang.ClassCastException: java.lang.Float cannot be cast to
> java.lang.String
> at
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.java:1100)
> at org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:105)
> at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1251)
> at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:1075)
> at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:775)
> at
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:754)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:429)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
>
[jira] [Updated] (METRON-1658) Upgrade bro to 2.5.4
[ https://issues.apache.org/jira/browse/METRON-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1658: Fix Version/s: 0.6.0 > Upgrade bro to 2.5.4 > > > Key: METRON-1658 > URL: https://issues.apache.org/jira/browse/METRON-1658 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Fix For: 0.6.0 > > > We're currently running Bro 2.5.2, and two releases have come out, both > fixing some security issues, and 2.5.4 also contains a couple of bugfixes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1620) Fixes for forensic clustering use case example
[ https://issues.apache.org/jira/browse/METRON-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1620: Fix Version/s: 0.6.0 > Fixes for forensic clustering use case example > -- > > Key: METRON-1620 > URL: https://issues.apache.org/jira/browse/METRON-1620 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.6.0 > > > ES mapping needed some adjustments. Change to dynamic template mapping so it > will work for non-existent indexes yet to be created. Make work with ES 5.6.x > data types. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1673) Fix Javadoc errors
[ https://issues.apache.org/jira/browse/METRON-1673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1673: Fix Version/s: 0.6.0 > Fix Javadoc errors > -- > > Key: METRON-1673 > URL: https://issues.apache.org/jira/browse/METRON-1673 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Our Javadocs have errors. They should be fixed. Ideally, building Javadoc > should be part of the build. However, we've held off for time restraints in > the build. [https://github.com/apache/metron/pull/1099] proposes a build > matrix for our build, and we could slot it into one of the sub builds without > these time constraints. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1659) The platform-info.sh should check for the vagrant hostmanager plugin
[ https://issues.apache.org/jira/browse/METRON-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1659: Fix Version/s: (was: Next + 1) 0.6.0 > The platform-info.sh should check for the vagrant hostmanager plugin > > > Key: METRON-1659 > URL: https://issues.apache.org/jira/browse/METRON-1659 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Trivial > Fix For: 0.6.0 > > > The platform-info.sh should check for vagrant the hostmanager plugin -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1657) Parser aggregation in storm
[ https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1657: Fix Version/s: 0.6.0 > Parser aggregation in storm > --- > > Key: METRON-1657 > URL: https://issues.apache.org/jira/browse/METRON-1657 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > Currently our parsing solution requires one storm topology per sensor. It has > been complained that this may be wasteful of resources and that, rather than > one storm topology per sensor, it would be advantageous to have multiple > sensors in the same topology. The benefit to this is that it would require > fewer storm slots. > The issue with this is that whenever we've aggregated functionality like this > before, we've run into issues appropriately being able to scale storm (e.g. > batch vs random access indexing in the same topology). The main point in > addressing this is to recommend that parsers with similar velocities and > complexity are grouped together. > Particularly for a first cut, leave the configuration mostly as-is, while > allowing for comma separated lists of sensors in start_parser_topology.sh > (e.g. bro,yaf creates a aggregated parser consisting of those two). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1684) Fix Markdown problems in 3rdPartyParser.md
[ https://issues.apache.org/jira/browse/METRON-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1684: Fix Version/s: 0.6.0 > Fix Markdown problems in 3rdPartyParser.md > -- > > Key: METRON-1684 > URL: https://issues.apache.org/jira/browse/METRON-1684 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.6.0 > > > The code blocks aren't properly indented, which causes some screwiness with > mvn clean site in the site-book in particular. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
