[jira] [Commented] (METRON-1233) Remove description of Global configuration from Enrichment doc
[ https://issues.apache.org/jira/browse/METRON-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463390#comment-16463390 ] ASF GitHub Bot commented on METRON-1233: Github user DimDroll commented on the issue: https://github.com/apache/metron/pull/789 @mmiklavc , I still don't see anything helpful specified about enrichment in here: https://metron.apache.org/current-book/metron-platform/metron-common/index.html I guess we can reference enrichment configuration in global configuration, but I still don't see sense of mentioning Global configuration in Enrichment section without anything relevant to enrichment in Global configuration. > Remove description of Global configuration from Enrichment doc > -- > > Key: METRON-1233 > URL: https://issues.apache.org/jira/browse/METRON-1233 > Project: Metron > Issue Type: Improvement >Reporter: Dima Kovalyov >Priority: Minor > > It confuses reader. Description of Global configuration is available ahead of > the enrichment and listed here: > https://metron.apache.org/current-book/metron-platform/metron-common/index.html > Global config is used for validation of the fields. > Enrichment is used for fields enrichment and probably can also be used for > validation of the fields (which is not covered in the doc). > From available doc I can see that they serve two different purposes. > So, I don't see any reason to mention global config in enrichment > description. What's the benefit of knowing about global configuration when > you want deep dive in enrichment? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...
Github user DimDroll commented on the issue: https://github.com/apache/metron/pull/789 @mmiklavc , I still don't see anything helpful specified about enrichment in here: https://metron.apache.org/current-book/metron-platform/metron-common/index.html I guess we can reference enrichment configuration in global configuration, but I still don't see sense of mentioning Global configuration in Enrichment section without anything relevant to enrichment in Global configuration. ---
[jira] [Commented] (METRON-1214) rpm build fails due to npm absence in Dockerfile
[
https://issues.apache.org/jira/browse/METRON-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463365#comment-16463365
]
ASF GitHub Bot commented on METRON-1214:
Github user DimDroll commented on the issue:
https://github.com/apache/metron/pull/775
It's been a while, sorry for ignorance. I am not able to replicate it now.
I appreciate your follow up on this.
It can be rejected and closed.
> rpm build fails due to npm absence in Dockerfile
> -
>
> Key: METRON-1214
> URL: https://issues.apache.org/jira/browse/METRON-1214
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Dima Kovalyov
>Priority: Minor
>
> I've received error in metron/metron-deployment/packaging/docker/rpm-docker
> that npm is not installed, which failed the whole RPM build. To fix it I've
> added npm to line 28 in Dockerfile:
> {code}
> RUN yum -y install asciidoc rpm-build rpm2cpio tar unzip xmlto zip rpmlint
> npm && yum clean all
> {code}
> Requirement of npm was introduced after ui and rest addition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #775: [METRON-1214] rpm build fails due to npm absence i...
Github user DimDroll closed the pull request at: https://github.com/apache/metron/pull/775 ---
[GitHub] metron issue #775: [METRON-1214] rpm build fails due to npm absence in Docke...
Github user DimDroll commented on the issue: https://github.com/apache/metron/pull/775 It's been a while, sorry for ignorance. I am not able to replicate it now. I appreciate your follow up on this. It can be rejected and closed. ---
[jira] [Commented] (METRON-1545) Upgrade Spring and Spring Boot
[ https://issues.apache.org/jira/browse/METRON-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462945#comment-16462945 ] ASF GitHub Bot commented on METRON-1545: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1008 I'm +1 by inspection. Thanks for the contribution! > Upgrade Spring and Spring Boot > -- > > Key: METRON-1545 > URL: https://issues.apache.org/jira/browse/METRON-1545 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > The metron-rest module depends on old versions of Spring and Spring Boot. We > should upgrade these to the latest release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1008: METRON-1545: Upgrade Spring and Spring Boot
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/1008 I'm +1 by inspection. Thanks for the contribution! ---
[jira] [Commented] (METRON-1545) Upgrade Spring and Spring Boot
[
https://issues.apache.org/jira/browse/METRON-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462907#comment-16462907
]
ASF GitHub Bot commented on METRON-1545:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1008#discussion_r185898854
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
---
@@ -87,13 +91,18 @@ public void configureJdbc(AuthenticationManagerBuilder
auth) throws Exception {
List activeProfiles =
Arrays.asList(environment.getActiveProfiles());
if (activeProfiles.contains(MetronRestConstants.DEV_PROFILE) ||
activeProfiles.contains(MetronRestConstants.TEST_PROFILE))
{
-auth.jdbcAuthentication().dataSource(dataSource)
-
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
+ auth.jdbcAuthentication().dataSource(dataSource)
+
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
} else {
auth.jdbcAuthentication().dataSource(dataSource);
}
}
+
+@Bean
+public PasswordEncoder passwordEncoder() {
+return NoOpPasswordEncoder.getInstance();
--- End diff --
I agree we should have that discussion anyways. I will start a thread on
that topic.
> Upgrade Spring and Spring Boot
> --
>
> Key: METRON-1545
> URL: https://issues.apache.org/jira/browse/METRON-1545
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> The metron-rest module depends on old versions of Spring and Spring Boot. We
> should upgrade these to the latest release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1545) Upgrade Spring and Spring Boot
[
https://issues.apache.org/jira/browse/METRON-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462892#comment-16462892
]
ASF GitHub Bot commented on METRON-1545:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1008#discussion_r185894824
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
---
@@ -87,13 +91,18 @@ public void configureJdbc(AuthenticationManagerBuilder
auth) throws Exception {
List activeProfiles =
Arrays.asList(environment.getActiveProfiles());
if (activeProfiles.contains(MetronRestConstants.DEV_PROFILE) ||
activeProfiles.contains(MetronRestConstants.TEST_PROFILE))
{
-auth.jdbcAuthentication().dataSource(dataSource)
-
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
+ auth.jdbcAuthentication().dataSource(dataSource)
+
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
} else {
auth.jdbcAuthentication().dataSource(dataSource);
}
}
+
+@Bean
+public PasswordEncoder passwordEncoder() {
+return NoOpPasswordEncoder.getInstance();
--- End diff --
Assuming it matches what was happening pre-upgrade, I'm fine with it for an
upgrade ticket.
Having a broader discussion outside this PR is a good idea. Would you be
able to kick off a discuss thread regarding current vs. proposed feature state?
It seems like that discussion should be had regardless of this PR.
> Upgrade Spring and Spring Boot
> --
>
> Key: METRON-1545
> URL: https://issues.apache.org/jira/browse/METRON-1545
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> The metron-rest module depends on old versions of Spring and Spring Boot. We
> should upgrade these to the latest release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1008: METRON-1545: Upgrade Spring and Spring Boot
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1008#discussion_r185894824
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
---
@@ -87,13 +91,18 @@ public void configureJdbc(AuthenticationManagerBuilder
auth) throws Exception {
List activeProfiles =
Arrays.asList(environment.getActiveProfiles());
if (activeProfiles.contains(MetronRestConstants.DEV_PROFILE) ||
activeProfiles.contains(MetronRestConstants.TEST_PROFILE))
{
-auth.jdbcAuthentication().dataSource(dataSource)
-
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
+ auth.jdbcAuthentication().dataSource(dataSource)
+
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
} else {
auth.jdbcAuthentication().dataSource(dataSource);
}
}
+
+@Bean
+public PasswordEncoder passwordEncoder() {
+return NoOpPasswordEncoder.getInstance();
--- End diff --
Assuming it matches what was happening pre-upgrade, I'm fine with it for an
upgrade ticket.
Having a broader discussion outside this PR is a good idea. Would you be
able to kick off a discuss thread regarding current vs. proposed feature state?
It seems like that discussion should be had regardless of this PR.
---
[jira] [Commented] (METRON-1545) Upgrade Spring and Spring Boot
[
https://issues.apache.org/jira/browse/METRON-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462844#comment-16462844
]
ASF GitHub Bot commented on METRON-1545:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1008#discussion_r185883201
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
---
@@ -87,13 +91,18 @@ public void configureJdbc(AuthenticationManagerBuilder
auth) throws Exception {
List activeProfiles =
Arrays.asList(environment.getActiveProfiles());
if (activeProfiles.contains(MetronRestConstants.DEV_PROFILE) ||
activeProfiles.contains(MetronRestConstants.TEST_PROFILE))
{
-auth.jdbcAuthentication().dataSource(dataSource)
-
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
+ auth.jdbcAuthentication().dataSource(dataSource)
+
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
} else {
auth.jdbcAuthentication().dataSource(dataSource);
}
}
+
+@Bean
+public PasswordEncoder passwordEncoder() {
+return NoOpPasswordEncoder.getInstance();
--- End diff --
The reason I used that particular encoder is to keep the system working the
same way it does now. Using a different encoder would significantly alter the
way we manage users and passwords. It will also force people to migrate all
passwords to a new encoding when upgrading.
I would like to see a broader discussion on our authentication strategy
outside of this PR. I suspect we will end up moving away from JDBC
authentication anyways and more towards a unified architecture that aligns with
other components in our stack.
> Upgrade Spring and Spring Boot
> --
>
> Key: METRON-1545
> URL: https://issues.apache.org/jira/browse/METRON-1545
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> The metron-rest module depends on old versions of Spring and Spring Boot. We
> should upgrade these to the latest release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1008: METRON-1545: Upgrade Spring and Spring Boot
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/1008#discussion_r185883201
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
---
@@ -87,13 +91,18 @@ public void configureJdbc(AuthenticationManagerBuilder
auth) throws Exception {
List activeProfiles =
Arrays.asList(environment.getActiveProfiles());
if (activeProfiles.contains(MetronRestConstants.DEV_PROFILE) ||
activeProfiles.contains(MetronRestConstants.TEST_PROFILE))
{
-auth.jdbcAuthentication().dataSource(dataSource)
-
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
+ auth.jdbcAuthentication().dataSource(dataSource)
+
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
} else {
auth.jdbcAuthentication().dataSource(dataSource);
}
}
+
+@Bean
+public PasswordEncoder passwordEncoder() {
+return NoOpPasswordEncoder.getInstance();
--- End diff --
The reason I used that particular encoder is to keep the system working the
same way it does now. Using a different encoder would significantly alter the
way we manage users and passwords. It will also force people to migrate all
passwords to a new encoding when upgrading.
I would like to see a broader discussion on our authentication strategy
outside of this PR. I suspect we will end up moving away from JDBC
authentication anyways and more towards a unified architecture that aligns with
other components in our stack.
---
[GitHub] metron pull request #1008: METRON-1545: Upgrade Spring and Spring Boot
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/1008#discussion_r185860632
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
---
@@ -87,13 +91,18 @@ public void configureJdbc(AuthenticationManagerBuilder
auth) throws Exception {
List activeProfiles =
Arrays.asList(environment.getActiveProfiles());
if (activeProfiles.contains(MetronRestConstants.DEV_PROFILE) ||
activeProfiles.contains(MetronRestConstants.TEST_PROFILE))
{
-auth.jdbcAuthentication().dataSource(dataSource)
-
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
-
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
+ auth.jdbcAuthentication().dataSource(dataSource)
+
.withUser("user").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user1").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("user2").password("password").roles(SECURITY_ROLE_USER).and()
+
.withUser("admin").password("password").roles(SECURITY_ROLE_USER,
SECURITY_ROLE_ADMIN);
} else {
auth.jdbcAuthentication().dataSource(dataSource);
}
}
+
+@Bean
+public PasswordEncoder passwordEncoder() {
+return NoOpPasswordEncoder.getInstance();
--- End diff --
Admitting that I have near total ignorance here, why are we using a
NoOpPasswordEncoder instead of something else? What are the implications of
this?
---
[jira] [Commented] (METRON-1545) Upgrade Spring and Spring Boot
[ https://issues.apache.org/jira/browse/METRON-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462701#comment-16462701 ] ASF GitHub Bot commented on METRON-1545: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1008 +1 Thanks for the upgrade and all the fixes. This look solid. I ran this up in CentOS 6 and validated the Alerts UI and ran a Service Check successfully. > Upgrade Spring and Spring Boot > -- > > Key: METRON-1545 > URL: https://issues.apache.org/jira/browse/METRON-1545 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > The metron-rest module depends on old versions of Spring and Spring Boot. We > should upgrade these to the latest release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1008: METRON-1545: Upgrade Spring and Spring Boot
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1008 +1 Thanks for the upgrade and all the fixes. This look solid. I ran this up in CentOS 6 and validated the Alerts UI and ran a Service Check successfully. ---
[jira] [Created] (METRON-1548) Alerts UI: Remove hardcoded source:type and other fields that may vary
Justin Leet created METRON-1548: --- Summary: Alerts UI: Remove hardcoded source:type and other fields that may vary Key: METRON-1548 URL: https://issues.apache.org/jira/browse/METRON-1548 Project: Metron Issue Type: Sub-task Reporter: Justin Leet In Solr, we use source.type instead of source:type (which was originally due to ES limitations, we'd prefer it to be common across both). However, it's hardcoded in the alerts UI to be source:type, so nothing in the UI has a source type and it breaks other things like metaalerts (which rely on source type for being able to pull the data together). Any other configs / queries that would cause similar problems should be updated appropriately. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1547) Solr Comment Fields
Justin Leet created METRON-1547: --- Summary: Solr Comment Fields Key: METRON-1547 URL: https://issues.apache.org/jira/browse/METRON-1547 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Right now the Solr schemas don't have comment fields defined. It'll get caught by the catch all with ignored type not multivalued. ES just handles this correctly out of the box, but we'll need to take care of it in Solr and document the schema restriction. This actually is probably fairly problematic in comparison to ES. Solr doesn't support an easy way of doing a complex structure without doing something a bit weird (like parsing a string representation) or miserable (nested document). This will be incompatible with the current comment update system (just using the patch() functionality). Preferably we need to add a new REST endpoint for comments specifically so that we can handle it without the frontend knowing the backend system. This also involves adjusting the UI to use the new REST API, along with testing for both ES and Solr. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1547) Solr Comment Fields
[ https://issues.apache.org/jira/browse/METRON-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1547: Issue Type: Sub-task (was: Bug) Parent: METRON-1416 > Solr Comment Fields > --- > > Key: METRON-1547 > URL: https://issues.apache.org/jira/browse/METRON-1547 > Project: Metron > Issue Type: Sub-task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > > Right now the Solr schemas don't have comment fields defined. It'll get > caught by the catch all with ignored type not multivalued. > ES just handles this correctly out of the box, but we'll need to take care of > it in Solr and document the schema restriction. > This actually is probably fairly problematic in comparison to ES. Solr > doesn't support an easy way of doing a complex structure without doing > something a bit weird (like parsing a string representation) or miserable > (nested document). > This will be incompatible with the current comment update system (just using > the patch() functionality). Preferably we need to add a new REST endpoint for > comments specifically so that we can handle it without the frontend knowing > the backend system. This also involves adjusting the UI to use the new REST > API, along with testing for both ES and Solr. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1533) Create KAFKA_FIND Stellar Function
[
https://issues.apache.org/jira/browse/METRON-1533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462387#comment-16462387
]
ASF GitHub Bot commented on METRON-1533:
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1000
Thanks for taking it for a test drive. I think all your observations are
explainable, but they all point out usability issues that I think I can improve
on.
1. Offsets
`KAFKA_FIND` 'sticks' on its consumer offset. It operates more like
`KAFKA_GET` than `KAFKA_TAIL`. This is how I described it in the docs.
> Finds messages that satisfy a given filter expression. Subsequent calls
will continue retrieving messages sequentially from the original offset.
When you first run `KAFKA_FIND`, its consumer offset will not be set. It
will pick-up from the end of the topic. When you run it again in the same
session, it will continue filtering from those same offsets, rather than going
to the end of the topic.
The `kafka-console-consumer` tool always seeks to the end when it is run.
In your test its likely that `kafka-console-consumer` and `KAFKA_FIND` are at
completely different offsets as you try to compare the two.
I had actually already been working on a version of this that always seeks
to the end and so behaves more like `KAFKA_TAIL` and `kafka-console-consumer`.
Per the use case I described in the PR, I think 'seek to end' makes more
sense. You make a change on a live stream and want to see the immediate
results. If `KAFKA_TAIL` 'sticks' on an earlier offset, you're not going to
see the most recent messages, which can be confusing for the user.
2. Timeouts
> How long will this command listen until it times out (or is it based on
number of messages read)? ... Is this configurable?
The command will poll for up to 5 seconds, by default. This can be
adjusted by defining a global property `stellar.kafka.max.wait`.
> Sometimes it returned an empty array immediately.
In this case, it probably pulled in messages from the topic, none of those
messages matched your filter, and so returned an empty array to you.
I probably need to look at the timeout logic under these conditions. It
should probably 'try harder' to find matching messages and not return
immediately. I'll take a look at this and see if it can be improved.
> Create KAFKA_FIND Stellar Function
> --
>
> Key: METRON-1533
> URL: https://issues.apache.org/jira/browse/METRON-1533
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Minor
>
> When creating enrichments, I often find that I want to validate that the
> enrichment I just created was successful on the live, incoming stream of
> telemetry. My workflow looks something like this.
> 1. Create and test the enrichment that I want to create.
> {code:java}
> [Stellar]>>> ip_src_addr := "72.34.49.86"
> 72.34.49.86
> [Stellar]>>> geo := GEO_GET(ip_src_addr)
> {country=US, dmaCode=803, city=Los Angeles, postalCode=90014,
> latitude=34.0438, location_point=34.0438,-118.2512, locID=5368361,
> longitude=-118.2512}
> {code}
> 2. That looks good to me. Now let's add that to my Bro telemetry.
> {code:java}
> [Stellar]>>> conf := SHELL_EDIT(conf)
> {
> "enrichment" : {
> "fieldMap": {
> "stellar": {
> "config": [
>"geo := GEO_GET(ip_src_addr)"
> ]
> }
> }
> },
> "threatIntel": {
> }
> }
> [Stellar]>>> CONFIG_PUT("ENRICHMENTS", e, "bro")
> {code}
>
> 3. It looks like that worked, but did that really work?
> At this point, I would run KAFKA_GET as many times as it takes to retrieve a
> Bro message. You would just have to get lucky and hope that the enrichment
> worked and secondly that you would pull down a Bro message (as opposed to a
> different sensor).
>
> I would rather have a function that lets me only pull back the messages that
> I care about. In this case I could either retrieve only Bro messages.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_GET('source.type', m) == 'bro')
> {code}
> Or I could look for messages that contain geolocation data.
> {code:java}
> KAFKA_FIND('indexing', m -> MAP_EXISTS('geo.city', m))
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1000: METRON-1533 Create KAFKA_FIND Stellar Function
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1000 Thanks for taking it for a test drive. I think all your observations are explainable, but they all point out usability issues that I think I can improve on. 1. Offsets `KAFKA_FIND` 'sticks' on its consumer offset. It operates more like `KAFKA_GET` than `KAFKA_TAIL`. This is how I described it in the docs. > Finds messages that satisfy a given filter expression. Subsequent calls will continue retrieving messages sequentially from the original offset. When you first run `KAFKA_FIND`, its consumer offset will not be set. It will pick-up from the end of the topic. When you run it again in the same session, it will continue filtering from those same offsets, rather than going to the end of the topic. The `kafka-console-consumer` tool always seeks to the end when it is run. In your test its likely that `kafka-console-consumer` and `KAFKA_FIND` are at completely different offsets as you try to compare the two. I had actually already been working on a version of this that always seeks to the end and so behaves more like `KAFKA_TAIL` and `kafka-console-consumer`. Per the use case I described in the PR, I think 'seek to end' makes more sense. You make a change on a live stream and want to see the immediate results. If `KAFKA_TAIL` 'sticks' on an earlier offset, you're not going to see the most recent messages, which can be confusing for the user. 2. Timeouts > How long will this command listen until it times out (or is it based on number of messages read)? ... Is this configurable? The command will poll for up to 5 seconds, by default. This can be adjusted by defining a global property `stellar.kafka.max.wait`. > Sometimes it returned an empty array immediately. In this case, it probably pulled in messages from the topic, none of those messages matched your filter, and so returned an empty array to you. I probably need to look at the timeout logic under these conditions. It should probably 'try harder' to find matching messages and not return immediately. I'll take a look at this and see if it can be improved. ---
