[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753000#comment-15753000
 ] 

ASF GitHub Bot commented on METRON-587:
---

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/374#discussion_r92731270
  
--- Diff: 
metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
 ---
@@ -153,9 +168,17 @@ public void test() throws Exception {
   fluxComponent.submitTopology();
 
   kafkaComponent.writeMessages(Constants.ENRICHMENT_TOPIC, 
inputMessages);
-  ProcessorResult>> result = 
runner.process(getProcessor(inputMessages));
+  ProcessorResult>> result = 
runner.process(getProcessor());
   // We expect failures, so we don't care if result returned failure 
or not
   List> docs = result.getResult();
+  // OTTO>>>
--- End diff --

*might* not want the OTTO there, unless you want to be immortalized in 
Metron code @ottobackwards  ;)  Lord knows the `StellarTest` has enough 
mentions of my name, though, so I can't talk.


> Integration tests should use common processor implementations where possible
> 
>
> Key: METRON-587
> URL: https://issues.apache.org/jira/browse/METRON-587
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> We have 7 or so integration tests that implement the Processor interface.  
> Much of the code in these implementations is the same.  It would be better 
> for the system if we had some common implementations of the Processor 
> interface.
> - This would make it easier to write correct tests
> - This would reduce errors from copy/paste test generation
> - This would clean up the codebase and make it easier to understand



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-587) Integration tests should use common processor implementations where possible

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752999#comment-15752999
 ] 

ASF GitHub Bot commented on METRON-587:
---

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/374#discussion_r92731374
  
--- Diff: 
metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
 ---
@@ -153,9 +168,17 @@ public void test() throws Exception {
   fluxComponent.submitTopology();
 
   kafkaComponent.writeMessages(Constants.ENRICHMENT_TOPIC, 
inputMessages);
-  ProcessorResult>> result = 
runner.process(getProcessor(inputMessages));
+  ProcessorResult>> result = 
runner.process(getProcessor());
   // We expect failures, so we don't care if result returned failure 
or not
   List> docs = result.getResult();
+  // OTTO>>>
--- End diff --

I also assume those printlns are debug code and should be removed.


> Integration tests should use common processor implementations where possible
> 
>
> Key: METRON-587
> URL: https://issues.apache.org/jira/browse/METRON-587
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> We have 7 or so integration tests that implement the Processor interface.  
> Much of the code in these implementations is the same.  It would be better 
> for the system if we had some common implementations of the Processor 
> interface.
> - This would make it easier to write correct tests
> - This would reduce errors from copy/paste test generation
> - This would clean up the codebase and make it easier to understand



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-627) Add HyperLogLogPlus implementation to Stellar

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752516#comment-15752516
 ] 

ASF GitHub Bot commented on METRON-627:
---

GitHub user mmiklavc opened a pull request:

https://github.com/apache/incubator-metron/pull/397

METRON-627: Add HyperLogLogPlus implementation to Stellar

This PR addresses https://issues.apache.org/jira/browse/METRON-627

Leverages the HLLP implementation from 
https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/cardinality/HyperLogLogPlus.java

4 new Stellar functions have been added that allow a user to initialize a 
cardinality estimator, add items, merge estimators, and calculate cardinality 
estimates.

### `HLLP_CARDINALITY`
  * Description: Returns HyperLogLogPlus-estimated cardinality for this set
  * Input:
* hyperLogLogPlus - the hllp set
  * Returns: Long value representing the cardinality for this set

### `HLLP_INIT`
  * Description: Initializes the set
  * Input:
* p (required) - the precision value for the normal set
* sp - the precision value for the sparse set. If sp is not specified 
the sparse set will be disabled.
  * Returns: A new HyperLogLogPlus set

### `HLLP_MERGE`
  * Description: Merge hllp sets together
  * Input:
* hllp1 - first hllp set
* hllp2 - second hllp set
* hllpn - additional sets to merge
  * Returns: A new merged HyperLogLogPlus estimator set

### `HLLP_OFFER`
  * Description: Add value to the set
  * Input:
* hyperLogLogPlus - the hllp set
* o - Object to add to the set
  * Returns: The HyperLogLogPlus set with a new object added

**Note:** Added new library to metron-common pom and added 3 new items to 
dependencies_with_url.csv.

**Testing**

Spun up the Stellar REPL in quick-dev. And verified that the function 
composition is working as expected and returning correct cardinality estimates 
for simple sparse set cases. For example:
```
[Stellar]>>> HLLP_CARDINALITY(HLLP_MERGE( 
HLLP_OFFER(HLLP_OFFER(HLLP_INIT(5, 6), "runnings"), "cool"), 
HLLP_OFFER(HLLP_OFFER(HLLP_INIT(5, 6), "bobsled"), "team")))
4
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/incubator-metron hyperloglog

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/397.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #397


commit afce30539f6996a607e85d3fd35aac5fcb5c19aa
Author: Michael Miklavcic 
Date:   2016-12-15T20:55:39Z

METRON-627: Add HyperLogLogPlus implementation to Stellar




> Add HyperLogLogPlus implementation to Stellar
> -
>
> Key: METRON-627
> URL: https://issues.apache.org/jira/browse/METRON-627
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>
> Calculating set cardinality can be a useful tool for a security analyst. For 
> instance, a large volume of non-unique src ip addresses hitting your network 
> may be an indication that you are currently under attack. There have been 
> many advancements in distinct value (DV) estimation over the years. We have 
> seen implementations evolve from K-Minimum-Values (KMV), to LogLog, to 
> HyperLogLog, and now to Google's much-improved HyperLogLogPlu algorithm. The 
> key improvements in this latest manifestation of the algorithm are:
> moves to a 64-bit hash
> handles sparse sets
> is more accurate with small cardinality
> This Jira tracks the effort to add a HyperLogLogPlus implementation to Metron.
> References:
> https://research.neustar.biz/2013/01/24/hyperloglog-googles-take-on-engineering-hll/
> http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-627) Add HyperLogLogPlus implementation to Stellar

2016-12-15 Thread Michael Miklavcic (JIRA) 
Michael Miklavcic created METRON-627:


 Summary: Add HyperLogLogPlus implementation to Stellar
 Key: METRON-627
 URL: https://issues.apache.org/jira/browse/METRON-627
 Project: Metron
  Issue Type: Improvement
Reporter: Michael Miklavcic


Calculating set cardinality can be a useful tool for a security analyst. For 
instance, a large volume of non-unique src ip addresses hitting your network 
may be an indication that you are currently under attack. There have been many 
advancements in distinct value (DV) estimation over the years. We have seen 
implementations evolve from K-Minimum-Values (KMV), to LogLog, to HyperLogLog, 
and now to Google's much-improved HyperLogLogPlu algorithm. The key 
improvements in this latest manifestation of the algorithm are:
moves to a 64-bit hash
handles sparse sets
is more accurate with small cardinality

This Jira tracks the effort to add a HyperLogLogPlus implementation to Metron.

References:
https://research.neustar.biz/2013/01/24/hyperloglog-googles-take-on-engineering-hll/
http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-622) Create a Metron Docker Compose application

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751902#comment-15751902
 ] 

ASF GitHub Bot commented on METRON-622:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/393
  
I will continue to validate this when that next push goes out. 


> Create a Metron Docker Compose application 
> ---
>
> Key: METRON-622
> URL: https://issues.apache.org/jira/browse/METRON-622
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> This is an alternative to the Vagrant quick-dev approach of creating a local 
> Metron environment for development and testing.  It is intended to be leaner 
> and easier to manage.  This compose application will contain separate images 
> for each of the services that Metron depends on as well as the Metron 
> services.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-622) Create a Metron Docker Compose application

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751897#comment-15751897
 ] 

ASF GitHub Bot commented on METRON-622:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/393
  
That is a start @cestella @merrimanr.  Enough to validate the PR.  For the 
intended use case - we should have as much documentation for this as the 
vagrant stuff.  So wiki peer entries etc.  At least I would think.



> Create a Metron Docker Compose application 
> ---
>
> Key: METRON-622
> URL: https://issues.apache.org/jira/browse/METRON-622
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> This is an alternative to the Vagrant quick-dev approach of creating a local 
> Metron environment for development and testing.  It is intended to be leaner 
> and easier to manage.  This compose application will contain separate images 
> for each of the services that Metron depends on as well as the Metron 
> services.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-590) Enable Use of Event Time in Profiler

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751880#comment-15751880
 ] 

ASF GitHub Bot commented on METRON-590:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/395
  
OK - I'll create a jira for that - and tag you guys for comments/editing?


> Enable Use of Event Time in Profiler
> 
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> There are at least two different times that are important to consider when 
> handling the telemetry messages received by Metron.  
> (1) Processing time is the time at which Metron processed the message.  
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event 
> times may remain close and consistent. When processing time differs from 
> event time the data produced by the Profiler may be inaccurate.  There are a 
> few scenarios under which these times might differ greatly which would 
> negatively impact the feature set produced by the Profiler.  
> (1) When the system has experienced an outage, for example, a scheduled 
> maintenance window. When restarted a high volume of messages will need to be 
> processed by the Profiler.  The output of the Profiler will indicate an 
> increase in activity, although no change in activity actually occurred on the 
> target network.  This could happen whether the outage was Metron itself or an 
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler, 
> the Profiler will attribute the activity to the time period in which it was 
> processed.  Obviously the activity should be attributed to the time period in 
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other 
> use cases where event time is preferred.  The Profiler should be enhanced to 
> allow it to produce profiles based on either processing time or event time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-622) Create a Metron Docker Compose application

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751859#comment-15751859
 ] 

ASF GitHub Bot commented on METRON-622:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/393
  
I suspect we should ensure that data is flowing through.  @merrimanr , 
maybe it'd be good to document in the `README.md` a worked example where you:
* spin up a docker cluster
* generate data using the data generation scripts
* look at the indices to see that there are data
* check Kibana for the data

Am I missing anything?


> Create a Metron Docker Compose application 
> ---
>
> Key: METRON-622
> URL: https://issues.apache.org/jira/browse/METRON-622
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> This is an alternative to the Vagrant quick-dev approach of creating a local 
> Metron environment for development and testing.  It is intended to be leaner 
> and easier to manage.  This compose application will contain separate images 
> for each of the services that Metron depends on as well as the Metron 
> services.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-622) Create a Metron Docker Compose application

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751850#comment-15751850
 ] 

ASF GitHub Bot commented on METRON-622:
---

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/393
  
In my experience I've found that if you run these commands:

eval $(docker-machine env metron-machine)
docker ps -a

and all containers are reported as up:

CONTAINER IDIMAGE   COMMAND  CREATED
  STATUS  PORTS
  NAMES
a2f28ac154f3metron_storm"/bin/bash /home/stor"   2 days ago
 Up 2 days   0.0.0.0:8000->8000/tcp,
0.0.0.0:8080-8081->8080-8081/tcp
   metron_storm_1
7f40995c1344metron_hbase"/bin/sh -c ./bin/sta"   2 days ago
 Up 2 days   8080/tcp, 8085/tcp, 9090/tcp, 9095/tcp,
16000/tcp, 16201/tcp, 16301/tcp, 0.0.0.0:16010->16010/tcp   metron_hbase_1
013a5b660f2dmetron_kibana   "/docker-entrypoint.s"   2 days ago
 Up 2 days   0.0.0.0:5601->5601/tcp
 metron_kibana_1
cdb8a20d1f2dmetron_mysql"docker-entrypoint.sh"   2 days ago
 Up 2 days   0.0.0.0:3306->3306/tcp
 metron_mysql_1
32f28acd2421elasticsearch:2.3   "/docker-entrypoint.s"   2 days ago
 Up 2 days   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
 metron_elasticsearch_1
6a8aaa5c3853metron_kafkazk  "/bin/sh -c ./bin/sta"   2 days ago
 Up 2 days   0.0.0.0:2181->2181/tcp, 0.0.0.0:9092->9092/tcp
 metron_kafkazk_1

It's a pretty good indication that everything is working.  If 1 or more
containers have a status of "Exited" that means something went wrong.

There are some UIs that are included as well and I agree we should add
those urls to the README.  Will get that pushed out soon.

Ryan


On Wed, Dec 14, 2016 at 3:53 PM, ottobackwards 
wrote:

> I have followed the instructions, and it looks like it all worked but I am
> not sure what to do next to test this out. The README is missing a usage
> example or something similar to the output of the ansible install that
> lists the services and ports
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> 
,
> or mute the thread
> 

> .
>



> Create a Metron Docker Compose application 
> ---
>
> Key: METRON-622
> URL: https://issues.apache.org/jira/browse/METRON-622
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> This is an alternative to the Vagrant quick-dev approach of creating a local 
> Metron environment for development and testing.  It is intended to be leaner 
> and easier to manage.  This compose application will contain separate images 
> for each of the services that Metron depends on as well as the Metron 
> services.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-590) Enable Use of Event Time in Profiler

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751840#comment-15751840
 ] 

ASF GitHub Bot commented on METRON-590:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/395
  
@ottobackwards Yeah, it's a very good point.  Backwards compatibility is 
going to be an issue going forward if we aren't quite careful.


> Enable Use of Event Time in Profiler
> 
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> There are at least two different times that are important to consider when 
> handling the telemetry messages received by Metron.  
> (1) Processing time is the time at which Metron processed the message.  
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event 
> times may remain close and consistent. When processing time differs from 
> event time the data produced by the Profiler may be inaccurate.  There are a 
> few scenarios under which these times might differ greatly which would 
> negatively impact the feature set produced by the Profiler.  
> (1) When the system has experienced an outage, for example, a scheduled 
> maintenance window. When restarted a high volume of messages will need to be 
> processed by the Profiler.  The output of the Profiler will indicate an 
> increase in activity, although no change in activity actually occurred on the 
> target network.  This could happen whether the outage was Metron itself or an 
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler, 
> the Profiler will attribute the activity to the time period in which it was 
> processed.  Obviously the activity should be attributed to the time period in 
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other 
> use cases where event time is preferred.  The Profiler should be enhanced to 
> allow it to produce profiles based on either processing time or event time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-590) Enable Use of Event Time in Profiler

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751824#comment-15751824
 ] 

ASF GitHub Bot commented on METRON-590:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/395
  
Not a bad idea @ottobackwards.  We are definitely going to have to think 
about change and dealing with deprecation as we move forward.  I think worthy 
of a separate discuss or JIRA, if you're motivated to drive that discussion.


> Enable Use of Event Time in Profiler
> 
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> There are at least two different times that are important to consider when 
> handling the telemetry messages received by Metron.  
> (1) Processing time is the time at which Metron processed the message.  
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event 
> times may remain close and consistent. When processing time differs from 
> event time the data produced by the Profiler may be inaccurate.  There are a 
> few scenarios under which these times might differ greatly which would 
> negatively impact the feature set produced by the Profiler.  
> (1) When the system has experienced an outage, for example, a scheduled 
> maintenance window. When restarted a high volume of messages will need to be 
> processed by the Profiler.  The output of the Profiler will indicate an 
> increase in activity, although no change in activity actually occurred on the 
> target network.  This could happen whether the outage was Metron itself or an 
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler, 
> the Profiler will attribute the activity to the time period in which it was 
> processed.  Obviously the activity should be attributed to the time period in 
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other 
> use cases where event time is preferred.  The Profiler should be enhanced to 
> allow it to produce profiles based on either processing time or event time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-590) Enable Use of Event Time in Profiler

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751787#comment-15751787
 ] 

ASF GitHub Bot commented on METRON-590:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/395
  
Could we have a way of marking parts of STELLAR as preview or beta?  Just 
get them out there, and have a review/feedback period.


> Enable Use of Event Time in Profiler
> 
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> There are at least two different times that are important to consider when 
> handling the telemetry messages received by Metron.  
> (1) Processing time is the time at which Metron processed the message.  
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event 
> times may remain close and consistent. When processing time differs from 
> event time the data produced by the Profiler may be inaccurate.  There are a 
> few scenarios under which these times might differ greatly which would 
> negatively impact the feature set produced by the Profiler.  
> (1) When the system has experienced an outage, for example, a scheduled 
> maintenance window. When restarted a high volume of messages will need to be 
> processed by the Profiler.  The output of the Profiler will indicate an 
> increase in activity, although no change in activity actually occurred on the 
> target network.  This could happen whether the outage was Metron itself or an 
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler, 
> the Profiler will attribute the activity to the time period in which it was 
> processed.  Obviously the activity should be attributed to the time period in 
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other 
> use cases where event time is preferred.  The Profiler should be enhanced to 
> allow it to produce profiles based on either processing time or event time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-590) Enable Use of Event Time in Profiler

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751758#comment-15751758
 ] 

ASF GitHub Bot commented on METRON-590:
---

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/395
  
Yes, your example makes sense.  I was not sure what the easiest way is to 
specify a timestamp.  In my example the offset was saying 2 months before now.  
But I like make that explicit (and more flexible) with `DATE_SUBTRACT`.  I'll 
crank on this and come up with something.


> Enable Use of Event Time in Profiler
> 
>
> Key: METRON-590
> URL: https://issues.apache.org/jira/browse/METRON-590
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> There are at least two different times that are important to consider when 
> handling the telemetry messages received by Metron.  
> (1) Processing time is the time at which Metron processed the message.  
> (2) Event time is the time at which the event actually occurred.
> If Metron is consuming live data and all is well, the processing and event 
> times may remain close and consistent. When processing time differs from 
> event time the data produced by the Profiler may be inaccurate.  There are a 
> few scenarios under which these times might differ greatly which would 
> negatively impact the feature set produced by the Profiler.  
> (1) When the system has experienced an outage, for example, a scheduled 
> maintenance window. When restarted a high volume of messages will need to be 
> processed by the Profiler.  The output of the Profiler will indicate an 
> increase in activity, although no change in activity actually occurred on the 
> target network.  This could happen whether the outage was Metron itself or an 
> upstream system that feeds data to Metron.
> (2) If the user attempts to replay historical telemetry through the Profiler, 
> the Profiler will attribute the activity to the time period in which it was 
> processed.  Obviously the activity should be attributed to the time period in 
> which the raw telemetry events originated in.
> There are some scenarios when processing time might be preferred and other 
> use cases where event time is preferred.  The Profiler should be enhanced to 
> allow it to produce profiles based on either processing time or event time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (METRON-626) Add Higher-Order Map Function to Stellar

2016-12-15 Thread Nick Allen (JIRA) 
Nick Allen created METRON-626:
-

 Summary: Add Higher-Order Map Function to Stellar
 Key: METRON-626
 URL: https://issues.apache.org/jira/browse/METRON-626
 Project: Metron
  Issue Type: Improvement
Reporter: Nick Allen


It would be extremely useful to have a 'map' higher-order function in Stellar. 
A 'flat map' might also be needed.

[https://en.wikipedia.org/wiki/Map_(higher-order_function)]

A few examples that I am thinking of.  Ignore my choice of syntax.  I am just 
trying to express the use cases here.

Fetch all of the profile measurements over the past 2 hours.  There may be 
many.  What is the mean of each of those measurements?  

{code}
stats := PROFILE_GET('p1', 'e1', 2, "HOURS")
stats.MAP( s -> STATS_MEAN(s))
{code}

Get profile measurements taken 1 week ago, 2 weeks ago, and 3 weeks ago, so 
that I can then compare them.

{code}
{ "1 week", "2 weeks", "3 weeks" }.MAP( lookback -> PROFILE_GET('p1','e1', 
lookback)
{code}






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-625) Parser Filters cannot be specified from the sensor config

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751632#comment-15751632
 ] 

ASF GitHub Bot commented on METRON-625:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/396
  
@ottobackwards There shouldn't be, no.


> Parser Filters cannot be specified from the sensor config
> -
>
> Key: METRON-625
> URL: https://issues.apache.org/jira/browse/METRON-625
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Parser filters cannot be specified because of a logic error in the prepare 
> for the ParserBolt.  Specifically, we are loading the filter from the config 
> only if the filter member variable is null.  Unfortunately, this will never 
> be null, because it is initialized to a default implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-625) Parser Filters cannot be specified from the sensor config

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751625#comment-15751625
 ] 

ASF GitHub Bot commented on METRON-625:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/396
  
So it there is nothing with that value out there so there are no 
serialization issues



> Parser Filters cannot be specified from the sensor config
> -
>
> Key: METRON-625
> URL: https://issues.apache.org/jira/browse/METRON-625
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Parser filters cannot be specified because of a logic error in the prepare 
> for the ParserBolt.  Specifically, we are loading the filter from the config 
> only if the filter member variable is null.  Unfortunately, this will never 
> be null, because it is initialized to a default implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-625) Parser Filters cannot be specified from the sensor config

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751616#comment-15751616
 ] 

ASF GitHub Bot commented on METRON-625:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/396
  
Are there upgrade issues with the QUERY change?


> Parser Filters cannot be specified from the sensor config
> -
>
> Key: METRON-625
> URL: https://issues.apache.org/jira/browse/METRON-625
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Parser filters cannot be specified because of a logic error in the prepare 
> for the ParserBolt.  Specifically, we are loading the filter from the config 
> only if the filter member variable is null.  Unfortunately, this will never 
> be null, because it is initialized to a default implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-625) Parser Filters cannot be specified from the sensor config

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751621#comment-15751621
 ] 

ASF GitHub Bot commented on METRON-625:
---

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/396
  
@ottobackwards I don't think so because it never worked (at least not in 
any of our apache releases; I checked yesterday).  Nobody could be depending on 
this functionality unless they fixed it like I have here.


> Parser Filters cannot be specified from the sensor config
> -
>
> Key: METRON-625
> URL: https://issues.apache.org/jira/browse/METRON-625
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Parser filters cannot be specified because of a logic error in the prepare 
> for the ParserBolt.  Specifically, we are loading the filter from the config 
> only if the filter member variable is null.  Unfortunately, this will never 
> be null, because it is initialized to a default implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (METRON-625) Parser Filters cannot be specified from the sensor config

2016-12-15 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/METRON-625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751175#comment-15751175
 ] 

ASF GitHub Bot commented on METRON-625:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/396
  
Where is QUERY documented?  I can't find it.  What I did find is this in 
the Parser readme:

Message Filter - a filter defining which messages can be dropped. This 
feature is only present in the Java paerer adapters

Maybe you can fix the spelling of Java paerer?


> Parser Filters cannot be specified from the sensor config
> -
>
> Key: METRON-625
> URL: https://issues.apache.org/jira/browse/METRON-625
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Parser filters cannot be specified because of a logic error in the prepare 
> for the ParserBolt.  Specifically, we are loading the filter from the config 
> only if the filter member variable is null.  Unfortunately, this will never 
> be null, because it is initialized to a default implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)