Re: Kafka error when upgrading to Metron 0.3.0

2017-01-09 Thread Michael Miklavcic 
Hi Tyler, You shouldn't have to skip the unit tests as we ensure every
commit runs with 100% of tests passing. There are rare instances that this
is not true, but in general all tests should pass. I just pulled down
latest master and re-ran the following successfully.

mvn clean install -PHDP-2.5.0.0

Which branch or commit are you working from, and what command did you run
for the build? Be aware that Maven will ignore misspelled profile names and
instead use the default profile without warning.

On Mon, Jan 9, 2017 at 10:32 AM, Tyler Moore  wrote:

> Seems like it's just the unit tests that are failing for pcap backend. Was
> able to build successfully using new came with -DskipTests option along
> with the profile argument.
>
> Regards,
>
> Tyler Moore
> IT Specialist
> tyler.math...@yahoo.com
> 248-909-2769
>
> > On Jan 9, 2017, at 02:29, Michael Miklavcic 
> wrote:
> >
> > Hi Tyler, I don't recall seeing any failures the last time I ran this. I
> > will take a look.
> >
> > On Jan 8, 2017 9:56 PM, "Tyler Moore"  wrote:
> >
> > Michael,
> >
> > I am receiving error when trying to build with HPD profile, have you had
> > any problems with pcap-backend tests failing when building with hdp
> profile?
> > Errors and log files provided below:
> >
> > Results :
> >
> > Failed tests:
> >  PcapTopologyIntegrationTest.testTimestampInKey:152->testTopology:388->
> assertInOrder:542
> > null
> >  PcapTopologyIntegrationTest.testTimestampInPacket:135->
> testTopology:388->assertInOrder:542
> > null
> >
> >
> >
> > Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> >
> > [INFO] 
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Metron . SUCCESS [
> > 0.639 s]
> > [INFO] metron-analytics ... SUCCESS [
> > 0.029 s]
> > [INFO] metron-maas-common . SUCCESS [
> > 8.771 s]
> > [INFO] metron-platform  SUCCESS [
> > 0.057 s]
> > [INFO] metron-test-utilities .. SUCCESS [
> > 0.976 s]
> > [INFO] metron-integration-test  SUCCESS [
> > 3.888 s]
> > [INFO] metron-maas-service  SUCCESS
> [01:19
> > min]
> > [INFO] metron-common .. SUCCESS [
> > 41.718 s]
> > [INFO] metron-statistics .. SUCCESS [
> > 30.713 s]
> > [INFO] metron-hbase ... SUCCESS [
> > 37.286 s]
> > [INFO] metron-profiler-common . SUCCESS [
> > 7.967 s]
> > [INFO] metron-profiler-client . SUCCESS [
> > 54.088 s]
> > [INFO] metron-profiler  SUCCESS
> [02:28
> > min]
> > [INFO] metron-writer .. SUCCESS [
> > 7.151 s]
> > [INFO] metron-enrichment .. SUCCESS [
> > 57.076 s]
> > [INFO] metron-indexing  SUCCESS [
> > 7.028 s]
> > [INFO] metron-solr  SUCCESS [
> > 36.695 s]
> > [INFO] metron-pcap  SUCCESS [
> > 1.123 s]
> > [INFO] metron-parsers . SUCCESS
> [01:58
> > min]
> > [INFO] metron-pcap-backend  FAILURE [
> > 49.928 s]
> > [INFO] metron-data-management . SKIPPED
> > [INFO] metron-api . SKIPPED
> > [INFO] metron-management .. SKIPPED
> > [INFO] elasticsearch-shaded ... SKIPPED
> > [INFO] metron-elasticsearch ... SKIPPED
> > [INFO] metron-deployment .. SKIPPED
> > [INFO] Metron Ambari Management Pack .. SKIPPED
> > [INFO] 
> > 
> > [INFO] BUILD FAILURE
> > [INFO] 
> > 
> > [INFO] Total time: 11:31 min
> > [INFO] Finished at: 2017-01-08T22:23:59-05:00
> > [INFO] Final Memory: 287M/6202M
> > [INFO] 
> > 
> > [ERROR] Failed to execute goal org.apache.maven.plugins:
> > maven-surefire-plugin:2.18:test (integration-tests) on project
> > metron-pcap-backend: There are test failures.
> >
> >
> > Regards,
> >
> > Tyler Moore
> > Software Engineer
> > Phone: 248-909-2769 <(248)%20909-2769>
> > Email: moore.ty...@goflyball.com
> >
> >
> > On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> >> Hey Tyler,
> >>
> >> Build Metron with the HDP profile to get the proper deps for this
> >> -PHDP-2.5.0.0
>

[GitHub] incubator-metron pull request #414: Metron-532 Define Profile Period When Ca...

2017-01-09 Thread mattf-horton 
GitHub user mattf-horton opened a pull request:

https://github.com/apache/incubator-metron/pull/414

Metron-532 Define Profile Period When Calling PROFILE_GET

This PR builds on top of @nickwallen 's work in METRON-529, which has 
already been committed to master. It adds an optional argument to PROFILE_GET, 
allowing run-time overrides of the profiler client global configuration 
parameters. The primary use case is when historical profiles have been created 
with a different profile configuration than is currently configured, and the 
analyst needing to access them does not want to change the global client 
configuration so as not to disrupt the work of other analysts working with 
current profiles.

This PR is essentially similar to the former proposed PR-345, except that 
the signature of the PROFILE_GET function has been changed to make `groups` be 
a List instead of a "varargs"-style sequence, so the new `config_overrides` Map 
argument can come after it.  This avoids a really hacky over-indulgence in 
polymorphism, and is consistent with discussion in PR-395.  I'm opening a new 
PR, rather than reopening the old one, because I also had to rebase to changes 
in master, and the forced push destroyed the history that PR-345 pointed at.

At suggestion of @cestella , scope of this work item was expanded to 
include checking for config changes at run time, in the PROFILE_GET Stellar 
function.  This is a much needed enhancement.

I also considerably improved the documentation of PROFILE_GET as regards 
the `groups` argument, with input from @nickwallen .  Thanks, Nick!

Testing: This build has passed Travis in my fork, and should shortly be 
reported passed here.  Since the only code changes were to the Stellar 
function, the corresponding junit tests are actually sufficient testing.  
However, I will go ahead and test them on a live single-host test platform, 
taking the opportunity to test the new Profiler RPMs from PR-413!  I'll report 
the results here tomorrow.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattf-horton/incubator-metron METRON-532

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/414.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #414


commit 837b7fa3656ddc6a408d69e6438d5270d7fa6eec
Author: mattf-horton 
Date:   2017-01-04T20:09:22Z

METRON-532 Define Profile Period When Calling PROFILE_GET
Rebase original PR-345 to current master.

commit 55a7c7aa0757bf55cfdd709063b11d7f4770525c
Author: mattf-horton 
Date:   2017-01-06T20:33:38Z

METRON-532 continued. Changed PROFILE_GET signature to make 'groups' 
argument a list; deprecate but still allow the former varargs-style usage. 
Improved documentation with input from Nick.

commit f7ef2c2fd438933677b661a20ec35f8cd98cba3f
Author: mattf-horton 
Date:   2017-01-06T23:40:12Z

METRON-532 fix unit tests for new PROFILE_GET signature, and explicitly 
test the deprecated older form.

commit 40ecab2798cca6ef96b73e543c2161a64a943f2f
Author: mattf-horton 
Date:   2017-01-09T19:31:32Z

Update metron-platform/metron-common/README.md to match new Stellar doc 
standard, and tweak the description a little more.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

2017-01-09 Thread Otto Fowler 
Just a heads up and hopefully not a distraction:

Per https://issues.apache.org/jira/browse/METRON-583, if we ever update to a 
never version of GROK, many of our existing rules are going to break.

I have a pull request out for this issue ( 
https://github.com/thekrakken/java-grok/pull/60 ) with them since November, but 
there has been no comment or movement on it.

On January 9, 2017 at 16:01:00, Michael Miklavcic (michael.miklav...@gmail.com) 
wrote:

This is a question primarily for the mentors.  

*Background*  
metron-common is currently depending on the openSOC github repo for hosting  
kraken artifacts. The original reason for this was that these jars are not  
hosted in Maven Central, and they were not reliably available in the Kraken  
repo. https://issues.apache.org/jira/browse/METRON-650 is tracking work  
around copying these artifacts to the Metron repo.  

Kraken source on openSOC - https://github.com/OpenSOC/kraken  
Krake maven repo on openSOC -  
https://github.com/OpenSOC/kraken/tree/mvn-repo  

*Ask*  
Create a new branch in incubator-metron to host any necessary maven  
artifacts. This branch would simply be incubator-metron/mvn-repo. This is  
similar to how we've hosted the asf-site.  

*Concerns/Questions*  

1. Can we host these jars/artifacts in this manner?  
2. Concerns regarding licensing?  
3. Do we need to also grab and host the source code?

[DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

2017-01-09 Thread Michael Miklavcic 
This is a question primarily for the mentors.

*Background*
metron-common is currently depending on the openSOC github repo for hosting
kraken artifacts. The original reason for this was that these jars are not
hosted in Maven Central, and they were not reliably available in the Kraken
repo. https://issues.apache.org/jira/browse/METRON-650 is tracking work
around copying these artifacts to the Metron repo.

Kraken source on openSOC - https://github.com/OpenSOC/kraken
Krake maven repo on openSOC -
https://github.com/OpenSOC/kraken/tree/mvn-repo

*Ask*
Create a new branch in incubator-metron to host any necessary maven
artifacts. This branch would simply be incubator-metron/mvn-repo. This is
similar to how we've hosted the asf-site.

*Concerns/Questions*

   1. Can we host these jars/artifacts in this manner?
   2. Concerns regarding licensing?
   3. Do we need to also grab and host the source code?

RE: Enrich enrichment

2017-01-09 Thread Carolyn Duby 
Dima - does this solve the enrichment you are trying to add?

Thanks
Carolyn



Sent from my Verizon, Samsung Galaxy smartphone


 Original message 
From: Ryan Merriman 
Date: 1/9/17 11:27 AM (GMT-08:00)
To: dev@metron.incubator.apache.org
Subject: Re: Enrich enrichment

We can already do what Carolyn suggests with Stellar enrichments, assuming
we get a Stellar function created for Geo enrichments (all enrichments
ideally).  Would be configured like this under enrichment section in
enrichment config:

"stellar" : {
  "config" : {
"geo_enriched_field" : "GEO_ENRICHMENT(field)"
"enriched_geo_enriched_field": "ENRICHMENT_GET('some_other_enrichment',
geo_enriched_field, 'enrichments', 'cf')"
  }
}

These statements are executed in order and can be grouped together.  As
Casey pointed out the limitation is that both would run in a single Storm
worker.  Would that be acceptable tradeoff?  Sure it would be ideal if we
could execute each one in separate workers but then we would have to
re-architect our topologies and our system would become much more complex.

Ryan


On Mon, Jan 9, 2017 at 12:59 PM, Carolyn Duby  wrote:

> Adding new topologies adds more processing requirements to the system.  It
> adds more topics (storage) and more producers and consumers to kafka
> (processing).
>
> I think what we need is a dependency of enrichments.  Maybe we need to
> either derive the dependencies using the Stellar (potentially not that
> easy) or allow the enrichment to specify the order or enrichment
> calculations.
> This will allow users to calculate more enrichment in the same topology.
>
> Thanks
> Carolyn
>
>
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>
>  Original message 
> From: Nick Allen 
> Date: 1/9/17 8:49 AM (GMT-08:00)
> To: dev@metron.incubator.apache.org
> Subject: Re: Enrich enrichment
>
> I agree that making it easy for the user to "enrich enrichments", as Dima
> put it, to an arbitrary depth, would be extremely useful for a lot of use
> cases. We've discussed the use case a little in the past in this thread
> [1].
>
> Re-purposing the "threat intel" phase gives us something that is feasible
> today, but only to a "depth" of 2.  We would also need to rename and
> redocument it so that users understand how they can leverage the two
> phases.  This seems like a minimally viable option if we want to head down
> this road.
>
> The other extreme might involve inferring the topology needed based on the
> user's configuration. If the user needs 3 phases, then we build a topology
> that supports 3 phases.  Under the covers instead of using Flux, we would
> use Storm's topology builder Java API to grok the configuration and build
> the topology(ies) that the user needs.
>
> I am not sure if we can infer this from the configuration as it exists
> today or if we would need to redefine the configuration somehow.  Like I
> said this is "extreme", but could give the user more expressive and
> intuitive options.
>
>
>
>
> ---
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-metron-
> dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSk
> Oc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E
>
>
>
> On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella  wrote:
>
> > I think that would be a good feature to add to have arbitrary number of
> > phases, though it might be tricky to code (the way I envisioned it would
> > involve a loop in storm, which is possible[1]), might have unintended
> > consequences to guarantees (e.g. updating enrichments might not be able
> to
> > be applied in realtime) and could be tricky to reason about
> > performance-wise.
> >
> > As it stands, the number of phases is a consequence of the topology
> > itself.  We do not currently have an architecture which would allow an
> > arbitrary number of phases without changing the flux file itself.  What
> you
> > can do, though, in a stellar enrichment is stack enrichments (e.g. depend
> > on previous enrichments) because it's just a list of stellar statements.
> > The consequence, of course, is that these statements get run within the
> > same worker, which is unfortunate, but may be a stopgap workaround.
> >
> > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8
> >
> > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
> > wrote:
> >
> > > Maybe the naming of the phases is misleading?  What if you could set up
> > an
> > > arbitrary number of stages, with defaults?
> > >
> > >
> > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com)
> wrote:
> > >
> > > You could do the geo enrichment normally and do a stellar hbase
> > enrichment
> > > in the threat Intel phase.
> > >
> > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman 
> wrote:
> > >
> > > > Hbase enrichments and geo enrichments are done in parallel so I would
> > > not
> > > > expect this to work. You

Re: Enrich enrichment

2017-01-09 Thread Ryan Merriman 
We can already do what Carolyn suggests with Stellar enrichments, assuming
we get a Stellar function created for Geo enrichments (all enrichments
ideally).  Would be configured like this under enrichment section in
enrichment config:

"stellar" : {
  "config" : {
"geo_enriched_field" : "GEO_ENRICHMENT(field)"
"enriched_geo_enriched_field": "ENRICHMENT_GET('some_other_enrichment',
geo_enriched_field, 'enrichments', 'cf')"
  }
}

These statements are executed in order and can be grouped together.  As
Casey pointed out the limitation is that both would run in a single Storm
worker.  Would that be acceptable tradeoff?  Sure it would be ideal if we
could execute each one in separate workers but then we would have to
re-architect our topologies and our system would become much more complex.

Ryan


On Mon, Jan 9, 2017 at 12:59 PM, Carolyn Duby  wrote:

> Adding new topologies adds more processing requirements to the system.  It
> adds more topics (storage) and more producers and consumers to kafka
> (processing).
>
> I think what we need is a dependency of enrichments.  Maybe we need to
> either derive the dependencies using the Stellar (potentially not that
> easy) or allow the enrichment to specify the order or enrichment
> calculations.
> This will allow users to calculate more enrichment in the same topology.
>
> Thanks
> Carolyn
>
>
>
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>
>  Original message 
> From: Nick Allen 
> Date: 1/9/17 8:49 AM (GMT-08:00)
> To: dev@metron.incubator.apache.org
> Subject: Re: Enrich enrichment
>
> I agree that making it easy for the user to "enrich enrichments", as Dima
> put it, to an arbitrary depth, would be extremely useful for a lot of use
> cases. We've discussed the use case a little in the past in this thread
> [1].
>
> Re-purposing the "threat intel" phase gives us something that is feasible
> today, but only to a "depth" of 2.  We would also need to rename and
> redocument it so that users understand how they can leverage the two
> phases.  This seems like a minimally viable option if we want to head down
> this road.
>
> The other extreme might involve inferring the topology needed based on the
> user's configuration. If the user needs 3 phases, then we build a topology
> that supports 3 phases.  Under the covers instead of using Flux, we would
> use Storm's topology builder Java API to grok the configuration and build
> the topology(ies) that the user needs.
>
> I am not sure if we can infer this from the configuration as it exists
> today or if we would need to redefine the configuration somehow.  Like I
> said this is "extreme", but could give the user more expressive and
> intuitive options.
>
>
>
>
> ---
> [1]
> http://mail-archives.apache.org/mod_mbox/incubator-metron-
> dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSk
> Oc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E
>
>
>
> On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella  wrote:
>
> > I think that would be a good feature to add to have arbitrary number of
> > phases, though it might be tricky to code (the way I envisioned it would
> > involve a loop in storm, which is possible[1]), might have unintended
> > consequences to guarantees (e.g. updating enrichments might not be able
> to
> > be applied in realtime) and could be tricky to reason about
> > performance-wise.
> >
> > As it stands, the number of phases is a consequence of the topology
> > itself.  We do not currently have an architecture which would allow an
> > arbitrary number of phases without changing the flux file itself.  What
> you
> > can do, though, in a stellar enrichment is stack enrichments (e.g. depend
> > on previous enrichments) because it's just a list of stellar statements.
> > The consequence, of course, is that these statements get run within the
> > same worker, which is unfortunate, but may be a stopgap workaround.
> >
> > *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8
> >
> > On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
> > wrote:
> >
> > > Maybe the naming of the phases is misleading?  What if you could set up
> > an
> > > arbitrary number of stages, with defaults?
> > >
> > >
> > > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com)
> wrote:
> > >
> > > You could do the geo enrichment normally and do a stellar hbase
> > enrichment
> > > in the threat Intel phase.
> > >
> > > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman 
> wrote:
> > >
> > > > Hbase enrichments and geo enrichments are done in parallel so I would
> > > not
> > > > expect this to work. You could do the Hbase enrichment as a threat
> > Intel
> > > > enrichment and that should work because enrichments and threat Intel
> > are
> > > > done in series.
> > > >
> > > >
> > > >
> > > > The ideal way would be to chain together Stellar enrichments but I
> > don't
> > > > think there is a geo enrichment

[GitHub] incubator-metron pull request #413: METRON-654 Create RPM Installer for Prof...

2017-01-09 Thread nickwallen 
GitHub user nickwallen opened a pull request:

https://github.com/apache/incubator-metron/pull/413

METRON-654 Create RPM Installer for Profiler

### [METRON-654](https://issues.apache.org/jira/browse/METRON-654)

The Profiler currently must be installed using the Ansible deployment 
scripts. Prior to getting the Profiler integrated with the rest of the Ambari 
installation process, we need to package the Profiler in an RPM.

### Testing

I tested this change by doing the following.

(1) Build the RPMs by running the following command.
```
cd metron-deployment
mvn clean package -Pbuild-rpms
```

(2) Stand-up a host running CentOS.  I used the Bento base that is used for 
the Quick Dev environment.

(3) Copy the RPMs to the host.
```
scp metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/metron-*.rpm 
vagrant@node1:/tmp/
```

(4) Install the RPMs.
```
[root@node1 ~]# rpm -ivh metron-*.rpm
Preparing...### 
[100%]
   1:metron-solr### [ 
11%]
   2:metron-profiler### [ 
22%]
   3:metron-pcap### [ 
33%]
   4:metron-parsers ### [ 
44%]
   5:metron-indexing### [ 
56%]
   6:metron-enrichment  ### [ 
67%]
   7:metron-elasticsearch   ### [ 
78%]
   8:metron-data-management ### [ 
89%]
   9:metron-common  ### 
[100%]
```

(5) Validate that all of the Profiler's required files exist and are valid.
```
[root@node1 ~]# ls -l /usr/metron/0.3.0/config/profiler.properties
-rw-r--r--. 1 root root 1257 Jan  3 14:59 
/usr/metron/0.3.0/config/profiler.properties

[root@node1 ~]# ls -l /usr/metron/0.3.0/bin/start_profiler_topology.sh
-rwxr-xr-x. 1 root root 1080 Nov 22 22:36 
/usr/metron/0.3.0/bin/start_profiler_topology.sh

[root@node1 ~]# ls -l /usr/metron/0.3.0/flux/profiler/remote.yaml
-rw-r--r--. 1 root root 3937 Jan  3 14:59 
/usr/metron/0.3.0/flux/profiler/remote.yaml

[root@node1 ~]# ls -l /usr/metron/0.3.0/lib/metron-profiler-0.3.0-uber.jar
-rw-r--r--. 1 root root 62959781 Jan  9 18:26 
/usr/metron/0.3.0/lib/metron-profiler-0.3.0-uber.jar
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-654

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/413.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #413


commit 615fa67a562edb6535a90f7f7314386f76da0341
Author: Nick Allen 
Date:   2017-01-09T19:15:08Z

METRON-654 Create RPM Installer for Profiler




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: Enrich enrichment

2017-01-09 Thread Carolyn Duby 
Adding new topologies adds more processing requirements to the system.  It adds 
more topics (storage) and more producers and consumers to kafka (processing).

I think what we need is a dependency of enrichments.  Maybe we need to either 
derive the dependencies using the Stellar (potentially not that easy) or allow 
the enrichment to specify the order or enrichment calculations.
This will allow users to calculate more enrichment in the same topology.

Thanks
Carolyn




Sent from my Verizon, Samsung Galaxy smartphone


 Original message 
From: Nick Allen 
Date: 1/9/17 8:49 AM (GMT-08:00)
To: dev@metron.incubator.apache.org
Subject: Re: Enrich enrichment

I agree that making it easy for the user to "enrich enrichments", as Dima
put it, to an arbitrary depth, would be extremely useful for a lot of use
cases. We've discussed the use case a little in the past in this thread
[1].

Re-purposing the "threat intel" phase gives us something that is feasible
today, but only to a "depth" of 2.  We would also need to rename and
redocument it so that users understand how they can leverage the two
phases.  This seems like a minimally viable option if we want to head down
this road.

The other extreme might involve inferring the topology needed based on the
user's configuration. If the user needs 3 phases, then we build a topology
that supports 3 phases.  Under the covers instead of using Flux, we would
use Storm's topology builder Java API to grok the configuration and build
the topology(ies) that the user needs.

I am not sure if we can infer this from the configuration as it exists
today or if we would need to redefine the configuration somehow.  Like I
said this is "extreme", but could give the user more expressive and
intuitive options.




---
[1]
http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSkOc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E



On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella  wrote:

> I think that would be a good feature to add to have arbitrary number of
> phases, though it might be tricky to code (the way I envisioned it would
> involve a loop in storm, which is possible[1]), might have unintended
> consequences to guarantees (e.g. updating enrichments might not be able to
> be applied in realtime) and could be tricky to reason about
> performance-wise.
>
> As it stands, the number of phases is a consequence of the topology
> itself.  We do not currently have an architecture which would allow an
> arbitrary number of phases without changing the flux file itself.  What you
> can do, though, in a stellar enrichment is stack enrichments (e.g. depend
> on previous enrichments) because it's just a list of stellar statements.
> The consequence, of course, is that these statements get run within the
> same worker, which is unfortunate, but may be a stopgap workaround.
>
> *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8
>
> On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
> wrote:
>
> > Maybe the naming of the phases is misleading?  What if you could set up
> an
> > arbitrary number of stages, with defaults?
> >
> >
> > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote:
> >
> > You could do the geo enrichment normally and do a stellar hbase
> enrichment
> > in the threat Intel phase.
> >
> > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:
> >
> > > Hbase enrichments and geo enrichments are done in parallel so I would
> > not
> > > expect this to work. You could do the Hbase enrichment as a threat
> Intel
> > > enrichment and that should work because enrichments and threat Intel
> are
> > > done in series.
> > >
> > >
> > >
> > > The ideal way would be to chain together Stellar enrichments but I
> don't
> > > think there is a geo enrichment function created yet. I think that
> > should
> > > be a Jira. I know someone is working on an update to how we do geo
> > > enrichments so I will file a follow on Jira if it's not included in the
> > > scope of that work.
> > >
> > >
> > >
> > > Ryan
> > >
> > >
> > >
> > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
> > > wrote:
> > >
> > > >
> > >
> > > > Is it possible to enrich enrichment?
> > >
> > > >
> > >
> > > > For example I have IP address, I enrich it with geo and get City
> name,
> > >
> > > > now I want to enrich City name with city crime level (assume I have
> > that
> > >
> > > > data). But when I do that it just does not work. I specify enrichment
> > >
> > > > like that:
> > >
> > > >> {
> > >
> > > >> "index" : "msexchange",
> > >
> > > >> "batchSize" : 5,
> > >
> > > >> "enrichment" : {
> > >
> > > >> "fieldMap" : {
> > >
> > > >> "geo" : [ "destination_ip", "source_ip" ],
> > >
> > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
> > >
> > > >> "hbaseEnrichment" : [

[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...

2017-01-09 Thread mattf-horton 
Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/400
  
Yes, it seems almost impossible on a VM.
I'm not in favor of including ifconfig output.  It doesn't tell you 
anything about bandwidth, and it produces a lot of info that I think is of 
negligible value for the purposes of this work item.
I'd say you've got enough here, and should proceed with review process.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Kafka error when upgrading to Metron 0.3.0

2017-01-09 Thread Tyler Moore 
Seems like it's just the unit tests that are failing for pcap backend. Was able 
to build successfully using new came with -DskipTests option along with the 
profile argument.

Regards,

Tyler Moore
IT Specialist
tyler.math...@yahoo.com
248-909-2769

> On Jan 9, 2017, at 02:29, Michael Miklavcic  
> wrote:
> 
> Hi Tyler, I don't recall seeing any failures the last time I ran this. I
> will take a look.
> 
> On Jan 8, 2017 9:56 PM, "Tyler Moore"  wrote:
> 
> Michael,
> 
> I am receiving error when trying to build with HPD profile, have you had
> any problems with pcap-backend tests failing when building with hdp profile?
> Errors and log files provided below:
> 
> Results :
> 
> Failed tests:
>  
> PcapTopologyIntegrationTest.testTimestampInKey:152->testTopology:388->assertInOrder:542
> null
>  
> PcapTopologyIntegrationTest.testTimestampInPacket:135->testTopology:388->assertInOrder:542
> null
> 
> 
> 
> Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
> 
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Metron . SUCCESS [
> 0.639 s]
> [INFO] metron-analytics ... SUCCESS [
> 0.029 s]
> [INFO] metron-maas-common . SUCCESS [
> 8.771 s]
> [INFO] metron-platform  SUCCESS [
> 0.057 s]
> [INFO] metron-test-utilities .. SUCCESS [
> 0.976 s]
> [INFO] metron-integration-test  SUCCESS [
> 3.888 s]
> [INFO] metron-maas-service  SUCCESS [01:19
> min]
> [INFO] metron-common .. SUCCESS [
> 41.718 s]
> [INFO] metron-statistics .. SUCCESS [
> 30.713 s]
> [INFO] metron-hbase ... SUCCESS [
> 37.286 s]
> [INFO] metron-profiler-common . SUCCESS [
> 7.967 s]
> [INFO] metron-profiler-client . SUCCESS [
> 54.088 s]
> [INFO] metron-profiler  SUCCESS [02:28
> min]
> [INFO] metron-writer .. SUCCESS [
> 7.151 s]
> [INFO] metron-enrichment .. SUCCESS [
> 57.076 s]
> [INFO] metron-indexing  SUCCESS [
> 7.028 s]
> [INFO] metron-solr  SUCCESS [
> 36.695 s]
> [INFO] metron-pcap  SUCCESS [
> 1.123 s]
> [INFO] metron-parsers . SUCCESS [01:58
> min]
> [INFO] metron-pcap-backend  FAILURE [
> 49.928 s]
> [INFO] metron-data-management . SKIPPED
> [INFO] metron-api . SKIPPED
> [INFO] metron-management .. SKIPPED
> [INFO] elasticsearch-shaded ... SKIPPED
> [INFO] metron-elasticsearch ... SKIPPED
> [INFO] metron-deployment .. SKIPPED
> [INFO] Metron Ambari Management Pack .. SKIPPED
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 11:31 min
> [INFO] Finished at: 2017-01-08T22:23:59-05:00
> [INFO] Final Memory: 287M/6202M
> [INFO] 
> 
> [ERROR] Failed to execute goal org.apache.maven.plugins:
> maven-surefire-plugin:2.18:test (integration-tests) on project
> metron-pcap-backend: There are test failures.
> 
> 
> Regards,
> 
> Tyler Moore
> Software Engineer
> Phone: 248-909-2769 <(248)%20909-2769>
> Email: moore.ty...@goflyball.com
> 
> 
> On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
> 
>> Hey Tyler,
>> 
>> Build Metron with the HDP profile to get the proper deps for this
>> -PHDP-2.5.0.0
>> 
>> Hopefully that works for you.
>> 
>> Best,
>> Mike
>> 
>> 
>>> On Thu, Jan 5, 2017 at 3:17 PM, Tyler Moore  wrote:
>>> 
>>> Hey all,
>>> 
>>> Wondering if there is a solution to the "Offset lags for kafka not
>>> supported for older versions. Please update kafka spout to latest
>> version."
>>> error seen when upgrading to metron 0.3.0?
>>> 
>>> I know it is due to kafka-storm dependency that needs updated, but what
>> is
>>> the best way to go about that? Is it as simple at changing the version in
>>> the metron pom.xml and provisioning?
>>> 
>>> Regards,
>>> 
>>> Tyler Moore
>>> Software Engineer
>>> Phone: 248-909-2769
>>> Email: moore.ty...@goflyball.com
>>> 
>>

Re: Enrich enrichment

2017-01-09 Thread Nick Allen 
I agree that making it easy for the user to "enrich enrichments", as Dima
put it, to an arbitrary depth, would be extremely useful for a lot of use
cases. We've discussed the use case a little in the past in this thread
[1].

Re-purposing the "threat intel" phase gives us something that is feasible
today, but only to a "depth" of 2.  We would also need to rename and
redocument it so that users understand how they can leverage the two
phases.  This seems like a minimally viable option if we want to head down
this road.

The other extreme might involve inferring the topology needed based on the
user's configuration. If the user needs 3 phases, then we build a topology
that supports 3 phases.  Under the covers instead of using Flux, we would
use Storm's topology builder Java API to grok the configuration and build
the topology(ies) that the user needs.

I am not sure if we can infer this from the configuration as it exists
today or if we would need to redefine the configuration somehow.  Like I
said this is "extreme", but could give the user more expressive and
intuitive options.




---
[1]
http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201610.mbox/%3CCAHSJ8NwJUiyp3YO6NVE4tfLoSSkOc6QG%2BMsAJSSDu%2B-wfct_vw%40mail.gmail.com%3E



On Mon, Jan 9, 2017 at 10:56 AM, Casey Stella  wrote:

> I think that would be a good feature to add to have arbitrary number of
> phases, though it might be tricky to code (the way I envisioned it would
> involve a loop in storm, which is possible[1]), might have unintended
> consequences to guarantees (e.g. updating enrichments might not be able to
> be applied in realtime) and could be tricky to reason about
> performance-wise.
>
> As it stands, the number of phases is a consequence of the topology
> itself.  We do not currently have an architecture which would allow an
> arbitrary number of phases without changing the flux file itself.  What you
> can do, though, in a stellar enrichment is stack enrichments (e.g. depend
> on previous enrichments) because it's just a list of stellar statements.
> The consequence, of course, is that these statements get run within the
> same worker, which is unfortunate, but may be a stopgap workaround.
>
> *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8
>
> On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
> wrote:
>
> > Maybe the naming of the phases is misleading?  What if you could set up
> an
> > arbitrary number of stages, with defaults?
> >
> >
> > On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote:
> >
> > You could do the geo enrichment normally and do a stellar hbase
> enrichment
> > in the threat Intel phase.
> >
> > On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:
> >
> > > Hbase enrichments and geo enrichments are done in parallel so I would
> > not
> > > expect this to work. You could do the Hbase enrichment as a threat
> Intel
> > > enrichment and that should work because enrichments and threat Intel
> are
> > > done in series.
> > >
> > >
> > >
> > > The ideal way would be to chain together Stellar enrichments but I
> don't
> > > think there is a geo enrichment function created yet. I think that
> > should
> > > be a Jira. I know someone is working on an update to how we do geo
> > > enrichments so I will file a follow on Jira if it's not included in the
> > > scope of that work.
> > >
> > >
> > >
> > > Ryan
> > >
> > >
> > >
> > > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
> > > wrote:
> > >
> > > >
> > >
> > > > Is it possible to enrich enrichment?
> > >
> > > >
> > >
> > > > For example I have IP address, I enrich it with geo and get City
> name,
> > >
> > > > now I want to enrich City name with city crime level (assume I have
> > that
> > >
> > > > data). But when I do that it just does not work. I specify enrichment
> > >
> > > > like that:
> > >
> > > >> {
> > >
> > > >> "index" : "msexchange",
> > >
> > > >> "batchSize" : 5,
> > >
> > > >> "enrichment" : {
> > >
> > > >> "fieldMap" : {
> > >
> > > >> "geo" : [ "destination_ip", "source_ip" ],
> > >
> > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
> > >
> > > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
> > >
> > > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
> > >
> > > >> },
> > >
> > > >> "fieldToTypeMap" : {
> > >
> > > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
> > >
> > > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
> > >
> > > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
> > >
> > > >> },
> > >
> > > >> "config" : { }
> > >
> > > >> },
> > >
> > > >> "threatIntel" : {
> > >
> > > >> "fieldMap" : { },
> > >
> > > >> "fieldToTypeMap" : { },
> > >
> > > >> "config" : { },
> > >
> > > >> "triageConfig" : {
> > >
> > > >> "riskLevelRules" : { },
> > >
> > > >> "aggregator" : "MAX",
>

Re: Enrich enrichment

2017-01-09 Thread Casey Stella 
You could do that, but at the moment, I believe the indexing topology is
looking for its configs in the same place in zookeeper, so that'd get in
the way of that.

On Mon, Jan 9, 2017 at 11:08 AM, Otto Fowler 
wrote:

> What about having multiple instances of the enrichment topology with
> different configurations?  Then you would just have the last one terminate
> to indexing?  Or something like that.
>
>
> On January 9, 2017 at 10:56:38, Casey Stella (ceste...@gmail.com) wrote:
>
> I think that would be a good feature to add to have arbitrary number of
> phases, though it might be tricky to code (the way I envisioned it would
> involve a loop in storm, which is possible[1]), might have unintended
> consequences to guarantees (e.g. updating enrichments might not be able to
> be applied in realtime) and could be tricky to reason about
> performance-wise.
>
> As it stands, the number of phases is a consequence of the topology
> itself.  We do not currently have an architecture which would allow an
> arbitrary number of phases without changing the flux file itself.  What you
> can do, though, in a stellar enrichment is stack enrichments (e.g. depend
> on previous enrichments) because it's just a list of stellar statements.
> The consequence, of course, is that these statements get run within the
> same worker, which is unfortunate, but may be a stopgap workaround.
>
> *1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8
>
> On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
> wrote:
>
>> Maybe the naming of the phases is misleading?  What if you could set up
>> an arbitrary number of stages, with defaults?
>>
>>
>> On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote:
>>
>> You could do the geo enrichment normally and do a stellar hbase enrichment
>> in the threat Intel phase.
>>
>> On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:
>>
>> > Hbase enrichments and geo enrichments are done in parallel so I would
>> not
>> > expect this to work. You could do the Hbase enrichment as a threat Intel
>> > enrichment and that should work because enrichments and threat Intel are
>> > done in series.
>> >
>> >
>> >
>> > The ideal way would be to chain together Stellar enrichments but I don't
>> > think there is a geo enrichment function created yet. I think that
>> should
>> > be a Jira. I know someone is working on an update to how we do geo
>> > enrichments so I will file a follow on Jira if it's not included in the
>> > scope of that work.
>> >
>> >
>> >
>> > Ryan
>> >
>> >
>> >
>> > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
>> > wrote:
>> >
>> > >
>> >
>> > > Is it possible to enrich enrichment?
>> >
>> > >
>> >
>> > > For example I have IP address, I enrich it with geo and get City name,
>> >
>> > > now I want to enrich City name with city crime level (assume I have
>> that
>> >
>> > > data). But when I do that it just does not work. I specify enrichment
>> >
>> > > like that:
>> >
>> > >> {
>> >
>> > >> "index" : "msexchange",
>> >
>> > >> "batchSize" : 5,
>> >
>> > >> "enrichment" : {
>> >
>> > >> "fieldMap" : {
>> >
>> > >> "geo" : [ "destination_ip", "source_ip" ],
>> >
>> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
>> >
>> > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
>> >
>> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
>> >
>> > >> },
>> >
>> > >> "fieldToTypeMap" : {
>> >
>> > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
>> >
>> > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
>> >
>> > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
>> >
>> > >> },
>> >
>> > >> "config" : { }
>> >
>> > >> },
>> >
>> > >> "threatIntel" : {
>> >
>> > >> "fieldMap" : { },
>> >
>> > >> "fieldToTypeMap" : { },
>> >
>> > >> "config" : { },
>> >
>> > >> "triageConfig" : {
>> >
>> > >> "riskLevelRules" : { },
>> >
>> > >> "aggregator" : "MAX",
>> >
>> > >> "aggregationConfig" : { }
>> >
>> > >> }
>> >
>> > >> },
>> >
>> > >> "configuration" : { }
>> >
>> > >> }
>> >
>> > > I tried all the ways how enrichment field can be entered just to be
>> sure
>> >
>> > > I do not mistype it.
>> >
>> > >
>> >
>> > > - Dima
>> >
>> >
>>
>>
>

Re: Enrich enrichment

2017-01-09 Thread Otto Fowler 
What about having multiple instances of the enrichment topology with
different configurations?  Then you would just have the last one terminate
to indexing?  Or something like that.


On January 9, 2017 at 10:56:38, Casey Stella (ceste...@gmail.com) wrote:

I think that would be a good feature to add to have arbitrary number of
phases, though it might be tricky to code (the way I envisioned it would
involve a loop in storm, which is possible[1]), might have unintended
consequences to guarantees (e.g. updating enrichments might not be able to
be applied in realtime) and could be tricky to reason about
performance-wise.

As it stands, the number of phases is a consequence of the topology
itself.  We do not currently have an architecture which would allow an
arbitrary number of phases without changing the flux file itself.  What you
can do, though, in a stellar enrichment is stack enrichments (e.g. depend
on previous enrichments) because it's just a list of stellar statements.
The consequence, of course, is that these statements get run within the
same worker, which is unfortunate, but may be a stopgap workaround.

*1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8

On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
wrote:

> Maybe the naming of the phases is misleading?  What if you could set up an
> arbitrary number of stages, with defaults?
>
>
> On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote:
>
> You could do the geo enrichment normally and do a stellar hbase enrichment
> in the threat Intel phase.
>
> On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:
>
> > Hbase enrichments and geo enrichments are done in parallel so I would not
> > expect this to work. You could do the Hbase enrichment as a threat Intel
> > enrichment and that should work because enrichments and threat Intel are
> > done in series.
> >
> >
> >
> > The ideal way would be to chain together Stellar enrichments but I don't
> > think there is a geo enrichment function created yet. I think that should
> > be a Jira. I know someone is working on an update to how we do geo
> > enrichments so I will file a follow on Jira if it's not included in the
> > scope of that work.
> >
> >
> >
> > Ryan
> >
> >
> >
> > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
> > wrote:
> >
> > >
> >
> > > Is it possible to enrich enrichment?
> >
> > >
> >
> > > For example I have IP address, I enrich it with geo and get City name,
> >
> > > now I want to enrich City name with city crime level (assume I have
> that
> >
> > > data). But when I do that it just does not work. I specify enrichment
> >
> > > like that:
> >
> > >> {
> >
> > >> "index" : "msexchange",
> >
> > >> "batchSize" : 5,
> >
> > >> "enrichment" : {
> >
> > >> "fieldMap" : {
> >
> > >> "geo" : [ "destination_ip", "source_ip" ],
> >
> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
> >
> > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
> >
> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
> >
> > >> },
> >
> > >> "fieldToTypeMap" : {
> >
> > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
> >
> > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
> >
> > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
> >
> > >> },
> >
> > >> "config" : { }
> >
> > >> },
> >
> > >> "threatIntel" : {
> >
> > >> "fieldMap" : { },
> >
> > >> "fieldToTypeMap" : { },
> >
> > >> "config" : { },
> >
> > >> "triageConfig" : {
> >
> > >> "riskLevelRules" : { },
> >
> > >> "aggregator" : "MAX",
> >
> > >> "aggregationConfig" : { }
> >
> > >> }
> >
> > >> },
> >
> > >> "configuration" : { }
> >
> > >> }
> >
> > > I tried all the ways how enrichment field can be entered just to be
> sure
> >
> > > I do not mistype it.
> >
> > >
> >
> > > - Dima
> >
> >
>
>

Re: Enrich enrichment

2017-01-09 Thread Casey Stella 
I think that would be a good feature to add to have arbitrary number of
phases, though it might be tricky to code (the way I envisioned it would
involve a loop in storm, which is possible[1]), might have unintended
consequences to guarantees (e.g. updating enrichments might not be able to
be applied in realtime) and could be tricky to reason about
performance-wise.

As it stands, the number of phases is a consequence of the topology
itself.  We do not currently have an architecture which would allow an
arbitrary number of phases without changing the flux file itself.  What you
can do, though, in a stellar enrichment is stack enrichments (e.g. depend
on previous enrichments) because it's just a list of stellar statements.
The consequence, of course, is that these statements get run within the
same worker, which is unfortunate, but may be a stopgap workaround.

*1. https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8

On Mon, Jan 9, 2017 at 10:48 AM, Otto Fowler 
wrote:

> Maybe the naming of the phases is misleading?  What if you could set up an
> arbitrary number of stages, with defaults?
>
>
> On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote:
>
> You could do the geo enrichment normally and do a stellar hbase enrichment
> in the threat Intel phase.
>
> On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:
>
> > Hbase enrichments and geo enrichments are done in parallel so I would
> not
> > expect this to work. You could do the Hbase enrichment as a threat Intel
> > enrichment and that should work because enrichments and threat Intel are
> > done in series.
> >
> >
> >
> > The ideal way would be to chain together Stellar enrichments but I don't
> > think there is a geo enrichment function created yet. I think that
> should
> > be a Jira. I know someone is working on an update to how we do geo
> > enrichments so I will file a follow on Jira if it's not included in the
> > scope of that work.
> >
> >
> >
> > Ryan
> >
> >
> >
> > > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
> > wrote:
> >
> > >
> >
> > > Is it possible to enrich enrichment?
> >
> > >
> >
> > > For example I have IP address, I enrich it with geo and get City name,
> >
> > > now I want to enrich City name with city crime level (assume I have
> that
> >
> > > data). But when I do that it just does not work. I specify enrichment
> >
> > > like that:
> >
> > >> {
> >
> > >> "index" : "msexchange",
> >
> > >> "batchSize" : 5,
> >
> > >> "enrichment" : {
> >
> > >> "fieldMap" : {
> >
> > >> "geo" : [ "destination_ip", "source_ip" ],
> >
> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
> >
> > >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
> >
> > >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
> >
> > >> },
> >
> > >> "fieldToTypeMap" : {
> >
> > >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
> >
> > >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
> >
> > >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
> >
> > >> },
> >
> > >> "config" : { }
> >
> > >> },
> >
> > >> "threatIntel" : {
> >
> > >> "fieldMap" : { },
> >
> > >> "fieldToTypeMap" : { },
> >
> > >> "config" : { },
> >
> > >> "triageConfig" : {
> >
> > >> "riskLevelRules" : { },
> >
> > >> "aggregator" : "MAX",
> >
> > >> "aggregationConfig" : { }
> >
> > >> }
> >
> > >> },
> >
> > >> "configuration" : { }
> >
> > >> }
> >
> > > I tried all the ways how enrichment field can be entered just to be
> sure
> >
> > > I do not mistype it.
> >
> > >
> >
> > > - Dima
> >
> >
>
>

Re: Enrich enrichment

2017-01-09 Thread Otto Fowler 
Maybe the naming of the phases is misleading?  What if you could set up an
arbitrary number of stages, with defaults?


On January 8, 2017 at 16:25:01, Casey Stella (ceste...@gmail.com) wrote:

You could do the geo enrichment normally and do a stellar hbase enrichment
in the threat Intel phase.

On Sun, Jan 8, 2017 at 16:22 Ryan Merriman  wrote:

> Hbase enrichments and geo enrichments are done in parallel so I would not
> expect this to work. You could do the Hbase enrichment as a threat Intel
> enrichment and that should work because enrichments and threat Intel are
> done in series.
>
>
>
> The ideal way would be to chain together Stellar enrichments but I don't
> think there is a geo enrichment function created yet. I think that should
> be a Jira. I know someone is working on an update to how we do geo
> enrichments so I will file a follow on Jira if it's not included in the
> scope of that work.
>
>
>
> Ryan
>
>
>
> > On Jan 8, 2017, at 2:31 PM, Dima Kovalyov 
> wrote:
>
> >
>
> > Is it possible to enrich enrichment?
>
> >
>
> > For example I have IP address, I enrich it with geo and get City name,
>
> > now I want to enrich City name with city crime level (assume I have
that
>
> > data). But when I do that it just does not work. I specify enrichment
>
> > like that:
>
> >> {
>
> >> "index" : "msexchange",
>
> >> "batchSize" : 5,
>
> >> "enrichment" : {
>
> >> "fieldMap" : {
>
> >> "geo" : [ "destination_ip", "source_ip" ],
>
> >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip.country" ],
>
> >> "hbaseEnrichment" : [ "enrichments:geo:destination_ip:country" ],
>
> >> "hbaseEnrichment" : [ "enrichments.geo.destination_ip:country" ]
>
> >> },
>
> >> "fieldToTypeMap" : {
>
> >> "enrichments.geo.destination_ip.country" : [ "city_crime_level" ],
>
> >> "enrichments:geo:destination_ip:country" : [ "city_crime_level" ],
>
> >> "enrichments.geo.destination_ip:country" : [ "city_crime_level" ]
>
> >> },
>
> >> "config" : { }
>
> >> },
>
> >> "threatIntel" : {
>
> >> "fieldMap" : { },
>
> >> "fieldToTypeMap" : { },
>
> >> "config" : { },
>
> >> "triageConfig" : {
>
> >> "riskLevelRules" : { },
>
> >> "aggregator" : "MAX",
>
> >> "aggregationConfig" : { }
>
> >> }
>
> >> },
>
> >> "configuration" : { }
>
> >> }
>
> > I tried all the ways how enrichment field can be entered just to be
sure
>
> > I do not mistype it.
>
> >
>
> > - Dima
>
>

[GitHub] incubator-metron pull request #407: METRON-643: Stellar function documentati...

2017-01-09 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/407


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #407: METRON-643: Stellar function documentation need...

2017-01-09 Thread cestella 
Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/407
  
@JonZeolla I agree; I like this approach. I will commit and update the 
`STATS_BIN` PR to conform to the new approach.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #400: METRON-636: Capture memory and cpu details as a...

2017-01-09 Thread anandsubbu 
Github user anandsubbu commented on the issue:

https://github.com/apache/incubator-metron/pull/400
  
Thanks @mattf-horton and @JonZeolla, I have now included the output of `df 
-h` . Here is a sample output on Mac.

```

Total System Memory = 16384 MB
Processor Model: Intel(R) Core(TM) i7-4870HQ CPU
Processor Speed: 2.50GHz
Total Physical Processors: 4
Total cores: 4
Disk information:
/dev/disk1 465Gi  338Gi  127Gi73% 88564601 33279109   73%   / 
```

I did attempt to check for commands to capture the NIC card / interface 
information, but I have to agree with @mattf-horton that it is non-trivial and 
can vary widely depending on environments. I considered including the output of 
`ifconfig` at least, however I was not sure if a user's IP address would fall 
under sensitive information category--which some might not want having 
captured/shared in logs (or would they?)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #404: METRON-624: Updated Comparison/Equality Evaluat...

2017-01-09 Thread cestella 
Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/404
  
@jjmeyer0 Yeah, I tend to agree.  I'll make a follow-on JIRA to add an 
`APPROX_EQUALS`.  I feel better about it now that we have an `ABS` function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---