Re: Define a function that can be used in Stellar

[Matt Foley]

Besides the example code Simon mentioned at 
https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
 ,
there is some documentation at 
http://metron.apache.org/current-book/metron-stellar/stellar-common/3rdPartyStellar.html
 

 

From: Nick Allen 
Reply-To: "user@metron.apache.org" 
Date: Wednesday, January 17, 2018 at 4:46 AM
To: "user@metron.apache.org" 
Subject: Re: Define a function that can be used in Stellar

 

 

 

If something we have already does not fit the bill, I would recommend creating 
that function in Java.   Since you described it as "a bit complex" and "the 
logic would be complicated" I don't see any value in defining something like 
this in Stellar with named functions. 

 

Best

 

 

 

 

On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball 
 wrote:

Have you looked at the recent TLSH functions in Stellar? We already have that 
for similarity preserving hashes. 

 

Simon

 


On 17 Jan 2018, at 12:35, Ali Nazemian  wrote:

It is a bit complex. We want to create a function that accepts a list of 
arguments for an asset and generate an asset identifier that can be used as a 
row_key for the enrichment store. The logic would be complicated, though. We 
may need to include some sort of similarity aware hash function as a part of 
this custom function.

 

On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen  wrote:

Ali - Can you describe the logic that you are trying to perform? That would be 
useful as a use case to help drive a discussion around creating named functions 
in Stellar. 

 

 

 

 

On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian  wrote:

Thanks, Simon. We have already got a script to deal with classpath management 
for the parsers. We should be able to use it for this extension as well. 

 

Yeah, I agree. It will be much easier to define functions on the fly and use 
them afterwards. It could be defined as Lambda or custom function. 

 

Regards,

Ali

 

 

 

On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball 
 wrote:

https://github.com/apache/metron/tree/master/metron-stellar/stellar-3rd-party-example
 gives good details on how to add a stellar function. 

 

Stellar will pick up an annotated function on its class path, so to add 
function there is no need to rebuild metron module, but you do need your 
modules on the classpath, and, pending 777, to deal with things like class path 
clash in your dependencies. 

 

Another idea worth discussion on the dev list is probably the notion of 
defining stellar functions in stellar, which would be a much simpler solution 
than custom java functions if you can already express you logic in stellar. 

 

Simon 

 



On 17 Jan 2018, at 10:37, Ali Nazemian  wrote:

 

Hi Simon, 

 

Yes, that is exactly what we are looking for. Is there any example regarding 
adding a Stellar function in Java? Hopefully, we don't need to rebuild the 
corresponding modules for this?

 

Cheers,

Ali

 

On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball 
 wrote:

At present you can certainly create custom stellar functions in Java. I’m 
guessing however that what you’re looking to do is create a kind of function 
that combines a number of stellar functions to avoid repetition, or to ensure 
consistency of certain parameters for example. Is that what you’re looking for? 
Maybe some sort of syntax to create a named stellar function similar to the way 
we create lambdas?

Simon


> On 17 Jan 2018, at 07:25, Ali Nazemian  wrote:
>
> Hi all,
>
> Is there any way that we can define a function that can be used rather than 
> duplicating a logic multiple times?
>
> Cheers,
> Ali



 

-- 

A.Nazemian

 



 

-- 

A.Nazemian



 

-- 

A.Nazemian

Re: Intro & Question

[Matt Foley]

BTW, any community member can open a jira, but to assign it to yourself, as the 
instructions say, requires being added to the list of contributors.  Just 
forward this thread to dev@ with a request to Casey Stella, our PMC Chair, to 
be added as a contributor.

From: Matt Foley <mfo...@hortonworks.com>
Reply-To: "user@metron.apache.org" <user@metron.apache.org>
Date: Wednesday, January 10, 2018 at 10:41 AM
To: "user@metron.apache.org" <user@metron.apache.org>, Ahmed Shah 
<ahmeds...@cmail.carleton.ca>
Subject: Re: Intro & Question

Ahmed, please see

https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines

Feel free to ask the dev community questions.  Suggest this discussion (as 
regards the contribution) be moved to 
d...@metron.apache.org<mailto:d...@metron.apache.org> instead of user@.

Cheers,
--Matt

From: Otto Fowler <ottobackwa...@gmail.com>
Reply-To: "user@metron.apache.org" <user@metron.apache.org>
Date: Wednesday, January 10, 2018 at 7:28 AM
To: Ahmed Shah <ahmeds...@cmail.carleton.ca>, "user@metron.apache.org" 
<user@metron.apache.org>
Subject: Re: Intro & Question

So, what would work would be:

1.  Create a jira like “Ability to deploy metron full dev to aws with vagrant”
With a description of the use case, and how the vagrant file will fill it.
2. create a pr, with the new file in metron-deployment/vagrant/aws
3. update the readme

I think



On January 10, 2018 at 10:16:03, Ahmed Shah 
(ahmeds...@cmail.carleton.ca<mailto:ahmeds...@cmail.carleton.ca>) wrote:

Would be glad to.

Where in github should I put it?


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>


From: Otto Fowler <ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>>
Sent: January 9, 2018 11:51 AM
To: Ahmed Shah; user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Re: Intro & Question

Any interest in submitting this?



On January 9, 2018 at 10:42:08, Ahmed Shah 
(ahmeds...@cmail.carleton.ca<mailto:ahmeds...@cmail.carleton.ca>) wrote:

Hello Srikanth,



Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: 
https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
  to deploy a single node to AWS.

Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile

You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID 
before running the file.



Hope it helps.



-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php>


From: Srikanth Nagarajan 
<s...@gandivanetworks.com<mailto:s...@gandivanetworks.com>>
Sent: January 9, 2018 2:39 AM
To: user@metron.apache.org<mailto:user@metron.apache.org>
Subject: Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building 
Metron to test in our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes, please 
share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the 
metron git-hub docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2  get 
an error

[WARNING]:  * Failed to parse 
/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin: 
Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an 
execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

__

Srikanth Nagarajan
Principal

Gandiva Networks Inc

732.690.1884 Mobile

s...@gandivanetworks.com<mailto:s...@gandivanetworks.com>

www.gandivanetworks.com<http://www.gandivanetworks.com/>

Please consider the environment before printing this. NOTICE: The information 
contained in this e-mail message is intended for addressee(s) only. If you have 
received this message in error please notify the sender.

Re: Intro & Question

[Matt Foley]

Ahmed, please see

https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines

Feel free to ask the dev community questions.  Suggest this discussion (as 
regards the contribution) be moved to 
d...@metron.apache.org instead of user@.

Cheers,
--Matt

From: Otto Fowler 
Reply-To: "user@metron.apache.org" 
Date: Wednesday, January 10, 2018 at 7:28 AM
To: Ahmed Shah , "user@metron.apache.org" 

Subject: Re: Intro & Question

So, what would work would be:

1.  Create a jira like “Ability to deploy metron full dev to aws with vagrant”
With a description of the use case, and how the vagrant file will fill it.
2. create a pr, with the new file in metron-deployment/vagrant/aws
3. update the readme

I think



On January 10, 2018 at 10:16:03, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Would be glad to.

Where in github should I put it?


-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com


From: Otto Fowler >
Sent: January 9, 2018 11:51 AM
To: Ahmed Shah; user@metron.apache.org
Subject: Re: Intro & Question

Any interest in submitting this?



On January 9, 2018 at 10:42:08, Ahmed Shah 
(ahmeds...@cmail.carleton.ca) wrote:

Hello Srikanth,



Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: 
https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform)
  to deploy a single node to AWS.

Our Vagrent file is here:

https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile

You can define your AWS Elastic IP,  Subnet ID, VPC, and Security Group ID 
before running the file.



Hope it helps.



-Ahmed
___
Ahmed Shah (PMP, M. Eng.)
Cybersecurity Analyst & Developer
GCR - Cybersecurity Operations Center
Carleton University - cugcr.com


From: Srikanth Nagarajan 
>
Sent: January 9, 2018 2:39 AM
To: user@metron.apache.org
Subject: Intro & Question


Hi

My name is Srikanth and work for a Cyber Security firm.   We are building 
Metron to test in our lab environment using AWS.

1. Is there a single VM version for Cloud install available ?   If yes, please 
share procedure.

2. During the Amazon-Ec2 install for the multi node version provided in the 
metron git-hub docs

https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2  get 
an error

[WARNING]:  * Failed to parse 
/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin: 
Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an 
execution

error: ERROR: "Forbidden", while: getting ElastiCache clusters

Any assistance would be appreciated.

Thanks

Srikanth

__

Srikanth Nagarajan
Principal

Gandiva Networks Inc

732.690.1884 Mobile

s...@gandivanetworks.com

www.gandivanetworks.com

Please consider the environment before printing this. NOTICE: The information 
contained in this e-mail message is intended for addressee(s) only. If you have 
received this message in error please notify the sender.

Re: Metron Rest with Kerberos support

[Matt Foley]

This change will be a little tricky, because the problem is in _indirect_ 
dependencies.  In case you’re not a maven expert, here are some more detailed 
instructions on how to do this.

 

If you are correct that only the 4.5.2 version is causing you problems, there 
are 3 instances, as seen in `mvn dependency:tree`:

 

metron-rest

    
org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE:compile

    org.apache.httpcomponents:httpclient:jar:4.5.2:compile

metron-enrichment

    com.maxmind.geoip2:geoip2:jar:2.8.0:compile

    org.apache.httpcomponents:httpclient:jar:4.5.2:compile

metron-elasticsearch

    metron-enrichment

    com.maxmind.geoip2:geoip2:jar:2.8.0:compile

    org.apache.httpcomponents:httpclient:jar:4.5.2:compile

 

It is not at all clear whether spring-security-kerberos-client and geoip2 will 
be agreeable with downgrading httpclient, but you can try.  Downgrading only to 
4.5.1 makes it more likely to succeed than if you needed a larger downgrade.

 

Because these are indirect dependencies, you must use the following construct 
in the pom.xml:

 

In metron-interface/metron-rest/pom.xml, WITHIN the declared dependency on 
org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE,
 add

       

    

    org.apache.httpcomponents

    httpclient

    

    

Then, at the SAME level as the dependency on spring-security-kerberos-client, 
add an additional dependency as

    

    org.apache.httpcomponents

    httpclient

    4.5.1

    

 

In metron-platform/metron-enrichment/pom.xml, WITHIN the declared dependency on 
com.maxmind.geoip2:geoip2:jar:2.8.0, there are already exclusions, so just add 
one more that says

    

    org.apache.httpcomponents

    httpclient

    

Then at the level of the dependency, add as before

    

    org.apache.httpcomponents

    httpclient

    4.5.1

    

 

You don’t need to change the pom for metron-elasticsearch, it will inherit the 
change from metron-enrichment.

 

Note:  If you trust IntelliJ that the httpclient really isn’t needed, you can 
insert the exclusions but leave out the new dependency declarations.  I have no 
opinion as to whether that will work with these two packages 
(spring-security-kerberos-client and geoip2), but if you try leaving out the 
4.5.1 dependency, then I would be sure to carefully test the functionality 
after the change.

 



The remainder of this message is general info about how to examine this 
category of problem:

 

Two useful commands are (executed from the root of a Metron code tree):

    mvn dependency:tree > deps.txt

and

    grep -r --include pom.xml -i HttpClient *

or

    grep -r --include pom.xml -B 3 -A 3 -i HttpClient *|more

 

The grep will show you direct dependencies, and exclusions.

The mvn dependency:tree command prints out all direct and (non-excluded) 
indirect dependencies.

The dependency tree was the starting point for my analysis above.

 

Hope this helps,

--Matt

 

From: Vipin Rathor 
Reply-To: "user@metron.apache.org" 
Date: Thursday, January 4, 2018 at 8:02 PM
To: "user@metron.apache.org" 
Subject: Re: Metron Rest with Kerberos support

 

So I build metron-rest jar and found that it does contain multiple httpclient 
classes : 

 

$ jar -tf target/metron-rest-0.4.1.jar | grep -i "HttpClient.class"

  org/apache/http/client/HttpClient.class

  org/apache/hadoop/hbase/shaded/org/apache/http/client/HttpClient.class

  org/apache/commons/httpclient/HttpClient.class

 

The first one is indeed a HttpClient v4.5.2 which is causing problem in your 
case.

 

>From the IntelliJ generated metron-rest.iml file,

$ grep httpclient metron-rest.iml





 

Interestingly, IntelliJ also reports that this library is not used at all in 
metron-rest project and hence can be removed.

Since HttpClient v4.5.2 is known to cause trouble with Kerberos, we should 
either remove it or downgrade it.

 

Thinking of opening a Metron bug.

 

@Simon/James, suggestions?

 

 

On Wed, Jan 3, 2018 at 6:33 PM, prakash r  wrote:

Hello Vipin 

 

I can see HttpClient related classes are loaded from metron-rest jar

 

[Loaded org.apache.http.client.HttpClient from 
file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]

[Loaded org.apache.http.impl.client.HttpClientBuilder from 
file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]

[Loaded org.apache.http.conn.HttpClientConnectionManager from 
file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]

[Loaded org.apache.http.impl.client.CloseableHttpClient from 
file:/usr/hcp/1.3.0.0-51/metron/lib/metron-rest-0.4.1.1.3.0.0-51.jar]

[Loaded 

Re: profiler Syntax error

[Matt Foley]

Good eye, Otto!

From: Otto Fowler <ottobackwa...@gmail.com>
Date: Wednesday, August 2, 2017 at 2:12 PM
To: Matt Foley <mfo...@hortonworks.com>, "d...@metron.apache.org" 
<d...@metron.apache.org>, "user@metron.apache.org" <user@metron.apache.org>
Subject: Re: profiler Syntax error

Can you try the rule without ‘in’ as a key?
“in” is a reserved word in Stellar.

What that error means is that the ANTLR lexer and parser cannot figure out what 
the statement means because of ambiguity, most likely introduced by the use of 
a keyword. (in is not a valid statement.





On August 2, 2017 at 16:49:49, Matt Foley 
(mfo...@hortonworks.com<mailto:mfo...@hortonworks.com>) wrote:
Unable to pars

Re: Integration of Honeeepi(honeypot sensor) with Metron

[Matt Foley]

The “top three” changed, so here are my preferred references for writing new 
parsers:

* 
https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry

* 
https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source

* 
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html
 

 

From: Matt Foley <mfo...@hortonworks.com> on behalf of Matt Foley 
<ma...@apache.org>
Date: Monday, July 31, 2017 at 2:39 PM
To: "user@metron.apache.org" <user@metron.apache.org>
Subject: Re: Integration of Honeeepi(honeypot sensor) with Metron

 

Hi Naveen,

 

Does Honeeepi produce a stream of logs and/or alerts, that you would like to 
process?

If not, you’ll need to define a “sensor” of sorts that will tell you when 
something interesting happens (or is happening) with the honeypot.  Metron does 
not help with that, although it can help compare normative with aberrational 
event streams, thereby identifying what is “interesting”, if Honeeepi itself 
does not do that.  The integration point with Metron will be the message stream 
from Honeeepi or that Honeeepi sensor, preferably piped into Kafka.

 

Next you need a parser for the logs from Kafka.  You may be able to write a 
Grok script for our generic Grok parser, otherwise you can write a Metron 
Parser module in Java.  Parsers are in the process of becoming plug-ins for 
Metron, but for now, the current way of creating new parsers can be found in 
the top three results when you google “apache metron writing a new parser”.  
Parsers convert messages of whatever format into a standard JSON format, which 
the rest of Metron knows how to deal with.

 

Now you’ve got your “integration”.  You still need to decide what to do with 
the message stream.  If you need to identify “interesting” vs “not interesting” 
events, you might plug in an ML model as one of your enrichers.  When you can 
filter for interesting events, you can “enrich” them by raising select info in 
the message body into the meta-data, or adding new meta-data based on 
associational lookups of existing fields.

 

Then you add Threat Intel and do threat recognition, and feed that into your 
Indexing and Alerting sub-systems.  That’s Metron in a thimble :-)  Suggest you 
read the entire site-book of Metron development information at 
https://metron.apache.org/current-book/index.html , especially the articles 
with architecture diagrams at 

* 
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html

* 
https://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html

* 
https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html

* 
https://metron.apache.org/current-book/metron-analytics/metron-maas-service/index.html

and the stuff about Profiling and Statistics at 

* 
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html

* 
https://metron.apache.org/current-book/metron-analytics/metron-profiler-client/index.html

* 
https://metron.apache.org/current-book/metron-analytics/metron-statistics/index.html

 

Hope this helps,

--Matt

 

From: Naveen Narayanasamy <naveennarayanas...@cmail.carleton.ca>
Reply-To: "user@metron.apache.org" <user@metron.apache.org>
Date: Monday, July 31, 2017 at 8:21 AM
To: "user@metron.apache.org" <user@metron.apache.org>
Subject: Integration of Honeeepi(honeypot sensor) with Metron

 

Hello all,

 

I would like to know the possibility of integrating Honeeepi omponents(Cowrie 
and Dionaea) with Apache Metron.As there are limited information available 
online,I would also like to know the different integration procedures that can 
be tried.

 

For more info about the honeeepi, please visit the link below

https://redmine.honeynet.org/projects/honeeepi/wiki

Wiki - Honeeepi - Honeynet Project Redmine
redmine.honeynet.org
Introduction¶ This project is about setting up honeypots with Raspberry Pi - a 
credit card sized ARM Linux box. !! Raspberry Pi¶ The Raspberry Pi is a credit 
sized ...
Any response will be appreciated!! 

 

Naveen

 

 

Re: Integration of Honeeepi(honeypot sensor) with Metron

[Matt Foley]

Hi Naveen,

 

Does Honeeepi produce a stream of logs and/or alerts, that you would like to 
process?

If not, you’ll need to define a “sensor” of sorts that will tell you when 
something interesting happens (or is happening) with the honeypot.  Metron does 
not help with that, although it can help compare normative with aberrational 
event streams, thereby identifying what is “interesting”, if Honeeepi itself 
does not do that.  The integration point with Metron will be the message stream 
from Honeeepi or that Honeeepi sensor, preferably piped into Kafka.

 

Next you need a parser for the logs from Kafka.  You may be able to write a 
Grok script for our generic Grok parser, otherwise you can write a Metron 
Parser module in Java.  Parsers are in the process of becoming plug-ins for 
Metron, but for now, the current way of creating new parsers can be found in 
the top three results when you google “apache metron writing a new parser”.  
Parsers convert messages of whatever format into a standard JSON format, which 
the rest of Metron knows how to deal with.

 

Now you’ve got your “integration”.  You still need to decide what to do with 
the message stream.  If you need to identify “interesting” vs “not interesting” 
events, you might plug in an ML model as one of your enrichers.  When you can 
filter for interesting events, you can “enrich” them by raising select info in 
the message body into the meta-data, or adding new meta-data based on 
associational lookups of existing fields.

 

Then you add Threat Intel and do threat recognition, and feed that into your 
Indexing and Alerting sub-systems.  That’s Metron in a thimble :-)  Suggest you 
read the entire site-book of Metron development information at 
https://metron.apache.org/current-book/index.html , especially the articles 
with architecture diagrams at 

* 
https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html

* 
https://metron.apache.org/current-book/metron-platform/metron-enrichment/index.html

* 
https://metron.apache.org/current-book/metron-platform/metron-indexing/index.html

* 
https://metron.apache.org/current-book/metron-analytics/metron-maas-service/index.html

and the stuff about Profiling and Statistics at 

* 
https://metron.apache.org/current-book/metron-analytics/metron-profiler/index.html

* 
https://metron.apache.org/current-book/metron-analytics/metron-profiler-client/index.html

* 
https://metron.apache.org/current-book/metron-analytics/metron-statistics/index.html

 

Hope this helps,

--Matt

 

From: Naveen Narayanasamy 
Reply-To: "user@metron.apache.org" 
Date: Monday, July 31, 2017 at 8:21 AM
To: "user@metron.apache.org" 
Subject: Integration of Honeeepi(honeypot sensor) with Metron

 

Hello all,

 

I would like to know the possibility of integrating Honeeepi omponents(Cowrie 
and Dionaea) with Apache Metron.As there are limited information available 
online,I would also like to know the different integration procedures that can 
be tried.

 

For more info about the honeeepi, please visit the link below

https://redmine.honeynet.org/projects/honeeepi/wiki

Wiki - Honeeepi - Honeynet Project Redmine
redmine.honeynet.org
Introduction¶ This project is about setting up honeypots with Raspberry Pi - a 
credit card sized ARM Linux box. !! Raspberry Pi¶ The Raspberry Pi is a credit 
sized ...
Any response will be appreciated!! 

 

Naveen

 

 

Re: [ANNOUNCE] Apache Metron 0.4.0 release

[Matt Foley]

BTW, if you’ve recently accessed any URIs under http://metron.apache.org/, you 
may need to hit “refresh” in your browser to see the new updated versions.

On 7/5/17, 1:35 PM, "Matt Foley" <ma...@apache.org> wrote:

Friends and Colleagues,
I’m happy to announce the completion and release of Apache Metron 0.4.0.
Besides a bunch of great new features, this is also our first release as a 
TLP.

The public website at http://metron.apache.org/ has been updated and has 
correct links to the new downloads and docs.

Full list of changes in this release is in the Release Notes:
http://www.apache.org/dyn/closer.cgi/metron/0.4.0/RELEASE_NOTES 

And the release point is tagged in github as “apache-metron-0.4.0-release”, 
viewable at:
https://github.com/apache/metron/tree/apache-metron-0.4.0-release
This is also the same has HEAD of Metron_0.4.0 branch.

Every release is a team effort, and this one with 140+ tasks and bug fixes, 
certainly involved the whole community. Congratulations and THANK YOU to 
everyone who contributed, whether ideas, code, reviews, documentation, or 
testing.

Best regards,
--Matt (as Release Manager)

[ANNOUNCE] Apache Metron 0.4.0 release

[Matt Foley]

Friends and Colleagues,
I’m happy to announce the completion and release of Apache Metron 0.4.0.
Besides a bunch of great new features, this is also our first release as a TLP.

The public website at http://metron.apache.org/ has been updated and has 
correct links to the new downloads and docs.

Full list of changes in this release is in the Release Notes:
http://www.apache.org/dyn/closer.cgi/metron/0.4.0/RELEASE_NOTES 

And the release point is tagged in github as “apache-metron-0.4.0-release”, 
viewable at:
https://github.com/apache/metron/tree/apache-metron-0.4.0-release
This is also the same has HEAD of Metron_0.4.0 branch.

Every release is a team effort, and this one with 140+ tasks and bug fixes, 
certainly involved the whole community. Congratulations and THANK YOU to 
everyone who contributed, whether ideas, code, reviews, documentation, or 
testing.

Best regards,
--Matt (as Release Manager)

Re: Metron current version and Docker

[Matt Foley]

This is funny/sad, because our Vagrant scripts, that you assert you cannot use, 
were intended to meet this need.  You can’t get much simpler than “vagrant up” 
to install a full-blown single-node Metron.

From: Otto Fowler <ottobackwa...@gmail.com>
Date: Monday, June 12, 2017 at 1:24 PM
To: "sml...@libero.it" <sml...@libero.it>, Matt Foley <ma...@apache.org>, 
"user@metron.apache.org" <user@metron.apache.org>
Subject: Re: Metron current version and Docker

I do not know of any outside of our vagrant scripts, that use ansible to deploy 
Metron onto a centos virtual box vm.



On June 12, 2017 at 08:12:44, sml...@libero.it<mailto:sml...@libero.it> 
(sml...@libero.it<mailto:sml...@libero.it>) wrote:

Hello Otto,

is there any VM Centos 6 with Metron installed that I can use for my tests with 
ML?

Thanks.

Simone
Il 12 giugno 2017 alle 14.05 Otto Fowler 
<ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> ha scritto:
We use centos 6 for our images and most deployments.  There are guides in the 
works for centos 7 manual setup with ambari in the works right now.  Centos 7 
is not supported by the full ansible setup at this time.





On June 12, 2017 at 06:08:40, sml...@libero.it<mailto:sml...@libero.it> 
(sml...@libero.it<mailto:sml...@libero.it>) wrote:


Hello,

I tried to install Metron 0.3.1 following  the guideline:

https://community.hortonworks.com/content/kbentry/88843/manually-installing-apache-metron-on-ubuntu-1404.html

but I got some errors.

One question about where I could find HDP 2.5.0 installation guideline for 
Ubuntu.

Finally, if you as community and developers of Metron use CentOS 7, maybe it 
would be better if I adapt myself to that distribution.

So, is there any ContOS 7 VM ready that I could download to test Metron 0.3.1 
and the MaaS option?

Maybe it would be faster than debug the installation in Ubuntu.

Thank you.

Best Regards,
Simone


Il 9 giugno 2017 alle 2.09 Otto Fowler 
<ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> ha scritto:
Sorry, I do not know, we use centos.




On June 8, 2017 at 18:12:50, sml...@libero.it<mailto:sml...@libero.it> 
(sml...@libero.it<mailto:sml...@libero.it>) wrote:


Hello Otto,

thanks for the link.

Do you know if it does work also with Ubuntu 16.04?

Just for curiosity, the post is quite recent but I do not understand the reason 
to use a distribution quite old.

Finally, I read in that guide that they do not use NIFI as source to record 
network packet.

Is there any guide to use NIFI with Metron?

Best Regards,

Simone
Il 8 giugno 2017 alle 22.25 Otto Fowler 
<ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> ha scritto:
https://community.hortonworks.com/content/kbentry/88843/manually-installing-apache-metron-on-ubuntu-1404.html


On June 8, 2017 at 15:46:09, sml...@libero.it<mailto:sml...@libero.it> 
(sml...@libero.it<mailto:sml...@libero.it>) wrote:


Hello Matt,

thanks for your email.

Yes, I said that I would use an Ubuntu VM to install Metron. I'm old school and 
I'm not so familiar with Docker.

Moreover, it seems that my CPU (MacBook) does not support virtualization.

On the other hand, I don't know if a VM with CentOS 7 on my machine could run 
Docker. I mean due to the same problem of virtualization on the host CPU.

Let assume that VM CentOS 7 support Docker, I'm not familiar with that 
distribution (I used several but never CentOS). Do you have rpm packages for 
the tools needed for Metron?

I would use the 0.4.0 version, but after several day I'm a bit frustrated 
because I stil didn't completely understand the tools chain to run Metron.

I asked also into the dev-mailing list.

My idea was to install NIFI as probe to catch network packets and fill in those 
into Metron.

Then, I still didn't understand where I should deploy ML model into Metron to 
run it as a service.

And finally, how to moidfy the ML to include it into a RESTful app.

I'm sorry for the long list of questions/issues. I knwo that it is not so 
elegant into a mailing list but I'm a step from give up Metron. Even if I know 
that it would be a mistake.

Thanks again if you could give some indications.

Simone




Il 8 giugno 2017 alle 20.40 Matt Foley 
<ma...@apache.org<mailto:ma...@apache.org>> ha scritto:

Hi Simone,

If I recall your previous email, you said you want to use an Ubuntu VM.  Can 
you use Centos 6 or 7 instead?



The reason I ask is that for Centos there is an “Ambari manual install” 
procedure, which does not require Docker, Vagrant, or Ansible on the server.  
In this scenario you just install Docker on your development machine (I use a 
Mac), build the Metron RPMs and Ambari MPack there, scp them to the server, and 
proceed with Ambari install.  This is in fact my main lab test method.



But with Ubuntu, I’m not aware of a documented procedure for Ambari manual 
install, only insta

Re: Metron components connectivity

[Matt Foley]

➢ It is a shame there isn’t a reporting tool for ambari, that can query all the 
configurations in the database and report out ports and hosts…….

There is.  If you configure a “happy” cluster with Ambari, then tell Ambari to 
generate a blueprint, that will tell you the actual as-built values for all 
configuration parameters controlled by Ambari.  Of course there’s a lot of 
other stuff too, but the full config is there.

Blueprint docs are at 
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
The API to export the current blueprint is at 
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Step1:CreateBlueprint
The structure of the blueprint and config elements are in 
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-BlueprintDetails

Cheers,


--Matt

From: Otto Fowler 
Reply-To: "user@metron.apache.org" 
Date: Tuesday, May 30, 2017 at 6:17 AM
To: Alex McLintock , "user@metron.apache.org" 

Subject: Re: Metron components connectivity

It is a shame there isn’t a reporting tool for ambari, that can query all the 
configurations in the database and report out ports and hosts…….


On May 30, 2017 at 08:34:42, Alex McLintock (a...@alephant.co.uk) wrote:
I have attempted to do this for just HDP and it is rather difficult. One of the 
main problems is that any documentation will point you towards default ports 
for Hadoop - which are not always the ports chosen by Hortonworks for HDP. You 
probably need to look mostly at the ports specified in your config files. 

I would suggest that you treat all HDP nodes as able to talk to each other 
across all ports - but limit anything which talks to those nodes. That is a lot 
easier. 


On 30 May 2017 at 10:49, Ali Nazemian  wrote:
Hi all, 

For deploying Metron in production, we need to specify all of the port and 
protocols connectivities. I was wondering how Metron components connected to 
each other. Is there any document available regarding the ports and 
connectivities of Metron components?

Regards,
Ali




--
Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to win £50

Re: Metron HBase conditional enrichment

[Matt Foley]

That sheds light.  Thanks, Nick.
--Matt

From: Nick Allen <n...@nickallen.org>
Reply-To: "user@metron.apache.org" <user@metron.apache.org>
Date: Thursday, May 25, 2017 at 3:06 PM
To: "user@metron.apache.org" <user@metron.apache.org>
Subject: Re: Metron HBase conditional enrichment

Each topology has its own uber-jar that is built from all of it's dependencies. 
 It's classpath is basically whatever is in the uber-jar.

That's why running with -pl against the project from which the uber jar is 
built should identify the Stellar functions available to each topology.

When running the REPL from a deployed instance of Metron it pulls in all of the 
jars deployed to /usr/metron/(version)/lib.

On May 25, 2017 3:29 PM, "Matt Foley" 
<mfo...@hortonworks.com<mailto:mfo...@hortonworks.com>> wrote:
Nick is correct that in any given environment, only Stellar functions defined 
in jars on the current classpath will be available.

When running with the maven exec:java plugin, as below, this means only jars 
declared (or transitively required) as dependencies to the given project.

In the installed environment, however, it is strictly a matter of the 
configured classpath.  I thought (I could be wrong), that all the metron jars 
are installed together (in /usr/metron//lib/* , in CentOS 7), and that 
all those jars will be added to the classpath for all topologies.

Nick, you’re much more familiar with the install stuff than I, please clarify 
if you can how the classpath is configured for the running topologies in the 
installed env.

At any rate, it is the classpath currently being run under that determines the 
availability of Stellar functions.
Thanks,
--Matt

From: Nick Allen <n...@nickallen.org<mailto:n...@nickallen.org>>
Reply-To: "user@metron.apache.org<mailto:user@metron.apache.org>" 
<user@metron.apache.org<mailto:user@metron.apache.org>>
Date: Thursday, May 25, 2017 at 12:05 PM
To: "user@metron.apache.org<mailto:user@metron.apache.org>" 
<user@metron.apache.org<mailto:user@metron.apache.org>>

Subject: Re: Metron HBase conditional enrichment

Correct me if I am wrong, Matt, but I believe that changing the project that 
you pass to the -pl switch will allow you to see exactly what Stellar functions 
would be available in each topology.  You just have to refer to whichever 
project drives the topology.

This might help answer Ali's previous question as to what functions are 
available where.  And of course, if some function is not available where it is 
needed, then it is a simply matter of changing the dependencies to make it 
available.

For example, these functions are available from the Profiler.

$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-analytics/metron-profiler
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be 
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
%functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, 
CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, 
FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, 
HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, 
IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, 
MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, 
OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, 
PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, 
PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, 
STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, 
STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, 
STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, 
STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, 
SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, 
TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, 
URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR


And these functions are available in the Enrichment topology.


$ mvn exec:java \
-Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
-pl metron-platform/metron-enrichment/
...
Stellar, Go!
Please note that functions are loading lazily in the background and will be 
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, 
CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD,

Re: Install Metron 0.4.0 on CentOS 7 with MySQL (MariaDB) for Metron REST.

[Matt Foley]

Hi Laurens,
Some related material is posted under 
https://cwiki.apache.org/confluence/display/METRON/Installation

We’d be happy to have another experience recorded, since you found the existing 
docs inadequate.
If you need help committing to the wiki, you can ping me.

Thanks,
--Matt

On 5/8/17, 3:08 PM, "Laurens Vets"  wrote:

Hi list,

I'm not sure where to post this, but I've got a simple document which 
explains installing Metron 0.4.0. I've been trying to install Metron 
0.4.0 in 3 VMs the past couple of days and with the help of Ryan, Jon & 
Otto succeeded today.
I've got Metron 0.4.0 installed on CentOS 7 with a MariaDB database for 
the REST interface.

What or where would be the best place to post this?