[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124709330
  
--- Diff: metron-analytics/metron-maas-service/README.md ---
@@ -138,7 +138,7 @@ Now that we have a deployed model, let's adjust the 
configurations for the Squid
 * Edit the squid parser configuration at 
`$METRON_HOME/config/zookeeper/parsers/squid.json` in your favorite text editor 
and add a new FieldTransformation to indicate a threat alert based on the model 
(note the addition of `is_malicious` and `is_alert`):
 ```
 {
-  "parserClassName": "org.apache.metron.parsers.GrokParser",
+  "parserClassName": "org.apache.metron.parsers.grok.GrokParser",
--- End diff --

Also, extensions installed using rest ( and the future ui I hope ;) ) can 
be uninstalled


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/530
  
I have added deployment readme files to document where things get deployed 
to.
This if for the built in system parser extensions.  The 3rd party 
extensions are deployed by the rest api in METRON-942, and should be documented 
there based on feedback


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124703942
  
--- Diff: metron-maven-archetypes/README.md ---
@@ -0,0 +1,56 @@
+# Metron Maven Archetypes
+
+These are [Apache Maven 
Archetypes](http://maven.apache.org/archetype/index.html) for use in creating 
extension components for the Apache Metron system
+Archetypes are helpful in creating maven based projects with the correct 
setup and dependency configurations for the target system, including providing 
sample implementations.
+
+### metron-maven-parser-extension-archetype
+This is an archetype for creating an Apache Metron Parser Extension
+
+ Use
+
+Build and install the archetype
+```
+$ mvn install
+```
+
+Create a directory to host your extension code
+
+```
+$ mkdir ~/src/my-parser-extension
+$ cd ~/src/my-parser-extension
+```
+
+Use the archetype to create your project
+
+```
+$ mvn archetype:generate -DarchetypeCatalog=local
+[Select the org.apache.metron:metron-maven-parser-extension-archetype 
(Apache Maven Parser Extension Archetype for Metron) entry]
+```
+
+Configure the project properties.  Ending up with something like this:
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124703981
  
--- Diff: bundles-lib/pom.xml ---
@@ -0,0 +1,179 @@
+

Re: Build failures

[Otto Fowler]

Also, if you are following an article or some documentation, can you tells
what those are?


On June 28, 2017 at 16:43:33, Ryan Merriman (merrim...@gmail.com) wrote:

Can you confirm you're on the master branch? I see "metron-streaming" in
your path to RestTestingUtil and that was changed a LONG time ago. You're
likely on a really old branch.

Ryan

On Wed, Jun 28, 2017 at 3:27 PM, Vasco Yordanov 

wrote:

> Hello , I just forked from github and it seems that "
Metron-Pcap_Service"
> is failing with following errors:
>
>
> [ERROR] /home/vasko/metron2/incubator-metron-fork/metron-streaming/
> Metron-Pcap_Service/src/main/java/org/apache/metron/
> pcapservice/RestTestingUtil.java:[212,5] cannot find symbol
> [ERROR] symbol: class ResponseEntity
> [ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
> [ERROR] /home/vasko/metron2/incubator-metron-fork/metron-streaming/
> Metron-Pcap_Service/src/main/java/org/apache/metron/
> pcapservice/RestTestingUtil.java:[212,63] cannot find symbol
> [ERROR] symbol: variable HttpMethod
> [ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
>
> Please advise ? Before I start changing pom files ,I 'd like to run this
> through you in case this is known issue. Thank you
> From: merrimanr 
> To: dev@metron.apache.org
> Sent: Wednesday, June 28, 2017 4:05 PM
> Subject: [GitHub] metron issue #620: Metron-988: UI for viewing alerts
> generated by Metron
>
> Github user merrimanr commented on the issue:
>
> https://github.com/apache/metron/pull/620
>
> Just tested again and I am able to now remove the first filter and
> properly filter on values with special characters (referrer field for
> example). I did another pass and found some trivial issues as well as a
> few non-trivial issues and have made comments.
>
> I think more thought needs to be put into the AlertService.search and
> AlertService.pollSearch functions. The AlertService.getAlert function is
> very clear to me: it requires a couple of clearly named parameters and I
> expect to get an 'Alert' type object back. The other functions in this
> service are not as clear. The search function for example takes in a
> QueryBuilder object which provides a generic javascript object as the
body
> for the post request. Then in return the post returns an Observable with
a
> generic javascript object. So essentially Typescript isn't being used
here
> when it should because it would make the search interface clearer.
>
> For example, I would prefer this function signature:
> `public search(searchRequest: SearchRequest):
> Observable`
>
> where SearchRequest and SearchResponse are model objects. The way it
> is now it's not easy to understand what is being sent and what is
expected
> back unless you've spent time tracing the search calls to where requests
> are built/response are processed and know all the source code well OR
> already has a lot of experience with the ES query syntax.
>
> The result of all this is that not having a clear contract between the
> search client/server will make developing a middle-tier more tedious.
>
>
> ---
> If your project is set up for it, you can reply to this email and have
your
> reply appear on GitHub as well. If your project does not have this
feature
> enabled and wishes so, or if the feature is enabled but not working,
please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
>
>

Re: Build failures

[Ryan Merriman]

Can you confirm you're on the master branch?  I see "metron-streaming" in
your path to RestTestingUtil and that was changed a LONG time ago.  You're
likely on a really old branch.

Ryan

On Wed, Jun 28, 2017 at 3:27 PM, Vasco Yordanov 
wrote:

> Hello , I just forked from github and it seems that " Metron-Pcap_Service"
> is failing with following errors:
>
>
> [ERROR] /home/vasko/metron2/incubator-metron-fork/metron-streaming/
> Metron-Pcap_Service/src/main/java/org/apache/metron/
> pcapservice/RestTestingUtil.java:[212,5] cannot find symbol
> [ERROR] symbol:   class ResponseEntity
> [ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
> [ERROR] /home/vasko/metron2/incubator-metron-fork/metron-streaming/
> Metron-Pcap_Service/src/main/java/org/apache/metron/
> pcapservice/RestTestingUtil.java:[212,63] cannot find symbol
> [ERROR] symbol:   variable HttpMethod
> [ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
>
> Please advise ? Before I start changing pom files ,I 'd like to run this
> through you in case this is known issue. Thank you
>   From: merrimanr 
>  To: dev@metron.apache.org
>  Sent: Wednesday, June 28, 2017 4:05 PM
>  Subject: [GitHub] metron issue #620: Metron-988: UI for viewing alerts
> generated by Metron
>
> Github user merrimanr commented on the issue:
>
> https://github.com/apache/metron/pull/620
>
> Just tested again and I am able to now remove the first filter and
> properly filter on values with special characters (referrer field for
> example).  I did another pass and found some trivial issues as well as a
> few non-trivial issues and have made comments.
>
> I think more thought needs to be put into the AlertService.search and
> AlertService.pollSearch functions.  The AlertService.getAlert function is
> very clear to me:  it requires a couple of clearly named parameters and I
> expect to get an 'Alert' type object back.  The other functions in this
> service are not as clear.  The search function for example takes in a
> QueryBuilder object which provides a generic javascript object as the body
> for the post request.  Then in return the post returns an Observable with a
> generic javascript object.  So essentially Typescript isn't being used here
> when it should because it would make the search interface clearer.
>
> For example, I would prefer this function signature:
> `public search(searchRequest: SearchRequest):
> Observable`
>
> where SearchRequest and SearchResponse are model objects.  The way it
> is now it's not easy to understand what is being sent and what is expected
> back unless you've spent time tracing the search calls to where requests
> are built/response are processed and know all the source code well OR
> already has a lot of experience with the ES query syntax.
>
> The result of all this is that not having a clear contract between the
> search client/server will make developing a middle-tier more tedious.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
>
>

Build failures

[Vasco Yordanov]

Hello , I just forked from github and it seems that " Metron-Pcap_Service" is 
failing with following errors:


[ERROR] 
/home/vasko/metron2/incubator-metron-fork/metron-streaming/Metron-Pcap_Service/src/main/java/org/apache/metron/pcapservice/RestTestingUtil.java:[212,5]
 cannot find symbol
[ERROR] symbol:   class ResponseEntity
[ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil
[ERROR] 
/home/vasko/metron2/incubator-metron-fork/metron-streaming/Metron-Pcap_Service/src/main/java/org/apache/metron/pcapservice/RestTestingUtil.java:[212,63]
 cannot find symbol
[ERROR] symbol:   variable HttpMethod
[ERROR] location: class org.apache.metron.pcapservice.RestTestingUtil

Please advise ? Before I start changing pom files ,I 'd like to run this 
through you in case this is known issue. Thank you
  From: merrimanr 
 To: dev@metron.apache.org 
 Sent: Wednesday, June 28, 2017 4:05 PM
 Subject: [GitHub] metron issue #620: Metron-988: UI for viewing alerts 
generated by Metron
   
Github user merrimanr commented on the issue:

    https://github.com/apache/metron/pull/620
  
    Just tested again and I am able to now remove the first filter and properly 
filter on values with special characters (referrer field for example).  I did 
another pass and found some trivial issues as well as a few non-trivial issues 
and have made comments.
    
    I think more thought needs to be put into the AlertService.search and 
AlertService.pollSearch functions.  The AlertService.getAlert function is very 
clear to me:  it requires a couple of clearly named parameters and I expect to 
get an 'Alert' type object back.  The other functions in this service are not 
as clear.  The search function for example takes in a QueryBuilder object which 
provides a generic javascript object as the body for the post request.  Then in 
return the post returns an Observable with a generic javascript object.  So 
essentially Typescript isn't being used here when it should because it would 
make the search interface clearer.
    
    For example, I would prefer this function signature:
    `public search(searchRequest: SearchRequest): Observable`
    
    where SearchRequest and SearchResponse are model objects.  The way it is 
now it's not easy to understand what is being sent and what is expected back 
unless you've spent time tracing the search calls to where requests are 
built/response are processed and know all the source code well OR already has a 
lot of experience with the ES query syntax.
    
    The result of all this is that not having a clear contract between the 
search client/server will make developing a middle-tier more tedious.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


   

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

[JonZeolla]

GitHub user JonZeolla reopened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/metron ~/metron-508/metron
cd ~/metron-508/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail 

[GitHub] metron issue #620: Metron-988: UI for viewing alerts generated by Metron

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/620
  
Just tested again and I am able to now remove the first filter and properly 
filter on values with special characters (referrer field for example).  I did 
another pass and found some trivial issues as well as a few non-trivial issues 
and have made comments.

I think more thought needs to be put into the AlertService.search and 
AlertService.pollSearch functions.  The AlertService.getAlert function is very 
clear to me:  it requires a couple of clearly named parameters and I expect to 
get an 'Alert' type object back.  The other functions in this service are not 
as clear.  The search function for example takes in a QueryBuilder object which 
provides a generic javascript object as the body for the post request.  Then in 
return the post returns an Observable with a generic javascript object.  So 
essentially Typescript isn't being used here when it should because it would 
make the search interface clearer.

 For example, I would prefer this function signature:
`public search(searchRequest: SearchRequest): Observable`

where SearchRequest and SearchResponse are model objects.  The way it is 
now it's not easy to understand what is being sent and what is expected back 
unless you've spent time tracing the search calls to where requests are 
built/response are processed and know all the source code well OR already has a 
lot of experience with the ES query syntax.

The result of all this is that not having a clear contract between the 
search client/server will make developing a middle-tier more tedious.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Apache Metron 0.4.0 release

[Anand Subramanian]

+1 (non-binding)

* Brought up Metron stack on 12-node CentOS7 openstack cluster
* Verify all services come up fine [PASS]
* Bro, YAF and snort - ingest into respective kafka topics and write indices 
[PASS]
* Add squid telemetry, ingest into kafka topic and write indices [PASS]
* Metron YAF Zeppelin dashboard with sample ingested YAF data [PASS]
* Management UI and REST Swagger UI sanity check [PASS]


-Anand 





On 6/28/17, 12:06 AM, "Matt Foley"  wrote:

>This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”.
>(Note: this is rc4 because the release candidate needed to be modified with 
>another commit after the rc3 tag was pushed to public.)
>
>Full list of changes in this release:
>https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES
>
>The tag/commit to be voted upon is:
>d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4)
>
>The source archive being voted upon can be found here:
>https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/apache-metron-0.4.0-rc4.tar.gz
>and in github at:
>https://github.com/apache/metron/tree/Metron_0.4.0 
>
>Other release files, signatures and digests can be found here:
>https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
>
>The release artifacts are signed with the following key:
>https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS
>pub   rsa4096/4169AA27ECB31663 2011-07-31 [SCEA]
>Key fingerprint = 7854 36A7 8258 6B71 829C  67A0 4169 AA27 ECB3 1663
>uid = Matthew Foley (CODE SIGNING KEY) 
>
>Please vote on releasing this package as Apache Metron 0.4.0.
>When voting, please list the actions taken to verify the release.
>
>Recommended build validation and verification instructions are posted here:
>https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds
>
>This vote will be open for at least 72 hours.  Please vote one of the 
>following responses:
>+1 Release this package as Apache Metron 0.4.0-RC4
>0 No opinion
>-1 Do not release this package because...
> 
>Thank you,
>--Matt
>(your friendly release manager)
>
>
>

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124629201
  
--- Diff: 
metron-interface/metron-alerts/src/app/shared/metron-table/metron-table-pagination/metron-table-pagination.component.scss
 ---
@@ -0,0 +1,30 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+i {
+  cursor: pointer;
+  margin: 0px 10px;
+}
+
+.disabled {
+  opacity: 0.3;
+  cursor: default;
+}
+
+span {
+  font-size: 14px;
+}
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124629155
  
--- Diff: 
metron-interface/metron-alerts/src/app/shared/metron-table/metron-table-pagination/metron-table-pagination.component.html
 ---
@@ -0,0 +1,20 @@
+
+
+  
+  
+   {{pagination.from + 1}} - {{pagination.total > (pagination.from + 
pagination.size) ? (pagination.from + pagination.size) : pagination.total}} of 
{{ pagination.total }} 
+  
+  
+
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124628897
  
--- Diff: 
metron-interface/metron-alerts/src/app/shared/list-group/list-group.component.scss
 ---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ul {
+  margin: 10px 0px;
+}
+.list-group-item {
+  span {
+padding-left: 25px;
+  }
+
+  i+span {
+padding-left: 10px;
+  }
+}
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124628855
  
--- Diff: 
metron-interface/metron-alerts/src/app/shared/list-group/list-group.component.html
 ---
@@ -0,0 +1,19 @@
+
+
+  
+
+{{ item.name }} 
+  
+
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124628476
  
--- Diff: 
metron-interface/metron-alerts/src/app/shared/collapse/collapse.component.html 
---
@@ -0,0 +1,31 @@
+
+
+  {{ data.getName() }}
+
+
+  
+ {{ 'No ' +  data.getName() + ' yet'}} 
+  
+  
+
+  {{ data.key | centerEllipses: strLength }}
+   
{{ data.doc_count }} 
+   
+
+  
+
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124627945
  
--- Diff: 
metron-interface/metron-alerts/src/app/service/cluster-metadata.service.ts ---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Injectable, Inject} from '@angular/core';
+import {Observable} from 'rxjs/Rx';
+import {Http, Headers, RequestOptions} from '@angular/http';
+import {HttpUtil} from '../utils/httpUtil';
+import {IAppConfig} from '../app.config.interface';
+import {APP_CONFIG} from '../app.config';
+import {MetadataUtil} from '../utils/metadata-utils';
+import {ColumnMetadata} from '../model/column-metadata';
+
+@Injectable()
+export class ClusterMetaDataService {
+  defaultHeaders = {'Content-Type': 'application/json', 
'X-Requested-With': 'XMLHttpRequest'};
+  url = '_cluster/state';
+
+  defaultColumnMetadata = [
+new ColumnMetadata('_id', 'string'),
+new ColumnMetadata('timestamp', 'date'),
+new ColumnMetadata('source:type', 'string'),
+new ColumnMetadata('ip_src_addr', 'ip'),
+new ColumnMetadata('enrichments:geo:ip_dst_addr:country', 'string'),
+new ColumnMetadata('ip_dst_addr', 'ip'),
+new ColumnMetadata('host', 'string'),
+new ColumnMetadata('alert_status', 'string')
+  ];
+
+  constructor(private http: Http, @Inject(APP_CONFIG) private config: 
IAppConfig) {
+  }
+
+  getDefaultColumns(): Observable {
+return Observable.create(observer => {
+  
observer.next(JSON.parse(JSON.stringify(this.defaultColumnMetadata)));
+  observer.complete();
+});
+  }
+
+  getColumnMetaData(): Observable {
--- End diff --

shouldn't this function return a ColumnMetadata typed object?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124627626
  
--- Diff: metron-interface/metron-alerts/src/app/service/alert.service.ts 
---
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import {Injectable, Inject, NgZone} from '@angular/core';
+import {Observable} from 'rxjs/Rx';
+import 'rxjs/add/observable/interval';
+import 'rxjs/add/operator/switchMap';
+import 'rxjs/add/operator/onErrorResumeNext';
+
+import {Alert} from '../model/alert';
+import {Http, Headers, RequestOptions} from '@angular/http';
+import {HttpUtil} from '../utils/httpUtil';
+import {IAppConfig} from '../app.config.interface';
+import {APP_CONFIG} from '../app.config';
+import {QueryBuilder} from '../model/query-builder';
+
+@Injectable()
+export class AlertService {
+
+  interval = 8;
+  defaultHeaders = {'Content-Type': 'application/json', 
'X-Requested-With': 'XMLHttpRequest'};
+
+  constructor(private http: Http,
+  @Inject(APP_CONFIG) private config: IAppConfig,
+  private ngZone: NgZone) {
+  }
+
+  public search(queryBuilder: QueryBuilder): Observable<{}> {
+let url = '/search/*,-*kibana/_search';
+return this.http.post(url, queryBuilder.getESSearchQuery(), new 
RequestOptions({headers: new Headers(this.defaultHeaders)}))
+  .map(HttpUtil.extractData)
+  .catch(HttpUtil.handleError);
+  }
+
+  public pollSearch(queryBuilder: QueryBuilder): Observable<{}> {
--- End diff --

same as above.  shouldn't the result of a search be a typed object?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124627293
  
--- Diff: metron-interface/metron-alerts/src/app/model/query-builder.ts ---
@@ -0,0 +1,160 @@
+import {Filter} from './filter';
+import {ColumnNamesService} from '../service/column-names.service';
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+export class QueryBuilder {
+  private _query = '*';
+  private _displayQuery = this._query;
+  private from = 0;
+  private size = 15;
+  private sort: {}[] = [{ timestamp: {order : 'desc', ignore_unmapped: 
true, unmapped_type: 'date'} }];
+  private aggs: {};
+  private _filters: Filter[] = [];
+
+  static fromJSON(obj: QueryBuilder): QueryBuilder {
+let queryBuilder = new QueryBuilder();
+queryBuilder._query = obj._query;
+queryBuilder._displayQuery = obj._displayQuery;
+queryBuilder.from = obj.from;
+queryBuilder.size = obj.size;
+queryBuilder.sort = obj.sort;
+queryBuilder.aggs = obj.aggs;
+queryBuilder._filters = obj._filters;
+queryBuilder.onSearchChange();
+
+return queryBuilder;
+  }
+
+  set query(value: string) {
+value = value.replace(/\\:/g, ':');
+this._query = value;
+this.updateFilters(this._query, false);
+this.onSearchChange();
+  }
+
+  get query(): string {
+return this._query;
+  }
+
+  set displayQuery(value: string) {
+this._displayQuery = value;
+this.updateFilters(this._displayQuery, true);
+this.onSearchChange();
+  }
+
+  get displayQuery(): string {
+return this._displayQuery;
+  }
+
+  get filters(): Filter[] {
+return this._filters;
+  }
+
+  addOrUpdateFilter(field: string, value: string) {
+let filter = this._filters.find(tFilter => tFilter.field === field);
+if (filter) {
+  filter.value = value;
+} else {
+  this._filters.push(new Filter(field, value));
+}
+
+this.onSearchChange();
+  }
+
+  asString(): string {
+let json = JSON.stringify(this.getESSearchQuery());
+json = json.replace(/"/g, '').replace(/^{/, '').replace(/}$/, '');
+
+return json;
+  }
+
+  generateSelect() {
+let select = this._filters.map(filter => {
+  return filter.field.replace(/:/g, '\\:') +
+  ':' +
+String(filter.value)
+  .replace(/[\*\+\-=~><\"\?^\${}\(\)\:\!\/[\]\\\s]/g, '\\$&') // 
replace single  special characters
+  .replace(/\|\|/g, '\\||') // replace ||
+  .replace(/\&\&/g, '\\&&'); // replace &&
+}).join(' AND ');
+return (select.length === 0) ? '*' : select;
+  }
+
+  generateSelectForDisplay() {
+let select = this._filters.map(filter => 
ColumnNamesService.getColumnDisplayValue(filter.field) + ':' + 
filter.value).join(' AND ');
+return (select.length === 0) ? '*' : select;
+  }
+
+  getESSearchQuery() {
+return {
+  query: { query_string: { query: this.generateSelect() } },
+  from: this.from,
+  size: this.size,
+  sort: this.sort,
+  aggs: {}
+};
+  }
+
+  onSearchChange() {
+this._query = this.generateSelect();
+this._displayQuery = this.generateSelectForDisplay();
+  }
+
+  removeFilter(field: string) {
+let filter = this._filters.find(tFilter => tFilter.field === field);
+this._filters.splice(this._filters.indexOf(filter), 1);
+
+this.onSearchChange();
+  }
+
+  setAggregations(value: any) {
--- End diff --

should aggregations be a model object?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at 

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124626694
  
--- Diff: metron-interface/metron-alerts/src/app/app.component.scss ---
@@ -0,0 +1,25 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+@import "../variables";
+
+.navbar
+{
+  background: $nav-bar-bg;
+  padding: 0rem 1rem;
+  max-height: 50px;
+  }
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124626493
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.scss
 ---
@@ -0,0 +1,22 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+@import "../../../variables";
+
+.form-title, .close-button {
+  color: $silver;
+}
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124626347
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/save-search/save-search.component.scss
 ---
@@ -0,0 +1,22 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+@import "../../../variables";
+
+.form-title, .close-button {
+  color: $silver;
+}
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124625631
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/configure-rows/configure-rows.component.html
 ---
@@ -0,0 +1,45 @@
+
+
+
+  
+Settings
+
+   REFRESH RATE 
+  
+5s   
+10s  
+15s  
+30s  
+1m   
+ 10m  
+1h   
+  
+   ROWS PER PAGE 
+  
+  10   
+  25   
+  50   
+  100  
+  250  
+  500  
+  1000  
+  
+
+   
+   
+
+
+  
+
--- End diff --

need a newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124625253
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
 ---
@@ -0,0 +1,371 @@
+import {Component, OnInit, ViewChild, ElementRef} from '@angular/core';
+import {Router, NavigationStart} from '@angular/router';
+import {Observable, Subscription} from 'rxjs/Rx';
+
+import {Alert} from '../../model/alert';
+import {AlertService} from '../../service/alert.service';
+import {QueryBuilder} from '../../model/query-builder';
+import {ConfigureTableService} from 
'../../service/configure-table.service';
+import {WorkflowService} from '../../service/workflow.service';
+import {ClusterMetaDataService} from 
'../../service/cluster-metadata.service';
+import {ColumnMetadata} from '../../model/column-metadata';
+import {SortEvent} from '../../shared/metron-table/metron-table.directive';
+import {Sort} from '../../utils/enums';
+import {Pagination} from '../../model/pagination';
+import {SaveSearchService} from '../../service/save-search.service';
+import {RefreshInterval} from '../configure-rows/configure-rows-enums';
+import {SaveSearch} from '../../model/save-search';
+import {TableMetadata} from '../../model/table-metadata';
+import {MetronDialogBox, DialogType} from '../../shared/metron-dialog-box';
+import {MetadataUtil} from '../../utils/metadata-utils';
+import {AlertSearchDirective} from 
'../../shared/directives/alert-search.directive';
+
+@Component({
+  selector: 'app-alerts-list',
+  templateUrl: './alerts-list.component.html',
+  styleUrls: ['./alerts-list.component.scss']
+})
+
+export class AlertsListComponent implements OnInit {
+
+  alertsColumns: ColumnMetadata[] = [];
+  alertsColumnsToDisplay: ColumnMetadata[] = [];
+  selectedAlerts: Alert[] = [];
+  alerts: any[] = [];
--- End diff --

should the type be Alert (and not any)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

[JonZeolla]

GitHub user JonZeolla reopened a pull request:

https://github.com/apache/metron/pull/586

METRON-508 Expand Elasticsearch templates to support the standard bro logs

## Contributor Comments
This PR makes it easier for someone with an existing bro install to send 
some of their log files into Metron, based off of a combination of the [bro 
documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and 
a fresh install of bro 2.5.  There are future plans to expand on this via 
[METRON-518](https://issues.apache.org/jira/browse/METRON-518) and 
[METRON-908](https://issues.apache.org/jira/browse/METRON-908).  Specifically, 
this attempts to provide initial support the default-on fields of the following 
logs:
 - 
[Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info)
 - 
[DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info)
 - 
[FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info)
 - 
[Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info)
 - 
[CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo)
 - 
[SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info)
 - 
[SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info)
 - 
[Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info)
 - 
[Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info)
 - 
[DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info)
 - 
[SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info)
 - 
[Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info)
 - 
[Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info)
 - 
[X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info)
 - 
[DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo)


## Testing
1.  Create a working directory and pull in this PR
```
mkdir ~/metron-508
git clone https://github.com/apache/metron ~/metron-508/metron
cd ~/metron-508/metron
git remote add jonzeolla https://github.com/jonzeolla/metron
git pull jonzeolla METRON-508
```
1.  Modify 
[this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20)
 to remove `sensors,` (to spin up the real sensors).
```
sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" 
metron-deployment/vagrant/full-dev-platform/Vagrantfile
```
1.  Start up full-dev.
```
cd metron-deployment/vagrant/full-dev-platform
vagrant up
```
1.  Set up the environment in full-dev.
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin
service monit stop && service sensor-stubs stop bro && broctl stop
yum -y install jq wireshark
```
1.  Configure kafka in local.bro.
```
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = 
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, 
SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, 
Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' 
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::debug = \"all\";" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >> 
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load 
policy/protocols/dhcp/known-devices-and-hostnames.bro' 
/usr/local/bro/share/bro/site/local.bro
```
1.  Monitor the bro kafka topic
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
```
1.  Monitor the storm logs.
```
# Open a new terminal
cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
# Look at the storm logs (The "failed to parse" errors for ip_src_addr 
and ip_dst_addr are expected, and should be addressed as a part of METRON-939)
tail 

[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...

[JonZeolla]

Github user JonZeolla closed the pull request at:

https://github.com/apache/metron/pull/586


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124623532
  
--- Diff: metron-interface/metron-alerts/scripts/package.json ---
@@ -0,0 +1,21 @@
+{
+  "name": "metron-management-ui-web-server",
--- End diff --

Was this copied from metron-config?  I think this file needs to be updated 
to match this module.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #620: Metron-988: UI for viewing alerts generated by Met...

[merrimanr]

Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/620#discussion_r124622851
  
--- Diff: metron-interface/metron-alerts/scripts/prepend_license_header.sh 
---
@@ -0,0 +1,42 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+LICENSE_HEADER="/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an \"AS IS\" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ "
+for file in ./dist/*.js
+do
+if !(grep -Fxq "$LICENSE_HEADER" $file)
+then
+echo "$LICENSE_HEADER$(cat $file)" > $file
+fi
+done
--- End diff --

Need a return character here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #618: METRON-999 Add virtualization support checks to pl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/618


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Question about Metron-Docker

[Otto Fowler]

Can you verify that it is working and that I’m just messing up running it?



On June 28, 2017 at 13:14:43, Kyle Richardson (kylerichards...@gmail.com) wrote:

I use it occasionally. I like it for quickly testing parser tweaks. 
Unfortunately I've had zero time lately so haven't run it off master for a 
while.  

-Kyle  

> On Jun 28, 2017, at 11:10 AM, Otto Fowler  wrote:  
>  
> Has anyone run it off master?  
> Is anyone using it?  
>  
>  

[GitHub] metron pull request #530: METRON-777 Metron Extension System and Parser Exte...

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/530#discussion_r124611122
  
--- Diff: metron-maven-archetypes/README.md ---
@@ -0,0 +1,56 @@
+# Metron Maven Archetypes
+
+These are [Apache Maven 
Archetypes](http://maven.apache.org/archetype/index.html) for use in creating 
extension components for the Apache Metron system
+Archetypes are helpful in creating maven based projects with the correct 
setup and dependency configurations for the target system, including providing 
sample implementations.
+
+### metron-maven-parser-extension-archetype
+This is an archetype for creating an Apache Metron Parser Extension
+
+ Use
+
+Build and install the archetype
+```
+$ mvn install
+```
+
+Create a directory to host your extension code
+
+```
+$ mkdir ~/src/my-parser-extension
+$ cd ~/src/my-parser-extension
+```
+
+Use the archetype to create your project
+
+```
+$ mvn archetype:generate -DarchetypeCatalog=local
+[Select the org.apache.metron:metron-maven-parser-extension-archetype 
(Apache Maven Parser Extension Archetype for Metron) entry]
+```
+
+Configure the project properties.  Ending up with something like this:
--- End diff --

Just some formatting tweaks - the properties run together as one line in 
the current layout. Maybe try this

```
Confirm properties configuration:

* groupId: org.someorg
* artifactId: someparser
* version: 1.0-SNAPSHOT
* package: org.someorg.parsers
* metronVersion: 0.4.0
* parserClassName: Nice
* parserName: nice
```

Which turns out like this:

Confirm properties configuration:

* groupId: org.someorg
* artifactId: someparser
* version: 1.0-SNAPSHOT
* package: org.someorg.parsers
* metronVersion: 0.4.0
* parserClassName: Nice
* parserName: nice


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Question about Metron-Docker

[Kyle Richardson]

I use it occasionally. I like it for quickly testing parser tweaks. 
Unfortunately I've had zero time lately so haven't run it off master for a 
while.

-Kyle

> On Jun 28, 2017, at 11:10 AM, Otto Fowler  wrote:
> 
> Has anyone run it off master?
> Is anyone using it?
> 
>