[jira] [Commented] (METRON-1510) Update Metron website to include info about github update subscription

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426486#comment-16426486
 ] 

ASF GitHub Bot commented on METRON-1510:


GitHub user anandsubbu opened a pull request:

https://github.com/apache/metron/pull/981

METRON-1510: Update Metron website to include info about github update 
subscription


## Contributor Comments
Update to include links for the 'Issues' mailing list. Have also changed 
Archives links to point to lists.apache.org, which has a better viewing than 
the earlier one. 

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-1510

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/981.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #981


commit e2889419a7fafe5b637fd6d49e829061c009b62c
Author: Anand Subramanian 
Date:   2018-04-05T04:59:37Z

Added links to Issues. Also changed Archives to point to lists.apache.org 
which has better look and feel.




> Update Metron website to include info about github update subscription
> --
>
> Key: METRON-1510
> URL: https://issues.apache.org/jira/browse/METRON-1510
> Project: Metron
>  Issue Type: Bug
>Reporter: Anand Subramanian
>Priority: Major
>
> Per the discussion 
> [here|https://lists.apache.org/thread.html/9a450a46701eb0924d431a042737b904ab344eb052f630b112a7e5a6@%3Cdev.metron.apache.org%3E],
>  github updates have been moved to a separate issues list at 
> https://lists.apache.org/list.html?issues@metron.apache.org
> The Metron website needs an update to reflect the same.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #981: METRON-1510: Update Metron website to include info...

[anandsubbu]

GitHub user anandsubbu opened a pull request:

https://github.com/apache/metron/pull/981

METRON-1510: Update Metron website to include info about github update 
subscription


## Contributor Comments
Update to include links for the 'Issues' mailing list. Have also changed 
Archives links to point to lists.apache.org, which has a better viewing than 
the earlier one. 

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anandsubbu/incubator-metron METRON-1510

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/981.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #981


commit e2889419a7fafe5b637fd6d49e829061c009b62c
Author: Anand Subramanian 
Date:   2018-04-05T04:59:37Z

Added links to Issues. Also changed Archives to point to lists.apache.org 
which has better look and feel.




---

[jira] [Created] (METRON-1510) Update Metron website to include info about github update subscription

[Anand Subramanian (JIRA)]

Anand Subramanian created METRON-1510:
-

 Summary: Update Metron website to include info about github update 
subscription
 Key: METRON-1510
 URL: https://issues.apache.org/jira/browse/METRON-1510
 Project: Metron
  Issue Type: Bug
Reporter: Anand Subramanian


Per the discussion 
[here|https://lists.apache.org/thread.html/9a450a46701eb0924d431a042737b904ab344eb052f630b112a7e5a6@%3Cdev.metron.apache.org%3E],
 github updates have been moved to a separate issues list at 
https://lists.apache.org/list.html?issues@metron.apache.org

The Metron website needs an update to reflect the same.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426480#comment-16426480
 ] 

ASF GitHub Bot commented on METRON-1434:


Github user lvets commented on the issue:

https://github.com/apache/metron/pull/916
  
@nickwallen, @ottobackwards , my 0.02$ :)

> For example, with everything in Metron today, you can stand-up a single 
node in AWS and use the Mpack to install Metron. It is not as "push button" 
simple as your contribution here, but it is "good enough" considering the 
resources we have in the community today.

Unfortunately, "push button" style setup is what most people want nowadays 
want to quickly set up a test environment and start playing around. I do 
understand that supporting different installation methods (including mine :p) 
adds a burden, but I think it is a burden we have to bear until Metron grows 
bigger.

> We should also consider that running Metron on a single node is a recipe 
for a horrible user experience. It should only be run on a single node for 
development purposes, which is something that we already do support. I would 
not recommend that anyone run Metron on a single node for any other purpose.

That's why AWS has a bunch of different VM types :) Maybe we should add a 
big fat warning somewhere saying "Your experience will be very bad if you not 
use at least an m5.4xlarge instance if you're going the single node way."

I think having quick & easy ways to setup Metron on a single node to play 
around is necessary until the Metron community grows...

Anyway, back to lurking mode for now.


> Ability to deploy Metron full dev as a single node (centos6) to AWS with 
> vagrant and AMI
> 
>
> Key: METRON-1434
> URL: https://issues.apache.org/jira/browse/METRON-1434
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
> Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 
> Cores, 64 bit] based on metron-deployment/development/centos6/
> ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS
> iii) AMI  "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: AWS, EC2, vagrant
> Fix For: 0.4.3
>
>
> The pull request has two potential solutions for deploying to Metron to AWS. 
> i) AWS Single Node Cluster Deployment Using Vagrant
> ii) AWS Single Node Cluster Deployment Using an AMI
> *i) AWS Single Node Cluster Deployment Using Vagrant:*
>  I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. 
> It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) 
> that was in git. I'm assuming the latest that was in git is 0.4.3 that is 
> progress.
> _Instructions:_
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant]
>  
> *ii) AWS Single Node Cluster Deployment Using an AMI:*
>  In addition to including instructions for deploying 0.4.3 using Vagrant on a 
> Mac I also included instructions for deploying to AWS using a preexisting AMI 
> (I made the following AMI public as a proof-of-concept: "GCR-Xetron 
> Demo"/"ami-93cb4ff7").  This might be the quicket option for deployment since 
> no preinstall requirments would be needed.
> _Instructions:_ 
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami]
>  
> PULL REQUEST CHECKLIST ITEMS
> *"For documentation related changes:"*
> Running cd site-book, mvn site produced "BUILD SUCCESS"
>  
> *"For code changes:"* 
> Was not sure where to run "mvn -q clean integration-test install && 
> dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not 
> changed. For i) the following code were newly added/modified:
> [Vagrant to work with aws]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile]
> [changing eth1 to eth0]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/inventory/group_vars/all]
>  
>  
> *"How the change may be verified: "*
> You should be able to see Ambari after following instructions 
> (http://:8080)
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #916: METRON-1434 - Ability to deploy Metron full dev as a sing...

[lvets]

Github user lvets commented on the issue:

https://github.com/apache/metron/pull/916
  
@nickwallen, @ottobackwards , my 0.02$ :)

> For example, with everything in Metron today, you can stand-up a single 
node in AWS and use the Mpack to install Metron. It is not as "push button" 
simple as your contribution here, but it is "good enough" considering the 
resources we have in the community today.

Unfortunately, "push button" style setup is what most people want nowadays 
want to quickly set up a test environment and start playing around. I do 
understand that supporting different installation methods (including mine :p) 
adds a burden, but I think it is a burden we have to bear until Metron grows 
bigger.

> We should also consider that running Metron on a single node is a recipe 
for a horrible user experience. It should only be run on a single node for 
development purposes, which is something that we already do support. I would 
not recommend that anyone run Metron on a single node for any other purpose.

That's why AWS has a bunch of different VM types :) Maybe we should add a 
big fat warning somewhere saying "Your experience will be very bad if you not 
use at least an m5.4xlarge instance if you're going the single node way."

I think having quick & easy ways to setup Metron on a single node to play 
around is necessary until the Metron community grows...

Anyway, back to lurking mode for now.


---

[jira] [Commented] (METRON-1509) Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 Single Node/Vagrant Method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426310#comment-16426310
 ] 

ASF GitHub Bot commented on METRON-1509:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/980
  
+1 Thanks @as22323!


> Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 Single 
> Node/Vagrant Method
> -
>
> Key: METRON-1509
> URL: https://issues.apache.org/jira/browse/METRON-1509
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: Next + 1
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
> Fix For: Next + 1
>
>
> Otto and Nickwallen in Jira:METRON-1434 & PR 
> [916|https://github.com/apache/metron/pull/916] suggested that a proposed 
> AWS/Vagrant method be put into a doc and/or be referred to in a doc. This 
> request updates a Metron doc to include an external community deployment 
> method as a reference in a doc. 
> This updates /metron-deployment/other-examples/README.md to include an 
> external reference to the following deployment method ""Metron 0.4.1 - AWS 
> EC2 Single Node Deployment with Vagrant"" 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #980: METRON-1509 Docs - metron-deployment/other-examples - Add...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/980
  
+1 Thanks @as22323!


---

[jira] [Commented] (METRON-1339) Stellar Shell: Should have a way to validate deployed functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426234#comment-16426234
 ] 

ASF GitHub Bot commented on METRON-1339:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
It has been long enough that **I** don't even like this PR any more.

-1


> Stellar Shell: Should have a way to validate deployed functions
> ---
>
> Key: METRON-1339
> URL: https://issues.apache.org/jira/browse/METRON-1339
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It is possible that the Stellar Language is changed during a release, meaning 
> stellar expressions that are in zookeeper are now invalid, and will not run.
> Currently users cannot verify this without just letting them fail and hunting 
> the errors down.
> The Stellar Shell should expose a '%' function that will verify that all 
> stellar functions compile and are syntactically correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
It has been long enough that **I** don't even like this PR any more.

-1


---

[jira] [Commented] (METRON-1339) Stellar Shell: Should have a way to validate deployed functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426206#comment-16426206
 ] 

ASF GitHub Bot commented on METRON-1339:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
The next attempt at this, if there is one, should start off with some sort 
of consensus first. And some agreement on initial scope.  This PR would have 
been smaller and less ambitious, if that were true.  Or at least everyone would 
have been on the same page as to the 'why' of certain things.
Lessons to learn :)



> Stellar Shell: Should have a way to validate deployed functions
> ---
>
> Key: METRON-1339
> URL: https://issues.apache.org/jira/browse/METRON-1339
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It is possible that the Stellar Language is changed during a release, meaning 
> stellar expressions that are in zookeeper are now invalid, and will not run.
> Currently users cannot verify this without just letting them fail and hunting 
> the errors down.
> The Stellar Shell should expose a '%' function that will verify that all 
> stellar functions compile and are syntactically correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
The next attempt at this, if there is one, should start off with some sort 
of consensus first. And some agreement on initial scope.  This PR would have 
been smaller and less ambitious, if that were true.  Or at least everyone would 
have been on the same page as to the 'why' of certain things.
Lessons to learn :)



---

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/873
  
Well, if it helped, cool, let’s keep it... was just curious about your 
use case and whether we should have a bolder scope or if there is anything like 
this that would help elsewhere in general performance debugging. I’m 
certainly not against this in any way. :)

> On 4 Apr 2018, at 22:21, ottobackwards  wrote:
> 
> I'm shelving this. Thinks for taking a look.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 



---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426199#comment-16426199
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/873
  
Well, if it helped, cool, let’s keep it... was just curious about your use 
case and whether we should have a bolder scope or if there is anything like 
this that would help elsewhere in general performance debugging. I’m certainly 
not against this in any way. :)

> On 4 Apr 2018, at 22:21, ottobackwards  wrote:
> 
> I'm shelving this. Thinks for taking a look.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 



> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #856: METRON-1339 Stellar Shell functionality to verify ...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/856


---

[jira] [Commented] (METRON-1339) Stellar Shell: Should have a way to validate deployed functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426195#comment-16426195
 ] 

ASF GitHub Bot commented on METRON-1339:


Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/856


> Stellar Shell: Should have a way to validate deployed functions
> ---
>
> Key: METRON-1339
> URL: https://issues.apache.org/jira/browse/METRON-1339
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It is possible that the Stellar Language is changed during a release, meaning 
> stellar expressions that are in zookeeper are now invalid, and will not run.
> Currently users cannot verify this without just letting them fail and hunting 
> the errors down.
> The Stellar Shell should expose a '%' function that will verify that all 
> stellar functions compile and are syntactically correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1339) Stellar Shell: Should have a way to validate deployed functions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426194#comment-16426194
 ] 

ASF GitHub Bot commented on METRON-1339:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
This has gone from a small thing to at least 'say' we have a way to check 
if we broke all your stellar stuff after upgrade, to stretching it based on 
feedback which was a mistake, to a run around to abandoned review status.

I'm going shelve this



> Stellar Shell: Should have a way to validate deployed functions
> ---
>
> Key: METRON-1339
> URL: https://issues.apache.org/jira/browse/METRON-1339
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It is possible that the Stellar Language is changed during a release, meaning 
> stellar expressions that are in zookeeper are now invalid, and will not run.
> Currently users cannot verify this without just letting them fail and hunting 
> the errors down.
> The Stellar Shell should expose a '%' function that will verify that all 
> stellar functions compile and are syntactically correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #856: METRON-1339 Stellar Shell functionality to verify stored ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/856
  
This has gone from a small thing to at least 'say' we have a way to check 
if we broke all your stellar stuff after upgrade, to stretching it based on 
feedback which was a mistake, to a run around to abandoned review status.

I'm going shelve this



---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426187#comment-16426187
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
I'm shelving this.  Thinks for taking a look.



> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
I'm shelving this.  Thinks for taking a look.



---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[Otto Fowler (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426185#comment-16426185
 ] 

Otto Fowler commented on METRON-1367:
-

While this is cool IMHO, there doesn't seem to be much support for it. I'm 
going to shelve it.

 

> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[ottobackwards]

Github user ottobackwards closed the pull request at:

https://github.com/apache/metron/pull/873


---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426180#comment-16426180
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
@simonellistonball, i have read and replied to you last about 100 times, 
but now I understand.  The PR makes it seem like this is a PR for a major 
instrumentation/sampling solution, when I meant it as a Shell utility to start, 
not a replacement for the existing stellar speed test, but for a different use.

I am sorry for the confusion.  I found this useful in working ON the 
stellar language and functions. But it is not Apache Sirona or anything.  I am 
sorry for the miss understanding.


Thanks





> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
@simonellistonball, i have read and replied to you last about 100 times, 
but now I understand.  The PR makes it seem like this is a PR for a major 
instrumentation/sampling solution, when I meant it as a Shell utility to start, 
not a replacement for the existing stellar speed test, but for a different use.

I am sorry for the confusion.  I found this useful in working ON the 
stellar language and functions. But it is not Apache Sirona or anything.  I am 
sorry for the miss understanding.


Thanks





---

[jira] [Commented] (METRON-1449) Set Zookeeper URL for Stellar Running in Zeppelin Notebook

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426159#comment-16426159
 ] 

ASF GitHub Bot commented on METRON-1449:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/931
  
The last two commits are refactorings I performed trying to track down what 
I describe in the PR description as a "Known Problem".  I need to pull each of 
those out as separate PRs.


> Set Zookeeper URL for Stellar Running in Zeppelin Notebook
> --
>
> Key: METRON-1449
> URL: https://issues.apache.org/jira/browse/METRON-1449
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> A user needs to be able to configure a Zookeeper URL for the Stellar session 
> backing their Zeppelin Notebook.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #931: METRON-1449 Set Zookeeper URL for Stellar Running in Zepp...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/931
  
The last two commits are refactorings I performed trying to track down what 
I describe in the PR description as a "Known Problem".  I need to pull each of 
those out as separate PRs.


---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426155#comment-16426155
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/873
  
Did it work out for you to find bottlenecks @ottobackwards ? I would be 
more inclined to do multi-run micro-benchmarking (I have a bunch of 
multi-sample perfidix runs on all the parser work I've done for example) rather 
than a one of timer. Would a more 'proper' run it a thousand times in the 
proper context of the topology code performance profiler be useful / relevant. 
Did you get the info you need from this as a quicker (and quite possibly good 
enough!) approach?


> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/873
  
Did it work out for you to find bottlenecks @ottobackwards ? I would be 
more inclined to do multi-run micro-benchmarking (I have a bunch of 
multi-sample perfidix runs on all the parser work I've done for example) rather 
than a one of timer. Would a more 'proper' run it a thousand times in the 
proper context of the topology code performance profiler be useful / relevant. 
Did you get the info you need from this as a quicker (and quite possibly good 
enough!) approach?


---

[jira] [Commented] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426153#comment-16426153
 ] 

ASF GitHub Bot commented on METRON-1434:


Github user as22323 commented on the issue:

https://github.com/apache/metron/pull/916
  
Thanks for the great feedback! 
Although I was hoping this would make it into the main build, I understand 
what you mean by the "Support Burden". 

I found a place in the Metron docs to refer to the AWS/Vagrantfile.
I created the following pull request that updates a Metron Doc. 
https://github.com/apache/metron/pull/980


> Ability to deploy Metron full dev as a single node (centos6) to AWS with 
> vagrant and AMI
> 
>
> Key: METRON-1434
> URL: https://issues.apache.org/jira/browse/METRON-1434
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
> Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 
> Cores, 64 bit] based on metron-deployment/development/centos6/
> ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS
> iii) AMI  "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: AWS, EC2, vagrant
> Fix For: 0.4.3
>
>
> The pull request has two potential solutions for deploying to Metron to AWS. 
> i) AWS Single Node Cluster Deployment Using Vagrant
> ii) AWS Single Node Cluster Deployment Using an AMI
> *i) AWS Single Node Cluster Deployment Using Vagrant:*
>  I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. 
> It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) 
> that was in git. I'm assuming the latest that was in git is 0.4.3 that is 
> progress.
> _Instructions:_
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant]
>  
> *ii) AWS Single Node Cluster Deployment Using an AMI:*
>  In addition to including instructions for deploying 0.4.3 using Vagrant on a 
> Mac I also included instructions for deploying to AWS using a preexisting AMI 
> (I made the following AMI public as a proof-of-concept: "GCR-Xetron 
> Demo"/"ami-93cb4ff7").  This might be the quicket option for deployment since 
> no preinstall requirments would be needed.
> _Instructions:_ 
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami]
>  
> PULL REQUEST CHECKLIST ITEMS
> *"For documentation related changes:"*
> Running cd site-book, mvn site produced "BUILD SUCCESS"
>  
> *"For code changes:"* 
> Was not sure where to run "mvn -q clean integration-test install && 
> dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not 
> changed. For i) the following code were newly added/modified:
> [Vagrant to work with aws]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile]
> [changing eth1 to eth0]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/inventory/group_vars/all]
>  
>  
> *"How the change may be verified: "*
> You should be able to see Ambari after following instructions 
> (http://:8080)
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #916: METRON-1434 - Ability to deploy Metron full dev as a sing...

[as22323]

Github user as22323 commented on the issue:

https://github.com/apache/metron/pull/916
  
Thanks for the great feedback! 
Although I was hoping this would make it into the main build, I understand 
what you mean by the "Support Burden". 

I found a place in the Metron docs to refer to the AWS/Vagrantfile.
I created the following pull request that updates a Metron Doc. 
https://github.com/apache/metron/pull/980


---

[jira] [Commented] (METRON-1509) Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 Single Node/Vagrant Method

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426147#comment-16426147
 ] 

ASF GitHub Bot commented on METRON-1509:


GitHub user as22323 opened a pull request:

https://github.com/apache/metron/pull/980

METRON-1509  Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 
Single Node/Vagrant Method

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LTW-GCR-CSOC/metron master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/980.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #980


commit 5e9777686b8fd25c53eec9505428ed50e69b122c
Author: as22323 
Date:   2018-04-04T20:26:52Z

Added Metron 0.4.1 - AWS EC2 Single Node Deployment

commit 224d2afe75a6dc4189aed3d54077b9468fdd7e27
Author: as22323 
Date:   2018-04-04T20:27:12Z

Update README.md

commit e3e2d8bd207393a0c4a8578cb2f8db598fd27f26
Author: as22323 
Date:   2018-04-04T20:28:36Z

Update README.md

commit 814cc7a3a71c7deede011179126bb6e333fb6f3f
Author: as22323 
Date:   2018-04-04T20:29:42Z

Update README.md




> Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 Single 
> Node/Vagrant Method
> -
>
> Key: METRON-1509
> URL: https://issues.apache.org/jira/browse/METRON-1509
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: Next + 1
>Reporter: ashah
>Priority: Minor
> Fix For: Next + 1
>
>
> Otto and Nickwallen in Jira:METRON-1434 & PR 
> [916|https://github.com/apache/metron/pull/916] suggested that a proposed 
> AWS/Vagrant method be put into a 

[GitHub] metron pull request #980: METRON-1509 Docs - metron-deployment/other-example...

[as22323]

GitHub user as22323 opened a pull request:

https://github.com/apache/metron/pull/980

METRON-1509  Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 
Single Node/Vagrant Method

## Contributor Comments
[Please place any comments here.  A description of the problem/enhancement, 
how to reproduce the issue, your testing methodology, etc.]


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LTW-GCR-CSOC/metron master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/980.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #980


commit 5e9777686b8fd25c53eec9505428ed50e69b122c
Author: as22323 
Date:   2018-04-04T20:26:52Z

Added Metron 0.4.1 - AWS EC2 Single Node Deployment

commit 224d2afe75a6dc4189aed3d54077b9468fdd7e27
Author: as22323 
Date:   2018-04-04T20:27:12Z

Update README.md

commit e3e2d8bd207393a0c4a8578cb2f8db598fd27f26
Author: as22323 
Date:   2018-04-04T20:28:36Z

Update README.md

commit 814cc7a3a71c7deede011179126bb6e333fb6f3f
Author: as22323 
Date:   2018-04-04T20:29:42Z

Update README.md




---

[jira] [Created] (METRON-1509) Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 Single Node/Vagrant Method

[ashah (JIRA)]

ashah created METRON-1509:
-

 Summary: Docs - metron-deployment/other-examples - Adding 0.4.1 
AWS EC2 Single Node/Vagrant Method
 Key: METRON-1509
 URL: https://issues.apache.org/jira/browse/METRON-1509
 Project: Metron
  Issue Type: Improvement
Affects Versions: Next + 1
Reporter: ashah
 Fix For: Next + 1


Otto and Nickwallen in Jira:METRON-1434 & PR 
[916|https://github.com/apache/metron/pull/916] suggested that a proposed 
AWS/Vagrant method be put into a doc and/or be referred to in a doc. This 
request updates a Metron doc to include an external community deployment method 
as a reference in a doc. 

This updates /metron-deployment/other-examples/README.md to include an external 
reference to the following deployment method ""Metron 0.4.1 - AWS EC2 Single 
Node Deployment with Vagrant"" 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426096#comment-16426096
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
I wrote this, because I found myself needing to quantify if there was a 
problem or bottleneck in a certain complex expression, and we had no way to 
find out.


> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1090) STELLAR Add Assignment capability to Stellar language

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425965#comment-16425965
 ] 

ASF GitHub Bot commented on METRON-1090:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
@nickwallen @cestella @mattf-horton , Sorry, I thought that question was 
more to @mattf-horton at the time. 

@cestella I'm not sure I understand what you are saying.  Are you saying 
that we should not do assignment until we overhaul the current mechanisms?  
That this is too much a 1/2 measure?


> STELLAR Add Assignment capability to Stellar language
> -
>
> Key: METRON-1090
> URL: https://issues.apache.org/jira/browse/METRON-1090
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Add support for =, +=, -=, *=, /= for variables in stellar.
> foo = 5  returns 5 AND sets foo to 5 if the VariableResolver supports it.
> The shell should be changed to assign variables using this syntax, because 
> exposing the left side to stellar allows catching errors for invalid keywords 
> in left side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
@nickwallen @cestella @mattf-horton , Sorry, I thought that question was 
more to @mattf-horton at the time. 

@cestella I'm not sure I understand what you are saying.  Are you saying 
that we should not do assignment until we overhaul the current mechanisms?  
That this is too much a 1/2 measure?


---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425950#comment-16425950
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
I think if you are writing or testing commands in the repl this would be 
useful.
If it were moved from stellar to common it could be integrated with the 
topologies. 
The reason I submitted it to commons was I thought it might be, but we 
don't have a 'common util' module, long run it should not be in stellar either


> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1462) Separate ES and Kibana from Metron Mpack

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425949#comment-16425949
 ] 

ASF GitHub Bot commented on METRON-1462:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/943
  
Awesome, thanks @anandsubbu and @nickwallen!


> Separate ES and Kibana from Metron Mpack
> 
>
> Key: METRON-1462
> URL: https://issues.apache.org/jira/browse/METRON-1462
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Need this to enable our ability to provide pluggable Solr support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/873
  
I think if you are writing or testing commands in the repl this would be 
useful.
If it were moved from stellar to common it could be integrated with the 
topologies. 
The reason I submitted it to commons was I thought it might be, but we 
don't have a 'common util' module, long run it should not be in stellar either


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/943
  
Awesome, thanks @anandsubbu and @nickwallen!


---

[jira] [Commented] (METRON-1503) Alerts are not getting populated in alerts UI when search engine is Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425913#comment-16425913
 ] 

ASF GitHub Bot commented on METRON-1503:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/975
  
Can this fix be tested in a unit or integration test?


> Alerts are not getting populated in alerts UI when search engine is Solr
> 
>
> Key: METRON-1503
> URL: https://issues.apache.org/jira/browse/METRON-1503
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> When navigating to the Alerts UI no alerts are displayed.  REST is returning 
> this error on a search call:
> {{Mar 27, 2018 6:51:50 AM org.apache.catalina.core.StandardWrapperValve 
> invoke SEVERE: Servlet.service() for servlet [dispatcherServlet] in context 
> with path [] threw exception [Request processing failed; nested exception is 
> org.apache.solr.common.SolrException: Collection not found: websphere] with 
> root cause org.apache.solr.common.SolrException: Collection not found: 
> websphere at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1401)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1094)
>  at 
> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
>  at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160) at 
> org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942) at 
> org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957) at 
> org.apache.metron.solr.dao.SolrSearchDao.search(SolrSearchDao.java:89) at 
> org.apache.metron.solr.dao.SolrDao.search(SolrDao.java:90) at 
> org.apache.metron.solr.dao.SolrMetaAlertDao.search(SolrMetaAlertDao.java:89) 
> at 
> org.apache.metron.rest.service.impl.SearchServiceImpl.search(SearchServiceImpl.java:73)
>  at 
> org.apache.metron.rest.controller.SearchController.search(SearchController.java:54)
>  at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source) at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>  at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
>  at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>  at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
>  at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
>  at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>  at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
>  at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
>  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
>  at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #975: METRON-1503: Alerts are not getting populated in alerts U...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/975
  
Can this fix be tested in a unit or integration test?


---

[jira] [Commented] (METRON-1504) Enriching missing values does not match the semantics between the new enrichment topology and old

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425905#comment-16425905
 ] 

ASF GitHub Bot commented on METRON-1504:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/976
  
I appreciate the check in the integration test that you added, but can we 
add a specific check for this in the unit test; `ParallelEnricherTest`?


> Enriching missing values does not match the semantics between the new 
> enrichment topology and old
> -
>
> Key: METRON-1504
> URL: https://issues.apache.org/jira/browse/METRON-1504
> Project: Metron
>  Issue Type: Bug
>Reporter: Casey Stella
>Priority: Major
>
> Currently if one tries to enrich an missing value (e.g. via a hbase 
> enrichment) with the new enrichment topology, you get NPEs.  The existing 
> semantics from the old enrichment topology is for no enrichment to happen.  
> The new enrichment topology should be made to match this semantic.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #976: METRON-1504: Enriching missing values does not match the ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/976
  
I appreciate the check in the integration test that you added, but can we 
add a specific check for this in the unit test; `ParallelEnricherTest`?


---

[jira] [Commented] (METRON-1090) STELLAR Add Assignment capability to Stellar language

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425889#comment-16425889
 ] 

ASF GitHub Bot commented on METRON-1090:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
> @cestella Should we perhaps separate parsing assignments (e.g. move 
StellarAssignment.from into the language like you've done here) from actually 
updating the variables.

@ottobackwards I didn't see  you respond to this.  I would agree with 
@cestella on this.


> STELLAR Add Assignment capability to Stellar language
> -
>
> Key: METRON-1090
> URL: https://issues.apache.org/jira/browse/METRON-1090
> Project: Metron
>  Issue Type: New Feature
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Add support for =, +=, -=, *=, /= for variables in stellar.
> foo = 5  returns 5 AND sets foo to 5 if the VariableResolver supports it.
> The shell should be changed to assign variables using this syntax, because 
> exposing the left side to stellar allows catching errors for invalid keywords 
> in left side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
> @cestella Should we perhaps separate parsing assignments (e.g. move 
StellarAssignment.from into the language like you've done here) from actually 
updating the variables.

@ottobackwards I didn't see  you respond to this.  I would agree with 
@cestella on this.


---

[jira] [Commented] (METRON-1367) Stellar should optionally instrument function calls

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425886#comment-16425886
 ] 

ASF GitHub Bot commented on METRON-1367:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/873
  
IMHO, I just don't see value in capturing timing information like this in 
the REPL.  Timing information like this would be super valuable when run on a 
live cluster in the topologies to help identify bottlenecks.  Is there a way 
that we could capture timings on a live cluster somehow?


> Stellar should optionally instrument function calls
> ---
>
> Key: METRON-1367
> URL: https://issues.apache.org/jira/browse/METRON-1367
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> It would be nice if stellar processor(s) could optionally instrument function 
> calls to test performance.
> The result would be some structure with the calls, the times and maybe some 
> context as a list or map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #873: METRON-1367 Stellar should have some instrumentation of f...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/873
  
IMHO, I just don't see value in capturing timing information like this in 
the REPL.  Timing information like this would be super valuable when run on a 
live cluster in the topologies to help identify bottlenecks.  Is there a way 
that we could capture timings on a live cluster somehow?


---

[jira] [Commented] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425883#comment-16425883
 ] 

ASF GitHub Bot commented on METRON-1434:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/916
  
I'd be perfectly happy to reference in Metron docs, external 
resources/projects like this that community members might find useful.  Good 
idea @ottobackwards 


> Ability to deploy Metron full dev as a single node (centos6) to AWS with 
> vagrant and AMI
> 
>
> Key: METRON-1434
> URL: https://issues.apache.org/jira/browse/METRON-1434
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
> Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 
> Cores, 64 bit] based on metron-deployment/development/centos6/
> ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS
> iii) AMI  "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: AWS, EC2, vagrant
> Fix For: 0.4.3
>
>
> The pull request has two potential solutions for deploying to Metron to AWS. 
> i) AWS Single Node Cluster Deployment Using Vagrant
> ii) AWS Single Node Cluster Deployment Using an AMI
> *i) AWS Single Node Cluster Deployment Using Vagrant:*
>  I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. 
> It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) 
> that was in git. I'm assuming the latest that was in git is 0.4.3 that is 
> progress.
> _Instructions:_
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant]
>  
> *ii) AWS Single Node Cluster Deployment Using an AMI:*
>  In addition to including instructions for deploying 0.4.3 using Vagrant on a 
> Mac I also included instructions for deploying to AWS using a preexisting AMI 
> (I made the following AMI public as a proof-of-concept: "GCR-Xetron 
> Demo"/"ami-93cb4ff7").  This might be the quicket option for deployment since 
> no preinstall requirments would be needed.
> _Instructions:_ 
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami]
>  
> PULL REQUEST CHECKLIST ITEMS
> *"For documentation related changes:"*
> Running cd site-book, mvn site produced "BUILD SUCCESS"
>  
> *"For code changes:"* 
> Was not sure where to run "mvn -q clean integration-test install && 
> dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not 
> changed. For i) the following code were newly added/modified:
> [Vagrant to work with aws]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile]
> [changing eth1 to eth0]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/inventory/group_vars/all]
>  
>  
> *"How the change may be verified: "*
> You should be able to see Ambari after following instructions 
> (http://:8080)
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #916: METRON-1434 - Ability to deploy Metron full dev as a sing...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/916
  
I'd be perfectly happy to reference in Metron docs, external 
resources/projects like this that community members might find useful.  Good 
idea @ottobackwards 


---

[jira] [Created] (METRON-1508) Indexing Fails to Write to Elasticsearch

[Nick Allen (JIRA)]

Nick Allen created METRON-1508:
--

 Summary: Indexing Fails to Write to Elasticsearch
 Key: METRON-1508
 URL: https://issues.apache.org/jira/browse/METRON-1508
 Project: Metron
  Issue Type: Bug
Affects Versions: 0.4.2
Reporter: Nick Allen


When spinning up the Ubuntu 14 development environment, the indexing topology 
fails to write to Elasticsearch.  This does not appear to be caused by resource 
constraints or OOM conditions.  The indexing topology reports the following 
error.
{code:java}
2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3] 
[ERROR] 
org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
configured nodes are available: 
[{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
at 
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
 ~[stormjar.jar:?]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
 ~[stormjar.jar:?]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
 ~[stormjar.jar:?]
at 
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
 ~[stormjar.jar:?]
at 
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
 [stormjar.jar:?]
at 
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
 [stormjar.jar:?]
at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
 [stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1508) Indexing Fails to Write to Elasticsearch

[Nick Allen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-1508:
---
Priority: Minor  (was: Critical)

> Indexing Fails to Write to Elasticsearch
> 
>
> Key: METRON-1508
> URL: https://issues.apache.org/jira/browse/METRON-1508
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Nick Allen
>Priority: Minor
>
> When spinning up the Ubuntu 14 development environment, the indexing topology 
> fails to write to Elasticsearch.  This does not appear to be caused by 
> resource constraints or OOM conditions.  The indexing topology reports the 
> following error.
> {code:java}
> 2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 3] 
> [ERROR] 
> org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
> configured nodes are available: 
> [{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
>   at 
> org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
>  ~[stormjar.jar:?]
>   at 
> org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
>  ~[stormjar.jar:?]
>   at 
> org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
>  ~[stormjar.jar:?]
>   at 
> org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
>  ~[stormjar.jar:?]
>   at 
> org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
>  ~[stormjar.jar:?]
>   at 
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
>  ~[stormjar.jar:?]
>   at 
> org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
>  ~[stormjar.jar:?]
>   at 
> org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
>  ~[stormjar.jar:?]
>   at 
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
>  [stormjar.jar:?]
>   at 
> org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
>  [stormjar.jar:?]
>   at 
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
>  [stormjar.jar:?]
>   at 
> org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at 
> org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at 
> org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
>  [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
>   at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425871#comment-16425871
 ] 

ASF GitHub Bot commented on METRON-1434:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/916
  
Why not add it to /contrib ?  Or can we have a way to like to it from docs 
or site?


> Ability to deploy Metron full dev as a single node (centos6) to AWS with 
> vagrant and AMI
> 
>
> Key: METRON-1434
> URL: https://issues.apache.org/jira/browse/METRON-1434
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
> Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 
> Cores, 64 bit] based on metron-deployment/development/centos6/
> ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS
> iii) AMI  "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: AWS, EC2, vagrant
> Fix For: 0.4.3
>
>
> The pull request has two potential solutions for deploying to Metron to AWS. 
> i) AWS Single Node Cluster Deployment Using Vagrant
> ii) AWS Single Node Cluster Deployment Using an AMI
> *i) AWS Single Node Cluster Deployment Using Vagrant:*
>  I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. 
> It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) 
> that was in git. I'm assuming the latest that was in git is 0.4.3 that is 
> progress.
> _Instructions:_
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant]
>  
> *ii) AWS Single Node Cluster Deployment Using an AMI:*
>  In addition to including instructions for deploying 0.4.3 using Vagrant on a 
> Mac I also included instructions for deploying to AWS using a preexisting AMI 
> (I made the following AMI public as a proof-of-concept: "GCR-Xetron 
> Demo"/"ami-93cb4ff7").  This might be the quicket option for deployment since 
> no preinstall requirments would be needed.
> _Instructions:_ 
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami]
>  
> PULL REQUEST CHECKLIST ITEMS
> *"For documentation related changes:"*
> Running cd site-book, mvn site produced "BUILD SUCCESS"
>  
> *"For code changes:"* 
> Was not sure where to run "mvn -q clean integration-test install && 
> dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not 
> changed. For i) the following code were newly added/modified:
> [Vagrant to work with aws]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile]
> [changing eth1 to eth0]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/inventory/group_vars/all]
>  
>  
> *"How the change may be verified: "*
> You should be able to see Ambari after following instructions 
> (http://:8080)
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #916: METRON-1434 - Ability to deploy Metron full dev as a sing...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/916
  
Why not add it to /contrib ?  Or can we have a way to like to it from docs 
or site?


---

[jira] [Commented] (METRON-1462) Separate ES and Kibana from Metron Mpack

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425870#comment-16425870
 ] 

ASF GitHub Bot commented on METRON-1462:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/943
  
I was able to replicate this same problem in master running on the ubuntu14 
dev environment.  This does not seem to be a problem caused by your PR.  I will 
open a separate defect for this.

+1 That being said, nice work!  


> Separate ES and Kibana from Metron Mpack
> 
>
> Key: METRON-1462
> URL: https://issues.apache.org/jira/browse/METRON-1462
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Need this to enable our ability to provide pluggable Solr support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/943
  
I was able to replicate this same problem in master running on the ubuntu14 
dev environment.  This does not seem to be a problem caused by your PR.  I will 
open a separate defect for this.

+1 That being said, nice work!  


---

[jira] [Commented] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425864#comment-16425864
 ] 

ASF GitHub Bot commented on METRON-1434:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/916
  
@as22323 Let me first say that this is a nice piece of work.  You've 
followed all the existing patterns and your contribution looks solid.  

But ultimately I am +0 on this.  I want others to speak up should they see 
value in this, but I'd argue that this should not be merged.  Please let me 
explain my reasoning.

**Support Burden**

IMHO, the 'surface area' to support in Metron has always been too large.  
We as a community spend too much time supporting non-core functionality, which 
impedes our progress in delivering features targeted toward the cybersecurity 
use case.

One key example of this are the multiple different deployment targets that 
we support.  This has been a difficult and time consuming support task over the 
history of the project. I've done plenty of work on this myself.

Coalescing Metron installation around the MPack was a major step forward 
for us.  This was intended to help remove some of that support burden.  With 
the MPack, much of that burden is shifted to Ambari.  

For example, with everything in Metron today, you can stand-up a single 
node in AWS and use the Mpack to install Metron.  It is not as "push button" 
simple as your contribution here, but it is "good enough" considering the 
resources we have in the community today.

**Uncommon Use Case**

We should also considering that running Metron on a single node is a recipe 
for a horrible user experience.  It should only be run on a single node for 
development purposes, which is something that we already do support. I would 
not recommend that anyone run Metron on a single node for any other purpose.

--

I'd prefer not too add to our ongoing support burden by merging this PR.  
What you've done is a great contribution and I'd love to see you publicly share 
and support this as a separate project, outside of Apache Metron, going forward.

If others agree or disagree with my reasoning, please do feel free to share.


> Ability to deploy Metron full dev as a single node (centos6) to AWS with 
> vagrant and AMI
> 
>
> Key: METRON-1434
> URL: https://issues.apache.org/jira/browse/METRON-1434
> Project: Metron
>  Issue Type: Improvement
>Affects Versions: 0.4.3
> Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 
> Cores, 64 bit] based on metron-deployment/development/centos6/
> ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS
> iii) AMI  "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge
>Reporter: ashah
>Assignee: ashah
>Priority: Minor
>  Labels: AWS, EC2, vagrant
> Fix For: 0.4.3
>
>
> The pull request has two potential solutions for deploying to Metron to AWS. 
> i) AWS Single Node Cluster Deployment Using Vagrant
> ii) AWS Single Node Cluster Deployment Using an AMI
> *i) AWS Single Node Cluster Deployment Using Vagrant:*
>  I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. 
> It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) 
> that was in git. I'm assuming the latest that was in git is 0.4.3 that is 
> progress.
> _Instructions:_
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant]
>  
> *ii) AWS Single Node Cluster Deployment Using an AMI:*
>  In addition to including instructions for deploying 0.4.3 using Vagrant on a 
> Mac I also included instructions for deploying to AWS using a preexisting AMI 
> (I made the following AMI public as a proof-of-concept: "GCR-Xetron 
> Demo"/"ami-93cb4ff7").  This might be the quicket option for deployment since 
> no preinstall requirments would be needed.
> _Instructions:_ 
> [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami]
>  
> PULL REQUEST CHECKLIST ITEMS
> *"For documentation related changes:"*
> Running cd site-book, mvn site produced "BUILD SUCCESS"
>  
> *"For code changes:"* 
> Was not sure where to run "mvn -q clean integration-test install && 
> dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not 
> changed. For i) the following code were newly added/modified:
> [Vagrant to work with aws]
> [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile]
> [changing eth1 to eth0]
> 

[GitHub] metron issue #916: METRON-1434 - Ability to deploy Metron full dev as a sing...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/916
  
@as22323 Let me first say that this is a nice piece of work.  You've 
followed all the existing patterns and your contribution looks solid.  

But ultimately I am +0 on this.  I want others to speak up should they see 
value in this, but I'd argue that this should not be merged.  Please let me 
explain my reasoning.

**Support Burden**

IMHO, the 'surface area' to support in Metron has always been too large.  
We as a community spend too much time supporting non-core functionality, which 
impedes our progress in delivering features targeted toward the cybersecurity 
use case.

One key example of this are the multiple different deployment targets that 
we support.  This has been a difficult and time consuming support task over the 
history of the project. I've done plenty of work on this myself.

Coalescing Metron installation around the MPack was a major step forward 
for us.  This was intended to help remove some of that support burden.  With 
the MPack, much of that burden is shifted to Ambari.  

For example, with everything in Metron today, you can stand-up a single 
node in AWS and use the Mpack to install Metron.  It is not as "push button" 
simple as your contribution here, but it is "good enough" considering the 
resources we have in the community today.

**Uncommon Use Case**

We should also considering that running Metron on a single node is a recipe 
for a horrible user experience.  It should only be run on a single node for 
development purposes, which is something that we already do support. I would 
not recommend that anyone run Metron on a single node for any other purpose.

--

I'd prefer not too add to our ongoing support burden by merging this PR.  
What you've done is a great contribution and I'd love to see you publicly share 
and support this as a separate project, outside of Apache Metron, going forward.

If others agree or disagree with my reasoning, please do feel free to share.


---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425824#comment-16425824
 ] 

ASF GitHub Bot commented on METRON-1507:


GitHub user simonellistonball opened a pull request:

https://github.com/apache/metron/pull/979

METRON-1507 Added support for vagrant cachier

## Contributor Comments
This is a Vagrant config change to enable the use of vagrant-cachier to 
reduce build times for the full dev platform. Time saving is heavily network 
dependent, but anecdotally I've found it saves me around 10 mins per provision 
of vagrant development environment. YMMV.

To test:

1. Install vagrant-cachier, and spin up centos and ubuntu vagrant 
development platforms.
2. Verify that packages are populated into your local (the client running 
vagrant) ~/.vagrant/cache directories.
3. vagrant destroy your development environments
4. spin up vagrant environments again, packages should be pulled from the 
cache instead of the network at this point.

Note that the vagrant-cachier plugin is not required, nor shipped, bundled 
or in any way distributed by this change. It does have a compatible license 
(MIT) but it is outside of the Metron project scope to install this, so there 
is no need to update license or dependency documentation.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/metron METRON-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/979.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #979


commit 2b6618d27903a6301834989f0b13f03258e88607
Author: Simon Elliston Ball 
Date:   2018-04-04T16:38:31Z

Added support for vagrant cachier




> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>   

[GitHub] metron pull request #979: METRON-1507 Added support for vagrant cachier

[simonellistonball]

GitHub user simonellistonball opened a pull request:

https://github.com/apache/metron/pull/979

METRON-1507 Added support for vagrant cachier

## Contributor Comments
This is a Vagrant config change to enable the use of vagrant-cachier to 
reduce build times for the full dev platform. Time saving is heavily network 
dependent, but anecdotally I've found it saves me around 10 mins per provision 
of vagrant development environment. YMMV.

To test:

1. Install vagrant-cachier, and spin up centos and ubuntu vagrant 
development platforms.
2. Verify that packages are populated into your local (the client running 
vagrant) ~/.vagrant/cache directories.
3. vagrant destroy your development environments
4. spin up vagrant environments again, packages should be pulled from the 
cache instead of the network at this point.

Note that the vagrant-cachier plugin is not required, nor shipped, bundled 
or in any way distributed by this change. It does have a compatible license 
(MIT) but it is outside of the Metron project scope to install this, so there 
is no need to update license or dependency documentation.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/metron METRON-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/979.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #979


commit 2b6618d27903a6301834989f0b13f03258e88607
Author: Simon Elliston Ball 
Date:   2018-04-04T16:38:31Z

Added support for vagrant cachier




---

[GitHub] metron pull request #978: METRON-1507 Added support for vagrant cachier

[simonellistonball]

Github user simonellistonball closed the pull request at:

https://github.com/apache/metron/pull/978


---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425816#comment-16425816
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user simonellistonball closed the pull request at:

https://github.com/apache/metron/pull/978


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425800#comment-16425800
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/978
  
Ooops... wrong branch. Let me sort out the PR. 


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1496) ChainLink Parser to reuse parser code at parserConfig level

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425802#comment-16425802
 ] 

ASF GitHub Bot commented on METRON-1496:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/969
  
@kevin91nl I think the discussion that you have helped spark here is worthy 
of a separate email discuss thread.  

This is going to take more thought and discussion in the community to land 
on a good solution.  What you've shown here are some great use cases that will 
help drive that discussion.  

I think there are others in the community with slightly different views on 
how this should be done.  Hopefully we can get everyone's minds working 
together in a discuss thread.




> ChainLink Parser to reuse parser code at parserConfig level
> ---
>
> Key: METRON-1496
> URL: https://issues.apache.org/jira/browse/METRON-1496
> Project: Metron
>  Issue Type: Improvement
>Reporter: Bas van de Lustgraaf
>Priority: Minor
>
> During the development of some custom parsers we wrote a couple of classes / 
> functions to make it possible to reuse code and assemble parser quicker at 
> java coding level.
> We took this idea one step further and created the so called ChainLinkParser.
> This parser gives user without any java knowledge the opportunity to assemble 
> parsers at parser configuration level.
> We would like to discuss the code and see if it can be submitted to the 
> project. We will create a PR during this week to submit the code for review 
> and discussion.
> Below you'll find an example of our Parser configuration for Suricata, which 
> is using our ChainParser. 
>  
> {noformat}
> {
>"parserClassName":"nl.qsight.chainparser.ChainParser",
>"sensorTopic":"suricata",
>"readMetadata":true,
>"mergeMetadata":true,
>"numWorkers":3,
>"numAckers":3,
>"spoutParallelism":6,
>"spoutNumTasks":6,
>"parserParallelism":20,
>"parserNumTasks":20,
>"errorWriterParallelism":1,
>"errorWriterNumTasks":1,
>"spoutConfig":{
>   "spout.firstPollOffsetStrategy":"LATEST"
>},
>"stormConfig":{
>   "topology.max.spout.pending":2000
>},
>"parserConfig":{
>   "chain":[
>  "parse_json",
>  "parse_username",
>  "rename_fields",
>  "parse_datetime"
>   ],
>   "parsers":{
>  "parse_json":{
> "class":"nl.qsight.links.io.JSONDecoderLink"
>  },
>  "parse_username":{
> "class":"nl.qsight.links.io.RegexLink",
> "pattern":"(?i)(user|username|log)[=:](\\w+)",
> "selector":{
>"username":"2"
> },
> "input":"{{payload_printable}}"
>  },
>  "rename_fields":{
> "class":"nl.qsight.links.fields.RenameLink",
> "rename":{
>"proto":"protocol",
>"dest_ip":"ip_dst_addr",
>"src_ip":"ip_src_addr",
>"dest_port":"ip_dst_port",
>"src_port":"ip_src_port"
> }
>  },
>  "parse_datetime":{
> "class":"nl.qsight.links.io.TimestampLink",
> "patterns":[
>[
>   
> "([0-9]{4})-([0-9]+)-([0-9]+)T([0-9]+):([0-9]+):([0-9]+).([0-9]+)([+-]{1}[0-9]{1,2}[:]?[0-9]{2})",
>   " MM dd HH mm ss SS Z",
>   "newest"
>]
> ],
> "input":"{{timestamp}}"
>  }
>   }
>}
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #969: METRON-1496: ChainLink Parser to reuse parser code at par...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/969
  
@kevin91nl I think the discussion that you have helped spark here is worthy 
of a separate email discuss thread.  

This is going to take more thought and discussion in the community to land 
on a good solution.  What you've shown here are some great use cases that will 
help drive that discussion.  

I think there are others in the community with slightly different views on 
how this should be done.  Hopefully we can get everyone's minds working 
together in a discuss thread.




---

[GitHub] metron issue #978: METRON-1507 Added support for vagrant cachier

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/978
  
Ooops... wrong branch. Let me sort out the PR. 


---

[GitHub] metron issue #978: METRON-1507 Added support for vagrant cachier

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/978
  
I think this is great.  I appreciate that the environments will continue to 
work with or without the Cachier plugin installed.  Anything we can do to 
speed-up the dev environments is a big win in my book.

One ask would be to embed a link to 'vagrant-cashier' in the README so that 
a user can easily find the plugin installation instructions.  I believe this is 
the right location; https://github.com/fgrehm/vagrant-cachier.






---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425786#comment-16425786
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/978
  
I think this is great.  I appreciate that the environments will continue to 
work with or without the Cachier plugin installed.  Anything we can do to 
speed-up the dev environments is a big win in my book.

One ask would be to embed a link to 'vagrant-cashier' in the README so that 
a user can easily find the plugin installation instructions.  I believe this is 
the right location; https://github.com/fgrehm/vagrant-cachier.






> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #974: METRON-1502: Upgrade Doxia plugin to 1.8

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/974
  
Just curious, is there some improved functionality we get out of the newer 
Doxia version?

Either way, this gets a +1 from me.  I rendered the site book and 
everything looks kosher.  Thanks for taking care of this.


---

[jira] [Commented] (METRON-1502) Upgrade Doxia plugin to 1.8

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425778#comment-16425778
 ] 

ASF GitHub Bot commented on METRON-1502:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/974
  
Just curious, is there some improved functionality we get out of the newer 
Doxia version?

Either way, this gets a +1 from me.  I rendered the site book and 
everything looks kosher.  Thanks for taking care of this.


> Upgrade Doxia plugin to 1.8
> ---
>
> Key: METRON-1502
> URL: https://issues.apache.org/jira/browse/METRON-1502
> Project: Metron
>  Issue Type: Improvement
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
>
> https://issues.apache.org/jira/browse/METRON-944 related to a pathological 
> parsing issues underlying the plugin.
>  
> That plugin has bene updated now and the related Jira on the underlying 
> issues is closed.  We should migrate to the newer version of the plugin.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/943
  
When I spin this up in Ubuntu, the Indexing topology is unable to write to 
Elasticsearch.  This may very well be a resource constraint issues in my 
environment, but I thought I'd mention it before I spend too much more time 
digging into this.
```
2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 
3] [ERROR] 
org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
configured nodes are available: 
[{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
at 
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
 ~[stormjar.jar:?]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
 ~[stormjar.jar:?]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
 ~[stormjar.jar:?]
at 
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
 ~[stormjar.jar:?]
at 
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
 [stormjar.jar:?]
at 
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
 [stormjar.jar:?]
at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
 [stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
```


---

[jira] [Commented] (METRON-1462) Separate ES and Kibana from Metron Mpack

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425760#comment-16425760
 ] 

ASF GitHub Bot commented on METRON-1462:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/943
  
When I spin this up in Ubuntu, the Indexing topology is unable to write to 
Elasticsearch.  This may very well be a resource constraint issues in my 
environment, but I thought I'd mention it before I spend too much more time 
digging into this.
```
2018-04-04 15:51:05.707 o.a.s.d.executor Thread-6-indexingBolt-executor[3 
3] [ERROR] 
org.elasticsearch.client.transport.NoNodeAvailableException: None of the 
configured nodes are available: 
[{#transport#-1}{rZcTXccfSPq4fH4IRLQ0zg}{node1}{127.0.1.1:9300}]
at 
org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:363)
 ~[stormjar.jar:?]
at 
org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408)
 ~[stormjar.jar:?]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80)
 ~[stormjar.jar:?]
at 
org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54)
 ~[stormjar.jar:?]
at 
org.apache.metron.elasticsearch.writer.ElasticsearchWriter.write(ElasticsearchWriter.java:92)
 ~[stormjar.jar:?]
at 
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:239)
 [stormjar.jar:?]
at 
org.apache.metron.writer.BulkWriterComponent.write(BulkWriterComponent.java:217)
 [stormjar.jar:?]
at 
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:236)
 [stormjar.jar:?]
at 
org.apache.storm.daemon.executor$fn__7590$tuple_action_fn__7592.invoke(executor.clj:730)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.daemon.executor$mk_task_receiver$fn__7511.invoke(executor.clj:462)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at 
org.apache.storm.daemon.executor$fn__7590$fn__7603$fn__7656.invoke(executor.clj:849)
 [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) 
[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
```


> Separate ES and Kibana from Metron Mpack
> 
>
> Key: METRON-1462
> URL: https://issues.apache.org/jira/browse/METRON-1462
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Need this to enable our ability to provide pluggable Solr support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425738#comment-16425738
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/978
  
The yum stuff only happens in the centos include in ansible, it's not 
necessary in the ubuntu version, which uses a different ansible include.

An extra test for the record... manual run of the ubuntu14 development 
environment: 

building ubuntu14 (measuring start to end of ansible logs)
without cache: 45m13
with cache populated: 28m57

so won't change the world, but will hopefully save some coffee during 
dev-test cycles. 


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #978: METRON-1507 Added support for vagrant cachier

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/978
  
The yum stuff only happens in the centos include in ansible, it's not 
necessary in the ubuntu version, which uses a different ansible include.

An extra test for the record... manual run of the ubuntu14 development 
environment: 

building ubuntu14 (measuring start to end of ansible logs)
without cache: 45m13
with cache populated: 28m57

so won't change the world, but will hopefully save some coffee during 
dev-test cycles. 


---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425735#comment-16425735
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/978
  
I didn't think ubuntu had yum repos.d in the image?


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #978: METRON-1507 Added support for vagrant cachier

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/978
  
I didn't think ubuntu had yum repos.d in the image?


---

[jira] [Commented] (METRON-1462) Separate ES and Kibana from Metron Mpack

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425710#comment-16425710
 ] 

ASF GitHub Bot commented on METRON-1462:


Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/943
  
My +1 still holds. Installed both mpacks on a 12-node cluster and was able 
to get bro indices to work successfully. Thanks @mmiklavc !


> Separate ES and Kibana from Metron Mpack
> 
>
> Key: METRON-1462
> URL: https://issues.apache.org/jira/browse/METRON-1462
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Need this to enable our ability to provide pluggable Solr support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/943
  
My +1 still holds. Installed both mpacks on a 12-node cluster and was able 
to get bro indices to work successfully. Thanks @mmiklavc !


---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425576#comment-16425576
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/978
  
The PR includes ubuntu support, the cachier would just cache debs from apt 
instead of rpms from yum. TBH I've tested centos more than ubuntu. The use is 
still the same though, should be transparent to the user.


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #978: METRON-1507 Added support for vagrant cachier

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/978
  
The PR includes ubuntu support, the cachier would just cache debs from apt 
instead of rpms from yum. TBH I've tested centos more than ubuntu. The use is 
still the same though, should be transparent to the user.


---

[GitHub] metron issue #978: METRON-1507 Added support for vagrant cachier

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/978
  
Should this work with ubuntu as well?  Would it have to be different?  Can 
you describe how it works? 


---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425572#comment-16425572
 ] 

ASF GitHub Bot commented on METRON-1507:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/978
  
Should this work with ubuntu as well?  Would it have to be different?  Can 
you describe how it works? 


> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>  Issue Type: Improvement
>Reporter: Simon Elliston Ball
>Priority: Trivial
>
> Add configuration in development vagrant files to support vagrant cachier 
> ([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
> install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #978: METRON-1507 Added support for vagrant cachier

[simonellistonball]

GitHub user simonellistonball opened a pull request:

https://github.com/apache/metron/pull/978

METRON-1507 Added support for vagrant cachier

## Contributor Comments
This is a Vagrant config change to enable the use of vagrant-cachier to 
reduce build times for the full dev platform. Time saving is heavily network 
dependent, but anecdotally I've found it saves me around 10 mins per provision 
of vagrant development environment. YMMV.

To test:

1. Install vagrant-cachier, and spin up centos and ubuntu vagrant 
development platforms.
2. Verify that packages are populated into your local (the client running 
vagrant) ~/.vagrant/cache directories.
3. vagrant destroy your development environments
4. spin up vagrant environments again, packages should be pulled from the 
cache instead of the network at this point.

Note that the vagrant-cachier plugin is not required, nor shipped, bundled 
or in any way distributed by this change. It does have a compatible license 
(MIT) but it is outside of the Metron project scope to install this, so there 
is no need to update license or dependency documentation.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/metron METRON-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/978.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #978


commit 006bb816622bb5422fb1e8b7d6015e0b9d7c2562
Author: Simon Elliston Ball 
Date:   2018-04-04T13:10:31Z

Added support for vagrant cachier




---

[jira] [Commented] (METRON-1507) Add support for Vagrant Cachier plugin if present

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425497#comment-16425497
 ] 

ASF GitHub Bot commented on METRON-1507:


GitHub user simonellistonball opened a pull request:

https://github.com/apache/metron/pull/978

METRON-1507 Added support for vagrant cachier

## Contributor Comments
This is a Vagrant config change to enable the use of vagrant-cachier to 
reduce build times for the full dev platform. Time saving is heavily network 
dependent, but anecdotally I've found it saves me around 10 mins per provision 
of vagrant development environment. YMMV.

To test:

1. Install vagrant-cachier, and spin up centos and ubuntu vagrant 
development platforms.
2. Verify that packages are populated into your local (the client running 
vagrant) ~/.vagrant/cache directories.
3. vagrant destroy your development environments
4. spin up vagrant environments again, packages should be pulled from the 
cache instead of the network at this point.

Note that the vagrant-cachier plugin is not required, nor shipped, bundled 
or in any way distributed by this change. It does have a compatible license 
(MIT) but it is outside of the Metron project scope to install this, so there 
is no need to update license or dependency documentation.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/simonellistonball/metron METRON-1507

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/978.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #978


commit 006bb816622bb5422fb1e8b7d6015e0b9d7c2562
Author: Simon Elliston Ball 
Date:   2018-04-04T13:10:31Z

Added support for vagrant cachier




> Add support for Vagrant Cachier plugin if present
> -
>
> Key: METRON-1507
> URL: https://issues.apache.org/jira/browse/METRON-1507
> Project: Metron
>   

[jira] [Created] (METRON-1507) Add support for Vagrant Cachier plugin if present

[Simon Elliston Ball (JIRA)]

Simon Elliston Ball created METRON-1507:
---

 Summary: Add support for Vagrant Cachier plugin if present
 Key: METRON-1507
 URL: https://issues.apache.org/jira/browse/METRON-1507
 Project: Metron
  Issue Type: Improvement
Reporter: Simon Elliston Ball


Add configuration in development vagrant files to support vagrant cachier 
([http://fgrehm.viewdocs.io/vagrant-cachier/)] to reduce development machine 
install times. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[wardbekker]

Github user wardbekker commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r179060961
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
 ---
@@ -107,32 +113,94 @@ public static String getBaseIndexName(String 
indexName) {
 return parts[0];
   }
 
-  public static TransportClient getClient(Map 
globalConfiguration, Map optionalSettings) {
+  /**
+   * Instantiates an Elasticsearch client based on es.client.class, if 
set. Defaults to
+   * org.elasticsearch.transport.client.PreBuiltTransportClient.
+   *
+   * @param globalConfiguration Metron global config
+   * @return
+   */
+  public static TransportClient getClient(Map 
globalConfiguration) {
+Set customESSettings = new HashSet<>();
+customESSettings.addAll(Arrays.asList("es.client.class", 
"es.xpack.username", "es.xpack.password.file"));
 Settings.Builder settingsBuilder = Settings.builder();
-settingsBuilder.put("cluster.name", 
globalConfiguration.get("es.clustername"));
-settingsBuilder.put("client.transport.ping_timeout","500s");
-if (optionalSettings != null) {
-  settingsBuilder.put(optionalSettings);
+Map esSettings = getEsSettings(globalConfiguration);
+for (Map.Entry entry : esSettings.entrySet()) {
+  String key = entry.getKey();
+  String value = entry.getValue();
+  if (!customESSettings.contains(key)) {
+settingsBuilder.put(key, value);
+  }
 }
-Settings settings = settingsBuilder.build();
-TransportClient client;
-try{
+settingsBuilder.put("cluster.name", 
globalConfiguration.get("es.clustername"));
+settingsBuilder.put("client.transport.ping_timeout", 
esSettings.getOrDefault("client.transport.ping_timeout","500s"));
+setXPackSecurityOrNone(settingsBuilder, esSettings);
+
+try {
   LOG.info("Number of available processors in Netty: {}", 
NettyRuntimeWrapper.availableProcessors());
   // Netty sets available processors statically and if an attempt is 
made to set it more than
   // once an IllegalStateException is thrown by 
NettyRuntime.setAvailableProcessors(NettyRuntime.java:87)
   // 
https://discuss.elastic.co/t/getting-availableprocessors-is-already-set-to-1-rejecting-1-illegalstateexception-exception/103082
   // 
https://discuss.elastic.co/t/elasticsearch-5-4-1-availableprocessors-is-already-set/88036
   System.setProperty("es.set.netty.runtime.available.processors", 
"false");
-  client = new PreBuiltTransportClient(settings);
-  for(HostnamePort hp : getIps(globalConfiguration)) {
+  TransportClient client = 
createTransportClient(settingsBuilder.build(), esSettings);
+  for (HostnamePort hp : getIps(globalConfiguration)) {
 client.addTransportAddress(
 new 
InetSocketTransportAddress(InetAddress.getByName(hp.hostname), hp.port)
 );
   }
-} catch (UnknownHostException exception){
+  return client;
+} catch (UnknownHostException exception) {
   throw new RuntimeException(exception);
 }
-return client;
+  }
+
+  private static Map getEsSettings(Map 
config) {
+return ConversionUtils
+.convertMap((Map) 
config.getOrDefault("es.client.settings", new HashMap()),
+String.class);
+  }
+
+  private static void setXPackSecurityOrNone(Settings.Builder 
settingsBuilder, Map esSettings) {
+if (esSettings.containsKey("es.xpack.password.file")) {
--- End diff --

Done


---

[jira] [Commented] (METRON-1465) X-pack support for Elasticsearch

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425180#comment-16425180
 ] 

ASF GitHub Bot commented on METRON-1465:


Github user wardbekker commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r179060961
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
 ---
@@ -107,32 +113,94 @@ public static String getBaseIndexName(String 
indexName) {
 return parts[0];
   }
 
-  public static TransportClient getClient(Map 
globalConfiguration, Map optionalSettings) {
+  /**
+   * Instantiates an Elasticsearch client based on es.client.class, if 
set. Defaults to
+   * org.elasticsearch.transport.client.PreBuiltTransportClient.
+   *
+   * @param globalConfiguration Metron global config
+   * @return
+   */
+  public static TransportClient getClient(Map 
globalConfiguration) {
+Set customESSettings = new HashSet<>();
+customESSettings.addAll(Arrays.asList("es.client.class", 
"es.xpack.username", "es.xpack.password.file"));
 Settings.Builder settingsBuilder = Settings.builder();
-settingsBuilder.put("cluster.name", 
globalConfiguration.get("es.clustername"));
-settingsBuilder.put("client.transport.ping_timeout","500s");
-if (optionalSettings != null) {
-  settingsBuilder.put(optionalSettings);
+Map esSettings = getEsSettings(globalConfiguration);
+for (Map.Entry entry : esSettings.entrySet()) {
+  String key = entry.getKey();
+  String value = entry.getValue();
+  if (!customESSettings.contains(key)) {
+settingsBuilder.put(key, value);
+  }
 }
-Settings settings = settingsBuilder.build();
-TransportClient client;
-try{
+settingsBuilder.put("cluster.name", 
globalConfiguration.get("es.clustername"));
+settingsBuilder.put("client.transport.ping_timeout", 
esSettings.getOrDefault("client.transport.ping_timeout","500s"));
+setXPackSecurityOrNone(settingsBuilder, esSettings);
+
+try {
   LOG.info("Number of available processors in Netty: {}", 
NettyRuntimeWrapper.availableProcessors());
   // Netty sets available processors statically and if an attempt is 
made to set it more than
   // once an IllegalStateException is thrown by 
NettyRuntime.setAvailableProcessors(NettyRuntime.java:87)
   // 
https://discuss.elastic.co/t/getting-availableprocessors-is-already-set-to-1-rejecting-1-illegalstateexception-exception/103082
   // 
https://discuss.elastic.co/t/elasticsearch-5-4-1-availableprocessors-is-already-set/88036
   System.setProperty("es.set.netty.runtime.available.processors", 
"false");
-  client = new PreBuiltTransportClient(settings);
-  for(HostnamePort hp : getIps(globalConfiguration)) {
+  TransportClient client = 
createTransportClient(settingsBuilder.build(), esSettings);
+  for (HostnamePort hp : getIps(globalConfiguration)) {
 client.addTransportAddress(
 new 
InetSocketTransportAddress(InetAddress.getByName(hp.hostname), hp.port)
 );
   }
-} catch (UnknownHostException exception){
+  return client;
+} catch (UnknownHostException exception) {
   throw new RuntimeException(exception);
 }
-return client;
+  }
+
+  private static Map getEsSettings(Map 
config) {
+return ConversionUtils
+.convertMap((Map) 
config.getOrDefault("es.client.settings", new HashMap()),
+String.class);
+  }
+
+  private static void setXPackSecurityOrNone(Settings.Builder 
settingsBuilder, Map esSettings) {
+if (esSettings.containsKey("es.xpack.password.file")) {
--- End diff --

Done


> X-pack support for Elasticsearch
> 
>
> Key: METRON-1465
> URL: https://issues.apache.org/jira/browse/METRON-1465
> Project: Metron
>  Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: Ward Bekker
>Priority: Major
> Fix For: 0.4.3
>
>
> Provide support for X-pack secured Elasticsearch clusters for the 
> Elasticsearch writer and the DAO used by the rest service.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)