date:20170425

http://stackoverflow.com/questions/22150209/maven-changes-the-order-of-plugins-of-different-profiles
https://issues.apache.org/jira/browse/MNG-2258

On April 25, 2017 at 22:33:58, Otto Fowler (ottobackwa...@gmail.com) wrote:

So, the copy resources configuration is still there, and not governed by
the profile.
Is it possible that since they both run in package, that they are out of
order?

On April 25, 2017 at 22:31:27, Otto Fowler (ottobackwa...@gmail.com) wrote:

Having said that, i see the problem.
I am very sorry

How do I go about resubmitting if I can fix it?

On April 25, 2017 at 22:29:56, Otto Fowler (ottobackwa...@gmail.com) wrote:

OK, I’ll take a look.  Sorry - I thought they read out of the RPMS dir and
verified that they were produced both ways.
I am not sure how my change would change the target copy.

On April 25, 2017 at 22:25:04, Justin Leet (justinjl...@gmail.com) wrote:

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler 
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: [DISCUSS] Regression introduced in Full Dev

So, the copy resources configuration is still there, and not governed by
the profile.
Is it possible that since they both run in package, that they are out of
order?

On April 25, 2017 at 22:31:27, Otto Fowler (ottobackwa...@gmail.com) wrote:

Having said that, i see the problem.
I am very sorry

How do I go about resubmitting if I can fix it?

On April 25, 2017 at 22:29:56, Otto Fowler (ottobackwa...@gmail.com) wrote:

OK, I’ll take a look.  Sorry - I thought they read out of the RPMS dir and
verified that they were produced both ways.
I am not sure how my change would change the target copy.

On April 25, 2017 at 22:25:04, Justin Leet (justinjl...@gmail.com) wrote:

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler 
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: [DISCUSS] Regression introduced in Full Dev

Having said that, i see the problem.
I am very sorry

How do I go about resubmitting if I can fix it?

On April 25, 2017 at 22:29:56, Otto Fowler (ottobackwa...@gmail.com) wrote:

OK, I’ll take a look.  Sorry - I thought they read out of the RPMS dir and
verified that they were produced both ways.
I am not sure how my change would change the target copy.

On April 25, 2017 at 22:25:04, Justin Leet (justinjl...@gmail.com) wrote:

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler 
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

Re: [DISCUSS] Regression introduced in Full Dev

2017-04-25 Thread Justin Leet

The problem seems to be that full/quick dev rely on having the RPMs in the
/target folder

>From metron-deployment/roles/metron-rpms/defaults:
metron_rpm_glob: "{{ playbook_dir
}}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm"

However, that PR appears to no longer copy to the /target folder run as
expected, so quick and full dev are unable to find the RPMs.

On Tue, Apr 25, 2017 at 9:26 PM, Otto Fowler 
wrote:

> Ryan, did you see why it was not working?
>
>
> On April 25, 2017 at 18:17:29, Ryan Merriman (merrim...@gmail.com) wrote:
>
> A regression was introduced recently that breaks full dev. I've narrowed
> down the commit that introduced it and have submitted a PR to revert that
> commit: https://github.com/apache/incubator-metron/pull/549.
>
> Given there has been confusion recently over our deployment build process,
> I think it's appropriate that we discuss and come to a consensus and on
> how this should work.
>
> Ryan
>

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #551: METRON-892 add docker to platform info

2017-04-25 Thread ottobackwards

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/551

METRON-892 add docker to platform info

## Contributor Comments
Now that docker is a requirement, we need to include the version in the 
platform info script

Tested on mac and centos6
run script on a handy machine
## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ NA] Have you included steps to reproduce the behavior or problem that 
is being changed or addressed?
- [ x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ NA] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [NA ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ NA] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [NA ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ NA] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-892

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/551.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #551


commit 23ba776582ffc5c03ec961f5c3d46fd5689bca98
Author: Otto Fowler 
Date:   2017-04-26T02:00:27Z

METRON-892 add docker to platform info




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: auto-install on bare metal

I failed at this today, but maybe it was the way I tried.
An example would be great.

On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:

Hi Dima,

The same Ansible playbooks that work for EC2 and Vagrant will work for bare
metal installations. The only difference is that you would need to
pre-provision your machines and hand-build your inventory file. The AWS
playbooks only provision the machines. All deployment of Metron is handled
(for all installation types) by the metron_full_install playbook [1].

-D...

[1]
https://github.com/apache/incubator-metron/blob/master/metron-deployment/playbooks/metron_full_install.yml

On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
wrote:

> Hello Metron Team,
>
> We have developed a script that performs auto-install of the Metron on
> bare metal machines, but still working on few issues here and there.
>
> I am curios as to what automate solutions we do have for Metron
> installation right now?
> The ones I am aware of are in
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/:

> a) AWS Ansible install (1 or 10 nodes)
> b) Vagrant local VM setup
>
> Is there any other solution available? Has anyone managed to use AWS
> Ansible playbooks for bare metal installation?
>
> - Dima
>
>
>

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/542#discussion_r113348853
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
 ---
@@ -263,7 +263,56 @@ public void exitVariable(StellarParser.VariableContext 
ctx) {
 
   @Override
   public void exitStringLiteral(StellarParser.StringLiteralContext ctx) {
-expression.tokenDeque.push(new Token<>(ctx.getText().substring(1, 
ctx.getText().length() - 1), String.class));
+String rawToken = ctx.getText();
+String literal = unquoteLiteral(rawToken);
+
+expression.tokenDeque.push(new Token<>(literal, String.class));
+  }
+
--- End diff --

Dude!  Great catch!  We're actually closer to JSON escaping because our 
strings can start with either `'` or `"`, so I used that escaping strategy and 
it worked just fine.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #542: METRON-873: Stellar string literals do n...

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/542#discussion_r113347347
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
 ---
@@ -263,7 +263,56 @@ public void exitVariable(StellarParser.VariableContext 
ctx) {
 
   @Override
   public void exitStringLiteral(StellarParser.StringLiteralContext ctx) {
-expression.tokenDeque.push(new Token<>(ctx.getText().substring(1, 
ctx.getText().length() - 1), String.class));
+String rawToken = ctx.getText();
+String literal = unquoteLiteral(rawToken);
+
+expression.tokenDeque.push(new Token<>(literal, String.class));
+  }
+
--- End diff --

hmm, let me try, good catch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: auto-install on bare metal

Hi Dima,

The same Ansible playbooks that work for EC2 and Vagrant will work for bare
metal installations. The only difference is that you would need to
pre-provision your machines and hand-build your inventory file. The AWS
playbooks only provision the machines. All deployment of Metron is handled
(for all installation types) by the metron_full_install playbook [1].

-D...

[1]
https://github.com/apache/incubator-metron/blob/master/metron-deployment/playbooks/metron_full_install.yml

On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
wrote:

> Hello Metron Team,
>
> We have developed a script that performs auto-install of the Metron on
> bare metal machines, but still working on few issues here and there.
>
> I am curios as to what automate solutions we do have for Metron
> installation right now?
> The ones I am aware of are in
> https://github.com/apache/incubator-metron/tree/master/metron-deployment/:
> a) AWS Ansible install (1 or 10 nodes)
> b) Vagrant local VM setup
>
> Is there any other solution available? Has anyone managed to use AWS
> Ansible playbooks for bare metal installation?
>
> - Dima
>
>
>

[GitHub] incubator-metron issue #550: METRON-890: Intermittent unit test errors in sh...

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/550
  
yep, that's the belief.  We're going to test it out here, in front of 
everyone. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: metron UI

2017-04-25 Thread Waterson, Kevin

Of course, I should disclaim, this is my own project, and not a Telstra related 
initiative.

Kev

-Original Message-
From: Waterson, Kevin [mailto:kevin.water...@team.telstra.com] 
Sent: Wednesday, 26 April 2017 8:37 AM
To: dev@metron.apache.org
Subject: RE: metron UI

HTML5/D3

Kev

-Original Message-
From: Ryan Merriman [mailto:merrim...@gmail.com]
Sent: Wednesday, 26 April 2017 8:24 AM
To: dev@metron.apache.org
Subject: Re: metron UI

Kevin,

Would be interested in learning more about the issues you have with Angular.  
Which technology would you prefer?

Ryan

On Tue, Apr 25, 2017 at 5:16 PM, Waterson, Kevin < 
kevin.water...@team.telstra.com> wrote:

> The use of Angular has me writing my own UI for Metron
>
> Kev
>
> -Original Message-
> From: Justin Leet [mailto:justinjl...@gmail.com]
> Sent: Tuesday, 25 April 2017 12:19 AM
> To: dev@metron.apache.org
> Subject: Re: metron UI
>
> To elaborate on Ryan's reply a bit, the UI is a fairly new component 
> in Metron and was merged into master not long ago.  It's definitely 
> open to being iterated and improved on, and getting feedback on 
> direction users would like it to go would be a great contribution in 
> and of itself (especially as it gets exercised and people want new 
> features or improvements).
>
> Justin
>
> On Mon, Apr 24, 2017 at 9:59 AM, Ryan Merriman 
> wrote:
>
> > What features would you like to see included?
> >
> > On Mon, Apr 24, 2017 at 8:51 AM, moshe jarusalem 
> wrote:
> >
> > > Hi All,
> > > I have recently run metron UI. I am a bit surprised because it has 
> > > very
> > few
> > > features such as configuring some ingestions and topologies.
> > >
> > > Have I not configured it properly or its features are limited at 
> > > this
> > time
> > > ?
> > >
> > > Regards,
> > >
> >
>

[GitHub] incubator-metron pull request #545: METRON-883 Capture Bro Plugin Enhancemen...

2017-04-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/545


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Regression introduced in Full Dev

2017-04-25 Thread Casey Stella

Yeah, I tend to agree that a rundown of the various methods and when you
would use them is in order. I will say that full-dev is especially
important to have working since it is required for validating PRs.
On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com  wrote:

> Can somebody map out all of the current methods and procedures to spin up
> Metron components?  I swear I find out about new ones every month.
> Metron-docker, the 4 vagrants, rpm-docker, ansible-docker, any others?
> Perhaps an agreed upon write up of when to use what would be helpful.
>
> Jon
>
> On Tue, Apr 25, 2017, 6:17 PM Ryan Merriman  wrote:
>
> > A regression was introduced recently that breaks full dev.  I've narrowed
> > down the commit that introduced it and have submitted a PR to revert that
> > commit:  https://github.com/apache/incubator-metron/pull/549.
> >
> > Given there has been confusion recently over our deployment build
> process,
> > I think it's appropriate that we discuss and come to a consensus  and on
> > how this should work.
> >
> > Ryan
> >
> --
>
> Jon
>

[GitHub] incubator-metron issue #550: METRON-890: Intermittent unit test errors in sh...

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/550
  
Is the belief here that killing the topologies first might allow full 
cluster shutdown and cleanup?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: metron UI

2017-04-25 Thread Ryan Merriman

Kevin,

Would be interested in learning more about the issues you have with
Angular.  Which technology would you prefer?

Ryan

On Tue, Apr 25, 2017 at 5:16 PM, Waterson, Kevin <
kevin.water...@team.telstra.com> wrote:

> The use of Angular has me writing my own UI for Metron
>
> Kev
>
> -Original Message-
> From: Justin Leet [mailto:justinjl...@gmail.com]
> Sent: Tuesday, 25 April 2017 12:19 AM
> To: dev@metron.apache.org
> Subject: Re: metron UI
>
> To elaborate on Ryan's reply a bit, the UI is a fairly new component in
> Metron and was merged into master not long ago.  It's definitely open to
> being iterated and improved on, and getting feedback on direction users
> would like it to go would be a great contribution in and of itself
> (especially as it gets exercised and people want new features or
> improvements).
>
> Justin
>
> On Mon, Apr 24, 2017 at 9:59 AM, Ryan Merriman 
> wrote:
>
> > What features would you like to see included?
> >
> > On Mon, Apr 24, 2017 at 8:51 AM, moshe jarusalem 
> wrote:
> >
> > > Hi All,
> > > I have recently run metron UI. I am a bit surprised because it has
> > > very
> > few
> > > features such as configuring some ingestions and topologies.
> > >
> > > Have I not configured it properly or its features are limited at
> > > this
> > time
> > > ?
> > >
> > > Regards,
> > >
> >
>

RE: metron UI

2017-04-25 Thread Waterson, Kevin

The use of Angular has me writing my own UI for Metron

Kev 

-Original Message-
From: Justin Leet [mailto:justinjl...@gmail.com] 
Sent: Tuesday, 25 April 2017 12:19 AM
To: dev@metron.apache.org
Subject: Re: metron UI

To elaborate on Ryan's reply a bit, the UI is a fairly new component in Metron 
and was merged into master not long ago.  It's definitely open to being 
iterated and improved on, and getting feedback on direction users would like it 
to go would be a great contribution in and of itself (especially as it gets 
exercised and people want new features or improvements).

Justin

On Mon, Apr 24, 2017 at 9:59 AM, Ryan Merriman  wrote:

> What features would you like to see included?
>
> On Mon, Apr 24, 2017 at 8:51 AM, moshe jarusalem  wrote:
>
> > Hi All,
> > I have recently run metron UI. I am a bit surprised because it has 
> > very
> few
> > features such as configuring some ingestions and topologies.
> >
> > Have I not configured it properly or its features are limited at 
> > this
> time
> > ?
> >
> > Regards,
> >
>

[DISCUSS] Regression introduced in Full Dev

2017-04-25 Thread Ryan Merriman

A regression was introduced recently that breaks full dev.  I've narrowed
down the commit that introduced it and have submitted a PR to revert that
commit:  https://github.com/apache/incubator-metron/pull/549.

Given there has been confusion recently over our deployment build process,
I think it's appropriate that we discuss and come to a consensus  and on
how this should work.

Ryan

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #549: METRON-889: Regression introduced in Ful...

GitHub user merrimanr opened a pull request:

https://github.com/apache/incubator-metron/pull/549

METRON-889: Regression introduced in Full Dev

## Contributor Comments
Full dev is currently broken because the RPM files are not being copied to 
vagrant.  I've determined the regression was introduced with 
https://github.com/apache/incubator-metron/commit/8c264f8e7345620c2a6c155d8d32e396ce30ec91.
  This PR reverts that commit until we can figure out the right way to achieve 
the desired outcome of that PR. 

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-889

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/549.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #549


commit 32959fb2fc6c988b4119e8c0e1a65bac702fe99c
Author: merrimanr 
Date:   2017-04-25T21:47:05Z

Revert "METRON-857 Metron should one unified docker build image 
(ottobackwards) closes apache/incubator-metron#543"

This reverts commit 8c264f8e7345620c2a6c155d8d32e396ce30ec91.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Storm topology.classpath setting

2017-04-25 Thread Justin Leet

topology.classpath is a topology level config, so Mike's potential
alternative is to migrate that down from Ambari into the actual topologies
we launch (assuming it works as expected).

Basically we're trading ease of use in Ambari vs. additional setup for each
topology that's run.

I'm in favor of pushing the config down to each topology (again, assuming
it works), despite the additional topology work, because it seems more
likely that users are going to see the issues from not restarting Storm and
Metron immediately after install/start than they are from missing configs
at the Flux level.  The topology config just becomes a bullet point in
instructions on creating a new topology, rather than a surprising and
nonintuitive issue resulting from install.

Once Metron installs and starts through Ambari, I would expect it to work
immediately.

On Tue, Apr 25, 2017 at 5:14 PM, Casey Stella  wrote:

> I don't know that I'm offended by either approach, but the way we have it
> now isn't offending me overly, honestly.
>
> Casey
>
> On Tue, Apr 25, 2017 at 5:11 PM, David Lyle  wrote:
>
> > I think you have one more restart than is strictly required but I get
> your
> > point. IIRC, when I did the original implementation there was a bit of a
> > wrinkle getting the right configs on the right classpath so I punted in
> > favor of topology.classpath.
> >
> > If you've got a more efficient way, I'm all for it.
> >
> > -D...
> >
> >
> > On Tue, Apr 25, 2017 at 4:55 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > We currently define topology.classpath at the global level for Storm.
> The
> > > reason for this feature was to enable setting the classpath to include
> > > things like hbase-site and core-site without modifying the topology jar
> > > files on the cluster. That made things simpler, but this also has the
> > > consequence that Metron's installation process via Ambari looks
> something
> > > like this:
> > >
> > >1. Install Metron
> > >2. Start Metron - part of install process
> > >3. Restart Storm  - bc of the new property that has been added that
> > >needs to be distributed
> > >4. Restart Metron - the topologies won't get the new classpath
> > otherwise
> > >
> > > We're in effect restarting Metron a couple times, which can take some
> > time.
> > >
> > > A possible alternative here is to set the classpath on the individual
> > > topologies. By doing so, we remove the need to restart Storm and Metron
> > > during the install. The down side is that this requires us to duplicate
> > > this setting for every flux file or topology, including newly-added
> > > topologies. What do folks think about which would be better?
> > >
> > > Best,
> > > Mike
> > >
> >
>

Re: Ambari Wizard: Repo Tab

Nm.  sorry.  I fixed it.



On April 25, 2017 at 16:42:05, Otto Fowler (ottobackwa...@gmail.com) wrote:

Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
the f://localrepo
even though the folder exists, there is no repodata/repomd.xml.

What is the command to create a local repo?



On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com) wrote:

I was going by the HW community page.

Ok, Let me try it



On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler 
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler 
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 93, in checked_call
> > >
> > > tries=tries, try_sleep=try_sleep)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 141, in _call_wrapper
> > >
> > > result = _call(command, **kwargs_copy)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 294, in _call
> > >
> > > raise Fail(err_msg)
> > >
> > > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum
> -d
> > 0
> > > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> > >
> > > stdout:
> > >
> > > 2017-04-25 14:12:48,669
> > >  - Using hadoop
> > conf
> > > dir: /usr/hdp/current/hadoop-client/conf
> > >
> > > 2017-04-25 14:12:48,671
> > >  -
> Group['metron']
> > {}
> > >
> > > 2017-04-25 14:12:48,673
> > >  - Adding group
> > > Group['metron']
> > >
> > > 2017-04-25

Re: Ambari Wizard: Repo Tab

2017-04-25 Thread Nick Allen

createrepo

On Tue, Apr 25, 2017 at 4:42 PM, Otto Fowler 
wrote:

> Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
> the f://localrepo
> even though the folder exists, there is no repodata/repomd.xml.
>
> What is the command to create a local repo?
>
>
>
> On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I was going by the HW community page.
>
> Ok, Let me try it
>
>
>
> On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:
>
> That would do it. It requires 2.4.2+. I would have sworn I put that in the
> README, but I must have only annotated the PR. :(
>
> I'll get that in asap.
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler 
> wrote:
>
> > 2.4.1 as a matter of fact
> >
> >
> > On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
> >
> > Any chance they're running Ambari < 2.4.2?
> >
> > -D...
> >
> >
> > On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Hey Otto, I don't have wizard screenshots in front of me right now to
> > say
> > > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > > anything like that, but next time I run through the install I can take
> > > another look.
> > >
> > > https://github.com/apache/incubator-metron/blob/master/
> > > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> > >
> > >
> > > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler  >
> > > wrote:
> > >
> > > > stderr:
> > > >
> > > > Traceback (most recent call last):
> > > >
> > > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > > >
> > > > Elasticsearch().execute()
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > libraries/script/script.py",
> > > > line 280, in execute
> > > >
> > > > method(env)
> > > >
> > > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > > >
> > > > self.install_packages(env)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > libraries/script/script.py",
> > > > line 567, in install_packages
> > > >
> > > > retry_count=agent_stack_retry_count)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/base.py",
> > > > line 155, in __init__
> > > >
> > > > self.env.run()
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/environment.py",
> > > > line 160, in run
> > > >
> > > > self.run_action(resource, action)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/environment.py",
> > > > line 124, in run_action
> > > >
> > > > provider_action()
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > > core/providers/package/__init__.py", line 54, in action_install
> > > >
> > > > self.install_package(package_name, self.resource.use_repos,
> > > > self.resource.skip_repos)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > > core/providers/package/yumrpm.py", line 49, in install_package
> > > >
> > > > self.checked_call_with_retries(cmd, sudo=True,
> > > > logoutput=self.get_logoutput())
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > > core/providers/package/__init__.py", line 83, in
> > > checked_call_with_retries
> > > >
> > > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > > >
> > > > code, out = func(cmd, **kwargs)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/shell.py",
> > > > line 71, in inner
> > > >
> > > > result = function(command, **kwargs)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/shell.py",
> > > > line 93, in checked_call
> > > >
> > > > tries=tries, try_sleep=try_sleep)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/shell.py",
> > > > line 141, in _call_wrapper
> > > >
> > > > result = _call(command, **kwargs_copy)
> > > >
> > > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/shell.py",
> > > > line 294, in _call
> > > >
> > > > raise Fail(err_msg)
> > > >
> > > > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum
> > -d
> > > 0
> > > > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> > > >
> > > > stdout:
> > > >
> > > > 2017-04-25 14:12:48,669
> > > >  - Using hadoop
> > > conf
> > > > dir:

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/541
  
# Testing Plan
## Preliminaries

* Please perform the following tests on the `full-dev` vagrant environment.
* Set an environment variable to indicate `METRON_HOME`:
`export METRON_HOME=/usr/metron/0.4.0` 


## Ensure Data Flows from the Indices
Ensure that with a basic full-dev we get data into the elasticsearch
indices and into HDFS.

## (Optional) Free Up Space on the virtual machine

First, let's free up some headroom on the virtual machine.  If you are 
running this on a
multinode cluster, you would not have to do this.
* Stop and disable Metron in Ambari
* Kill monit via `service monit stop`
* From ambari, stop the metron service
* Kill the sensors via `service sensor-stubs stop`

## Install and start pycapa 
```
# set env vars
export PYCAPA_HOME=/opt/pycapa
export PYTHON27_HOME=/opt/rh/python27/root

# Install these packages via yum (RHEL, CentOS)
yum -y install epel-release centos-release-scl 
yum -y install "@Development tools" python27 python27-scldevel 
python27-python-virtualenv libpcap-devel libselinux-python

# Setup directories
mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME

#Grab pycapa from git 
cd ~
git clone https://github.com/apache/incubator-metron.git
cp -R ~/incubator-metron/metron-sensors/pycapa* $PYCAPA_HOME

# Create virtualenv
export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv

# Build it
cd ${PYCAPA_HOME}/pycapa
# activate the virtualenv
source ${PYCAPA_HOME}/pycapa-venv/bin/activate
pip install -r requirements.txt
python setup.py install

# Run it
cd ${PYCAPA_HOME}/pycapa-venv/bin
pycapa --producer --topic pcap -i eth1 -k node1:6667
```
## Ensure pycapa can write to HDFS
* Ensure that `/apps/metron/pcap` exists and can be written to by the
  storm user.  If not, then:
```
sudo su - hdfs
hadoop fs -mkdir -p /apps/metron/pcap
hadoop fs -chown metron:hadoop /apps/metron/pcap
hadoop fs -chmod 775 /apps/metron/pcap
exit
``` 
* Start the pcap topology via `$METRON_HOME/bin/start_pcap_topology.sh`
* Watch the topology in the Storm UI and kill the packet capture utility 
from before, when the number of packets ingested is over 3k.  Ensure that at at 
least 3 files exist on HDFS by running `hadoop fs -ls /apps/metron/pcap`


Note that if your MR job fails because of a lack of user directory for 
`root`, then the following will create the directory appropriately:
```
sudo su - hdfs
hadoop fs -mkdir /user/root
hadoop fs -chown root:hadoop /user/root
hadoop fs -chmod 755 /user/root
exit
```
### Regression Test

 Fixed 
* Run a fixed pcap query by executing a command similar to the following:
```
$METRON_HOME/bin/pcap_query.sh fixed --ip_dst_port 8080 -st "20170425" -df 
"MMdd"
```
* Verify the MR job finishes successfully. Upon completion, you should see 
multiple files named with relatively current datestamps in your current 
directory, e.g. pcap-data-20160617160549737+.pcap
* Copy the files to your local machine and verify you can them it in 
Wireshark. Open the files and ensure that they contain only packets to the 
destination port of 8080.

 Stellar 
* Run a fixed pcap query by executing a command similar to the following:
```
$METRON_HOME/bin/pcap_query.sh query --query "ip_dst_port == 8080" -st 
"20170425" -df "MMdd"
```
* Verify the MR job finishes successfully. Upon completion, you should see 
multiple files named with relatively current datestamps in your current 
directory, e.g. pcap-data-20160617160549737+.pcap
* Copy the files to your local machine and verify you can them it in 
Wireshark. Open the files and ensure that they contain only packets to the 
destination port of 8080.

### Binary Payload Search : Strings

 Fixed 
* Run a fixed pcap query by executing a command similar to the following:
```
$METRON_HOME/bin/pcap_query.sh fixed --ip_dst_port 8080 --packet_filter 
"\`persist\`" -st "20170425" -df "MMdd"
```
* Verify the MR job finishes successfully. Upon completion, you should see 
multiple files named with relatively current datestamps in your current 
directory, e.g. pcap-data-20160617160549737+.pcap
* Copy the files to your local machine and verify you can them it in 
Wireshark. Open the files and ensure that they contain only packets to the 
destination port of 8080 and are api calls involving ` 
/api/v1/persist/wizard-data` in Ambari.

 Stellar

Re: Quick Dev - Atlas Images

If it is docker, then it is just the Dockerfiles.
I think you can have vagrant use docker as a back end too right?



On April 25, 2017 at 14:34:14, Nick Allen (n...@nickallen.org) wrote:

>> I hadn't really reasoned about the notion of a "released" Quick Dev
image,
but I can see a lot of value in having a versioned sandbox type image- but
maybe not quick dev, maybe not even Vagrant? We could actually pre-package
everything needed and save some provisioning time on released versions.

I really like the idea. I think it would be very beneficial to have a
single pre-packaged image for each release that users can download and take
new features for a spin.

If we do stick with Vagrant for this I think Atlas works just fine. Who
else is going to host a 5.1 GB image for us? :) Although I am very open to
alternative implementations of this idea.


>> I always thought of Quick Dev as a developer tool, so our obligation is
to make it work with the current master and any branches currently used by
devs.

Would be interested to get other opinions on this. I am good with making
that assumption. Whatever the community agrees on for Quick Dev though,
we should document it as such. Right now, I think it would be reasonable
to assume that a user would download a release and expect to be able to
launch Quick Dev.









On Mon, Apr 24, 2017 at 11:51 AM, David Lyle  wrote:

> I think it's a really good idea. There is some complexity:
>
> a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> allow -SNAPSHOT in their release number scheme. That is, we'll have
> different versions of the image that work with a 0.4.1-SNAPSHOT Metron
and
> some released versions of Metron that won't require a new image (A guy's
> gotta believe). I think that can be easily worked around.
>
> b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> likely the wrong place to host it.
>
> I always thought of Quick Dev as a developer tool, so our obligation is
to
> make it work with the current master and any branches currently used by
> devs. I hadn't really reasoned about the notion of a "released" Quick Dev
> image, but I can see a lot of value in having a versioned sandbox type
> image- but maybe not quick dev, maybe not even Vagrant? We could actually
> pre-package everything needed and save some provisioning time on released
> versions. It'd just come up ready to go. I think, should we want one, we
> should release it as a convenience binary signed and hosted alongside the
> other release artifacts. Meantime, we could keep the incremental versions
> of Quick Dev in Atlas.
>
> Anyway, I think it's a really interesting notion.
>
> -D...
>
>
> On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen  wrote:
>
> > Right now, we have the images that get pushed to Atlas for Quick Dev
> >  versioned
> > independently from the rest of Metron. We currently have versions 0.1.0
> > and 0.2.0.
> >
> > What happens when a user downloads an official release of Metron, like
> > 0.3.1, and attempts to run Quick Dev? I would assume that the code
would
> > download the latest image version, which we may have been updated since
> the
> > release. This would cause it to fail for the release version. Am I
> wrong?
> >
> > If we had the Atlas images follow Metron's versioning scheme, would
this
> > solve the problem? Are there other cons of doing this?
> >
> > Thanks
> >
>

Re: Ambari Wizard: Repo Tab

I was going by the HW community page.

Ok, Let me try it



On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler 
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler 
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 93, in checked_call
> > >
> > > tries=tries, try_sleep=try_sleep)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 141, in _call_wrapper
> > >
> > > result = _call(command, **kwargs_copy)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 294, in _call
> > >
> > > raise Fail(err_msg)
> > >
> > > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum
> -d
> > 0
> > > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> > >
> > > stdout:
> > >
> > > 2017-04-25 14:12:48,669
> > >  - Using hadoop
> > conf
> > > dir: /usr/hdp/current/hadoop-client/conf
> > >
> > > 2017-04-25 14:12:48,671
> > >  -
> Group['metron']
> > {}
> > >
> > > 2017-04-25 14:12:48,673
> > >  - Adding group
> > > Group['metron']
> > >
> > > 2017-04-25 14:12:48,740
> > >  - Group['livy']
> {}
> > >
> > > 2017-04-25 14:12:48,741
> > >  - Adding group
> > > Group['livy']
> > >
> > > 2017-04-25 14:12:48,774
> > >  -
> > > Group['elasticsearch'] {}
> > >
> > > 2017-04-25 14:12:48,775

Re: Ambari Wizard: Repo Tab

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler 
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler 
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 93, in checked_call
> > >
> > > tries=tries, try_sleep=try_sleep)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 141, in _call_wrapper
> > >
> > > result = _call(command, **kwargs_copy)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 294, in _call
> > >
> > > raise Fail(err_msg)
> > >
> > > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum
> -d
> > 0
> > > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> > >
> > > stdout:
> > >
> > > 2017-04-25 14:12:48,669
> > >  - Using hadoop
> > conf
> > > dir: /usr/hdp/current/hadoop-client/conf
> > >
> > > 2017-04-25 14:12:48,671
> > >  -
> Group['metron']
> > {}
> > >
> > > 2017-04-25 14:12:48,673
> > >  - Adding group
> > > Group['metron']
> > >
> > > 2017-04-25 14:12:48,740
> > >  - Group['livy']
> {}
> > >
> > > 2017-04-25 14:12:48,741
> > >  - Adding group
> > > Group['livy']
> > >
> > > 2017-04-25 14:12:48,774
> > >  -
> > > Group['elasticsearch'] {}
> > >
> > > 2017-04-25 14:12:48,775
> > >  - Adding group
> > > Group['elasticsearch']
> > >
> > > 2017-04-25

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/531

I would love to see Metron have a solution for both approaches - ingesting
DHCP server logs, as well as DHCP observations based on network traffic. Like
@ottobackwards mentioned, not everyone can get the right
infrastructure/viewpoint on their network to run something like Bro and get the
DHCP traffic to their sensors to be processed.

I have definitely sent more than just DNS and HTTP from Bro to Metron and
it has been properly ingested, but to date I haven't done DHCP. Like
@simonellistonball and @nickwallen mentioned, both the parser and the kafka
plugin are setup to handle new bro logs quite well, and a while back I worked
on updating Metron's support for more Bro sources via
[METRON-508](https://github.com/JonZeolla/incubator-metron/commit/736cc39525f9f08f6e781faea2610e893327e74c).
I just never had a chance to test it, so I haven't yet opened a PR.

Once #545 and #547 get merged into master, and I'm able to finish
[METRON-813](https://issues.apache.org/jira/browse/METRON-813), I would be
happy to work on anything related to Bro and DHCP logs at scale, including
finishing up METRON-508. I have two hardware bro environments and my larger
one currently sees about 7 million DHCP observations/day and sends ~30,000
messages per second into Metron.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
It would take some work but I think it could be done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Quick Dev - Atlas Images

2017-04-25 Thread Michael Miklavcic

I may have missed a discuss thread on this, but it looks like we are no
longer using -SNAPSHOT in our current dev master
https://github.com/apache/incubator-metron/blob/master/pom.xml#L19

On Tue, Apr 25, 2017 at 12:34 PM, Nick Allen  wrote:

> >>  I hadn't really reasoned about the notion of a "released" Quick Dev
> image,
> but I can see a lot of value in having a versioned sandbox type image- but
> maybe not quick dev, maybe not even Vagrant? We could actually pre-package
> everything needed and save some provisioning time on released versions.
>
> I really like the idea.  I think it would be very beneficial to have a
> single pre-packaged image for each release that users can download and take
> new features for a spin.
>
> If we do stick with Vagrant for this I think Atlas works just fine.  Who
> else is going to host a 5.1 GB image for us? :)  Although I am very open to
> alternative implementations of this idea.
>
>
> >> I always thought of Quick Dev as a developer tool, so our obligation is
> to make it work with the current master and any branches currently used by
> devs.
>
> Would be interested to get other opinions on this.  I am good with making
> that assumption.   Whatever the community agrees on for Quick Dev though,
> we should document it as such.  Right now, I think it would be reasonable
> to assume that a user would download a release and expect to be able to
> launch Quick Dev.
>
>
>
>
>
>
>
>
>
> On Mon, Apr 24, 2017 at 11:51 AM, David Lyle  wrote:
>
> > I think it's a really good idea. There is some complexity:
> >
> > a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> > allow -SNAPSHOT in their release number scheme. That is, we'll have
> > different versions of the image that work with a 0.4.1-SNAPSHOT Metron
> and
> > some released versions of Metron that won't require a new image (A guy's
> > gotta believe). I think that can be easily worked around.
> >
> > b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> > likely the wrong place to host it.
> >
> > I always thought of Quick Dev as a developer tool, so our obligation is
> to
> > make it work with the current master and any branches currently used by
> > devs. I hadn't really reasoned about the notion of a "released" Quick Dev
> > image, but I can see a lot of value in having a versioned sandbox type
> > image- but maybe not quick dev, maybe not even Vagrant? We could actually
> > pre-package everything needed and save some provisioning time on released
> > versions. It'd just come up ready to go. I think, should we want one, we
> > should release it as a convenience binary signed and hosted alongside the
> > other release artifacts. Meantime, we could keep the incremental versions
> > of Quick Dev in Atlas.
> >
> > Anyway, I think it's a really interesting notion.
> >
> > -D...
> >
> >
> > On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen  wrote:
> >
> > > Right now, we have the images that get pushed to Atlas for Quick Dev
> > >  versioned
> > > independently from the rest of Metron.  We currently have versions
> 0.1.0
> > > and 0.2.0.
> > >
> > > What happens when a user downloads an official release of Metron, like
> > > 0.3.1, and attempts to run Quick Dev?  I would assume that the code
> would
> > > download the latest image version, which we may have been updated since
> > the
> > > release.  This would cause it to fail for the release version.  Am I
> > wrong?
> > >
> > > If we had the Atlas images follow Metron's versioning scheme, would
> this
> > > solve the problem?  Are there other cons of doing this?
> > >
> > > Thanks
> > >
> >
>

Re: Ambari Wizard: Repo Tab

2.4.1 as a matter of fact


On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:

Any chance they're running Ambari < 2.4.2?

-D...


On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Hey Otto, I don't have wizard screenshots in front of me right now to say
> for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> anything like that, but next time I run through the install I can take
> another look.
>
> https://github.com/apache/incubator-metron/blob/master/
> metron-deployment/packaging/ambari/metron-mpack/src/main/
> resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
>
>
> On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler 
> wrote:
>
> > stderr:
> >
> > Traceback (most recent call last):
> >
> > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> >
> > Elasticsearch().execute()
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> libraries/script/script.py",
> > line 280, in execute
> >
> > method(env)
> >
> > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> >
> > self.install_packages(env)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> libraries/script/script.py",
> > line 567, in install_packages
> >
> > retry_count=agent_stack_retry_count)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/base.py",
> > line 155, in __init__
> >
> > self.env.run()
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/environment.py",
> > line 160, in run
> >
> > self.run_action(resource, action)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/environment.py",
> > line 124, in run_action
> >
> > provider_action()
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/__init__.py", line 54, in action_install
> >
> > self.install_package(package_name, self.resource.use_repos,
> > self.resource.skip_repos)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/yumrpm.py", line 49, in install_package
> >
> > self.checked_call_with_retries(cmd, sudo=True,
> > logoutput=self.get_logoutput())
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/__init__.py", line 83, in
> checked_call_with_retries
> >
> > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/providers/package/__init__.py", line 91, in _call_with_retries
> >
> > code, out = func(cmd, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 71, in inner
> >
> > result = function(command, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 93, in checked_call
> >
> > tries=tries, try_sleep=try_sleep)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 141, in _call_wrapper
> >
> > result = _call(command, **kwargs_copy)
> >
> > File "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > line 294, in _call
> >
> > raise Fail(err_msg)
> >
> > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d
> 0
> > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> >
> > stdout:
> >
> > 2017-04-25 14:12:48,669
> >  - Using hadoop
> conf
> > dir: /usr/hdp/current/hadoop-client/conf
> >
> > 2017-04-25 14:12:48,671
> >  - Group['metron']
> {}
> >
> > 2017-04-25 14:12:48,673
> >  - Adding group
> > Group['metron']
> >
> > 2017-04-25 14:12:48,740
> >  - Group['livy']
{}
> >
> > 2017-04-25 14:12:48,741
> >  - Adding group
> > Group['livy']
> >
> > 2017-04-25 14:12:48,774
> >  -
> > Group['elasticsearch'] {}
> >
> > 2017-04-25 14:12:48,775
> >  - Adding group
> > Group['elasticsearch']
> >
> > 2017-04-25 14:12:48,812
> >  - Group['spark']
> {}
> >
> > 2017-04-25 14:12:48,812
> >  - Adding group
> > Group['spark']
> >
> > 2017-04-25 14:12:48,848
> >  -
> Group['zeppelin']
> > {}
> >
> > 2017-04-25 14:12:48,849
> >  - Adding group
> > Group['zeppelin']
> >
> > 2017-04-25 14:12:48,884
> >

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

2017-04-25 Thread JonZeolla

GitHub user JonZeolla opened a pull request:

https://github.com/apache/incubator-metron/pull/547

METRON-858 bro-plugin-kafka is throwing segfaults

## Contributor Comments
This PR is a follow-on of #545.  Please DO NOT MERGE until the outstanding 
items are all completed.

 Outstanding items:
 - [ ] Thoroughly test example 3
 - [ ] Test at scale

The primary change here resolves a thread safety issue that is only seen 
when under load.  It has been reported in numerous places, but I've seen it 
best documented [here](https://github.com/bro/bro-plugins/issues/43).

### Testing
The following steps can be used to validate the PR.  (Mostly extracted from 
METRON-883's testing steps)

1.  Create a working directory.
```
mkdir metron-858
cd metron-858
```
1.  Launch a CentOS host.
```
vagrant init bento/centos-6.7
vagrant up
vagrant ssh
```
1.  Install some dependencies.
```
sudo su -
yum -y install epel-release
yum -y install "@Development tools" java-1.8.0-openjdk cmake 
libpcap-devel openssl-devel python-devel
```
1.  Create a new `HDP.repo` Yum repository; this will allow us to install 
Kafka.
```
cat << EOF > /etc/yum.repos.d/HDP.repo
[HDP-2.5]
name=HDP-2.5

baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0
path=/
enabled=1
gpgcheck=0
EOF
```
1.  Install and start Kafka.
```
yum -y install kafka
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
zookeeper-server start
kafka start
```
1.  Install Librdkafka 0.9.4.
```
wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz  -O - 
| tar -xz
cd librdkafka-0.9.4/
./configure --prefix=/usr
make
make install
```
1.  Add Librdkafka to our default load path.
```
echo "/usr/lib" >> /etc/ld.so.conf.d/bro-plugin.conf
ldconfig -v
```
1.  Build and install Bro.
```
yum -y install cmake libpcap-devel openssl-devel python-devel
wget https://www.bro.org/downloads/release/bro-2.4.1.tar.gz -O 
~/bro-2.4.1.tar.gz
tar -xzf ~/bro-2.4.1.tar.gz -C ~
cd ~/bro-2.4.1
./configure --prefix=/usr
make
make install
```
1.  Fetch the code from this PR.
```
git clone https://github.com/apache/incubator-metron ~/incubator-metron
cd ~/incubator-metron
git pull origin pull/XXX/head
```
1.  Install the Bro Plugin.
```
cd metron-sensors/bro-plugin-kafka
./configure --bro-dist=/root/bro-2.4.1 
--install-root=/usr/lib/bro/plugins/ --with-librdkafka=/usr
make
make install
```
1.  Modify your `/usr/share/bro/site/local.bro`:
```
cat << EOF >> /usr/share/bro/site/local.bro

@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table( ["metadata.broker.list"] = 
"localhost:9092" );
EOF
```
1.  Create a virtual interface called `tap0` to listen on.
```
yum install -y tunctl
tunctl -p
ifconfig tap0 10.0.0.1 up
ip link set tap0 promisc on
```
1.  Configure Bro to listen on virtual interface.
```
sed -i 's/eth0/tap0/g' /usr/etc/node.cfg
```
1.  Create a Kafka topic called `bro`.
```
kafka-topics.sh --zookeeper localhost:2181 --create --topic bro 
--partitions 1 --replication-factor 1
```
1.  Make sure the Bro changes are installed and start Bro.
```
broctl deploy
```
1.  Grab an example pcap file and replay some packet data through `tap0`. 
Keep this running in a separate session.
```
yum -y install tcpreplay
wget 
https://github.com/apache/incubator-metron/raw/master/metron-deployment/roles/sensor-test-mode/files/example.pcap
tcpreplay -i tap0 --loop=0 --stats=5 example.pcap
```
1.  Ensure that data is hitting the `bro` topic in Kafka.
```
# kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro 
--from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is 
expected to increase from one, then you should configure the number of parallel 
GC threads appropriately using -XX:ParallelGCThreads=N
{metadata.broker.list=localhost:9092, request.timeout.ms=3, 
client.id=console-consumer-99442, security.protocol=PLAINTEXT}
{"dns":

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

2017-04-25 Thread mattf-horton

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
@merrimanr but is the update of README.md part of the component build?  Or 
can it be made so?  That way if a developer makes changes/additions in the 
annotations, but misses re-committing the README, git will remind them after 
the build.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

2017-04-25 Thread Michael Miklavcic

Hey Otto, I don't have wizard screenshots in front of me right now to say
for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
anything like that, but next time I run through the install I can take
another look.

https://github.com/apache/incubator-metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml


On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler 
wrote:

> stderr:
>
> Traceback (most recent call last):
>
>   File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> 2.3.3/package/scripts/elastic_slave.py", line 71, in 
>
> Elasticsearch().execute()
>
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 280, in execute
>
> method(env)
>
>   File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> 2.3.3/package/scripts/elastic_slave.py", line 32, in install
>
> self.install_packages(env)
>
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 567, in install_packages
>
> retry_count=agent_stack_retry_count)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
>
> self.env.run()
>
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
>
> self.run_action(resource, action)
>
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
>
> provider_action()
>
>   File "/usr/lib/python2.6/site-packages/resource_management/
> core/providers/package/__init__.py", line 54, in action_install
>
> self.install_package(package_name, self.resource.use_repos,
> self.resource.skip_repos)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/
> core/providers/package/yumrpm.py", line 49, in install_package
>
> self.checked_call_with_retries(cmd, sudo=True,
> logoutput=self.get_logoutput())
>
>   File "/usr/lib/python2.6/site-packages/resource_management/
> core/providers/package/__init__.py", line 83, in checked_call_with_retries
>
> return self._call_with_retries(cmd, is_checked=True, **kwargs)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/
> core/providers/package/__init__.py", line 91, in _call_with_retries
>
> code, out = func(cmd, **kwargs)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 71, in inner
>
> result = function(command, **kwargs)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 93, in checked_call
>
> tries=tries, try_sleep=try_sleep)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 141, in _call_wrapper
>
> result = _call(command, **kwargs_copy)
>
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 294, in _call
>
> raise Fail(err_msg)
>
> resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0
> -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
>
> stdout:
>
> 2017-04-25 14:12:48,669
>  - Using hadoop conf
> dir: /usr/hdp/current/hadoop-client/conf
>
> 2017-04-25 14:12:48,671
>  - Group['metron'] {}
>
> 2017-04-25 14:12:48,673
>  - Adding group
> Group['metron']
>
> 2017-04-25 14:12:48,740
>  - Group['livy'] {}
>
> 2017-04-25 14:12:48,741
>  - Adding group
> Group['livy']
>
> 2017-04-25 14:12:48,774
>  -
> Group['elasticsearch'] {}
>
> 2017-04-25 14:12:48,775
>  - Adding group
> Group['elasticsearch']
>
> 2017-04-25 14:12:48,812
>  - Group['spark'] {}
>
> 2017-04-25 14:12:48,812
>  - Adding group
> Group['spark']
>
> 2017-04-25 14:12:48,848
>  - Group['zeppelin']
> {}
>
> 2017-04-25 14:12:48,849
>  - Adding group
> Group['zeppelin']
>
> 2017-04-25 14:12:48,884
>  - Group['hadoop'] {}
>
> 2017-04-25 14:12:48,885
>  - Adding group
> Group['hadoop']
>
> 2017-04-25 14:12:48,921
>  - Group['kibana'] {}
>
> 2017-04-25 14:12:48,922
>  - Adding group
> Group['kibana']
>
> 2017-04-25 14:12:48,959
>  -

Re: Quick Dev - Atlas Images

2017-04-25 Thread Nick Allen

>>  I hadn't really reasoned about the notion of a "released" Quick Dev image,
but I can see a lot of value in having a versioned sandbox type image- but
maybe not quick dev, maybe not even Vagrant? We could actually pre-package
everything needed and save some provisioning time on released versions.

I really like the idea.  I think it would be very beneficial to have a
single pre-packaged image for each release that users can download and take
new features for a spin.

If we do stick with Vagrant for this I think Atlas works just fine.  Who
else is going to host a 5.1 GB image for us? :)  Although I am very open to
alternative implementations of this idea.


>> I always thought of Quick Dev as a developer tool, so our obligation is
to make it work with the current master and any branches currently used by
devs.

Would be interested to get other opinions on this.  I am good with making
that assumption.   Whatever the community agrees on for Quick Dev though,
we should document it as such.  Right now, I think it would be reasonable
to assume that a user would download a release and expect to be able to
launch Quick Dev.









On Mon, Apr 24, 2017 at 11:51 AM, David Lyle  wrote:

> I think it's a really good idea. There is some complexity:
>
> a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> allow -SNAPSHOT in their release number scheme. That is, we'll have
> different versions of the image that work with a 0.4.1-SNAPSHOT Metron and
> some released versions of Metron that won't require a new image (A guy's
> gotta believe). I think that can be easily worked around.
>
> b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> likely the wrong place to host it.
>
> I always thought of Quick Dev as a developer tool, so our obligation is to
> make it work with the current master and any branches currently used by
> devs. I hadn't really reasoned about the notion of a "released" Quick Dev
> image, but I can see a lot of value in having a versioned sandbox type
> image- but maybe not quick dev, maybe not even Vagrant? We could actually
> pre-package everything needed and save some provisioning time on released
> versions. It'd just come up ready to go. I think, should we want one, we
> should release it as a convenience binary signed and hosted alongside the
> other release artifacts. Meantime, we could keep the incremental versions
> of Quick Dev in Atlas.
>
> Anyway, I think it's a really interesting notion.
>
> -D...
>
>
> On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen  wrote:
>
> > Right now, we have the images that get pushed to Atlas for Quick Dev
> >  versioned
> > independently from the rest of Metron.  We currently have versions 0.1.0
> > and 0.2.0.
> >
> > What happens when a user downloads an official release of Metron, like
> > 0.3.1, and attempts to run Quick Dev?  I would assume that the code would
> > download the latest image version, which we may have been updated since
> the
> > release.  This would cause it to fail for the release version.  Am I
> wrong?
> >
> > If we had the Atlas images follow Metron's versioning scheme, would this
> > solve the problem?  Are there other cons of doing this?
> >
> > Thanks
> >
>

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/incubator-metron/pull/531
  
The Bro parsers is actually pretty generic, and will take whatever json bro 
dumps out. From a quick inspection you should just need to configure the bro 
instance to send out dhcp, and in theory it should just work. (I've not tested 
that though)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

stderr:

Traceback (most recent call last):

  File
"/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py",
line 71, in 

Elasticsearch().execute()

  File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute

method(env)

  File
"/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_slave.py",
line 32, in install

self.install_packages(env)

  File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 567, in install_packages

retry_count=agent_stack_retry_count)

  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 155, in __init__

self.env.run()

  File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 160, in run

self.run_action(resource, action)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 124, in run_action

provider_action()

  File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
line 54, in action_install

self.install_package(package_name, self.resource.use_repos,
self.resource.skip_repos)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py",
line 49, in install_package

self.checked_call_with_retries(cmd, sudo=True,
logoutput=self.get_logoutput())

  File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
line 83, in checked_call_with_retries

return self._call_with_retries(cmd, is_checked=True, **kwargs)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
line 91, in _call_with_retries

code, out = func(cmd, **kwargs)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
71, in inner

result = function(command, **kwargs)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
93, in checked_call

tries=tries, try_sleep=try_sleep)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
141, in _call_wrapper

result = _call(command, **kwargs_copy)

  File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
294, in _call

raise Fail(err_msg)

resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0
-e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do

stdout:

2017-04-25 14:12:48,669
 - Using hadoop conf
dir: /usr/hdp/current/hadoop-client/conf

2017-04-25 14:12:48,671
 - Group['metron'] {}

2017-04-25 14:12:48,673
 - Adding group
Group['metron']

2017-04-25 14:12:48,740
 - Group['livy'] {}

2017-04-25 14:12:48,741
 - Adding group
Group['livy']

2017-04-25 14:12:48,774
 -
Group['elasticsearch'] {}

2017-04-25 14:12:48,775
 - Adding group
Group['elasticsearch']

2017-04-25 14:12:48,812
 - Group['spark'] {}

2017-04-25 14:12:48,812
 - Adding group
Group['spark']

2017-04-25 14:12:48,848
 - Group['zeppelin'] {}

2017-04-25 14:12:48,849
 - Adding group
Group['zeppelin']

2017-04-25 14:12:48,884
 - Group['hadoop'] {}

2017-04-25 14:12:48,885
 - Adding group
Group['hadoop']

2017-04-25 14:12:48,921
 - Group['kibana'] {}

2017-04-25 14:12:48,922
 - Adding group
Group['kibana']

2017-04-25 14:12:48,959
 - Group['users'] {}

2017-04-25 14:12:48,960
 - User['hive']
{'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}

2017-04-25 14:12:48,961
 - Adding user
User['hive']

2017-04-25 14:12:49,227
 - User['storm']
{'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}

2017-04-25 14:12:49,227
 - Adding user
User['storm']

2017-04-25 14:12:49,388
 - User['zookeeper']
{'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}

2017-04-25

Re: Status of METRON-153

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361

This was broken when the rpm builds were introduced ( docker in docker no
no ).
This should work now ( i can indeed build the rpms etc and run ansible
inside of docker now with METRON-857 landed ) with
centos 6x ( although I have not tried it since my testing cluster is centos
7 right now ).




On April 25, 2017 at 13:30:02, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

We didn't come up with another approach Otto. I closed the PR at the time
bc I didn't think we wanted to break with Centos 6 yet, and my PR probably
would have done so. Can you elaborate on "prior capability wrt centos 6.x
work again" a bit?

On Apr 25, 2017 10:59 AM, "Otto Fowler"  wrote:

> Also, I can’t find any list discussion of the issue or other approaches.
>
>
> On April 25, 2017 at 12:48:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> What I am looking for is the prior capability wrt centos 6.x work again (
> which I think my
> pr fixing the docker in docker issue would do ), and have it extended to
> work with 7.x.
>
> I don’t think re-writing all the stuff that is in metron roles is very
> attractive.
>
>
> On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:
>
> So, the current Ansible deployment actually does use blueprints. It
> constructs them from one of small_cluster.yml or single_node_vm.yml using
> the ambari_custer_state module.
>
> Ambari blueprints [1] and the Ambari Install Wizard are actually two
> separate things that are only related because both are available via
> Ambari.
>
> Currently, you should be able to create a blueprint and instantiate a
> cluster on Centos7 without using Ansible at all, though if you wished to,
> you could realize your blueprint in the same way that small_cluster and
> single_node_vm do, or alternatively create some Ansible roles to make the
> REST calls that Ambari requires to stand up a cluster.
>
> -D...
>
>
> [1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>
> On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler 
> wrote:
>
> > Let me clarify, that this is support for automated blueprint deployment
> > with ansible, not the ambari wizard.
> >
> >
> >
> > On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > Just tagging on here to indicate my interest in this - in order to have
> > someone other than me manage the OSs in my Metron cluster, I must run
on
> > RHEL 7. I assume that will be common across many enterprises.
> > Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a
> rough
> > go and I dropped it down on my priority list.
> >
> > Jon
> >
> > On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/METRON-153
> > >
> > > What is the other approach that is mentioned here? Was it
implemented?
> > > The ansible build still does not support this and I was thinking of
> > looking
> > > at it.
> > >
> > --
> >
> > Jon
> >
>

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
I think src/main is fine.  @mattf-horton there are no new files generated, 
this simply updates the metron-rest/README.md file.  In theory you don't have 
to use it and it's not part of any automated workflow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #546: METRON-886 ansible-docker : need to have...

2017-04-25 Thread ottobackwards

GitHub user ottobackwards opened a pull request:

https://github.com/apache/incubator-metron/pull/546

METRON-886 ansible-docker : need to have trailing slash for directory in 
copy for ansible.cfg

## Contributor Comments
On a centos 6 host using docker-io, the COPY fails without the trailing 
slash

Test:
1. build and run the ansible-docker on your normal platform ( mac os x etc )
Should work as before
2. build and run the ansible-docker image on a centos 6.9 machine, with 
latest docker-io installed
Should not fail with cannot stat error on rootfs/root/ansible.cfg


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [ x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/incubator-metron METRON-886

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/546.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #546


commit c9685f9e4ff331a078a34cca3a6e8a0932dd8e5c
Author: Otto Fowler 
Date:   2017-04-25T14:02:56Z

need to have trailing slash for directory in copy




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Status of METRON-153

2017-04-25 Thread Matt Foley

FWIW, we know that several of the changes in the (recently committed)
METRON-634 Mpack bug fixes and improvements, not impacting Ambari 
database (PR#532)
are necessary for a smooth install on Centos7.  These changes are for the 
manual install wizard, but all the non-“description” changes would be needed in 
the blueprint also.  And for that matter the “description” changes may be 
desirable for post-install config management.

Thanks,
--Matt

On 4/25/17, 10:29 AM, "Michael Miklavcic"  wrote:

We didn't come up with another approach Otto. I closed the PR at the time
bc I didn't think we wanted to break with Centos 6 yet, and my PR probably
would have done so. Can you elaborate on "prior capability wrt centos 6.x
work again" a bit?

On Apr 25, 2017 10:59 AM, "Otto Fowler"  wrote:

> Also, I can’t find any list discussion of the issue or other approaches.
>
>
> On April 25, 2017 at 12:48:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> What I am looking for is the prior capability wrt centos 6.x work again (
> which I think my
> pr fixing the docker in docker issue would do ), and have it extended to
> work with 7.x.
>
> I don’t think re-writing all the stuff that is in metron roles is very
> attractive.
>
>
> On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:
>
> So, the current Ansible deployment actually does use blueprints. It
> constructs them from one of small_cluster.yml or single_node_vm.yml using
> the ambari_custer_state module.
>
> Ambari blueprints [1] and the Ambari Install Wizard are actually two
> separate things that are only related because both are available via
> Ambari.
>
> Currently, you should be able to create a blueprint and instantiate a
> cluster on Centos7 without using Ansible at all, though if you wished to,
> you could realize your blueprint in the same way that small_cluster and
> single_node_vm do, or alternatively create some Ansible roles to make the
> REST calls that Ambari requires to stand up a cluster.
>
> -D...
>
>
> [1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>
> On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler 
> wrote:
>
> > Let me clarify, that this is support for automated blueprint deployment
> > with ansible, not the ambari wizard.
> >
> >
> >
> > On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > Just tagging on here to indicate my interest in this - in order to have
> > someone other than me manage the OSs in my Metron cluster, I must run on
> > RHEL 7. I assume that will be common across many enterprises.
> > Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a
> rough
> > go and I dropped it down on my priority list.
> >
> > Jon
> >
> > On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/METRON-153
> > >
> > > What is the other approach that is mentioned here? Was it implemented?
> > > The ansible build still does not support this and I was thinking of
> > looking
> > > at it.
> > >
> > --
> >
> > Jon
> >
>

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/531
  
unless of course someone can't use bro for some reason



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Ambari Wizard: Repo Tab

Hi,
Should I see the repo tab in the ambari wizard for metron at this point?
Asking for a friend who can’t get the wizard to work :)

Re: Status of METRON-153

2017-04-25 Thread Michael Miklavcic

We didn't come up with another approach Otto. I closed the PR at the time
bc I didn't think we wanted to break with Centos 6 yet, and my PR probably
would have done so. Can you elaborate on "prior capability wrt centos 6.x
work again" a bit?

On Apr 25, 2017 10:59 AM, "Otto Fowler"  wrote:

> Also, I can’t find any list discussion of the issue or other approaches.
>
>
> On April 25, 2017 at 12:48:46, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> What I am looking for is the prior capability wrt centos 6.x work again (
> which I think my
> pr fixing the docker in docker issue would do ), and have it extended to
> work with 7.x.
>
> I don’t think re-writing all the stuff that is in metron roles is very
> attractive.
>
>
> On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:
>
> So, the current Ansible deployment actually does use blueprints. It
> constructs them from one of small_cluster.yml or single_node_vm.yml using
> the ambari_custer_state module.
>
> Ambari blueprints [1] and the Ambari Install Wizard are actually two
> separate things that are only related because both are available via
> Ambari.
>
> Currently, you should be able to create a blueprint and instantiate a
> cluster on Centos7 without using Ansible at all, though if you wished to,
> you could realize your blueprint in the same way that small_cluster and
> single_node_vm do, or alternatively create some Ansible roles to make the
> REST calls that Ambari requires to stand up a cluster.
>
> -D...
>
>
> [1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>
> On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler 
> wrote:
>
> > Let me clarify, that this is support for automated blueprint deployment
> > with ansible, not the ambari wizard.
> >
> >
> >
> > On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > Just tagging on here to indicate my interest in this - in order to have
> > someone other than me manage the OSs in my Metron cluster, I must run on
> > RHEL 7. I assume that will be common across many enterprises.
> > Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a
> rough
> > go and I dropped it down on my priority list.
> >
> > Jon
> >
> > On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/METRON-153
> > >
> > > What is the other approach that is mentioned here? Was it implemented?
> > > The ansible build still does not support this and I was thinking of
> > looking
> > > at it.
> > >
> > --
> >
> > Jon
> >
>

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/531
  

We also have a `JSONMapParser` that was contributed after the original Bro 
parser.  The data coming out of the Bro plugin can be configured to be JSON.  
That's how we typically use it anyways.  Can we just use the `JSONMapParser` to 
parse Bro records?   

I cannot think of any reason it wouldn't work off the top of my head, but I 
am not sure.  Then you wouldn't even have to write any code at all.

And also with PR #545 we can now send different Bro record types to their 
own Kafka topics. I have included an example in the README on how you can do 
that.  I only mention that if that becomes a problem for the `JSONMapParser` 
and what you are trying to achieve.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
Agreed this isn't Kerberos related. I just wasn't clear what the file was 
and why it is in test. Any reason the utility can't go in src/main along with 
the readme? Separate Jira, of course, if it's even reasonable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
Should that be in the test src directory? 
`metron-interface/metron-rest/src/test/resources/README.vm`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
The REST API is documented with annotations.  This is standard practice and 
powers the in context documentation in the Swagger UI.  It was requested that 
this documentation also be included in the README so README.vm (a velocity 
template) is provided to generate the API README content from the annotations.

The catch is that you have to keep the READMEs in sync outside of the API 
section.  Or, without it, you would have to keep the README in sync with the 
annotations.  In practice I'm not sure which one is more worse, it's annoying 
either way.  My personal preference would be to reference Swagger for REST 
documentation (and not duplicate it) but I understand why we included it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/531
  
> As an alternative method for getting DHCP data out of pcap, you might 
consider the existing Bro sensor, which essentially does what dhcpdump does...

The current Bro parser only supports HTTP and DNS records coming out of 
Bro.  Bro does a lot more indeed, but we don't have parsers for all of that 
yet.   

I think enhancing the existing Bro parser to support DHCP records would be 
an excellent approach.  The Bro plugin that we have can be configured to land 
any of the record types in Kafka, so that would not be a problem.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Status of METRON-153

Also, I can’t find any list discussion of the issue or other approaches.

On April 25, 2017 at 12:48:46, Otto Fowler (ottobackwa...@gmail.com) wrote:

What I am looking for is the prior capability wrt centos 6.x work again (
which I think my
pr fixing the docker in docker issue would do ), and have it extended to
work with 7.x.

I don’t think re-writing all the stuff that is in metron roles is very
attractive.

On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:

So, the current Ansible deployment actually does use blueprints. It
constructs them from one of small_cluster.yml or single_node_vm.yml using
the ambari_custer_state module.

Ambari blueprints [1] and the Ambari Install Wizard are actually two
separate things that are only related because both are available via
Ambari.

Currently, you should be able to create a blueprint and instantiate a
cluster on Centos7 without using Ansible at all, though if you wished to,
you could realize your blueprint in the same way that small_cluster and
single_node_vm do, or alternatively create some Ansible roles to make the
REST calls that Ambari requires to stand up a cluster.

-D...

[1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler 
wrote:

> Let me clarify, that this is support for automated blueprint deployment
> with ansible, not the ambari wizard.
>
>
>
> On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Just tagging on here to indicate my interest in this - in order to have
> someone other than me manage the OSs in my Metron cluster, I must run on
> RHEL 7. I assume that will be common across many enterprises.
> Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
> go and I dropped it down on my priority list.
>
> Jon
>
> On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-153
> >
> > What is the other approach that is mentioned here? Was it implemented?
> > The ansible build still does not support this and I was thinking of
> looking
> > at it.
> >
> --
>
> Jon
>

Re: metron_full_install playbook with small_cluster and multiple hosts

I have a 5 node cluster ( with deploy machine/docker ).  I want used to be
able
to run the full install with a custom inventory for this cluster to deploy.

I am trying it again ( post docker fix ), but with centos 7.

EC2 doesn’t work for everyone.  unless you are saying I can use that to a
local cluster.

On April 25, 2017 at 12:32:17, David Lyle (dlyle65...@gmail.com) wrote:

Hi Otto,

I don't use a custom inventory that I create, but I do run an EC2 build
very frequently, most recently on Friday. That does use the small_cluster
file.

1. It's used because the Ansible Vagrant provisioner doesn't add localhost
(as it should) to the inventory, so [local] works around that. If you're
running under Vagrant, you'll want to set it to 127.0.0.1 or omit it
otherwise.
2. Are you sure? I used run_once and local_action to prevent that when I
originally did it. Through multiple distributed deployments, I only saw one
build being executed. I don't know what you mean by "set to become", Become
is set to false because I didn't want to sudo to execute the build.

-D...

On Tue, Apr 25, 2017 at 10:51 AM, Otto Fowler 
wrote:

> Now that I have docker sorted with building the rpms, I’m trying to
deploy
> using a custom inventory and metron_full_install.
> So the question is, has anyone else tried to do this recently? I may be
> doing something wrong here with a couple of things.
>
> 1. what is the [local] host type in the inventory hosts file for? what
> should it be set to?
> 2. metron_build is set to become for ALL hosts, so it is trying to build
on
> all the hosts. That isn’t right, it should build locally not on the
hosts.
>

Re: Status of METRON-153

What I am looking for is the prior capability wrt centos 6.x work again (
which I think my
pr fixing the docker in docker issue would do ), and have it extended to
work with 7.x.

I don’t think re-writing all the stuff that is in metron roles is very
attractive.

On April 25, 2017 at 12:10:47, David Lyle (dlyle65...@gmail.com) wrote:

So, the current Ansible deployment actually does use blueprints. It
constructs them from one of small_cluster.yml or single_node_vm.yml using
the ambari_custer_state module.

Ambari blueprints [1] and the Ambari Install Wizard are actually two
separate things that are only related because both are available via
Ambari.

Currently, you should be able to create a blueprint and instantiate a
cluster on Centos7 without using Ansible at all, though if you wished to,
you could realize your blueprint in the same way that small_cluster and
single_node_vm do, or alternatively create some Ansible roles to make the
REST calls that Ambari requires to stand up a cluster.

-D...

[1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler 
wrote:

> Let me clarify, that this is support for automated blueprint deployment
> with ansible, not the ambari wizard.
>
>
>
> On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Just tagging on here to indicate my interest in this - in order to have
> someone other than me manage the OSs in my Metron cluster, I must run on
> RHEL 7. I assume that will be common across many enterprises.
> Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a
rough
> go and I dropped it down on my priority list.
>
> Jon
>
> On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-153
> >
> > What is the other approach that is mentioned here? Was it implemented?
> > The ansible build still does not support this and I was thinking of
> looking
> > at it.
> >
> --
>
> Jon
>

Re: metron_full_install playbook with small_cluster and multiple hosts

Hi Otto,

I don't use a custom inventory that I create, but I do run an EC2 build
very frequently, most recently on Friday. That does use the small_cluster
file.

1. It's used because the Ansible Vagrant provisioner doesn't add localhost
(as it should) to the inventory, so [local] works around that. If you're
running under Vagrant, you'll want to set it to 127.0.0.1 or omit it
otherwise.
2. Are you sure? I used run_once and local_action to prevent that when I
originally did it. Through multiple distributed deployments, I only saw one
build being executed. I don't know what you mean by "set to become", Become
is set to false because I didn't want to sudo to execute the build.

-D...

On Tue, Apr 25, 2017 at 10:51 AM, Otto Fowler 
wrote:

> Now that I have docker sorted with building the rpms, I’m trying to deploy
> using a custom inventory and metron_full_install.
> So the question is, has anyone else tried to do this recently?  I may be
> doing something wrong here with a couple of things.
>
> 1. what is the [local] host type in the inventory hosts file for?  what
> should it be set to?
> 2. metron_build is set to become for ALL hosts, so it is trying to build on
> all the hosts.  That isn’t right, it should build locally not on the hosts.
>

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread simonellistonball

Github user simonellistonball commented on the issue:

https://github.com/apache/incubator-metron/pull/531
  
As an alternative method for getting DHCP data out of pcap, you might 
consider the existing Bro sensor, which essentially does what dhcpdump does, 
but for a wider range of protocols, in a more sophisticated way. We also 
already have a built in parser. 

That said it would great to have this parser too for people not looking for 
the full range of bro. 

The multi-line aspect may not be an issue. The boundary for Metron is the 
Kafka message, not really the line, so if you can split the log into multi-line 
chunks prior to kafka, potentially with something like NiFi based on a 
delimiter. The way to do this is to use nifi to insert a true delimiter (not 
end of line) and then use the SplitContent to send individual log entries via 
kafka. It's a little heavy, but solves the multi-line problem as long as you're 
not going to crazy levels of throughput e.g. hundreds of thousands of EPS.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser

2017-04-25 Thread basvdl

Github user basvdl commented on the issue:

https://github.com/apache/incubator-metron/pull/531
  
@nickwallen, these are indeed the options we have discussed...

> I am going to lay out all of the possibilities that I can think of just 
so that we don't leave any stone unturned.
(1) Alter the Source of Telemetry - ...
(2) Use an Alternative Source of Telemetry - ...
(3) Reunite lines at the parser - ...
(4) Transport Mechanism - ...

1. Alter the Source of Telemetry - I agree with you that this is the least 
preferred method.

2. Use an Alternative Source of Telemetry - The alternative I've looked 
into was `tcpdump`, but this is less detailed.

3. Reunite lines at the parser - This will not give you a reliable 
solution, mainly due to the reason you have given: 'We cannot rely on ordering 
of the messages'

4. Transport Mechanism - In our case we are shipping the log using 
(Mi)NiFi. We could look into a custom NiFi processor.

Another option that just came as a brainwave, maybe we can develop a kind 
of yaf / yafscii solution. Where you pipe the output of DHCPDump into the stdin 
of a `DHCPDumpToSingleLine` which will stitch the lines together and output 
single line events to disk.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #535: METRON-859: Use REST application with Kerberos

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/535
  
@merrimanr Just curious, what's the difference between a vm file and an md 
file?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Status of METRON-153

So, the current Ansible deployment actually does use blueprints. It
constructs them from one of small_cluster.yml or single_node_vm.yml using
the ambari_custer_state module.

Ambari blueprints [1] and the Ambari Install Wizard are actually two
separate things that are only related because both are available via
Ambari.

Currently, you should be able to create a blueprint and instantiate a
cluster on Centos7 without using Ansible at all, though if you wished to,
you could realize your blueprint in the same way that small_cluster and
single_node_vm do, or alternatively create some Ansible roles to make the
REST calls that Ambari requires to stand up a cluster.

-D...

[1] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints

On Tue, Apr 25, 2017 at 11:35 AM, Otto Fowler 
wrote:

> Let me clarify, that this is support for automated blueprint deployment
> with ansible, not the ambari wizard.
>
>
>
> On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> Just tagging on here to indicate my interest in this - in order to have
> someone other than me manage the OSs in my Metron cluster, I must run on
> RHEL 7. I assume that will be common across many enterprises.
> Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
> go and I dropped it down on my priority list.
>
> Jon
>
> On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
> wrote:
>
> > https://issues.apache.org/jira/browse/METRON-153
> >
> > What is the other approach that is mentioned here? Was it implemented?
> > The ansible build still does not support this and I was thinking of
> looking
> > at it.
> >
> --
>
> Jon
>

[GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/507
  
Merge with master again


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Status of METRON-153

Let me clarify, that this is support for automated blueprint deployment
with ansible, not the ambari wizard.

On April 25, 2017 at 11:31:37, zeo...@gmail.com (zeo...@gmail.com) wrote:

Just tagging on here to indicate my interest in this - in order to have
someone other than me manage the OSs in my Metron cluster, I must run on
RHEL 7. I assume that will be common across many enterprises.
Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
go and I dropped it down on my priority list.

Jon

On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
wrote:

> https://issues.apache.org/jira/browse/METRON-153
>
> What is the other approach that is mentioned here? Was it implemented?
> The ansible build still does not support this and I was thinking of
looking
> at it.
>
-- 

Jon

Re: Status of METRON-153

2017-04-25 Thread zeo...@gmail.com

Just tagging on here to indicate my interest in this - in order to have
someone other than me manage the OSs in my Metron cluster, I must run on
RHEL 7.  I assume that will be common across many enterprises.
Semi-recentlyI took a stab at CentOS 7 support but it was a bit of a rough
go and I dropped it down on my priority list.

Jon

On Tue, Apr 25, 2017 at 11:28 AM Otto Fowler 
wrote:

> https://issues.apache.org/jira/browse/METRON-153
>
> What is the other approach that is mentioned here?  Was it implemented?
> The ansible build still does not support this and I was thinking of looking
> at it.
>
-- 

Jon

Status of METRON-153

https://issues.apache.org/jira/browse/METRON-153

What is the other approach that is mentioned here?  Was it implemented?
The ansible build still does not support this and I was thinking of looking
at it.

[GitHub] incubator-metron pull request #521: METRON-835 Use Profiler with Kerberos

2017-04-25 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/521


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #521: METRON-835 Use Profiler with Kerberos