Re: Reliable place to get latest release info for script

[Dima Kovalyov]

Thank you Matt,

That is great reply and exactly what I am looking for until 777 PR is in
progress.

- Dima

On 04/27/2017 01:46 AM, Matt Foley wrote:
> Hi Dima,
> Regardless of the github release area, etc., the official release repository 
> for Apache Metron are via the Apache mirrors at
>   http://www.apache.org/dyn/closer.cgi/incubator/metron/
> soon to be moved to
>   http://www.apache.org/dyn/closer.cgi/metron/
>
> Please go there, pick a mirror, and browse that repository, eg at
>   http://mirrors.gigenet.com/apache/incubator/metron/ 
> The numbered top-level subdirectories under these URIs represent released 
> versions of Metron.  By looking at those you can determine the currently 
> available releases.  For a given mirror, you can script the determination.
>
> The corresponding location for Release Candidates is
>   https://dist.apache.org/repos/dist/dev/incubator/metron/
> soon to be moved to
>   https://dist.apache.org/repos/dist/dev/metron/
>
> DO NOT DOWNLOAD LARGE OR FREQUENT AMOUNTS OF CONTENT FROM 
> dist.apache.org/repos/dist/dev/ .  It is not mirrored, and is only intended 
> for developer use.
>
> Again, the numbering system is obvious.  The 0.4.0-RC1 is here, and RC2 will 
> be going up shortly.
>
> Once you’ve determined the version you want, you can deduce the github tag 
> and clone from it, if that is more convenient than using the tarball from the 
> Apache release repo.
>
> Hope this helps,
> --Matt
>
>
> On 4/26/17, 11:09 AM, "Dima Kovalyov"  wrote:
>
> Otto, had a look at 777 PR. Yes, it looks like something we would use.
> When do you think it will be released?
> 
> And in meantime the question still stands, is there a way to get current 
> release id/branch for the script?
> Thank you.
> 
> - Dima
> 
> On 04/26/2017 10:56 PM, Otto Fowler wrote:
> No,  Take a look at METRON-777 PR.
> What I want is an extension mechanism, where you can:
> 
> 1. create a project from a metron maven archetype
> 2. produce an extension ( parser, enrichment, stellar lib, ?? )
> 3. install that extension through the management UI and configure
> 
> All without touching the metron code.
> 
> 
> 
> 
> On April 26, 2017 at 12:38:28, Dima Kovalyov 
> (dima.koval...@sstech.us) wrote:
> 
> Regarding parsers and enrichment, new java based topologies for bluecoat, 
> msexchange, msserver, asa, etc.
> 
> Batching, we use NiFi + Kylo for both stream and batch, but batch data 
> send to java based tool that passes data through parsers/enrichments classes 
> and stores in Hive.
> 
> So what you suggesting is to have internal repo with all our changes 
> alone and to merge it with Apache Metron whenever we need to get full build?
> 
> - Dima
> 
> On 04/26/2017 09:09 PM, Otto Fowler wrote:
> Thanks!
> 
> Parsers, and Enrichments, we have a plan or idea about. When you say 
> extends batch processing… can you say what tech/component/part of stack you 
> extended?
> 
> New topologies?  New WriterBolts? etc?
> 
> I would like as complete a picture as possible of the things for which 
> anyone would say:
> “If I could write this outside the metron tree, i could not have to 
> maintain a private company fork”
> 
> 
> 
> 
> 
> On April 26, 2017 at 11:22:21, Dima Kovalyov 
> (dima.koval...@sstech.us) wrote:
> 
> Otto,
> 
> Yes, we developed custom parsers and enrichments (in future we seek to 
> open source them, but have no time to up them to the decent level currently). 
> We also merged our custom tools that extends Metron batching processing and 
> store data in HDFS and Hive to be used by tableau, zeppelin, etc.
> 
> Is that answers your question? Let me know if you want to know more.
> 
> - Dima
> 
> On 04/26/2017 08:29 PM, Otto Fowler wrote:
> Can I ask Dima, as comfortable you are in describing, what areas do you 
> make changes or additions to that require you to maintain a fork?
> 
> Custom parsers?  Enrichments? Other?
> 
> I have done some work, and have more planned and lined up to try to 
> eliminate the requirement to develop in the metron tree to extend the 
> product, and would be interested in your cases.
> 
> 
> 
> On April 26, 2017 at 10:46:03, Dima Kovalyov 
> (dima.koval...@sstech.us) wrote:
> 
> Hello,
> 
> We want to merge latest metron release branch with out develpoment every
> time it is get released. What is the path we should take here in order
> to identify what release happened successfully?
> 
> Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
> (which will take a while) following links already have 0.4.0:
> 

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113576968
  
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -94,6 +95,52 @@ event bro_init()
 }
 ```
 
+### Example 3
+
+As documented in 
[METRON-285](https://issues.apache.org/jira/browse/METRON-285) and 
[METRON-286](https://issues.apache.org/jira/browse/METRON-286), various 
components in Metron do not currently support IPv6.  Because of this, you may 
not want to send bro logs that contain IPv6 source or destination IPs into 
Metron.  In this example, we are assuming a somewhat standard bro configuration 
for sending logs into a Metron cluster, such that:
+ * Each type of bro log is sent to the `bro` topic, but is tagged with the 
appropriate log type (such as `http`, `dns`, or `conn`).  This is done by 
setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving 
it unset), and by setting `tag_json` to true.
+ * The Kafka writer is set appropriately to send logs to the `bro` Kafka 
topic being used in your Metron cluster.  This requires that your `kafka_conf` 
and `$config` tables are appropriately configured.
+
--- End diff --

My goal was just to be explicit.  I can take another stab at it tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.

commit 420da379cb78341c6e55a234e26cfc6d2546ed41
Author: cstella 
Date:   2017-04-26T14:51:12Z

Updating controller integration test to not be intermittently stupid.

commit 31156dc6cd2ecfddf50074554a4c83c6f2063eaf
Author: cstella 
Date:   2017-04-26T17:42:09Z

Dump threads when we can't figure things out.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Thanks Nick.  I don’t think we catch exceptions at all from the execute
calls.  I wonder what the best practice is?
I can’t seem to find the ambari mpack programming documentation ;)


On April 26, 2017 at 16:50:18, Nick Allen (n...@nickallen.org) wrote:

I can create the JIRA.  I capture the logs.

On Wed, Apr 26, 2017 at 4:48 PM, Nick Allen  wrote:

> Yes, Otto.  I just experienced that myself.  It is a bug that we should
> create a JIRA for.
>
> I was able to work around it by just using "start" instead of "restart".
>
> On Wed, Apr 26, 2017 at 3:41 PM, Otto Fowler 
> wrote:
>
>> Ok, I did ifconfig, i mean ip a and got the right values
>> that and the brackets and it looks like es is up.
>>
>> Now my next problem is prob. a bug.
>>
>> Ambari thinks it needs to restart ‘metron’.
>> the enrichment restart crashes because we don’t catch the storm exception
>> that you are stopping a topology that was not running.
>>
>> The question is - is the state wrong _and_ the exception catch or not.
>>
>> I am guessing I should start a Jira for that one?
>>
>>
>>
>> On April 26, 2017 at 15:31:05, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> So now I’m getting errors for ETH0
>> I think that is one of the things that has to change on centos7?
>> Can you check what you have?
>>
>>
>>
>> On April 26, 2017 at 15:20:18, Nick Allen (n...@nickallen.org) wrote:
>>
>> Yes, I am also running on CentOS 7.
>>
>> On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler 
>> wrote:
>>
>>> Would i have to change that on Centos 7?
>>>
>>>
>>> On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
>>>
>>> Check on the `network_hosts` setting. I just ran into the same issue. I
>>> had to add brackets around it otherwise the file was not valid YAML and
>>> it
>>> would crash trying to load the config file.
>>>
>>> network.host: ["_lo:ipv4_","_eth0:ipv4_"]
>>>
>>> On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler 
>>> wrote:
>>>
>>> > I think it is literally crashing trying to load the config file
>>> >
>>> >
>>> >
>>> > On April 26, 2017 at 15:00:25, Michael Miklavcic (
>>> > michael.miklav...@gmail.com) wrote:
>>> >
>>> > A couple more things you should check, given this setup:
>>> >
>>> > - 1 master node (not as data node)
>>> > - 2 data nodes
>>> >
>>> > I'd use something like this:
>>> >
>>> > - gateway_recover_after_data_nodes=1 or 2
>>> > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
>>> > the index - if you set this value to 2, it would need 1 more data node
>>> > to
>>> > attain 1 orig index + 2 replicas)
>>> >
>>> >
>>> >
>>> > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler >> >
>>> > wrote:
>>> >
>>> > > Actually never mind. I am not sure how to read the exception. Is it
>>> > > failing to read the configuration?
>>> > >
>>> > >
>>> > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > So the elasticsearch service is disabled. That is the issue. When ES
>>> > > calls get status ( service elasticsearch status ) it fails.
>>> > > I don’t know systemd too well, but does that mean we didn’t do the
>>> equiv
>>> > > of chkconfig on?
>>> > >
>>> > >
>>> > >
>>> > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > Getting the same error in ES as before from what I can see
>>> > >
>>> > >
>>> > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > I still can’t keep ES running, it dies.
>>> > > Also, I’m getting exceptions from enrichment trying to restart when
>>> the
>>> > > topology wasn’t running.
>>> > >
>>> > > Is there some coordination in this work? Should I just log jiras?
>>> > > If someone is already sussing out ES, we can sync up
>>> > >
>>> > >
>>> > >
>>> > > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
>>> > wrote:
>>> > >
>>> > > I'm also interested to know why that's important at such a small
>>> scale.
>>> > >
>>> > > Jon
>>> > >
>>> > > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 
>>> > > wrote:
>>> > >
>>> > > > I am following
>>> > > >
>>> > > > https://community.hortonworks.com/articles/60805/deploying-
>>> > > a-fresh-metron-cluster-using-ambari-serv.html
>>> > > > I DO have master and data node together.
>>> > > > Why is that a problem?
>>> > > >
>>> > > > I will try again with master and data node separate.
>>> > > >
>>> > > >
>>> > > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
>>> > > > michael.miklav...@gmail.com) wrote:
>>> > > >
>>> > > > Hey Otto,
>>> > > >
>>> > > > How do you have the ES nodes configured? For a base install I would
>>> > > setup 1
>>> > > > master (NOT as data node) and 2 data nodes (NOT on the same node
>>> as the
>>> > > > master). This is the install configuration I got working. You can
>>> also
>>> > > > modify some configuration 

Re: Ambari Wizard: Repo Tab

[Nick Allen]

I can create the JIRA.  I capture the logs.

On Wed, Apr 26, 2017 at 4:48 PM, Nick Allen  wrote:

> Yes, Otto.  I just experienced that myself.  It is a bug that we should
> create a JIRA for.
>
> I was able to work around it by just using "start" instead of "restart".
>
> On Wed, Apr 26, 2017 at 3:41 PM, Otto Fowler 
> wrote:
>
>> Ok, I did ifconfig, i mean ip a and got the right values
>> that and the brackets and it looks like es is up.
>>
>> Now my next problem is prob. a bug.
>>
>> Ambari thinks it needs to restart ‘metron’.
>> the enrichment restart crashes because we don’t catch the storm exception
>> that you are stopping a topology that was not running.
>>
>> The question is - is the state wrong _and_ the exception catch or not.
>>
>> I am guessing I should start a Jira for that one?
>>
>>
>>
>> On April 26, 2017 at 15:31:05, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> So now I’m getting errors for ETH0
>> I think that is one of the things that has to change on centos7?
>> Can you check what you have?
>>
>>
>>
>> On April 26, 2017 at 15:20:18, Nick Allen (n...@nickallen.org) wrote:
>>
>> Yes, I am also running on CentOS 7.
>>
>> On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler 
>> wrote:
>>
>>> Would i have to change that on Centos 7?
>>>
>>>
>>> On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
>>>
>>> Check on the `network_hosts` setting. I just ran into the same issue. I
>>> had to add brackets around it otherwise the file was not valid YAML and
>>> it
>>> would crash trying to load the config file.
>>>
>>> network.host: ["_lo:ipv4_","_eth0:ipv4_"]
>>>
>>> On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler 
>>> wrote:
>>>
>>> > I think it is literally crashing trying to load the config file
>>> >
>>> >
>>> >
>>> > On April 26, 2017 at 15:00:25, Michael Miklavcic (
>>> > michael.miklav...@gmail.com) wrote:
>>> >
>>> > A couple more things you should check, given this setup:
>>> >
>>> > - 1 master node (not as data node)
>>> > - 2 data nodes
>>> >
>>> > I'd use something like this:
>>> >
>>> > - gateway_recover_after_data_nodes=1 or 2
>>> > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
>>> > the index - if you set this value to 2, it would need 1 more data node
>>> > to
>>> > attain 1 orig index + 2 replicas)
>>> >
>>> >
>>> >
>>> > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler >> >
>>> > wrote:
>>> >
>>> > > Actually never mind. I am not sure how to read the exception. Is it
>>> > > failing to read the configuration?
>>> > >
>>> > >
>>> > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > So the elasticsearch service is disabled. That is the issue. When ES
>>> > > calls get status ( service elasticsearch status ) it fails.
>>> > > I don’t know systemd too well, but does that mean we didn’t do the
>>> equiv
>>> > > of chkconfig on?
>>> > >
>>> > >
>>> > >
>>> > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > Getting the same error in ES as before from what I can see
>>> > >
>>> > >
>>> > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
>>> > > wrote:
>>> > >
>>> > > I still can’t keep ES running, it dies.
>>> > > Also, I’m getting exceptions from enrichment trying to restart when
>>> the
>>> > > topology wasn’t running.
>>> > >
>>> > > Is there some coordination in this work? Should I just log jiras?
>>> > > If someone is already sussing out ES, we can sync up
>>> > >
>>> > >
>>> > >
>>> > > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
>>> > wrote:
>>> > >
>>> > > I'm also interested to know why that's important at such a small
>>> scale.
>>> > >
>>> > > Jon
>>> > >
>>> > > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 
>>> > > wrote:
>>> > >
>>> > > > I am following
>>> > > >
>>> > > > https://community.hortonworks.com/articles/60805/deploying-
>>> > > a-fresh-metron-cluster-using-ambari-serv.html
>>> > > > I DO have master and data node together.
>>> > > > Why is that a problem?
>>> > > >
>>> > > > I will try again with master and data node separate.
>>> > > >
>>> > > >
>>> > > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
>>> > > > michael.miklav...@gmail.com) wrote:
>>> > > >
>>> > > > Hey Otto,
>>> > > >
>>> > > > How do you have the ES nodes configured? For a base install I would
>>> > > setup 1
>>> > > > master (NOT as data node) and 2 data nodes (NOT on the same node
>>> as the
>>> > > > master). This is the install configuration I got working. You can
>>> also
>>> > > > modify some configuration properties around master node as data
>>> node,
>>> > > index
>>> > > > replicas, and gateway recovery to get it working differently, but
>>> this
>>> > is
>>> > > > what will work OOTB with the default config settings from the
>>> mpack. If
>>> > > > 

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113557275
  
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -94,6 +95,60 @@ event bro_init()
 }
 ```
 
+### Example 3
+
+As documented in 
[METRON-285](https://issues.apache.org/jira/browse/METRON-285) and 
[METRON-286](https://issues.apache.org/jira/browse/METRON-286), various 
components in Metron do not currently support IPv6.  Because of this, you may 
not want to send bro logs that contain IPv6 source or destination IPs into 
Metron.  In this example, we are assuming a somewhat standard bro configuration 
for sending logs into a Metron cluster, such that:
+ * Each type of bro log is sent to the `bro` topic, but is tagged with the 
appropriate log type (such as `http`, `dns`, or `conn`).  This is done by 
setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving 
it unset), and by setting `tag_json` to true.
+ * The Kafka writer is set appropriately to send logs to the `bro` Kafka 
topic being used in your Metron cluster.  This requires that your `kafka_conf` 
and `$config` tables are appropriately configured.
+
+```
+@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "bro";
--- End diff --

We have to set `topic_name` to empty string otherwise `logs-to-kafka.bro` 
will create its own filters.

```
redef Kafka::topic_name = "";
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I can’t reproduce now, and the log rolled, so I’m not going to log it until
I see it again.



On April 26, 2017 at 15:47:48, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

I thought we had addressed that at some point, so if it's blowing up still
on restart when parsers are down, then I would file a Jira.

On Wed, Apr 26, 2017 at 1:41 PM, Otto Fowler 
wrote:

> Ok, I did ifconfig, i mean ip a and got the right values
> that and the brackets and it looks like es is up.
>
> Now my next problem is prob. a bug.
>
> Ambari thinks it needs to restart ‘metron’.
> the enrichment restart crashes because we don’t catch the storm exception
> that you are stopping a topology that was not running.
>
> The question is - is the state wrong _and_ the exception catch or not.
>
> I am guessing I should start a Jira for that one?
>
>
>
> On April 26, 2017 at 15:31:05, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So now I’m getting errors for ETH0
> I think that is one of the things that has to change on centos7?
> Can you check what you have?
>
>
>
> On April 26, 2017 at 15:20:18, Nick Allen (n...@nickallen.org) wrote:
>
> Yes, I am also running on CentOS 7.
>
> On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler 
> wrote:
>
> > Would i have to change that on Centos 7?
> >
> >
> > On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
> >
> > Check on the `network_hosts` setting. I just ran into the same issue. I
> > had to add brackets around it otherwise the file was not valid YAML and
> it
> > would crash trying to load the config file.
> >
> > network.host: ["_lo:ipv4_","_eth0:ipv4_"]
> >
> > On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler 
> > wrote:
> >
> > > I think it is literally crashing trying to load the config file
> > >
> > >
> > >
> > > On April 26, 2017 at 15:00:25, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > A couple more things you should check, given this setup:
> > >
> > > - 1 master node (not as data node)
> > > - 2 data nodes
> > >
> > > I'd use something like this:
> > >
> > > - gateway_recover_after_data_nodes=1 or 2
> > > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies
of
> > > the index - if you set this value to 2, it would need 1 more data
node
> > > to
> > > attain 1 orig index + 2 replicas)
> > >
> > >
> > >
> > > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler  >
> > > wrote:
> > >
> > > > Actually never mind. I am not sure how to read the exception. Is it
> > > > failing to read the configuration?
> > > >
> > > >
> > > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)

> > > > wrote:
> > > >
> > > > So the elasticsearch service is disabled. That is the issue. When
ES
> > > > calls get status ( service elasticsearch status ) it fails.
> > > > I don’t know systemd too well, but does that mean we didn’t do the
> > equiv
> > > > of chkconfig on?
> > > >
> > > >
> > > >
> > > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)

> > > > wrote:
> > > >
> > > > Getting the same error in ES as before from what I can see
> > > >
> > > >
> > > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)

> > > > wrote:
> > > >
> > > > I still can’t keep ES running, it dies.
> > > > Also, I’m getting exceptions from enrichment trying to restart when
> the
> > > > topology wasn’t running.
> > > >
> > > > Is there some coordination in this work? Should I just log jiras?
> > > > If someone is already sussing out ES, we can sync up
> > > >
> > > >
> > > >
> > > > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
> > > wrote:
> > > >
> > > > I'm also interested to know why that's important at such a small
> scale.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 

> > > > wrote:
> > > >
> > > > > I am following
> > > > >
> > > > > https://community.hortonworks.com/articles/60805/deploying-
> > > > a-fresh-metron-cluster-using-ambari-serv.html
> > > > > I DO have master and data node together.
> > > > > Why is that a problem?
> > > > >
> > > > > I will try again with master and data node separate.
> > > > >
> > > > >
> > > > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
> > > > > michael.miklav...@gmail.com) wrote:
> > > > >
> > > > > Hey Otto,
> > > > >
> > > > > How do you have the ES nodes configured? For a base install I
would
> > > > setup 1
> > > > > master (NOT as data node) and 2 data nodes (NOT on the same node
as
> > the
> > > > > master). This is the install configuration I got working. You can
> > also
> > > > > modify some configuration properties around master node as data
> node,
> > > > index
> > > > > replicas, and gateway recovery to get it working differently, but
> > this
> > > is
> > > > > what will work OOTB with the default config settings from the
> mpack.
> > If
> > > > > you've already 

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.

commit 420da379cb78341c6e55a234e26cfc6d2546ed41
Author: cstella 
Date:   2017-04-26T14:51:12Z

Updating controller integration test to not be intermittently stupid.

commit 31156dc6cd2ecfddf50074554a4c83c6f2063eaf
Author: cstella 
Date:   2017-04-26T17:42:09Z

Dump threads when we can't figure things out.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Michael Miklavcic]

I thought we had addressed that at some point, so if it's blowing up still
on restart when parsers are down, then I would file a Jira.

On Wed, Apr 26, 2017 at 1:41 PM, Otto Fowler 
wrote:

> Ok, I did ifconfig, i mean ip a and got the right values
> that and the brackets and it looks like es is up.
>
> Now my next problem is prob. a bug.
>
> Ambari thinks it needs to restart ‘metron’.
> the enrichment restart crashes because we don’t catch the storm exception
> that you are stopping a topology that was not running.
>
> The question is - is the state wrong _and_ the exception catch or not.
>
> I am guessing I should start a Jira for that one?
>
>
>
> On April 26, 2017 at 15:31:05, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So now I’m getting errors for ETH0
> I think that is one of the things that has to change on centos7?
> Can you check what you have?
>
>
>
> On April 26, 2017 at 15:20:18, Nick Allen (n...@nickallen.org) wrote:
>
> Yes, I am also running on CentOS 7.
>
> On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler 
> wrote:
>
> > Would i have to change that on Centos 7?
> >
> >
> > On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
> >
> > Check on the `network_hosts` setting. I just ran into the same issue. I
> > had to add brackets around it otherwise the file was not valid YAML and
> it
> > would crash trying to load the config file.
> >
> > network.host: ["_lo:ipv4_","_eth0:ipv4_"]
> >
> > On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler 
> > wrote:
> >
> > > I think it is literally crashing trying to load the config file
> > >
> > >
> > >
> > > On April 26, 2017 at 15:00:25, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > A couple more things you should check, given this setup:
> > >
> > > - 1 master node (not as data node)
> > > - 2 data nodes
> > >
> > > I'd use something like this:
> > >
> > > - gateway_recover_after_data_nodes=1 or 2
> > > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
> > > the index - if you set this value to 2, it would need 1 more data node
> > > to
> > > attain 1 orig index + 2 replicas)
> > >
> > >
> > >
> > > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler  >
> > > wrote:
> > >
> > > > Actually never mind. I am not sure how to read the exception. Is it
> > > > failing to read the configuration?
> > > >
> > > >
> > > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
> > > > wrote:
> > > >
> > > > So the elasticsearch service is disabled. That is the issue. When ES
> > > > calls get status ( service elasticsearch status ) it fails.
> > > > I don’t know systemd too well, but does that mean we didn’t do the
> > equiv
> > > > of chkconfig on?
> > > >
> > > >
> > > >
> > > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
> > > > wrote:
> > > >
> > > > Getting the same error in ES as before from what I can see
> > > >
> > > >
> > > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
> > > > wrote:
> > > >
> > > > I still can’t keep ES running, it dies.
> > > > Also, I’m getting exceptions from enrichment trying to restart when
> the
> > > > topology wasn’t running.
> > > >
> > > > Is there some coordination in this work? Should I just log jiras?
> > > > If someone is already sussing out ES, we can sync up
> > > >
> > > >
> > > >
> > > > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
> > > wrote:
> > > >
> > > > I'm also interested to know why that's important at such a small
> scale.
> > > >
> > > > Jon
> > > >
> > > > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 
> > > > wrote:
> > > >
> > > > > I am following
> > > > >
> > > > > https://community.hortonworks.com/articles/60805/deploying-
> > > > a-fresh-metron-cluster-using-ambari-serv.html
> > > > > I DO have master and data node together.
> > > > > Why is that a problem?
> > > > >
> > > > > I will try again with master and data node separate.
> > > > >
> > > > >
> > > > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
> > > > > michael.miklav...@gmail.com) wrote:
> > > > >
> > > > > Hey Otto,
> > > > >
> > > > > How do you have the ES nodes configured? For a base install I would
> > > > setup 1
> > > > > master (NOT as data node) and 2 data nodes (NOT on the same node as
> > the
> > > > > master). This is the install configuration I got working. You can
> > also
> > > > > modify some configuration properties around master node as data
> node,
> > > > index
> > > > > replicas, and gateway recovery to get it working differently, but
> > this
> > > is
> > > > > what will work OOTB with the default config settings from the
> mpack.
> > If
> > > > > you've already setup a master node and a data node on the same
> host,
> > > > we'll
> > > > > need to re-install.
> > > > >
> > > > > Mike
> > > > >
> > > > >
> > > > > On Wed, Apr 26, 2017 at 7:02 AM, 

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.

commit 420da379cb78341c6e55a234e26cfc6d2546ed41
Author: cstella 
Date:   2017-04-26T14:51:12Z

Updating controller integration test to not be intermittently stupid.

commit 31156dc6cd2ecfddf50074554a4c83c6f2063eaf
Author: cstella 
Date:   2017-04-26T17:42:09Z

Dump threads when we can't figure things out.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Nick Allen]

Yes, I am also running on CentOS 7.

On Wed, Apr 26, 2017 at 3:19 PM, Otto Fowler 
wrote:

> Would i have to change that on Centos 7?
>
>
> On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:
>
> Check on the `network_hosts` setting. I just ran into the same issue. I
> had to add brackets around it otherwise the file was not valid YAML and it
> would crash trying to load the config file.
>
> network.host: ["_lo:ipv4_","_eth0:ipv4_"]
>
> On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler 
> wrote:
>
> > I think it is literally crashing trying to load the config file
> >
> >
> >
> > On April 26, 2017 at 15:00:25, Michael Miklavcic (
> > michael.miklav...@gmail.com) wrote:
> >
> > A couple more things you should check, given this setup:
> >
> > - 1 master node (not as data node)
> > - 2 data nodes
> >
> > I'd use something like this:
> >
> > - gateway_recover_after_data_nodes=1 or 2
> > - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
> > the index - if you set this value to 2, it would need 1 more data node
> > to
> > attain 1 orig index + 2 replicas)
> >
> >
> >
> > On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler 
> > wrote:
> >
> > > Actually never mind. I am not sure how to read the exception. Is it
> > > failing to read the configuration?
> > >
> > >
> > > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
> > > wrote:
> > >
> > > So the elasticsearch service is disabled. That is the issue. When ES
> > > calls get status ( service elasticsearch status ) it fails.
> > > I don’t know systemd too well, but does that mean we didn’t do the
> equiv
> > > of chkconfig on?
> > >
> > >
> > >
> > > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
> > > wrote:
> > >
> > > Getting the same error in ES as before from what I can see
> > >
> > >
> > > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
> > > wrote:
> > >
> > > I still can’t keep ES running, it dies.
> > > Also, I’m getting exceptions from enrichment trying to restart when
> the
> > > topology wasn’t running.
> > >
> > > Is there some coordination in this work? Should I just log jiras?
> > > If someone is already sussing out ES, we can sync up
> > >
> > >
> > >
> > > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
> > wrote:
> > >
> > > I'm also interested to know why that's important at such a small
> scale.
> > >
> > > Jon
> > >
> > > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 
> > > wrote:
> > >
> > > > I am following
> > > >
> > > > https://community.hortonworks.com/articles/60805/deploying-
> > > a-fresh-metron-cluster-using-ambari-serv.html
> > > > I DO have master and data node together.
> > > > Why is that a problem?
> > > >
> > > > I will try again with master and data node separate.
> > > >
> > > >
> > > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
> > > > michael.miklav...@gmail.com) wrote:
> > > >
> > > > Hey Otto,
> > > >
> > > > How do you have the ES nodes configured? For a base install I would
> > > setup 1
> > > > master (NOT as data node) and 2 data nodes (NOT on the same node as
> the
> > > > master). This is the install configuration I got working. You can
> also
> > > > modify some configuration properties around master node as data
> node,
> > > index
> > > > replicas, and gateway recovery to get it working differently, but
> this
> > is
> > > > what will work OOTB with the default config settings from the mpack.
> If
> > > > you've already setup a master node and a data node on the same host,
> > > we'll
> > > > need to re-install.
> > > >
> > > > Mike
> > > >
> > > >
> > > > On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler <
> ottobackwa...@gmail.com>
> > > > wrote:
> > > >
> > > > > File
> > > > > "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > > line
> > > > > 293, in _call
> > > > >
> > > > > raise ExecutionFailed(err_msg, code, out, err)
> > > > >
> > > > > ExecutionFailed: Execution of 'service elasticsearch status'
> returned
> > > 3.
> > > > ●
> > > > > elasticsearch.service - Elasticsearch
> > > > >
> > > > > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > > > > disabled;
> > > > > vendor preset: disabled)
> > > > >
> > > > > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58
> EDT
> > > > > ; 10h ago
> > > > >
> > > > > Docs: http://www.elastic.co
> > > > >
> > > > > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
>
> > > > > -Des.pidfile=${PID_DIR}/elasticsearch.pid
> > -Des.default.path.home=${ES_
> > > > > HOME}
> > > > > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
>
> > > > > -Des.default.path.conf=${CONF_DIR} (code=exited,
> status=1/FAILURE)
> > > > >
> > > > > Process: 16819
> > > > > ExecStartPre=/usr/share/elasticsearch/bin/
> > > 

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Would i have to change that on Centos 7?


On April 26, 2017 at 15:12:38, Nick Allen (n...@nickallen.org) wrote:

Check on the `network_hosts` setting. I just ran into the same issue. I
had to add brackets around it otherwise the file was not valid YAML and it
would crash trying to load the config file.

network.host: ["_lo:ipv4_","_eth0:ipv4_"]

On Wed, Apr 26, 2017 at 3:09 PM, Otto Fowler 
wrote:

> I think it is literally crashing trying to load the config file
>
>
>
> On April 26, 2017 at 15:00:25, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> A couple more things you should check, given this setup:
>
> - 1 master node (not as data node)
> - 2 data nodes
>
> I'd use something like this:
>
> - gateway_recover_after_data_nodes=1 or 2
> - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
> the index - if you set this value to 2, it would need 1 more data node
> to
> attain 1 orig index + 2 replicas)
>
>
>
> On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler 
> wrote:
>
> > Actually never mind. I am not sure how to read the exception. Is it
> > failing to read the configuration?
> >
> >
> > On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > So the elasticsearch service is disabled. That is the issue. When ES
> > calls get status ( service elasticsearch status ) it fails.
> > I don’t know systemd too well, but does that mean we didn’t do the
equiv
> > of chkconfig on?
> >
> >
> >
> > On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Getting the same error in ES as before from what I can see
> >
> >
> > On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > I still can’t keep ES running, it dies.
> > Also, I’m getting exceptions from enrichment trying to restart when the
> > topology wasn’t running.
> >
> > Is there some coordination in this work? Should I just log jiras?
> > If someone is already sussing out ES, we can sync up
> >
> >
> >
> > On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com)
> wrote:
> >
> > I'm also interested to know why that's important at such a small scale.
> >
> > Jon
> >
> > On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 
> > wrote:
> >
> > > I am following
> > >
> > > https://community.hortonworks.com/articles/60805/deploying-
> > a-fresh-metron-cluster-using-ambari-serv.html
> > > I DO have master and data node together.
> > > Why is that a problem?
> > >
> > > I will try again with master and data node separate.
> > >
> > >
> > > On April 26, 2017 at 10:41:59, Michael Miklavcic (
> > > michael.miklav...@gmail.com) wrote:
> > >
> > > Hey Otto,
> > >
> > > How do you have the ES nodes configured? For a base install I would
> > setup 1
> > > master (NOT as data node) and 2 data nodes (NOT on the same node as
the
> > > master). This is the install configuration I got working. You can
also
> > > modify some configuration properties around master node as data node,
> > index
> > > replicas, and gateway recovery to get it working differently, but
this
> is
> > > what will work OOTB with the default config settings from the mpack.
If
> > > you've already setup a master node and a data node on the same host,
> > we'll
> > > need to re-install.
> > >
> > > Mike
> > >
> > >
> > > On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 

> > > wrote:
> > >
> > > > File
> > > > "/usr/lib/python2.6/site-packages/resource_management/
> core/shell.py",
> > > line
> > > > 293, in _call
> > > >
> > > > raise ExecutionFailed(err_msg, code, out, err)
> > > >
> > > > ExecutionFailed: Execution of 'service elasticsearch status'
returned
> > 3.
> > > ●
> > > > elasticsearch.service - Elasticsearch
> > > >
> > > > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > > > disabled;
> > > > vendor preset: disabled)
> > > >
> > > > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58
EDT
> > > > ; 10h ago
> > > >
> > > > Docs: http://www.elastic.co
> > > >
> > > > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > > > -Des.pidfile=${PID_DIR}/elasticsearch.pid
> -Des.default.path.home=${ES_
> > > > HOME}
> > > > -Des.default.path.logs=${LOG_DIR}
-Des.default.path.data=${DATA_DIR}
> > > > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> > > >
> > > > Process: 16819
> > > > ExecStartPre=/usr/share/elasticsearch/bin/
> > elasticsearch-systemd-pre-exec
> > > > (code=exited, status=0/SUCCESS)
> > > >
> > > > Main PID: 16821 (code=exited, status=1/FAILURE)
> > > >
> > > >
> > > >
> > > > Apr 25 22:14:58  >
> > > > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]:
> at
> > > > org.elasticsearch.common.settings.Settings$Builder.
> > > > loadFromStream(Settings.java:1080)
> > > >
> > > > Apr 

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[JonZeolla]

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113538317
  
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -94,6 +95,52 @@ event bro_init()
 }
 ```
 
+### Example 3
+
+As documented in 
[METRON-285](https://issues.apache.org/jira/browse/METRON-285) and 
[METRON-286](https://issues.apache.org/jira/browse/METRON-286), various 
components in Metron do not currently support IPv6.  Because of this, you may 
not want to send bro logs that contain IPv6 source or destination IPs into 
Metron.  In this example, we are assuming a somewhat standard bro configuration 
for sending logs into a Metron cluster, such that:
+ * Each type of bro log is sent to the `bro` topic, but is tagged with the 
appropriate log type (such as `http`, `dns`, or `conn`).  This is done by 
setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving 
it unset), and by setting `tag_json` to true.
+ * The Kafka writer is set appropriately to send logs to the `bro` Kafka 
topic being used in your Metron cluster.  This requires that your `kafka_conf` 
and `$config` tables are appropriately configured.
+
--- End diff --

Yes.  Do you think it's too wordy?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/incubator-metron/pull/459
  
@justinleet , I agree with @dlyle65535 that if the intermittent error rate 
is demonstrably no worse (aprx) than master, that you should proceed with this 
commit on the reasonable assumption that you're not making the problem worse.  
This is a good contribution and I'd like to see it go in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #535: METRON-859: Use REST application with Ke...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/535


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Michael Miklavcic]

A couple more things you should check, given this setup:

   - 1 master node (not as data node)
   - 2 data nodes

I'd use something like this:

   - gateway_recover_after_data_nodes=1 or 2
   - index_number_of_replicas=0 or 1 (2 data nodes, with 2 total copies of
   the index - if you set this value to 2, it would need 1 more data node to
   attain 1 orig index + 2 replicas)



On Wed, Apr 26, 2017 at 12:02 PM, Otto Fowler 
wrote:

> Actually never mind.  I am not sure how to read the exception. Is it
> failing to read the configuration?
>
>
> On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> So the elasticsearch service is disabled.  That is the issue.  When ES
> calls get status ( service elasticsearch status ) it fails.
> I don’t know systemd too well, but does that mean we didn’t do the equiv
> of chkconfig on?
>
>
>
> On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Getting the same error in ES as before from what I can see
>
>
> On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I still can’t keep ES running, it dies.
> Also, I’m getting exceptions from enrichment trying to restart when the
> topology wasn’t running.
>
> Is there some coordination in this work?  Should I just log jiras?
> If someone is already sussing out ES, we can sync up
>
>
>
> On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:
>
> I'm also interested to know why that's important at such a small scale.
>
> Jon
>
> On Wed, Apr 26, 2017, 10:51 AM Otto Fowler 
> wrote:
>
> > I am following
> >
> > https://community.hortonworks.com/articles/60805/deploying-
> a-fresh-metron-cluster-using-ambari-serv.html
> > I DO have master and data node together.
> > Why is that a problem?
> >
> > I will try again with master and data node separate.
> >
> >
> > On April 26, 2017 at 10:41:59, Michael Miklavcic (
> > michael.miklav...@gmail.com) wrote:
> >
> > Hey Otto,
> >
> > How do you have the ES nodes configured? For a base install I would
> setup 1
> > master (NOT as data node) and 2 data nodes (NOT on the same node as the
> > master). This is the install configuration I got working. You can also
> > modify some configuration properties around master node as data node,
> index
> > replicas, and gateway recovery to get it working differently, but this is
> > what will work OOTB with the default config settings from the mpack. If
> > you've already setup a master node and a data node on the same host,
> we'll
> > need to re-install.
> >
> > Mike
> >
> >
> > On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> > wrote:
> >
> > > File
> > > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> > line
> > > 293, in _call
> > >
> > > raise ExecutionFailed(err_msg, code, out, err)
> > >
> > > ExecutionFailed: Execution of 'service elasticsearch status' returned
> 3.
> > ●
> > > elasticsearch.service - Elasticsearch
> > >
> > > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > > disabled;
> > > vendor preset: disabled)
> > >
> > > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > > ; 10h ago
> > >
> > > Docs: http://www.elastic.co
> > >
> > > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > > HOME}
> > > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> > >
> > > Process: 16819
> > > ExecStartPre=/usr/share/elasticsearch/bin/
> elasticsearch-systemd-pre-exec
> > > (code=exited, status=0/SUCCESS)
> > >
> > > Main PID: 16821 (code=exited, status=1/FAILURE)
> > >
> > >
> > >
> > > Apr 25 22:14:58 
> > > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > > org.elasticsearch.common.settings.Settings$Builder.
> > > loadFromStream(Settings.java:1080)
> > >
> > > Apr 25 22:14:58 
> > > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > > org.elasticsearch.common.settings.Settings$Builder.
> > > loadFromPath(Settings.java:1067)
> > >
> > > Apr 25 22:14:58 
> > > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > > prepareEnvironment(InternalSettingsPreparer.java:88)
> > >
> > > Apr 25 22:14:58 
> > > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > > org.elasticsearch.bootstrap.Bootstrap.initialSettings(
> Bootstrap.java:202)
> > >
> > > Apr 25 22:14:58 
> > > 

Re: Reliable place to get latest release info for script

[Dima Kovalyov]

Otto, had a look at 777 PR. Yes, it looks like something we would use.
When do you think it will be released?

And in meantime the question still stands, is there a way to get current 
release id/branch for the script?
Thank you.

- Dima

On 04/26/2017 10:56 PM, Otto Fowler wrote:
No,  Take a look at METRON-777 PR.
What I want is an extension mechanism, where you can:

1. create a project from a metron maven archetype
2. produce an extension ( parser, enrichment, stellar lib, ?? )
3. install that extension through the management UI and configure

All without touching the metron code.




On April 26, 2017 at 12:38:28, Dima Kovalyov 
(dima.koval...@sstech.us) wrote:

Regarding parsers and enrichment, new java based topologies for bluecoat, 
msexchange, msserver, asa, etc.

Batching, we use NiFi + Kylo for both stream and batch, but batch data send to 
java based tool that passes data through parsers/enrichments classes and stores 
in Hive.

So what you suggesting is to have internal repo with all our changes alone and 
to merge it with Apache Metron whenever we need to get full build?

- Dima

On 04/26/2017 09:09 PM, Otto Fowler wrote:
Thanks!

Parsers, and Enrichments, we have a plan or idea about. When you say extends 
batch processing… can you say what tech/component/part of stack you extended?

New topologies?  New WriterBolts? etc?

I would like as complete a picture as possible of the things for which anyone 
would say:
“If I could write this outside the metron tree, i could not have to maintain a 
private company fork”





On April 26, 2017 at 11:22:21, Dima Kovalyov 
(dima.koval...@sstech.us) wrote:

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open 
source them, but have no time to up them to the decent level currently). We 
also merged our custom tools that extends Metron batching processing and store 
data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:
Can I ask Dima, as comfortable you are in describing, what areas do you make 
changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to eliminate 
the requirement to develop in the metron tree to extend the product, and would 
be interested in your cases.



On April 26, 2017 at 10:46:03, Dima Kovalyov 
(dima.koval...@sstech.us) wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

[GitHub] incubator-metron issue #541: METRON-870: Add filtering by packet payload to ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/541
  
Yeah, no `0x` for specifying the regex syntax.  I'll update the function 
docs to point to the syntax guide.  Also, I'm going to give a bit of a better 
effort at the testing too.  Good catches all around.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Actually never mind.  I am not sure how to read the exception. Is it
failing to read the configuration?


On April 26, 2017 at 14:01:26, Otto Fowler (ottobackwa...@gmail.com) wrote:

So the elasticsearch service is disabled.  That is the issue.  When ES
calls get status ( service elasticsearch status ) it fails.
I don’t know systemd too well, but does that mean we didn’t do the equiv of
chkconfig on?



On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com) wrote:

Getting the same error in ES as before from what I can see


On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler  wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > ; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > 

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

So the elasticsearch service is disabled.  That is the issue.  When ES
calls get status ( service elasticsearch status ) it fails.
I don’t know systemd too well, but does that mean we didn’t do the equiv of
chkconfig on?



On April 26, 2017 at 13:54:10, Otto Fowler (ottobackwa...@gmail.com) wrote:

Getting the same error in ES as before from what I can see


On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler  wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > ; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service: main process exited, code=exited,
status=1/FAILURE
> >
> > Apr 25 22:14:58 
> > 

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Getting the same error in ES as before from what I can see


On April 26, 2017 at 13:47:09, Otto Fowler (ottobackwa...@gmail.com) wrote:

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler  wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > ; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service: main process exited, code=exited,
status=1/FAILURE
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> > elasticsearch.service entered failed state.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service failed.
> >
> > INFO 2017-04-26 08:55:19,962
> > 

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I still can’t keep ES running, it dies.
Also, I’m getting exceptions from enrichment trying to restart when the
topology wasn’t running.

Is there some coordination in this work?  Should I just log jiras?
If someone is already sussing out ES, we can sync up



On April 26, 2017 at 11:35:03, zeo...@gmail.com (zeo...@gmail.com) wrote:

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler  wrote:

> I am following
>
>
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup
1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node,
index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host,
we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned
3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > ; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> >
ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> >
org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]:
Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service: main process exited, code=exited,
status=1/FAILURE
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> > elasticsearch.service entered failed state.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service failed.
> >
> > INFO 2017-04-26 08:55:19,962
> >  Contr
> >
> >
> > On April 26, 2017 at 08:27:27, Otto Fowler (ottobackwa...@gmail.com)
> > 

Re: Quick Dev - Atlas Images

[David Lyle]

It's not an anti-Ansible sentiment, it's more of an
anti-using-Ansible-as-a-general-purpose-installer sentiment. Ansible is
fantastic in a constrained environment where the OS, Python versions, and
Ansible versions are known a priori and won't change without the Ansible
maintainer's knowledge. Ambari is WAY harder to use at first, but
basically, we're trading more effort up front for a (hopefully) lower
barrier to entry and less support burden going forward.

-D...


On Wed, Apr 26, 2017 at 1:16 PM, Otto Fowler 
wrote:

> I know there is a lot of anti-ansible sentiment.  But having gone
> spelunking through the MPack scripts and the metron.spec to more -777 let
> me just say I missed ansible’s flexibility, and documentation.
>
>
> On April 26, 2017 at 12:12:08, Nick Allen (n...@nickallen.org) wrote:
>
> > I think you can have vagrant use docker as a back end too right?
>
> I don't know about using Docker as a backend for Vagrant.  I don't think
> Vagrant is our major sticking point.  I think Ansible is the problem.  We
> have a lot of deployment functionality still in Ansible.  Much of that has
> been moved to Ambari which helps a ton, but we have a fair amount left
> AFAIK.
>
> > If it is docker, then it is just the Dockerfiles.
>
> Agreed, the storage size would be lighter weight with Docker.  But there
> are a lot of other comparisons to make when thinking about Docker too. Many
> of which I don't know the answers to right now.
>
>- Would Docker offer a more "production-like" environment? In that each
>component can run on isolated components?
>- Can we avoid some of the resource constraints that we currently have
>in running single-node Metron?
>- Can we avoid re-writing the entire deployment process?
>- How does the "time to start" compare for a "canned" release image as
>Dave suggested?
>
>
>
>
>
>
>
> On Tue, Apr 25, 2017 at 4:09 PM, Otto Fowler 
> wrote:
>
> > If it is docker, then it is just the Dockerfiles.
> > I think you can have vagrant use docker as a back end too right?
> >
> >
> >
> > On April 25, 2017 at 14:34:14, Nick Allen (n...@nickallen.org) wrote:
> >
> > >> I hadn't really reasoned about the notion of a "released" Quick Dev
> > image,
> > but I can see a lot of value in having a versioned sandbox type image-
> but
> > maybe not quick dev, maybe not even Vagrant? We could actually
> pre-package
> > everything needed and save some provisioning time on released versions.
> >
> > I really like the idea. I think it would be very beneficial to have a
> > single pre-packaged image for each release that users can download and
> take
> > new features for a spin.
> >
> > If we do stick with Vagrant for this I think Atlas works just fine. Who
> > else is going to host a 5.1 GB image for us? :) Although I am very open
> to
> > alternative implementations of this idea.
> >
> >
> > >> I always thought of Quick Dev as a developer tool, so our obligation
> is
> > to make it work with the current master and any branches currently used
> by
> > devs.
> >
> > Would be interested to get other opinions on this. I am good with making
> > that assumption. Whatever the community agrees on for Quick Dev though,
> > we should document it as such. Right now, I think it would be reasonable
> > to assume that a user would download a release and expect to be able to
> > launch Quick Dev.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Apr 24, 2017 at 11:51 AM, David Lyle 
> wrote:
> >
> > > I think it's a really good idea. There is some complexity:
> > >
> > > a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> > > allow -SNAPSHOT in their release number scheme. That is, we'll have
> > > different versions of the image that work with a 0.4.1-SNAPSHOT Metron
> > and
> > > some released versions of Metron that won't require a new image (A
> guy's
> > > gotta believe). I think that can be easily worked around.
> > >
> > > b) If the Quick Dev image becomes one of our release artifacts, Atlas
> is
> > > likely the wrong place to host it.
> > >
> > > I always thought of Quick Dev as a developer tool, so our obligation is
> > to
> > > make it work with the current master and any branches currently used by
> > > devs. I hadn't really reasoned about the notion of a "released" Quick
> Dev
> > > image, but I can see a lot of value in having a versioned sandbox type
> > > image- but maybe not quick dev, maybe not even Vagrant? We could
> actually
> > > pre-package everything needed and save some provisioning time on
> released
> > > versions. It'd just come up ready to go. I think, should we want one,
> we
> > > should release it as a convenience binary signed and hosted alongside
> the
> > > other release artifacts. Meantime, we could keep the incremental
> versions
> > > of Quick Dev in Atlas.
> > >
> > > Anyway, I think it's a really interesting notion.
> > >
> > > -D...
> > >
> > >
> > > On Mon, Apr 

Re: Reliable place to get latest release info for script

[Otto Fowler]

No,  Take a look at METRON-777 PR.
What I want is an extension mechanism, where you can:

1. create a project from a metron maven archetype
2. produce an extension ( parser, enrichment, stellar lib, ?? )
3. install that extension through the management UI and configure

All without touching the metron code.



On April 26, 2017 at 12:38:28, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Regarding parsers and enrichment, new java based topologies for bluecoat,
msexchange, msserver, asa, etc.

Batching, we use NiFi + Kylo for both stream and batch, but batch data send
to java based tool that passes data through parsers/enrichments classes and
stores in Hive.

So what you suggesting is to have internal repo with all our changes alone
and to merge it with Apache Metron whenever we need to get full build?

- Dima

On 04/26/2017 09:09 PM, Otto Fowler wrote:

Thanks!

Parsers, and Enrichments, we have a plan or idea about. When you say
extends batch processing… can you say what tech/component/part of stack you
extended?

New topologies?  New WriterBolts? etc?

I would like as complete a picture as possible of the things for which
anyone would say:
“If I could write this outside the metron tree, i could not have to
maintain a private company fork”




On April 26, 2017 at 11:22:21, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open
source them, but have no time to up them to the decent level currently). We
also merged our custom tools that extends Metron batching processing and
store data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:

Can I ask Dima, as comfortable you are in describing, what areas do you
make changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to
eliminate the requirement to develop in the metron tree to extend the
product, and would be interested in your cases.


On April 26, 2017 at 10:46:03, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: Quick Dev - Atlas Images

[Otto Fowler]

I know there is a lot of anti-ansible sentiment.  But having gone
spelunking through the MPack scripts and the metron.spec to more -777 let
me just say I missed ansible’s flexibility, and documentation.


On April 26, 2017 at 12:12:08, Nick Allen (n...@nickallen.org) wrote:

> I think you can have vagrant use docker as a back end too right?

I don't know about using Docker as a backend for Vagrant.  I don't think
Vagrant is our major sticking point.  I think Ansible is the problem.  We
have a lot of deployment functionality still in Ansible.  Much of that has
been moved to Ambari which helps a ton, but we have a fair amount left
AFAIK.

> If it is docker, then it is just the Dockerfiles.

Agreed, the storage size would be lighter weight with Docker.  But there
are a lot of other comparisons to make when thinking about Docker too. Many
of which I don't know the answers to right now.

   - Would Docker offer a more "production-like" environment? In that each
   component can run on isolated components?
   - Can we avoid some of the resource constraints that we currently have
   in running single-node Metron?
   - Can we avoid re-writing the entire deployment process?
   - How does the "time to start" compare for a "canned" release image as
   Dave suggested?







On Tue, Apr 25, 2017 at 4:09 PM, Otto Fowler 
wrote:

> If it is docker, then it is just the Dockerfiles.
> I think you can have vagrant use docker as a back end too right?
>
>
>
> On April 25, 2017 at 14:34:14, Nick Allen (n...@nickallen.org) wrote:
>
> >> I hadn't really reasoned about the notion of a "released" Quick Dev
> image,
> but I can see a lot of value in having a versioned sandbox type image- but
> maybe not quick dev, maybe not even Vagrant? We could actually pre-package
> everything needed and save some provisioning time on released versions.
>
> I really like the idea. I think it would be very beneficial to have a
> single pre-packaged image for each release that users can download and take
> new features for a spin.
>
> If we do stick with Vagrant for this I think Atlas works just fine. Who
> else is going to host a 5.1 GB image for us? :) Although I am very open to
> alternative implementations of this idea.
>
>
> >> I always thought of Quick Dev as a developer tool, so our obligation is
> to make it work with the current master and any branches currently used by
> devs.
>
> Would be interested to get other opinions on this. I am good with making
> that assumption. Whatever the community agrees on for Quick Dev though,
> we should document it as such. Right now, I think it would be reasonable
> to assume that a user would download a release and expect to be able to
> launch Quick Dev.
>
>
>
>
>
>
>
>
>
> On Mon, Apr 24, 2017 at 11:51 AM, David Lyle  wrote:
>
> > I think it's a really good idea. There is some complexity:
> >
> > a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> > allow -SNAPSHOT in their release number scheme. That is, we'll have
> > different versions of the image that work with a 0.4.1-SNAPSHOT Metron
> and
> > some released versions of Metron that won't require a new image (A guy's
> > gotta believe). I think that can be easily worked around.
> >
> > b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> > likely the wrong place to host it.
> >
> > I always thought of Quick Dev as a developer tool, so our obligation is
> to
> > make it work with the current master and any branches currently used by
> > devs. I hadn't really reasoned about the notion of a "released" Quick Dev
> > image, but I can see a lot of value in having a versioned sandbox type
> > image- but maybe not quick dev, maybe not even Vagrant? We could actually
> > pre-package everything needed and save some provisioning time on released
> > versions. It'd just come up ready to go. I think, should we want one, we
> > should release it as a convenience binary signed and hosted alongside the
> > other release artifacts. Meantime, we could keep the incremental versions
> > of Quick Dev in Atlas.
> >
> > Anyway, I think it's a really interesting notion.
> >
> > -D...
> >
> >
> > On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen  wrote:
> >
> > > Right now, we have the images that get pushed to Atlas for Quick Dev
> > >  versioned
> > > independently from the rest of Metron. We currently have versions 0.1.0
> > > and 0.2.0.
> > >
> > > What happens when a user downloads an official release of Metron, like
> > > 0.3.1, and attempts to run Quick Dev? I would assume that the code
> would
> > > download the latest image version, which we may have been updated since
> > the
> > > release. This would cause it to fail for the release version. Am I
> > wrong?
> > >
> > > If we had the Atlas images follow Metron's versioning scheme, would
> this
> > > solve the problem? Are there other cons of doing this?

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.

commit 420da379cb78341c6e55a234e26cfc6d2546ed41
Author: cstella 
Date:   2017-04-26T14:51:12Z

Updating controller integration test to not be intermittently stupid.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reliable place to get latest release info for script

[Dima Kovalyov]

Regarding parsers and enrichment, new java based topologies for bluecoat, 
msexchange, msserver, asa, etc.

Batching, we use NiFi + Kylo for both stream and batch, but batch data send to 
java based tool that passes data through parsers/enrichments classes and stores 
in Hive.

So what you suggesting is to have internal repo with all our changes alone and 
to merge it with Apache Metron whenever we need to get full build?

- Dima

On 04/26/2017 09:09 PM, Otto Fowler wrote:
Thanks!

Parsers, and Enrichments, we have a plan or idea about. When you say extends 
batch processing… can you say what tech/component/part of stack you extended?

New topologies?  New WriterBolts? etc?

I would like as complete a picture as possible of the things for which anyone 
would say:
“If I could write this outside the metron tree, i could not have to maintain a 
private company fork”





On April 26, 2017 at 11:22:21, Dima Kovalyov 
(dima.koval...@sstech.us) wrote:

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open 
source them, but have no time to up them to the decent level currently). We 
also merged our custom tools that extends Metron batching processing and store 
data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:
Can I ask Dima, as comfortable you are in describing, what areas do you make 
changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to eliminate 
the requirement to develop in the metron tree to extend the product, and would 
be interested in your cases.



On April 26, 2017 at 10:46:03, Dima Kovalyov 
(dima.koval...@sstech.us) wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: Quick Dev - Atlas Images

[Nick Allen]

> I think you can have vagrant use docker as a back end too right?

I don't know about using Docker as a backend for Vagrant.  I don't think
Vagrant is our major sticking point.  I think Ansible is the problem.  We
have a lot of deployment functionality still in Ansible.  Much of that has
been moved to Ambari which helps a ton, but we have a fair amount left
AFAIK.

> If it is docker, then it is just the Dockerfiles.

Agreed, the storage size would be lighter weight with Docker.  But there
are a lot of other comparisons to make when thinking about Docker too. Many
of which I don't know the answers to right now.

   - Would Docker offer a more "production-like" environment? In that each
   component can run on isolated components?
   - Can we avoid some of the resource constraints that we currently have
   in running single-node Metron?
   - Can we avoid re-writing the entire deployment process?
   - How does the "time to start" compare for a "canned" release image as
   Dave suggested?







On Tue, Apr 25, 2017 at 4:09 PM, Otto Fowler 
wrote:

> If it is docker, then it is just the Dockerfiles.
> I think you can have vagrant use docker as a back end too right?
>
>
>
> On April 25, 2017 at 14:34:14, Nick Allen (n...@nickallen.org) wrote:
>
> >> I hadn't really reasoned about the notion of a "released" Quick Dev
> image,
> but I can see a lot of value in having a versioned sandbox type image- but
> maybe not quick dev, maybe not even Vagrant? We could actually pre-package
> everything needed and save some provisioning time on released versions.
>
> I really like the idea. I think it would be very beneficial to have a
> single pre-packaged image for each release that users can download and
> take
> new features for a spin.
>
> If we do stick with Vagrant for this I think Atlas works just fine. Who
> else is going to host a 5.1 GB image for us? :) Although I am very open to
> alternative implementations of this idea.
>
>
> >> I always thought of Quick Dev as a developer tool, so our obligation is
> to make it work with the current master and any branches currently used by
> devs.
>
> Would be interested to get other opinions on this. I am good with making
> that assumption. Whatever the community agrees on for Quick Dev though,
> we should document it as such. Right now, I think it would be reasonable
> to assume that a user would download a release and expect to be able to
> launch Quick Dev.
>
>
>
>
>
>
>
>
>
> On Mon, Apr 24, 2017 at 11:51 AM, David Lyle 
> wrote:
>
> > I think it's a really good idea. There is some complexity:
> >
> > a) Image releases do not map 1:1 with Metron releases and Atlas doesn't
> > allow -SNAPSHOT in their release number scheme. That is, we'll have
> > different versions of the image that work with a 0.4.1-SNAPSHOT Metron
> and
> > some released versions of Metron that won't require a new image (A guy's
> > gotta believe). I think that can be easily worked around.
> >
> > b) If the Quick Dev image becomes one of our release artifacts, Atlas is
> > likely the wrong place to host it.
> >
> > I always thought of Quick Dev as a developer tool, so our obligation is
> to
> > make it work with the current master and any branches currently used by
> > devs. I hadn't really reasoned about the notion of a "released" Quick
> Dev
> > image, but I can see a lot of value in having a versioned sandbox type
> > image- but maybe not quick dev, maybe not even Vagrant? We could
> actually
> > pre-package everything needed and save some provisioning time on
> released
> > versions. It'd just come up ready to go. I think, should we want one, we
> > should release it as a convenience binary signed and hosted alongside
> the
> > other release artifacts. Meantime, we could keep the incremental
> versions
> > of Quick Dev in Atlas.
> >
> > Anyway, I think it's a really interesting notion.
> >
> > -D...
> >
> >
> > On Mon, Apr 24, 2017 at 11:26 AM, Nick Allen 
> wrote:
> >
> > > Right now, we have the images that get pushed to Atlas for Quick Dev
> > >  versioned
> > > independently from the rest of Metron. We currently have versions
> 0.1.0
> > > and 0.2.0.
> > >
> > > What happens when a user downloads an official release of Metron, like
> > > 0.3.1, and attempts to run Quick Dev? I would assume that the code
> would
> > > download the latest image version, which we may have been updated
> since
> > the
> > > release. This would cause it to fail for the release version. Am I
> > wrong?
> > >
> > > If we had the Atlas images follow Metron's versioning scheme, would
> this
> > > solve the problem? Are there other cons of doing this?
> > >
> > > Thanks
> > >
> >
>
>

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.

commit 420da379cb78341c6e55a234e26cfc6d2546ed41
Author: cstella 
Date:   2017-04-26T14:51:12Z

Updating controller integration test to not be intermittently stupid.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reliable place to get latest release info for script

[Otto Fowler]

Thanks!

Parsers, and Enrichments, we have a plan or idea about. When you say
extends batch processing… can you say what tech/component/part of stack you
extended?

New topologies?  New WriterBolts? etc?

I would like as complete a picture as possible of the things for which
anyone would say:
“If I could write this outside the metron tree, i could not have to
maintain a private company fork”




On April 26, 2017 at 11:22:21, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open
source them, but have no time to up them to the decent level currently). We
also merged our custom tools that extends Metron batching processing and
store data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:

Can I ask Dima, as comfortable you are in describing, what areas do you
make changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to
eliminate the requirement to develop in the metron tree to extend the
product, and would be interested in your cases.


On April 26, 2017 at 10:46:03, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: Ambari Wizard: Repo Tab

[zeo...@gmail.com]

I'm also interested to know why that's important at such a small scale.

Jon

On Wed, Apr 26, 2017, 10:51 AM Otto Fowler  wrote:

> I am following
>
> https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
> I DO have master and data node together.
> Why is that a problem?
>
> I will try again with master and data node separate.
>
>
> On April 26, 2017 at 10:41:59, Michael Miklavcic (
> michael.miklav...@gmail.com) wrote:
>
> Hey Otto,
>
> How do you have the ES nodes configured? For a base install I would setup 1
> master (NOT as data node) and 2 data nodes (NOT on the same node as the
> master). This is the install configuration I got working. You can also
> modify some configuration properties around master node as data node, index
> replicas, and gateway recovery to get it working differently, but this is
> what will work OOTB with the default config settings from the mpack. If
> you've already setup a master node and a data node on the same host, we'll
> need to re-install.
>
> Mike
>
>
> On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
> wrote:
>
> > File
> > "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line
> > 293, in _call
> >
> > raise ExecutionFailed(err_msg, code, out, err)
> >
> > ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
> ●
> > elasticsearch.service - Elasticsearch
> >
> > Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> > disabled;
> > vendor preset: disabled)
> >
> > Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> > ; 10h ago
> >
> > Docs: http://www.elastic.co
> >
> > Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> > -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> > HOME}
> > -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> > -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
> >
> > Process: 16819
> > ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> > (code=exited, status=0/SUCCESS)
> >
> > Main PID: 16821 (code=exited, status=1/FAILURE)
> >
> >
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromStream(Settings.java:1080)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.common.settings.Settings$Builder.
> > loadFromPath(Settings.java:1067)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.node.internal.InternalSettingsPreparer.
> > prepareEnvironment(InternalSettingsPreparer.java:88)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> > org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> > to
> > the log for complete error details.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service: main process exited, code=exited, status=1/FAILURE
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> > elasticsearch.service entered failed state.
> >
> > Apr 25 22:14:58 
> > ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> > elasticsearch.service failed.
> >
> > INFO 2017-04-26 08:55:19,962
> >  Contr
> >
> >
> > On April 26, 2017 at 08:27:27, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Can you describe what ES issues you are working on?
> > Fixing the repos got everything installed, but my ES components don’t
> stay
> > running.
> > I need to harvest the errors.
> >
> >
> >
> > On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com)
> > wrote:
> >
> > Nm. sorry. I fixed it.
> >
> >
> >
> 

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113484992
  
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -94,6 +95,52 @@ event bro_init()
 }
 ```
 
+### Example 3
+
+As documented in 
[METRON-285](https://issues.apache.org/jira/browse/METRON-285) and 
[METRON-286](https://issues.apache.org/jira/browse/METRON-286), various 
components in Metron do not currently support IPv6.  Because of this, you may 
not want to send bro logs that contain IPv6 source or destination IPs into 
Metron.  In this example, we are assuming a somewhat standard bro configuration 
for sending logs into a Metron cluster, such that:
+ * Each type of bro log is sent to the `bro` topic, but is tagged with the 
appropriate log type (such as `http`, `dns`, or `conn`).  This is done by 
setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving 
it unset), and by setting `tag_json` to true.
+ * The Kafka writer is set appropriately to send logs to the `bro` Kafka 
topic being used in your Metron cluster.  This requires that your `kafka_conf` 
and `$config` tables are appropriately configured.
+
--- End diff --

The effect of this paragraph is saying, "this is like example 1, but 
excludes IPv6", right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113483664
  
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -94,6 +95,52 @@ event bro_init()
 }
 ```
 
+### Example 3
+
+As documented in 
[METRON-285](https://issues.apache.org/jira/browse/METRON-285) and 
[METRON-286](https://issues.apache.org/jira/browse/METRON-286), various 
components in Metron do not currently support IPv6.  Because of this, you may 
not want to send bro logs that contain IPv6 source or destination IPs into 
Metron.  In this example, we are assuming a somewhat standard bro configuration 
for sending logs into a Metron cluster, such that:
+ * Each type of bro log is sent to the `bro` topic, but is tagged with the 
appropriate log type (such as `http`, `dns`, or `conn`).  This is done by 
setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving 
it unset), and by setting `tag_json` to true.
+ * The Kafka writer is set appropriately to send logs to the `bro` Kafka 
topic being used in your Metron cluster.  This requires that your `kafka_conf` 
and `$config` tables are appropriately configured.
+
+```
+@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "bro";
+redef Kafka::tag_json = T;
+redef Kafka::kafka_conf = table(
+["metadata.broker.list"] = "localhost:9092"
+);
+
+
+event bro_init() =-5
+{
+# handles HTTP
+Log::add_filter(HTTP::LOG, [$name = "kafka-http",
+$writer = Log::WRITER_KAFKAWRITER,
+$pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || 
|rec$id$resp_h| == 128 )); },
+$config = table(["stream_id"] = fmt("%s", HTTP::LOG))
+]);
+
+# handles DNS
+Log::add_filter(DNS::LOG, [$name = "kafka-dns",
+$writer = Log::WRITER_KAFKAWRITER,
+$pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || 
|rec$id$resp_h| == 128 )); },
+$config = table(["stream_id"] = fmt("%s", DNS::LOG))
+]);
+
+# handles Conn
+Log::add_filter(Conn::LOG, [$name = "kafka-conn",
+$writer = Log::WRITER_KAFKAWRITER,
+$pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 || 
|rec$id$resp_h| == 128 )); },
+$config = table(["stream_id"] = fmt("%s", Conn::LOG))
+]);
+}
--- End diff --

With this script, I would expect to find a total of 6 log filters having 
been created.  The first 3 created by `Bro/Kafka/logs-to-kafka.bro` and then 
the last 3 created by your `bro_init()` function.  To avoid this, I think what 
you want to do something more like this...

```
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::topic_name = "";
redef Kafka::tag_json = T;

event bro_init() =-5
{
# handles HTTP
Log::add_filter(HTTP::LOG, [
$name = "kafka-http",
$writer = Log::WRITER_KAFKAWRITER,
$pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || 
|rec$id$resp_h| == 128 )); },
$config = table(
["stream_id"] = fmt("%s", HTTP::LOG),
["metadata.broker.list"] = "localhost:9092"
)
]);

# handles DNS
Log::add_filter(DNS::LOG, [
$name = "kafka-dns",
$writer = Log::WRITER_KAFKAWRITER,
$pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || 
|rec$id$resp_h| == 128 )); },
$config = table(
["stream_id"] = fmt("%s", DNS::LOG),
["metadata.broker.list"] = "localhost:9092"
)
]);
}
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113485511
  
--- Diff: metron-sensors/bro-plugin-kafka/configure.plugin ---
@@ -31,7 +31,7 @@ plugin_option()
 {
   case "$1" in
 --with-librdkafka=*)
-  append_cache_entry LibRdKafka_ROOT_DIR PATH $optarg
+  append_cache_entry LibRDKafka_ROOT_DIR PATH $optarg
--- End diff --

Good catch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/547#discussion_r113484619
  
--- Diff: metron-sensors/bro-plugin-kafka/README.md ---
@@ -43,6 +43,7 @@ The following examples highlight different ways that the 
plugin can be used.  Si
 The goal in this example is to send all HTTP and DNS records to a Kafka 
topic named `bro`. 
  * Any configuration value accepted by librdkafka can be added to the 
`kafka_conf` configuration table.  
  * By defining `topic_name` all records will be sent to the same Kafka 
topic.
+ * By providing a set of logs via `logs_to_send`.
--- End diff --

This doesn't sound like a complete thought to me.  Maybe this?

"Defining `logs_to_send` will ensure that only HTTP and DNS records are 
sent."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.

commit 420da379cb78341c6e55a234e26cfc6d2546ed41
Author: cstella 
Date:   2017-04-26T14:51:12Z

Updating controller integration test to not be intermittently stupid.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reliable place to get latest release info for script

[Dima Kovalyov]

Otto,

Yes, we developed custom parsers and enrichments (in future we seek to open 
source them, but have no time to up them to the decent level currently). We 
also merged our custom tools that extends Metron batching processing and store 
data in HDFS and Hive to be used by tableau, zeppelin, etc.

Is that answers your question? Let me know if you want to know more.

- Dima

On 04/26/2017 08:29 PM, Otto Fowler wrote:
Can I ask Dima, as comfortable you are in describing, what areas do you make 
changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to eliminate 
the requirement to develop in the metron tree to extend the product, and would 
be interested in your cases.



On April 26, 2017 at 10:46:03, Dima Kovalyov 
(dima.koval...@sstech.us) wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: Reliable place to get latest release info for script

[Otto Fowler]

Can I ask Dima, as comfortable you are in describing, what areas do you
make changes or additions to that require you to maintain a fork?

Custom parsers?  Enrichments? Other?

I have done some work, and have more planned and lined up to try to
eliminate the requirement to develop in the metron tree to extend the
product, and would be interested in your cases.


On April 26, 2017 at 10:46:03, Dima Kovalyov (dima.koval...@sstech.us)
wrote:

Hello,

We want to merge latest metron release branch with out develpoment every
time it is get released. What is the path we should take here in order
to identify what release happened successfully?

Latest release is 0.3.1 currently, but since you're preparing for 0.4.0
(which will take a while) following links already have 0.4.0:
https://github.com/apache/incubator-metron/blob/master/pom.xml
https://github.com/apache/incubator-metron/releases

The only place I have found with 0.3.1 (current latest release) is:
https://metron.apache.org/documentation/#releases
But there is no way for a script to figure what commit or branch to pick
up from github for merging.

Can you please suggest?
Thank you.

- Dima

Re: auto-install on bare metal

[Otto Fowler]

Well, let’s be sure what Dima’s looking for.

That site is most def. what I’m looking for however.


On April 26, 2017 at 10:29:36, Nick Allen (n...@nickallen.org) wrote:

> But that still requires to pre-install Ambari first, right?

No. Just like what happens when deploying "Full Dev", Ansible will install
Ambari.

On Wed, Apr 26, 2017 at 10:25 AM, Dima Kovalyov 
wrote:

> But that still requires to pre-install Ambari first, right?
>
> - Dima
>
> On 04/26/2017 07:54 PM, Nick Allen wrote:
> > Ok, then I must have totally misunderstood what you're looking for.
> Sorry.
> >
> > On Wed, Apr 26, 2017 at 9:59 AM, Otto Fowler 
> > wrote:
> >
> >> Right, I think this : https://cwiki.apache.org/
> confluence/pages/viewpage.
> >> action?pageId=65144361 is the flow,
> >> but I need to verify it post recent changes to allow building in
docker
> >> again.
> >>
> >>
> >> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
> >>
> >> Here is an example of how you might do that. I created this quite a
> while
> >> ago, but it shows you the structure and how you could manage multiple
> >> environments with this method.
> >>
> >> https://github.com/nickwallen/metron-environments
> >>
> >> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
> >> wrote:
> >>
> >>> I failed at this today, but maybe it was the way I tried.
> >>> An example would be great.
> >>>
> >>>
> >>>
> >>> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com)
> wrote:
> >>>
> >>> Hi Dima,
> >>>
> >>> The same Ansible playbooks that work for EC2 and Vagrant will work
for
> >> bare
> >>> metal installations. The only difference is that you would need to
> >>> pre-provision your machines and hand-build your inventory file. The
AWS
> >>> playbooks only provision the machines. All deployment of Metron is
> >> handled
> >>> (for all installation types) by the metron_full_install playbook [1].
> >>>
> >>> -D...
> >>>
> >>> [1]
> >>> https://github.com/apache/incubator-metron/blob/master/
> >>> metron-deployment/playbooks/metron_full_install.yml
> >>>
> >>> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov <
> dima.koval...@sstech.us>
> >>> wrote:
> >>>
>  Hello Metron Team,
> 
>  We have developed a script that performs auto-install of the Metron
on
>  bare metal machines, but still working on few issues here and there.
> 
>  I am curios as to what automate solutions we do have for Metron
>  installation right now?
>  The ones I am aware of are in
>  https://github.com/apache/incubator-metron/tree/master/
> >>> metron-deployment/:
> >>>
>  a) AWS Ansible install (1 or 10 nodes)
>  b) Vagrant local VM setup
> 
>  Is there any other solution available? Has anyone managed to use AWS
>  Ansible playbooks for bare metal installation?
> 
>  - Dima
> 
> 
> 
> >>
>
>

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

I am following
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
I DO have master and data node together.
Why is that a problem?

I will try again with master and data node separate.


On April 26, 2017 at 10:41:59, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Hey Otto,

How do you have the ES nodes configured? For a base install I would setup 1
master (NOT as data node) and 2 data nodes (NOT on the same node as the
master). This is the install configuration I got working. You can also
modify some configuration properties around master node as data node, index
replicas, and gateway recovery to get it working differently, but this is
what will work OOTB with the default config settings from the mpack. If
you've already setup a master node and a data node on the same host, we'll
need to re-install.

Mike


On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
wrote:

> File
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line
> 293, in _call
>
> raise ExecutionFailed(err_msg, code, out, err)
>
> ExecutionFailed: Execution of 'service elasticsearch status' returned 3.
●
> elasticsearch.service - Elasticsearch
>
> Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> disabled;
> vendor preset: disabled)
>
> Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> ; 10h ago
>
> Docs: http://www.elastic.co
>
> Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> HOME}
> -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
>
> Process: 16819
> ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> (code=exited, status=0/SUCCESS)
>
> Main PID: 16821 (code=exited, status=1/FAILURE)
>
>
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.common.settings.Settings$Builder.
> loadFromStream(Settings.java:1080)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.common.settings.Settings$Builder.
> loadFromPath(Settings.java:1067)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.node.internal.InternalSettingsPreparer.
> prepareEnvironment(InternalSettingsPreparer.java:88)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> to
> the log for complete error details.
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> elasticsearch.service: main process exited, code=exited, status=1/FAILURE
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> elasticsearch.service entered failed state.
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> elasticsearch.service failed.
>
> INFO 2017-04-26 08:55:19,962
>  Contr
>
>
> On April 26, 2017 at 08:27:27, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Can you describe what ES issues you are working on?
> Fixing the repos got everything installed, but my ES components don’t
stay
> running.
> I need to harvest the errors.
>
>
>
> On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Nm. sorry. I fixed it.
>
>
>
> On April 25, 2017 at 16:42:05, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
> the f://localrepo
> even though the folder exists, there is no repodata/repomd.xml.
>
> What is the command to create a local repo?
>
>
>
> On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I was going by the HW 

[GitHub] incubator-metron issue #552: METRON-857 FIX: Full_Dev regression: use the pr...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/552
  
lol beat me to it


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/541#discussion_r113468302
  
--- Diff: 
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java
 ---
@@ -21,76 +21,131 @@
 import com.google.common.base.Joiner;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.metron.common.Constants;
+import org.apache.metron.common.dsl.MapVariableResolver;
 import org.apache.metron.common.dsl.VariableResolver;
 import org.apache.metron.pcap.PacketInfo;
 import org.apache.metron.pcap.PcapHelper;
 import org.apache.metron.pcap.filter.PcapFilter;
 import org.apache.metron.pcap.filter.PcapFilterConfigurator;
 import org.apache.metron.pcap.filter.PcapFilters;
 import org.apache.metron.pcap.filter.PcapFieldResolver;
+import org.apache.metron.pcap.pattern.ByteArrayMatchingUtil;
 
+import javax.xml.bind.DatatypeConverter;
 import java.util.EnumMap;
 import java.util.Map;
+import java.util.concurrent.ExecutionException;
 
 
 public class FixedPcapFilter implements PcapFilter {
 
-  public static class Configurator implements 
PcapFilterConfigurator> {
+  public static class Configurator implements 
PcapFilterConfigurator> {
 @Override
-public void addToConfig(EnumMap fields, 
Configuration conf) {
-  for (Map.Entry kv : fields.entrySet()) {
-conf.set(kv.getKey().getName(), kv.getValue());
+public void addToConfig(Map fields, Configuration 
conf) {
+  for (Map.Entry kv : fields.entrySet()) {
+conf.set(kv.getKey(), kv.getValue());
   }
   conf.set(PCAP_FILTER_NAME_CONF, PcapFilters.FIXED.name());
 }
 
 @Override
-public String queryToString(EnumMap fields) {
+public String queryToString(Map fields) {
   return (fields == null ? "" : Joiner.on("_").join(fields.values()));
 }
   }
 
+  private String packetFilter;
   private String srcAddr;
   private Integer srcPort;
   private String dstAddr;
   private Integer dstPort;
   private String protocol;
   private boolean includesReverseTraffic = false;
+  private boolean doHeaderFiltering = false;
 
   @Override
   public void configure(Iterable> config) {
 for (Map.Entry kv : config) {
   if (kv.getKey().equals(Constants.Fields.DST_ADDR.getName())) {
+System.out.println("Processing: " + kv.getKey() + " => " + 
kv.getValue());
--- End diff --

Nah, leave them alone. I forgot about that bit of annoyance in MR jobs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: auto-install on bare metal

[Dima Kovalyov]

Thank you Nick,

Yes, I was referring to a 
link 
that Jon and Otto provided, their method requires Ambari to be setup seems like.

We will test your solution, thanks.

- Dima

On 04/26/2017 07:59 PM, Nick Allen wrote:

But that still requires to pre-install Ambari first, right?



No.  Just like what happens when deploying "Full Dev", Ansible will install
Ambari.

On Wed, Apr 26, 2017 at 10:25 AM, Dima Kovalyov 

wrote:



But that still requires to pre-install Ambari first, right?

- Dima

On 04/26/2017 07:54 PM, Nick Allen wrote:


Ok, then I must have totally misunderstood what you're looking for.


Sorry.



On Wed, Apr 26, 2017 at 9:59 AM, Otto Fowler 

wrote:



Right, I think this : https://cwiki.apache.org/


confluence/pages/viewpage.


action?pageId=65144361  is the flow,
but I need to verify it post recent changes to allow building in docker
again.


On April 26, 2017 at 09:54:26, Nick Allen 
(n...@nickallen.org) wrote:

Here is an example of how you might do that. I created this quite a


while


ago, but it shows you the structure and how you could manage multiple
environments with this method.

https://github.com/nickwallen/metron-environments

On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 

wrote:



I failed at this today, but maybe it was the way I tried.
An example would be great.



On April 25, 2017 at 20:11:26, David Lyle 
(dlyle65...@gmail.com)


wrote:



Hi Dima,

The same Ansible playbooks that work for EC2 and Vagrant will work for


bare


metal installations. The only difference is that you would need to
pre-provision your machines and hand-build your inventory file. The AWS
playbooks only provision the machines. All deployment of Metron is


handled


(for all installation types) by the metron_full_install playbook [1].

-D...

[1]
https://github.com/apache/incubator-metron/blob/master/
metron-deployment/playbooks/metron_full_install.yml

On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov <


dima.koval...@sstech.us>


wrote:



Hello Metron Team,

We have developed a script that performs auto-install of the Metron on
bare metal machines, but still working on few issues here and there.

I am curios as to what automate solutions we do have for Metron
installation right now?
The ones I am aware of are in
https://github.com/apache/incubator-metron/tree/master/


metron-deployment/:



a) AWS Ansible install (1 or 10 nodes)
b) Vagrant local VM setup

Is there any other solution available? Has anyone managed to use AWS
Ansible playbooks for bare metal installation?

- Dima

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/541#discussion_r113467882
  
--- Diff: metron-platform/metron-pcap-backend/README.md ---
@@ -127,3 +130,23 @@ usage: Query filter options
  -q,--query Query string to use as a filter
  -st,--start_time   (required) Packet start time range.
 ```
+
+The Query filter's `--query` argument specifies the Stellar expression to
+execute on each packet.  To interact with the packet, a few variables are 
exposed:
+* `packet` : The packet data (a `byte[]`)
+* `ip_src_addr` : The source address for the packet (a `String`)
+* `ip_src_port` : The source port for the packet (an `Integer`)
+* `ip_dst_addr` : The destination address for the packet (a `String`)
+* `ip_dst_port` : The destination port for the packet (an `Integer`)
+
+ Binary Regex
+
+Filtering can be done both by the packet header as well as via a binary 
regular expression
+which can be run on the packet payload itself.  This filter can be 
specified via:
+* The `-pf` or `--packet_filter` options for the fixed query filter
+* The `BYTEARRAY_MATCH(pattern, data)` Stellar function.
--- End diff --

yep


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #541: METRON-870: Add filtering by packet payl...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/541#discussion_r113467857
  
--- Diff: 
metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/filter/fixed/FixedPcapFilter.java
 ---
@@ -21,76 +21,131 @@
 import com.google.common.base.Joiner;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.metron.common.Constants;
+import org.apache.metron.common.dsl.MapVariableResolver;
 import org.apache.metron.common.dsl.VariableResolver;
 import org.apache.metron.pcap.PacketInfo;
 import org.apache.metron.pcap.PcapHelper;
 import org.apache.metron.pcap.filter.PcapFilter;
 import org.apache.metron.pcap.filter.PcapFilterConfigurator;
 import org.apache.metron.pcap.filter.PcapFilters;
 import org.apache.metron.pcap.filter.PcapFieldResolver;
+import org.apache.metron.pcap.pattern.ByteArrayMatchingUtil;
 
+import javax.xml.bind.DatatypeConverter;
 import java.util.EnumMap;
 import java.util.Map;
+import java.util.concurrent.ExecutionException;
 
 
 public class FixedPcapFilter implements PcapFilter {
 
-  public static class Configurator implements 
PcapFilterConfigurator> {
+  public static class Configurator implements 
PcapFilterConfigurator> {
 @Override
-public void addToConfig(EnumMap fields, 
Configuration conf) {
-  for (Map.Entry kv : fields.entrySet()) {
-conf.set(kv.getKey().getName(), kv.getValue());
+public void addToConfig(Map fields, Configuration 
conf) {
+  for (Map.Entry kv : fields.entrySet()) {
+conf.set(kv.getKey(), kv.getValue());
   }
   conf.set(PCAP_FILTER_NAME_CONF, PcapFilters.FIXED.name());
 }
 
 @Override
-public String queryToString(EnumMap fields) {
+public String queryToString(Map fields) {
   return (fields == null ? "" : Joiner.on("_").join(fields.values()));
 }
   }
 
+  private String packetFilter;
   private String srcAddr;
   private Integer srcPort;
   private String dstAddr;
   private Integer dstPort;
   private String protocol;
   private boolean includesReverseTraffic = false;
+  private boolean doHeaderFiltering = false;
 
   @Override
   public void configure(Iterable> config) {
 for (Map.Entry kv : config) {
   if (kv.getKey().equals(Constants.Fields.DST_ADDR.getName())) {
+System.out.println("Processing: " + kv.getKey() + " => " + 
kv.getValue());
--- End diff --

It's in a MR job, those printlns are getting captured in the map stdout 
log.  I can make them logger logs though if it's more comfortable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #552: METRON-857 FIX: Full_Dev regression: use...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/552


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Michael Miklavcic]

Hey Otto,

How do you have the ES nodes configured? For a base install I would setup 1
master (NOT as data node) and 2 data nodes (NOT on the same node as the
master). This is the install configuration I got working. You can also
modify some configuration properties around master node as data node, index
replicas, and gateway recovery to get it working differently, but this is
what will work OOTB with the default config settings from the mpack. If
you've already setup a master node and a data node on the same host, we'll
need to re-install.

Mike


On Wed, Apr 26, 2017 at 7:02 AM, Otto Fowler 
wrote:

>   File
> "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line
> 293, in _call
>
> raise ExecutionFailed(err_msg, code, out, err)
>
> ExecutionFailed: Execution of 'service elasticsearch status' returned 3. ●
> elasticsearch.service - Elasticsearch
>
>Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service;
> disabled;
> vendor preset: disabled)
>
>Active: failed (Result: exit-code) since Tue 2017-04-25 22:14:58 EDT
> ; 10h ago
>
>  Docs: http://www.elastic.co
>
>   Process: 16821 ExecStart=/usr/share/elasticsearch/bin/elasticsearch
> -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_
> HOME}
> -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR}
> -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
>
>   Process: 16819
> ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec
> (code=exited, status=0/SUCCESS)
>
>  Main PID: 16821 (code=exited, status=1/FAILURE)
>
>
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.common.settings.Settings$Builder.
> loadFromStream(Settings.java:1080)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.common.settings.Settings$Builder.
> loadFromPath(Settings.java:1067)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.node.internal.InternalSettingsPreparer.
> prepareEnvironment(InternalSettingsPreparer.java:88)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com elasticsearch[16821]: Refer
> to
> the log for complete error details.
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> elasticsearch.service: main process exited, code=exited, status=1/FAILURE
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]: Unit
> elasticsearch.service entered failed state.
>
> Apr 25 22:14:58 
> ccs-fx-met-poc-86.dev.industrialdefender.com systemd[1]:
> elasticsearch.service failed.
>
> INFO 2017-04-26 08:55:19,962
>  Contr
>
>
> On April 26, 2017 at 08:27:27, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Can you describe what ES issues you are working on?
> Fixing the repos got everything installed, but my ES components don’t stay
> running.
> I need to harvest the errors.
>
>
>
> On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Nm.  sorry.  I fixed it.
>
>
>
> On April 25, 2017 at 16:42:05, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
> the f://localrepo
> even though the folder exists, there is no repodata/repomd.xml.
>
> What is the command to create a local repo?
>
>
>
> On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> I was going by the HW community page.
>
> Ok, Let me try it
>
>
>
> On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:
>
> That would do it. It requires 2.4.2+. I would have sworn I put that in the
> README, but I must have only annotated the PR. :(
>
> I'll get that in asap.
>
> -D...
>
>
> On Tue, Apr 

Re: auto-install on bare metal

[Nick Allen]

> But that still requires to pre-install Ambari first, right?

No.  Just like what happens when deploying "Full Dev", Ansible will install
Ambari.

On Wed, Apr 26, 2017 at 10:25 AM, Dima Kovalyov 
wrote:

> But that still requires to pre-install Ambari first, right?
>
> - Dima
>
> On 04/26/2017 07:54 PM, Nick Allen wrote:
> > Ok, then I must have totally misunderstood what you're looking for.
> Sorry.
> >
> > On Wed, Apr 26, 2017 at 9:59 AM, Otto Fowler 
> > wrote:
> >
> >> Right, I think this : https://cwiki.apache.org/
> confluence/pages/viewpage.
> >> action?pageId=65144361  is the flow,
> >> but I need to verify it post recent changes to allow building in docker
> >> again.
> >>
> >>
> >> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
> >>
> >> Here is an example of how you might do that. I created this quite a
> while
> >> ago, but it shows you the structure and how you could manage multiple
> >> environments with this method.
> >>
> >> https://github.com/nickwallen/metron-environments
> >>
> >> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
> >> wrote:
> >>
> >>> I failed at this today, but maybe it was the way I tried.
> >>> An example would be great.
> >>>
> >>>
> >>>
> >>> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com)
> wrote:
> >>>
> >>> Hi Dima,
> >>>
> >>> The same Ansible playbooks that work for EC2 and Vagrant will work for
> >> bare
> >>> metal installations. The only difference is that you would need to
> >>> pre-provision your machines and hand-build your inventory file. The AWS
> >>> playbooks only provision the machines. All deployment of Metron is
> >> handled
> >>> (for all installation types) by the metron_full_install playbook [1].
> >>>
> >>> -D...
> >>>
> >>> [1]
> >>> https://github.com/apache/incubator-metron/blob/master/
> >>> metron-deployment/playbooks/metron_full_install.yml
> >>>
> >>> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov <
> dima.koval...@sstech.us>
> >>> wrote:
> >>>
>  Hello Metron Team,
> 
>  We have developed a script that performs auto-install of the Metron on
>  bare metal machines, but still working on few issues here and there.
> 
>  I am curios as to what automate solutions we do have for Metron
>  installation right now?
>  The ones I am aware of are in
>  https://github.com/apache/incubator-metron/tree/master/
> >>> metron-deployment/:
> >>>
>  a) AWS Ansible install (1 or 10 nodes)
>  b) Vagrant local VM setup
> 
>  Is there any other solution available? Has anyone managed to use AWS
>  Ansible playbooks for bare metal installation?
> 
>  - Dima
> 
> 
> 
> >>
>
>

Re: auto-install on bare metal

[Dima Kovalyov]

But that still requires to pre-install Ambari first, right?

- Dima

On 04/26/2017 07:54 PM, Nick Allen wrote:
> Ok, then I must have totally misunderstood what you're looking for.  Sorry.
>
> On Wed, Apr 26, 2017 at 9:59 AM, Otto Fowler 
> wrote:
>
>> Right, I think this : https://cwiki.apache.org/confluence/pages/viewpage.
>> action?pageId=65144361  is the flow,
>> but I need to verify it post recent changes to allow building in docker
>> again.
>>
>>
>> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
>>
>> Here is an example of how you might do that. I created this quite a while
>> ago, but it shows you the structure and how you could manage multiple
>> environments with this method.
>>
>> https://github.com/nickwallen/metron-environments
>>
>> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
>> wrote:
>>
>>> I failed at this today, but maybe it was the way I tried.
>>> An example would be great.
>>>
>>>
>>>
>>> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
>>>
>>> Hi Dima,
>>>
>>> The same Ansible playbooks that work for EC2 and Vagrant will work for
>> bare
>>> metal installations. The only difference is that you would need to
>>> pre-provision your machines and hand-build your inventory file. The AWS
>>> playbooks only provision the machines. All deployment of Metron is
>> handled
>>> (for all installation types) by the metron_full_install playbook [1].
>>>
>>> -D...
>>>
>>> [1]
>>> https://github.com/apache/incubator-metron/blob/master/
>>> metron-deployment/playbooks/metron_full_install.yml
>>>
>>> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
>>> wrote:
>>>
 Hello Metron Team,

 We have developed a script that performs auto-install of the Metron on
 bare metal machines, but still working on few issues here and there.

 I am curios as to what automate solutions we do have for Metron
 installation right now?
 The ones I am aware of are in
 https://github.com/apache/incubator-metron/tree/master/
>>> metron-deployment/:
>>>
 a) AWS Ansible install (1 or 10 nodes)
 b) Vagrant local VM setup

 Is there any other solution available? Has anyone managed to use AWS
 Ansible playbooks for bare metal installation?

 - Dima



>>

Re: auto-install on bare metal

[Nick Allen]

Ok, then I must have totally misunderstood what you're looking for.  Sorry.

On Wed, Apr 26, 2017 at 9:59 AM, Otto Fowler 
wrote:

> Right, I think this : https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=65144361  is the flow,
> but I need to verify it post recent changes to allow building in docker
> again.
>
>
> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
>
> Here is an example of how you might do that. I created this quite a while
> ago, but it shows you the structure and how you could manage multiple
> environments with this method.
>
> https://github.com/nickwallen/metron-environments
>
> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
> wrote:
>
> > I failed at this today, but maybe it was the way I tried.
> > An example would be great.
> >
> >
> >
> > On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
> >
> > Hi Dima,
> >
> > The same Ansible playbooks that work for EC2 and Vagrant will work for
> bare
> > metal installations. The only difference is that you would need to
> > pre-provision your machines and hand-build your inventory file. The AWS
> > playbooks only provision the machines. All deployment of Metron is
> handled
> > (for all installation types) by the metron_full_install playbook [1].
> >
> > -D...
> >
> > [1]
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/playbooks/metron_full_install.yml
> >
> > On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
>
> > wrote:
> >
> > > Hello Metron Team,
> > >
> > > We have developed a script that performs auto-install of the Metron on
> > > bare metal machines, but still working on few issues here and there.
> > >
> > > I am curios as to what automate solutions we do have for Metron
> > > installation right now?
> > > The ones I am aware of are in
> > > https://github.com/apache/incubator-metron/tree/master/
> > metron-deployment/:
> >
> > > a) AWS Ansible install (1 or 10 nodes)
> > > b) Vagrant local VM setup
> > >
> > > Is there any other solution available? Has anyone managed to use AWS
> > > Ansible playbooks for bare metal installation?
> > >
> > > - Dima
> > >
> > >
> > >
> >
>
>

Re: Normalization topology or separate normalization bolt for parsing topology

[Casey Stella]

Yeah, we definitely don't want to rewrite parsing in Stellar.  I would
expect the job of the parser, however, to handle structural issues.  In my
mind, parsing is about transforming structures into fields and the role of
the field transformations are to transform values.  There's obvious overlap
there wherein parsers may do some normalizations/transformations (i.e. look
how grok handles timestamps), but it almost always gets us into trouble
when parsers do even moderately complex value transformations.

As I type this, though, I think I see your point.  What you really want is
to chain parsers, have a pre-parser to bring you 80% of the way there and
hammer out all the structural issues so you might be able to use a more
generic parser down the chain.  I have often thought that maybe we should
expose parsers as Stellar functions which take raw data and emit whole
messages.  This would allow us to compose parsers, so imagine the above
example where you've written a stellar function to normalize the input and
you're then passing it to a CSV parser, you could run
"CSV_PARSE(ALI_NORMALIZE(message))" where you'd otherwise specify a parser.

As for speed, the stellar expression would get compiled into a java object,
so it shouldn't be appreciable overhead since we no longer lex and parse
for every message.

Is this kinda how you were seeing it?

On Wed, Apr 26, 2017 at 9:51 AM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> The challenge there I suspect is going to be that you essentially end up
> with the actual parser doing very little of value, and then effectively
> trying to write a parser in stellar against a few broad strings, which
> would likely give you all sorts of performance problems.
>
> One solution is to write a very defensive and flexible parser, but that
> would tend to be time consuming.
>
> There is also something to be said for doing some basic transformation
> before the parser topic kafka in something like nifi, but again,
> performance can be an issue there.
>
> If the noise is about broken structure for example, maybe a simple
> pre-process step as part of your parser would make sense, e.g. stripping
> syslog headers, or character set conversion, removing very broken bits as
> part of the parse method.
>
> In terms of normalisation post-parse, I agree, that 100% a job for
> Stellar, and the fieldTransformations capability. Something I would like to
> see would be a means to use that transformation step to map to a well known
> (though loosely enforced) schema provided by a governance framework, but
> that is a much bigger topic of conversation.
>
> Not of course that not everything has to be parsed just because it’s in
> the message. A relatively loose fitting parser which pulls out the relevant
> data for the use case would be fine, and likely a lot more tolerant of
> noise than something that felt the need for every field. We do after all
> store the original_string for you if you really absolutely have to had
> everything, so a more schema-on-read philosophy certainly applies and will
> likely side-step a lot of your issues.
>
> Simon
>
> > On 26 Apr 2017, at 14:37, Casey Stella  wrote:
> >
> > Ok, that's another story.  h, we don't generally pre-parse becuase we
> > try to not assume any particular format there (i.e. it could be strings,
> > could be byte arrays).  Maybe the right answer is to pass the raw,
> > non-normalized data (best effort tyep of thing) through the parser and do
> > the normalization post-parse..or is there a problem with that?
> >
> > On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian 
> wrote:
> >
> >> Hi Casey,
> >>
> >> It is actually pre-parse process, not a post-parse one. These type of
> >> noises affect the position of an attribute for example and give us
> parsing
> >> exception. The timestamp example was not a good one because that is
> >> actually a post-parse exception.
> >>
> >> On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella 
> wrote:
> >>
> >>> So, further transformation post-parse was one of the motivating reasons
> >> for
> >>> Stellar (to do that transformation post-parse).  Is there a capability
> >> that
> >>> it's lacking that we can add to fit your usecase?
> >>>
> >>> On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian 
> >>> wrote:
> >>>
>  I've created a Jira ticket regarding this feature.
> 
>  https://issues.apache.org/jira/browse/METRON-893
> 
> 
>  On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian  >
>  wrote:
> 
> > Currently, we are using normal regex at the Java source code to
> >> handle
> > those situations. However, it would be nice to have a separate bolt
> >> and
> > deal with them separately. Yeah, I can create a Jira issue regarding
>  that.
> > The main reason I am asking for such a feature is the fact that lack
> >> of
> > such a feature makes the process 

[GitHub] incubator-metron pull request #551: METRON-892 add docker to platform info

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/551


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: auto-install on bare metal

[zeo...@gmail.com]

I can verify that I've used
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361 to
install Metron on a bare metal cluster before the docker requirement was
imposed.

Jon

On Wed, Apr 26, 2017 at 9:59 AM Otto Fowler  wrote:

> Right, I think this :
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361
> is the flow,
> but I need to verify it post recent changes to allow building in docker
> again.
>
>
> On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:
>
> Here is an example of how you might do that. I created this quite a while
> ago, but it shows you the structure and how you could manage multiple
> environments with this method.
>
> https://github.com/nickwallen/metron-environments
>
> On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
> wrote:
>
> > I failed at this today, but maybe it was the way I tried.
> > An example would be great.
> >
> >
> >
> > On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
> >
> > Hi Dima,
> >
> > The same Ansible playbooks that work for EC2 and Vagrant will work for
> bare
> > metal installations. The only difference is that you would need to
> > pre-provision your machines and hand-build your inventory file. The AWS
> > playbooks only provision the machines. All deployment of Metron is
> handled
> > (for all installation types) by the metron_full_install playbook [1].
> >
> > -D...
> >
> > [1]
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/playbooks/metron_full_install.yml
> >
> > On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
> > wrote:
> >
> > > Hello Metron Team,
> > >
> > > We have developed a script that performs auto-install of the Metron on
> > > bare metal machines, but still working on few issues here and there.
> > >
> > > I am curios as to what automate solutions we do have for Metron
> > > installation right now?
> > > The ones I am aware of are in
> > > https://github.com/apache/incubator-metron/tree/master/
> > metron-deployment/:
> >
> > > a) AWS Ansible install (1 or 10 nodes)
> > > b) Vagrant local VM setup
> > >
> > > Is there any other solution available? Has anyone managed to use AWS
> > > Ansible playbooks for bare metal installation?
> > >
> > > - Dima
> > >
> > >
> > >
> >
>
-- 

Jon

Re: Normalization topology or separate normalization bolt for parsing topology

[Casey Stella]

Ok, this may be easier with a couple of examples:

*Simple Example : Downstream Processing is Independent of Normalization*

Pretend we have a data format that is CSV and the first field, let's call
it 'input_dname' is supposed to be a domain name, but sometimes you get IP
addresses.  In the situation where you get IP addresses, let's say you want
to remove the field.  Rather than doing that in the parser, you could just
emit the raw data for that field, ip address or domain name, and then in a
field transformation you could run a field transformation:

'input_dname' : "if IS_IP(input_dname) then null else input_dname"

*Intermediate Example:* *Downstream Processing is Independent of
Normalization*

Same situation, but now we have a new field called "input_tld" in which you
pull out the TLD of input_dname.  BUT you can't, because it may or may not
be a proper domain name and, furthermore, it may have spaces around it.  In
that situation, I'd suggest adding just *not* adding the field until the
field transformation and doing the following as field transformations:
'input_dname' : "if IS_IP(input_dname) then null else TRIM(input_dname)"
'input_tld' : "DOMAIN_TO_TLD(input_dname)"

If your situation doesn't fit there, could you give us an example like
above?

On Wed, Apr 26, 2017 at 9:43 AM, Ali Nazemian  wrote:

> Having Stellar function for the normalization is very cool actually.
>
> Casey, how are you going to deal with normalization after the parsing if
> that noise affects the parsing? For some reason, the incoming data do not
> look like in the way that has to be.
>
> On Wed, Apr 26, 2017 at 11:37 PM, Casey Stella  wrote:
>
> > Ok, that's another story.  h, we don't generally pre-parse becuase we
> > try to not assume any particular format there (i.e. it could be strings,
> > could be byte arrays).  Maybe the right answer is to pass the raw,
> > non-normalized data (best effort tyep of thing) through the parser and do
> > the normalization post-parse..or is there a problem with that?
> >
> > On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian 
> > wrote:
> >
> > > Hi Casey,
> > >
> > > It is actually pre-parse process, not a post-parse one. These type of
> > > noises affect the position of an attribute for example and give us
> > parsing
> > > exception. The timestamp example was not a good one because that is
> > > actually a post-parse exception.
> > >
> > > On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella 
> > wrote:
> > >
> > > > So, further transformation post-parse was one of the motivating
> reasons
> > > for
> > > > Stellar (to do that transformation post-parse).  Is there a
> capability
> > > that
> > > > it's lacking that we can add to fit your usecase?
> > > >
> > > > On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian  >
> > > > wrote:
> > > >
> > > > > I've created a Jira ticket regarding this feature.
> > > > >
> > > > > https://issues.apache.org/jira/browse/METRON-893
> > > > >
> > > > >
> > > > > On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian <
> > alinazem...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Currently, we are using normal regex at the Java source code to
> > > handle
> > > > > > those situations. However, it would be nice to have a separate
> bolt
> > > and
> > > > > > deal with them separately. Yeah, I can create a Jira issue
> > regarding
> > > > > that.
> > > > > > The main reason I am asking for such a feature is the fact that
> > lack
> > > of
> > > > > > such a feature makes the process of creating some parser for the
> > > > > community
> > > > > > a little painful for us. We need to maintain two different
> > versions,
> > > > one
> > > > > > for community another for the internal use case. Clearly, noise
> is
> > an
> > > > > > inevitable part of real world use cases.
> > > > > >
> > > > > > Cheers,
> > > > > > Ali
> > > > > >
> > > > > > On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler <
> > > ottobackwa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> Are you doing this cleansing all in the parser or are you using
> > any
> > > > > >> Stellar to do it?
> > > > > >> Can you create a jira?
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On April 26, 2017 at 08:59:16, Ali Nazemian (
> > alinazem...@gmail.com)
> > > > > >> wrote:
> > > > > >>
> > > > > >> Hi all,
> > > > > >>
> > > > > >>
> > > > > >> We are facing certain use cases in Metron production that happen
> > to
> > > be
> > > > > >> related to noisy stream. For example, a wrong timestamp,
> duplicate
> > > > > >> hostname/IP address, etc. To deal with the normalization we have
> > > added
> > > > > an
> > > > > >> additional step for the corresponding parsers to do the data
> > > cleaning.
> > > > > >> Clearly, parsing is a standard factor which is mostly related to
> > the
> > > > > >> device
> > > > > >> that is generating the data and can be used for 

Re: auto-install on bare metal

[Otto Fowler]

Right, I think this :
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361
is the flow,
but I need to verify it post recent changes to allow building in docker
again.


On April 26, 2017 at 09:54:26, Nick Allen (n...@nickallen.org) wrote:

Here is an example of how you might do that. I created this quite a while
ago, but it shows you the structure and how you could manage multiple
environments with this method.

https://github.com/nickwallen/metron-environments

On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
wrote:

> I failed at this today, but maybe it was the way I tried.
> An example would be great.
>
>
>
> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
>
> Hi Dima,
>
> The same Ansible playbooks that work for EC2 and Vagrant will work for
bare
> metal installations. The only difference is that you would need to
> pre-provision your machines and hand-build your inventory file. The AWS
> playbooks only provision the machines. All deployment of Metron is
handled
> (for all installation types) by the metron_full_install playbook [1].
>
> -D...
>
> [1]
> https://github.com/apache/incubator-metron/blob/master/
> metron-deployment/playbooks/metron_full_install.yml
>
> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
> wrote:
>
> > Hello Metron Team,
> >
> > We have developed a script that performs auto-install of the Metron on
> > bare metal machines, but still working on few issues here and there.
> >
> > I am curios as to what automate solutions we do have for Metron
> > installation right now?
> > The ones I am aware of are in
> > https://github.com/apache/incubator-metron/tree/master/
> metron-deployment/:
>
> > a) AWS Ansible install (1 or 10 nodes)
> > b) Vagrant local VM setup
> >
> > Is there any other solution available? Has anyone managed to use AWS
> > Ansible playbooks for bare metal installation?
> >
> > - Dima
> >
> >
> >
>

Re: auto-install on bare metal

[Nick Allen]

Here is an example of how you might do that.  I created this quite a while
ago, but it shows you the structure and how you could manage multiple
environments with this method.

https://github.com/nickwallen/metron-environments

On Tue, Apr 25, 2017 at 9:46 PM, Otto Fowler 
wrote:

> I failed at this today, but maybe it was the way I tried.
> An example would be great.
>
>
>
> On April 25, 2017 at 20:11:26, David Lyle (dlyle65...@gmail.com) wrote:
>
> Hi Dima,
>
> The same Ansible playbooks that work for EC2 and Vagrant will work for bare
> metal installations. The only difference is that you would need to
> pre-provision your machines and hand-build your inventory file. The AWS
> playbooks only provision the machines. All deployment of Metron is handled
> (for all installation types) by the metron_full_install playbook [1].
>
> -D...
>
> [1]
> https://github.com/apache/incubator-metron/blob/master/
> metron-deployment/playbooks/metron_full_install.yml
>
> On Tue, Apr 25, 2017 at 7:37 PM, Dima Kovalyov 
> wrote:
>
> > Hello Metron Team,
> >
> > We have developed a script that performs auto-install of the Metron on
> > bare metal machines, but still working on few issues here and there.
> >
> > I am curios as to what automate solutions we do have for Metron
> > installation right now?
> > The ones I am aware of are in
> > https://github.com/apache/incubator-metron/tree/master/
> metron-deployment/:
>
> > a) AWS Ansible install (1 or 10 nodes)
> > b) Vagrant local VM setup
> >
> > Is there any other solution available? Has anyone managed to use AWS
> > Ansible playbooks for bare metal installation?
> >
> > - Dima
> >
> >
> >
>

Re: Normalization topology or separate normalization bolt for parsing topology

[Nick Allen]

> For some reason, the incoming data do not look like in the way that has
to be.

In my mind that would be something for your parser to handle.

On Wed, Apr 26, 2017 at 9:43 AM, Ali Nazemian  wrote:

> Having Stellar function for the normalization is very cool actually.
>
> Casey, how are you going to deal with normalization after the parsing if
> that noise affects the parsing? For some reason, the incoming data do not
> look like in the way that has to be.
>
> On Wed, Apr 26, 2017 at 11:37 PM, Casey Stella  wrote:
>
> > Ok, that's another story.  h, we don't generally pre-parse becuase we
> > try to not assume any particular format there (i.e. it could be strings,
> > could be byte arrays).  Maybe the right answer is to pass the raw,
> > non-normalized data (best effort tyep of thing) through the parser and do
> > the normalization post-parse..or is there a problem with that?
> >
> > On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian 
> > wrote:
> >
> > > Hi Casey,
> > >
> > > It is actually pre-parse process, not a post-parse one. These type of
> > > noises affect the position of an attribute for example and give us
> > parsing
> > > exception. The timestamp example was not a good one because that is
> > > actually a post-parse exception.
> > >
> > > On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella 
> > wrote:
> > >
> > > > So, further transformation post-parse was one of the motivating
> reasons
> > > for
> > > > Stellar (to do that transformation post-parse).  Is there a
> capability
> > > that
> > > > it's lacking that we can add to fit your usecase?
> > > >
> > > > On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian  >
> > > > wrote:
> > > >
> > > > > I've created a Jira ticket regarding this feature.
> > > > >
> > > > > https://issues.apache.org/jira/browse/METRON-893
> > > > >
> > > > >
> > > > > On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian <
> > alinazem...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Currently, we are using normal regex at the Java source code to
> > > handle
> > > > > > those situations. However, it would be nice to have a separate
> bolt
> > > and
> > > > > > deal with them separately. Yeah, I can create a Jira issue
> > regarding
> > > > > that.
> > > > > > The main reason I am asking for such a feature is the fact that
> > lack
> > > of
> > > > > > such a feature makes the process of creating some parser for the
> > > > > community
> > > > > > a little painful for us. We need to maintain two different
> > versions,
> > > > one
> > > > > > for community another for the internal use case. Clearly, noise
> is
> > an
> > > > > > inevitable part of real world use cases.
> > > > > >
> > > > > > Cheers,
> > > > > > Ali
> > > > > >
> > > > > > On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler <
> > > ottobackwa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> Are you doing this cleansing all in the parser or are you using
> > any
> > > > > >> Stellar to do it?
> > > > > >> Can you create a jira?
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On April 26, 2017 at 08:59:16, Ali Nazemian (
> > alinazem...@gmail.com)
> > > > > >> wrote:
> > > > > >>
> > > > > >> Hi all,
> > > > > >>
> > > > > >>
> > > > > >> We are facing certain use cases in Metron production that happen
> > to
> > > be
> > > > > >> related to noisy stream. For example, a wrong timestamp,
> duplicate
> > > > > >> hostname/IP address, etc. To deal with the normalization we have
> > > added
> > > > > an
> > > > > >> additional step for the corresponding parsers to do the data
> > > cleaning.
> > > > > >> Clearly, parsing is a standard factor which is mostly related to
> > the
> > > > > >> device
> > > > > >> that is generating the data and can be used for the same type of
> > > > device
> > > > > >> everywhere, but normalization is very production dependent and
> > there
> > > > is
> > > > > >> no
> > > > > >> point of mixing normalization with parsing. It would be nice to
> > > have a
> > > > > >> sperate bolt in a parsing topologies to dedicate to production
> > > > > >> related cleaning process. In that case, eveybody can easily
> > > contribute
> > > > > to
> > > > > >> Metron community with additional parsers without being worried
> > about
> > > > > >> mixing
> > > > > >> parsers and data cleaning process.
> > > > > >>
> > > > > >>
> > > > > >> Regards,
> > > > > >>
> > > > > >> Ali
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > A.Nazemian
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > A.Nazemian
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > A.Nazemian
> > >
> >
>
>
>
> --
> A.Nazemian
>

[GitHub] incubator-metron issue #552: METRON-857 FIX: Full_Dev regression: use the pr...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/552
  
Ditto. +1, thanks Otto!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Normalization topology or separate normalization bolt for parsing topology

[Ali Nazemian]

Having Stellar function for the normalization is very cool actually.

Casey, how are you going to deal with normalization after the parsing if
that noise affects the parsing? For some reason, the incoming data do not
look like in the way that has to be.

On Wed, Apr 26, 2017 at 11:37 PM, Casey Stella  wrote:

> Ok, that's another story.  h, we don't generally pre-parse becuase we
> try to not assume any particular format there (i.e. it could be strings,
> could be byte arrays).  Maybe the right answer is to pass the raw,
> non-normalized data (best effort tyep of thing) through the parser and do
> the normalization post-parse..or is there a problem with that?
>
> On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian 
> wrote:
>
> > Hi Casey,
> >
> > It is actually pre-parse process, not a post-parse one. These type of
> > noises affect the position of an attribute for example and give us
> parsing
> > exception. The timestamp example was not a good one because that is
> > actually a post-parse exception.
> >
> > On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella 
> wrote:
> >
> > > So, further transformation post-parse was one of the motivating reasons
> > for
> > > Stellar (to do that transformation post-parse).  Is there a capability
> > that
> > > it's lacking that we can add to fit your usecase?
> > >
> > > On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian 
> > > wrote:
> > >
> > > > I've created a Jira ticket regarding this feature.
> > > >
> > > > https://issues.apache.org/jira/browse/METRON-893
> > > >
> > > >
> > > > On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian <
> alinazem...@gmail.com>
> > > > wrote:
> > > >
> > > > > Currently, we are using normal regex at the Java source code to
> > handle
> > > > > those situations. However, it would be nice to have a separate bolt
> > and
> > > > > deal with them separately. Yeah, I can create a Jira issue
> regarding
> > > > that.
> > > > > The main reason I am asking for such a feature is the fact that
> lack
> > of
> > > > > such a feature makes the process of creating some parser for the
> > > > community
> > > > > a little painful for us. We need to maintain two different
> versions,
> > > one
> > > > > for community another for the internal use case. Clearly, noise is
> an
> > > > > inevitable part of real world use cases.
> > > > >
> > > > > Cheers,
> > > > > Ali
> > > > >
> > > > > On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler <
> > ottobackwa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> Are you doing this cleansing all in the parser or are you using
> any
> > > > >> Stellar to do it?
> > > > >> Can you create a jira?
> > > > >>
> > > > >>
> > > > >>
> > > > >> On April 26, 2017 at 08:59:16, Ali Nazemian (
> alinazem...@gmail.com)
> > > > >> wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >>
> > > > >> We are facing certain use cases in Metron production that happen
> to
> > be
> > > > >> related to noisy stream. For example, a wrong timestamp, duplicate
> > > > >> hostname/IP address, etc. To deal with the normalization we have
> > added
> > > > an
> > > > >> additional step for the corresponding parsers to do the data
> > cleaning.
> > > > >> Clearly, parsing is a standard factor which is mostly related to
> the
> > > > >> device
> > > > >> that is generating the data and can be used for the same type of
> > > device
> > > > >> everywhere, but normalization is very production dependent and
> there
> > > is
> > > > >> no
> > > > >> point of mixing normalization with parsing. It would be nice to
> > have a
> > > > >> sperate bolt in a parsing topologies to dedicate to production
> > > > >> related cleaning process. In that case, eveybody can easily
> > contribute
> > > > to
> > > > >> Metron community with additional parsers without being worried
> about
> > > > >> mixing
> > > > >> parsers and data cleaning process.
> > > > >>
> > > > >>
> > > > >> Regards,
> > > > >>
> > > > >> Ali
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > A.Nazemian
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > > >
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>



-- 
A.Nazemian

[GitHub] incubator-metron pull request #549: METRON-889: Regression introduced in Ful...

[merrimanr]

Github user merrimanr closed the pull request at:

https://github.com/apache/incubator-metron/pull/549


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Normalization topology or separate normalization bolt for parsing topology

[Casey Stella]

Ok, that's another story.  h, we don't generally pre-parse becuase we
try to not assume any particular format there (i.e. it could be strings,
could be byte arrays).  Maybe the right answer is to pass the raw,
non-normalized data (best effort tyep of thing) through the parser and do
the normalization post-parse..or is there a problem with that?

On Wed, Apr 26, 2017 at 9:33 AM, Ali Nazemian  wrote:

> Hi Casey,
>
> It is actually pre-parse process, not a post-parse one. These type of
> noises affect the position of an attribute for example and give us parsing
> exception. The timestamp example was not a good one because that is
> actually a post-parse exception.
>
> On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella  wrote:
>
> > So, further transformation post-parse was one of the motivating reasons
> for
> > Stellar (to do that transformation post-parse).  Is there a capability
> that
> > it's lacking that we can add to fit your usecase?
> >
> > On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian 
> > wrote:
> >
> > > I've created a Jira ticket regarding this feature.
> > >
> > > https://issues.apache.org/jira/browse/METRON-893
> > >
> > >
> > > On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian 
> > > wrote:
> > >
> > > > Currently, we are using normal regex at the Java source code to
> handle
> > > > those situations. However, it would be nice to have a separate bolt
> and
> > > > deal with them separately. Yeah, I can create a Jira issue regarding
> > > that.
> > > > The main reason I am asking for such a feature is the fact that lack
> of
> > > > such a feature makes the process of creating some parser for the
> > > community
> > > > a little painful for us. We need to maintain two different versions,
> > one
> > > > for community another for the internal use case. Clearly, noise is an
> > > > inevitable part of real world use cases.
> > > >
> > > > Cheers,
> > > > Ali
> > > >
> > > > On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler <
> ottobackwa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> Are you doing this cleansing all in the parser or are you using any
> > > >> Stellar to do it?
> > > >> Can you create a jira?
> > > >>
> > > >>
> > > >>
> > > >> On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com)
> > > >> wrote:
> > > >>
> > > >> Hi all,
> > > >>
> > > >>
> > > >> We are facing certain use cases in Metron production that happen to
> be
> > > >> related to noisy stream. For example, a wrong timestamp, duplicate
> > > >> hostname/IP address, etc. To deal with the normalization we have
> added
> > > an
> > > >> additional step for the corresponding parsers to do the data
> cleaning.
> > > >> Clearly, parsing is a standard factor which is mostly related to the
> > > >> device
> > > >> that is generating the data and can be used for the same type of
> > device
> > > >> everywhere, but normalization is very production dependent and there
> > is
> > > >> no
> > > >> point of mixing normalization with parsing. It would be nice to
> have a
> > > >> sperate bolt in a parsing topologies to dedicate to production
> > > >> related cleaning process. In that case, eveybody can easily
> contribute
> > > to
> > > >> Metron community with additional parsers without being worried about
> > > >> mixing
> > > >> parsers and data cleaning process.
> > > >>
> > > >>
> > > >> Regards,
> > > >>
> > > >> Ali
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > A.Nazemian
> > > >
> > >
> > >
> > >
> > > --
> > > A.Nazemian
> > >
> >
>
>
>
> --
> A.Nazemian
>

[GitHub] incubator-metron issue #552: METRON-857 FIX: Full_Dev regression: use the pr...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/incubator-metron/pull/552
  
Ran up full dev with this change and it is working again.  Thanks for 
addressing this so quickly @ottobackwards. +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Normalization topology or separate normalization bolt for parsing topology

[Ali Nazemian]

Hi Casey,

It is actually pre-parse process, not a post-parse one. These type of
noises affect the position of an attribute for example and give us parsing
exception. The timestamp example was not a good one because that is
actually a post-parse exception.

On Wed, Apr 26, 2017 at 11:28 PM, Casey Stella  wrote:

> So, further transformation post-parse was one of the motivating reasons for
> Stellar (to do that transformation post-parse).  Is there a capability that
> it's lacking that we can add to fit your usecase?
>
> On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian 
> wrote:
>
> > I've created a Jira ticket regarding this feature.
> >
> > https://issues.apache.org/jira/browse/METRON-893
> >
> >
> > On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian 
> > wrote:
> >
> > > Currently, we are using normal regex at the Java source code to handle
> > > those situations. However, it would be nice to have a separate bolt and
> > > deal with them separately. Yeah, I can create a Jira issue regarding
> > that.
> > > The main reason I am asking for such a feature is the fact that lack of
> > > such a feature makes the process of creating some parser for the
> > community
> > > a little painful for us. We need to maintain two different versions,
> one
> > > for community another for the internal use case. Clearly, noise is an
> > > inevitable part of real world use cases.
> > >
> > > Cheers,
> > > Ali
> > >
> > > On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler  >
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> Are you doing this cleansing all in the parser or are you using any
> > >> Stellar to do it?
> > >> Can you create a jira?
> > >>
> > >>
> > >>
> > >> On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com)
> > >> wrote:
> > >>
> > >> Hi all,
> > >>
> > >>
> > >> We are facing certain use cases in Metron production that happen to be
> > >> related to noisy stream. For example, a wrong timestamp, duplicate
> > >> hostname/IP address, etc. To deal with the normalization we have added
> > an
> > >> additional step for the corresponding parsers to do the data cleaning.
> > >> Clearly, parsing is a standard factor which is mostly related to the
> > >> device
> > >> that is generating the data and can be used for the same type of
> device
> > >> everywhere, but normalization is very production dependent and there
> is
> > >> no
> > >> point of mixing normalization with parsing. It would be nice to have a
> > >> sperate bolt in a parsing topologies to dedicate to production
> > >> related cleaning process. In that case, eveybody can easily contribute
> > to
> > >> Metron community with additional parsers without being worried about
> > >> mixing
> > >> parsers and data cleaning process.
> > >>
> > >>
> > >> Regards,
> > >>
> > >> Ali
> > >>
> > >>
> > >
> > >
> > > --
> > > A.Nazemian
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>



-- 
A.Nazemian

Re: Normalization topology or separate normalization bolt for parsing topology

[Casey Stella]

So, having stellar operate on the whole message is definitely something
that would be cool.  That being said, it's also nice to motivate the
construction of functions to do simple transformations/normalizations.
That way, common useful capabilities may be reused all the places Stellar
is used (which is all over the place at this point).

If we had some example normalizations, we might be able to address the gaps
and it'd be a win-win. :)

On Wed, Apr 26, 2017 at 9:28 AM, Otto Fowler 
wrote:

> What if you could implement your cleaning in Stellar functions, which would
> be in libraries that were loaded as plugins and available to all your
> parsers?
>
> my_field = ALI_CLEANMYFIELD(my_field)
>
> The idea would be:
>
> * Metron has an archetype for creating stellar ‘libraries’
> * You write your stellar functions and the unit/integration tests for them,
> and maintain that project outside the metron tree ( as hopefully you will
> be able to do soon with parsers -METRON-777, METRON-258 )
> * You use the metron management UI to install your stellar libraries
> * You call your stellar functions from your parser configuration
>
>
>
> On April 26, 2017 at 09:11:25, Ali Nazemian (alinazem...@gmail.com) wrote:
>
> Currently, we are using normal regex at the Java source code to handle
> those situations. However, it would be nice to have a separate bolt and
> deal with them separately. Yeah, I can create a Jira issue regarding that.
> The main reason I am asking for such a feature is the fact that lack of
> such a feature makes the process of creating some parser for the community
> a little painful for us. We need to maintain two different versions, one
> for community another for the internal use case. Clearly, noise is an
> inevitable part of real world use cases.
>
> Cheers,
> Ali
>
> On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler 
> wrote:
>
> > Hi,
> >
> > Are you doing this cleansing all in the parser or are you using any
> > Stellar to do it?
> > Can you create a jira?
> >
> >
> >
> > On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com)
> wrote:
> >
> > Hi all,
> >
> >
> > We are facing certain use cases in Metron production that happen to be
> > related to noisy stream. For example, a wrong timestamp, duplicate
> > hostname/IP address, etc. To deal with the normalization we have added an
> > additional step for the corresponding parsers to do the data cleaning.
> > Clearly, parsing is a standard factor which is mostly related to the
> device
> > that is generating the data and can be used for the same type of device
> > everywhere, but normalization is very production dependent and there is
> no
> > point of mixing normalization with parsing. It would be nice to have a
> > sperate bolt in a parsing topologies to dedicate to production
> > related cleaning process. In that case, eveybody can easily contribute to
> > Metron community with additional parsers without being worried about
> mixing
> > parsers and data cleaning process.
> >
> >
> > Regards,
> >
> > Ali
> >
> >
>
>
> --
> A.Nazemian
>

Re: Normalization topology or separate normalization bolt for parsing topology

[Casey Stella]

So, further transformation post-parse was one of the motivating reasons for
Stellar (to do that transformation post-parse).  Is there a capability that
it's lacking that we can add to fit your usecase?

On Wed, Apr 26, 2017 at 9:24 AM, Ali Nazemian  wrote:

> I've created a Jira ticket regarding this feature.
>
> https://issues.apache.org/jira/browse/METRON-893
>
>
> On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian 
> wrote:
>
> > Currently, we are using normal regex at the Java source code to handle
> > those situations. However, it would be nice to have a separate bolt and
> > deal with them separately. Yeah, I can create a Jira issue regarding
> that.
> > The main reason I am asking for such a feature is the fact that lack of
> > such a feature makes the process of creating some parser for the
> community
> > a little painful for us. We need to maintain two different versions, one
> > for community another for the internal use case. Clearly, noise is an
> > inevitable part of real world use cases.
> >
> > Cheers,
> > Ali
> >
> > On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler 
> > wrote:
> >
> >> Hi,
> >>
> >> Are you doing this cleansing all in the parser or are you using any
> >> Stellar to do it?
> >> Can you create a jira?
> >>
> >>
> >>
> >> On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com)
> >> wrote:
> >>
> >> Hi all,
> >>
> >>
> >> We are facing certain use cases in Metron production that happen to be
> >> related to noisy stream. For example, a wrong timestamp, duplicate
> >> hostname/IP address, etc. To deal with the normalization we have added
> an
> >> additional step for the corresponding parsers to do the data cleaning.
> >> Clearly, parsing is a standard factor which is mostly related to the
> >> device
> >> that is generating the data and can be used for the same type of device
> >> everywhere, but normalization is very production dependent and there is
> >> no
> >> point of mixing normalization with parsing. It would be nice to have a
> >> sperate bolt in a parsing topologies to dedicate to production
> >> related cleaning process. In that case, eveybody can easily contribute
> to
> >> Metron community with additional parsers without being worried about
> >> mixing
> >> parsers and data cleaning process.
> >>
> >>
> >> Regards,
> >>
> >> Ali
> >>
> >>
> >
> >
> > --
> > A.Nazemian
> >
>
>
>
> --
> A.Nazemian
>

Re: Normalization topology or separate normalization bolt for parsing topology

[Ali Nazemian]

I've created a Jira ticket regarding this feature.

https://issues.apache.org/jira/browse/METRON-893


On Wed, Apr 26, 2017 at 11:11 PM, Ali Nazemian 
wrote:

> Currently, we are using normal regex at the Java source code to handle
> those situations. However, it would be nice to have a separate bolt and
> deal with them separately. Yeah, I can create a Jira issue regarding that.
> The main reason I am asking for such a feature is the fact that lack of
> such a feature makes the process of creating some parser for the community
> a little painful for us. We need to maintain two different versions, one
> for community another for the internal use case. Clearly, noise is an
> inevitable part of real world use cases.
>
> Cheers,
> Ali
>
> On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler 
> wrote:
>
>> Hi,
>>
>> Are you doing this cleansing all in the parser or are you using any
>> Stellar to do it?
>> Can you create a jira?
>>
>>
>>
>> On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com)
>> wrote:
>>
>> Hi all,
>>
>>
>> We are facing certain use cases in Metron production that happen to be
>> related to noisy stream. For example, a wrong timestamp, duplicate
>> hostname/IP address, etc. To deal with the normalization we have added an
>> additional step for the corresponding parsers to do the data cleaning.
>> Clearly, parsing is a standard factor which is mostly related to the
>> device
>> that is generating the data and can be used for the same type of device
>> everywhere, but normalization is very production dependent and there is
>> no
>> point of mixing normalization with parsing. It would be nice to have a
>> sperate bolt in a parsing topologies to dedicate to production
>> related cleaning process. In that case, eveybody can easily contribute to
>> Metron community with additional parsers without being worried about
>> mixing
>> parsers and data cleaning process.
>>
>>
>> Regards,
>>
>> Ali
>>
>>
>
>
> --
> A.Nazemian
>



-- 
A.Nazemian

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

GitHub user cestella reopened a pull request:

https://github.com/apache/incubator-metron/pull/550

METRON-890: Intermittent unit test errors in shutting down Storm in memory 
component

## Contributor Comments
Cross your fingers.  This may or may not work.  Please don't merge until 
this runs at least 10 times in a row in travis.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?


 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
intermittent_unit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/550.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #550


commit a26c2275d14586bbbfc509214c0fa0907a01422a
Author: cstella 
Date:   2017-04-25T22:10:21Z

Cross your fingers.

commit 9706b46befd8e85cefad70c193e1e2fb5a71331c
Author: cstella 
Date:   2017-04-25T22:38:31Z

Updating spout.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #550: METRON-890: Intermittent unit test error...

[cestella]

Github user cestella closed the pull request at:

https://github.com/apache/incubator-metron/pull/550


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #548: METRON-888: Update README to show requir...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/548


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Normalization topology or separate normalization bolt for parsing topology

[Ali Nazemian]

Currently, we are using normal regex at the Java source code to handle
those situations. However, it would be nice to have a separate bolt and
deal with them separately. Yeah, I can create a Jira issue regarding that.
The main reason I am asking for such a feature is the fact that lack of
such a feature makes the process of creating some parser for the community
a little painful for us. We need to maintain two different versions, one
for community another for the internal use case. Clearly, noise is an
inevitable part of real world use cases.

Cheers,
Ali

On Wed, Apr 26, 2017 at 11:04 PM, Otto Fowler 
wrote:

> Hi,
>
> Are you doing this cleansing all in the parser or are you using any
> Stellar to do it?
> Can you create a jira?
>
>
>
> On April 26, 2017 at 08:59:16, Ali Nazemian (alinazem...@gmail.com) wrote:
>
> Hi all,
>
>
> We are facing certain use cases in Metron production that happen to be
> related to noisy stream. For example, a wrong timestamp, duplicate
> hostname/IP address, etc. To deal with the normalization we have added an
> additional step for the corresponding parsers to do the data cleaning.
> Clearly, parsing is a standard factor which is mostly related to the
> device
> that is generating the data and can be used for the same type of device
> everywhere, but normalization is very production dependent and there is no
> point of mixing normalization with parsing. It would be nice to have a
> sperate bolt in a parsing topologies to dedicate to production
> related cleaning process. In that case, eveybody can easily contribute to
> Metron community with additional parsers without being worried about
> mixing
> parsers and data cleaning process.
>
>
> Regards,
>
> Ali
>
>


-- 
A.Nazemian

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/459
  
For the errors on master:

[master_error.txt](https://github.com/apache/incubator-metron/files/958364/master_error.txt)
 
   4 BUILD FAILURE
   8 BUILD SUCCESS

Interestingly, it's two PcapTopologyIntegrationTest tests errors and two 
intermittent unit test failures.  I'm unsure if there are additional errors if 
things run longer. I believe the Pcap error has occurred on my branch, but I 
haven't seen the unit test failures before.

I apparently don't still have a recent log of the branch with errors.  
There is one where the branch ran 4 times successfully, but given that I've 
seen errors, I'm inclined to just run things on the branch overnight again and 
see where we end up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/459
  
@cestella I wish, I'd have been more than willing to ignore issues if it 
was.  I'm scraping together a more concise version of the some of the logs and 
will post them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #549: METRON-889: Regression introduced in Full Dev

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/549
  
A PR was opened up at https://github.com/apache/incubator-metron/pull/552.  
Given that both involve spinning up quick/full dev, I'm personally inclined to 
hold off on the revert, and just make sure that one works.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #552: METRON-857 FIX: Full_Dev regression: use the pr...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/552
  
Appreciate it, sorry for the trouble



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ambari Wizard: Repo Tab

[Otto Fowler]

Can you describe what ES issues you are working on?
Fixing the repos got everything installed, but my ES components don’t stay
running.
I need to harvest the errors.



On April 25, 2017 at 16:46:00, Otto Fowler (ottobackwa...@gmail.com) wrote:

Nm.  sorry.  I fixed it.



On April 25, 2017 at 16:42:05, Otto Fowler (ottobackwa...@gmail.com) wrote:

Ok, now I see the repos in the ‘pick version’ screen, but it is erring on
the f://localrepo
even though the folder exists, there is no repodata/repomd.xml.

What is the command to create a local repo?



On April 25, 2017 at 16:05:17, Otto Fowler (ottobackwa...@gmail.com) wrote:

I was going by the HW community page.

Ok, Let me try it



On April 25, 2017 at 16:04:07, David Lyle (dlyle65...@gmail.com) wrote:

That would do it. It requires 2.4.2+. I would have sworn I put that in the
README, but I must have only annotated the PR. :(

I'll get that in asap.

-D...


On Tue, Apr 25, 2017 at 3:44 PM, Otto Fowler 
wrote:

> 2.4.1 as a matter of fact
>
>
> On April 25, 2017 at 15:29:43, David Lyle (dlyle65...@gmail.com) wrote:
>
> Any chance they're running Ambari < 2.4.2?
>
> -D...
>
>
> On Tue, Apr 25, 2017 at 3:23 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Hey Otto, I don't have wizard screenshots in front of me right now to
> say
> > for sure, but I do see a repoinfo.xml in the mpack. I haven't run into
> > anything like that, but next time I run through the install I can take
> > another look.
> >
> > https://github.com/apache/incubator-metron/blob/master/
> > metron-deployment/packaging/ambari/metron-mpack/src/main/
> > resources/addon-services/ELASTICSEARCH/2.3.3/repos/repoinfo.xml
> >
> >
> > On Tue, Apr 25, 2017 at 12:16 PM, Otto Fowler 
> > wrote:
> >
> > > stderr:
> > >
> > > Traceback (most recent call last):
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 71, in 
> > >
> > > Elasticsearch().execute()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 280, in execute
> > >
> > > method(env)
> > >
> > > File "/var/lib/ambari-agent/cache/common-services/ELASTICSEARCH/
> > > 2.3.3/package/scripts/elastic_slave.py", line 32, in install
> > >
> > > self.install_packages(env)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > libraries/script/script.py",
> > > line 567, in install_packages
> > >
> > > retry_count=agent_stack_retry_count)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/base.py",
> > > line 155, in __init__
> > >
> > > self.env.run()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 160, in run
> > >
> > > self.run_action(resource, action)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/environment.py",
> > > line 124, in run_action
> > >
> > > provider_action()
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 54, in action_install
> > >
> > > self.install_package(package_name, self.resource.use_repos,
> > > self.resource.skip_repos)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/yumrpm.py", line 49, in install_package
> > >
> > > self.checked_call_with_retries(cmd, sudo=True,
> > > logoutput=self.get_logoutput())
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 83, in
> > checked_call_with_retries
> > >
> > > return self._call_with_retries(cmd, is_checked=True, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > > core/providers/package/__init__.py", line 91, in _call_with_retries
> > >
> > > code, out = func(cmd, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 71, in inner
> > >
> > > result = function(command, **kwargs)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 93, in checked_call
> > >
> > > tries=tries, try_sleep=try_sleep)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 141, in _call_wrapper
> > >
> > > result = _call(command, **kwargs_copy)
> > >
> > > File "/usr/lib/python2.6/site-packages/resource_management/
> > core/shell.py",
> > > line 294, in _call
> > >
> > > raise Fail(err_msg)
> > >
> > > resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum
> -d
> > 0
> > > -e 0 -y install elasticsearch-2.3.3' returned 1. Error: Nothing to do
> > >
> > > stdout:
> > >
> > > 2017-04-25 14:12:48,669
> > >  - Using hadoop
> > conf
> > > dir: /usr/hdp/current/hadoop-client/conf
> > >
> > > 2017-04-25 14:12:48,671
> 

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/459
  
I did submit a PR that is in flight and somewhat promising at cleaning up 
the intermittent storm shutdown errors.  Are those what you are seeing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Regression introduced in Full Dev

[David Lyle]

I just filter on 'to:(iss...@metron.incubator.apache.org)' and skip the
inbox. There's an interesting (and useful to me anyway) side-effect in
Gmail- issues I'm watching still end up in my inbox because JIRA also
emails me directly.

-D...


On Wed, Apr 26, 2017 at 8:13 AM, Otto Fowler 
wrote:

> It did, I didn’t see it until later in the night though, all my jira spam
> goes into one folder
> and honestly, nifi issues is flooding it.  I’ll have to split that out.
>
> Is there a metron issues list?
>
>
> On April 26, 2017 at 08:08:59, David Lyle (dlyle65...@gmail.com) wrote:
>
> Thanks Otto, the original JIRA is good. I reopened it yesterday when I had
> the issue. I was hoping it would have emailed you.
>
> -D...
>
>
> On Wed, Apr 26, 2017 at 8:04 AM, zeo...@gmail.com 
> wrote:
>
> > Interesting. I found it via pony mail -
> > https://lists.apache.org/thread.html/82e194ad8f8b8378676a28c09b074f
> > 45dee82820ead6ff8ee8fbebcc@
> > 
> >
> > But nothing in my inbox. I suspected it was @metron.incubator.apache.org
> > vs @metron.apache.org but when I attempted to subscribe to the top
> level
> > mailing list I was told I'm already subscribed. Same with User.
> >
> > Jon
> >
> > On Wed, Apr 26, 2017, 7:39 AM Justin Leet 
> wrote:
> >
> > > I have it (and had it yesterday). Subject is: "[DISCUSS] The various
> > > methods and incantations to spin up Metron".
> > >
> > > On Wed, Apr 26, 2017 at 7:33 AM, zeo...@gmail.com 
> > > wrote:
> > >
> > > > Yeah, I don't see the other thread either. Stuck in the outbox
> Casey?
> > > >
> > > > Jon
> > > >
> > > > On Wed, Apr 26, 2017, 6:53 AM Otto Fowler 
> > > wrote:
> > > >
> > > > > What other thread?
> > > > >
> > > > >
> > > > > On April 25, 2017 at 19:56:56, Casey Stella (ceste...@gmail.com)
> > > wrote:
> > > > >
> > > > > Ok, I spun up that discussion in another thread. Hopefully we can
> get
> > > > some
> > > > > better sense about the various ways to spin up metron and a
> > centralized
> > > > > place to direct people to along with with guidance on when some
> > > approach
> > > > > would be better than another.
> > > > >
> > > > > I'll be honest, I've totally lost track and never really consider
> > > > anything
> > > > > outside of full-dev anymore since it's the one that is generally
> > stable
> > > > > (quick-dev gets out of date quickly because mpack changes cause it
> to
> > > get
> > > > > stale) and is sufficient for validating PRs. Most of the other
> ones
> > > tend
> > > > > to either not have all of the system spun up (i.e. the hadoop
> > > components)
> > > > > and therefore end up with me having to test in full-dev anyway or
> > just
> > > > > weren't apparent to me and have unknown pros and cons. ;)
> > > > >
> > > > > On Tue, Apr 25, 2017 at 7:21 PM, Casey Stella 
>
> > > > wrote:
> > > > >
> > > > > > Yeah, I tend to agree that a rundown of the various methods and
> > when
> > > > you
> > > > > > would use them is in order. I will say that full-dev is
> especially
> > > > > > important to have working since it is required for validating
> PRs.
> > > > > >
> > > > > > On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com 
>
> > > > wrote:
> > > > > >
> > > > > >> Can somebody map out all of the current methods and procedures
> to
> > > spin
> > > > > up
> > > > > >> Metron components? I swear I find out about new ones every
> month.
> > > > > >> Metron-docker, the 4 vagrants, rpm-docker, ansible-docker, any
> > > others?
> > > > > >> Perhaps an agreed upon write up of when to use what would be
> > > helpful.
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >> On Tue, Apr 25, 2017, 6:17 PM Ryan Merriman <
> merrim...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> > A regression was introduced recently that breaks full dev.
> I've
> > > > > >> narrowed
> > > > > >> > down the commit that introduced it and have submitted a PR to
> > > revert
> > > > > >> that
> > > > > >> > commit: https://github.com/apache/incubator-metron/pull/549.
> > > > > >> >
> > > > > >> > Given there has been confusion recently over our deployment
> > build
> > > > > >> process,
> > > > > >> > I think it's appropriate that we discuss and come to a
> consensus
> > > and
> > > > > on
> > > > > >> > how this should work.
> > > > > >> >
> > > > > >> > Ryan
> > > > > >> >
> > > > > >> --
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> > --
> >
> > Jon
> >
>
>

Re: [DISCUSS] Regression introduced in Full Dev

[zeo...@gmail.com]

Interesting.  I found it via pony mail -
https://lists.apache.org/thread.html/82e194ad8f8b8378676a28c09b074f45dee82820ead6ff8ee8fbebcc@


But nothing in my inbox.  I suspected it was @metron.incubator.apache.org
vs @metron.apache.org but when I attempted to subscribe to the top level
mailing list I was told I'm already subscribed.  Same with User.

Jon

On Wed, Apr 26, 2017, 7:39 AM Justin Leet  wrote:

> I have it (and had it yesterday). Subject is:  "[DISCUSS] The various
> methods and incantations to spin up Metron".
>
> On Wed, Apr 26, 2017 at 7:33 AM, zeo...@gmail.com 
> wrote:
>
> > Yeah, I don't see the other thread either.  Stuck in the outbox Casey?
> >
> > Jon
> >
> > On Wed, Apr 26, 2017, 6:53 AM Otto Fowler 
> wrote:
> >
> > > What other thread?
> > >
> > >
> > > On April 25, 2017 at 19:56:56, Casey Stella (ceste...@gmail.com)
> wrote:
> > >
> > > Ok, I spun up that discussion in another thread. Hopefully we can get
> > some
> > > better sense about the various ways to spin up metron and a centralized
> > > place to direct people to along with with guidance on when some
> approach
> > > would be better than another.
> > >
> > > I'll be honest, I've totally lost track and never really consider
> > anything
> > > outside of full-dev anymore since it's the one that is generally stable
> > > (quick-dev gets out of date quickly because mpack changes cause it to
> get
> > > stale) and is sufficient for validating PRs. Most of the other ones
> tend
> > > to either not have all of the system spun up (i.e. the hadoop
> components)
> > > and therefore end up with me having to test in full-dev anyway or just
> > > weren't apparent to me and have unknown pros and cons. ;)
> > >
> > > On Tue, Apr 25, 2017 at 7:21 PM, Casey Stella 
> > wrote:
> > >
> > > > Yeah, I tend to agree that a rundown of the various methods and when
> > you
> > > > would use them is in order. I will say that full-dev is especially
> > > > important to have working since it is required for validating PRs.
> > > >
> > > > On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com 
> > wrote:
> > > >
> > > >> Can somebody map out all of the current methods and procedures to
> spin
> > > up
> > > >> Metron components? I swear I find out about new ones every month.
> > > >> Metron-docker, the 4 vagrants, rpm-docker, ansible-docker, any
> others?
> > > >> Perhaps an agreed upon write up of when to use what would be
> helpful.
> > > >>
> > > >> Jon
> > > >>
> > > >> On Tue, Apr 25, 2017, 6:17 PM Ryan Merriman 
> > > wrote:
> > > >>
> > > >> > A regression was introduced recently that breaks full dev. I've
> > > >> narrowed
> > > >> > down the commit that introduced it and have submitted a PR to
> revert
> > > >> that
> > > >> > commit: https://github.com/apache/incubator-metron/pull/549.
> > > >> >
> > > >> > Given there has been confusion recently over our deployment build
> > > >> process,
> > > >> > I think it's appropriate that we discuss and come to a consensus
> and
> > > on
> > > >> > how this should work.
> > > >> >
> > > >> > Ryan
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > >
> > --
> >
> > Jon
> >
>
-- 

Jon

[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/459
  
I think you're suffering from the same intermittent issues we have on 
master. Concur that you didn't add anything that obviously would create a 
problem and you may have helped. Maybe measure the pass/fail rate of N runs on 
this branch and N runs on master and see if they're similar? What was the fail 
rate for your current testing, btw?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Regression introduced in Full Dev

[Otto Fowler]

So I have a fix, vagrant full installs for me, with the rpm’s in
/localrepo.  I do have issues
post deploy cluster, but I think they are resource related.

How should I go about this?  What Jira would I post the PR against?  The
same as the original?



On April 26, 2017 at 07:39:07, Justin Leet (justinjl...@gmail.com) wrote:

I have it (and had it yesterday). Subject is: "[DISCUSS] The various
methods and incantations to spin up Metron".

On Wed, Apr 26, 2017 at 7:33 AM, zeo...@gmail.com  wrote:

> Yeah, I don't see the other thread either. Stuck in the outbox Casey?
>
> Jon
>
> On Wed, Apr 26, 2017, 6:53 AM Otto Fowler 
wrote:
>
> > What other thread?
> >
> >
> > On April 25, 2017 at 19:56:56, Casey Stella (ceste...@gmail.com) wrote:
> >
> > Ok, I spun up that discussion in another thread. Hopefully we can get
> some
> > better sense about the various ways to spin up metron and a centralized
> > place to direct people to along with with guidance on when some
approach
> > would be better than another.
> >
> > I'll be honest, I've totally lost track and never really consider
> anything
> > outside of full-dev anymore since it's the one that is generally stable
> > (quick-dev gets out of date quickly because mpack changes cause it to
get
> > stale) and is sufficient for validating PRs. Most of the other ones
tend
> > to either not have all of the system spun up (i.e. the hadoop
components)
> > and therefore end up with me having to test in full-dev anyway or just
> > weren't apparent to me and have unknown pros and cons. ;)
> >
> > On Tue, Apr 25, 2017 at 7:21 PM, Casey Stella 
> wrote:
> >
> > > Yeah, I tend to agree that a rundown of the various methods and when
> you
> > > would use them is in order. I will say that full-dev is especially
> > > important to have working since it is required for validating PRs.
> > >
> > > On Tue, Apr 25, 2017 at 18:56 zeo...@gmail.com 
> wrote:
> > >
> > >> Can somebody map out all of the current methods and procedures to
spin
> > up
> > >> Metron components? I swear I find out about new ones every month.
> > >> Metron-docker, the 4 vagrants, rpm-docker, ansible-docker, any
others?
> > >> Perhaps an agreed upon write up of when to use what would be
helpful.
> > >>
> > >> Jon
> > >>
> > >> On Tue, Apr 25, 2017, 6:17 PM Ryan Merriman 
> > wrote:
> > >>
> > >> > A regression was introduced recently that breaks full dev. I've
> > >> narrowed
> > >> > down the commit that introduced it and have submitted a PR to
revert
> > >> that
> > >> > commit: https://github.com/apache/incubator-metron/pull/549.
> > >> >
> > >> > Given there has been confusion recently over our deployment build
> > >> process,
> > >> > I think it's appropriate that we discuss and come to a consensus
and
> > on
> > >> > how this should work.
> > >> >
> > >> > Ryan
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> >
> --
>
> Jon
>

Re: metron UI

[Raghu Mitra Kandikonda]

Yeah I do agree there is a technical debt. But it might be true for any 
framework :). 

If you wish to extend the existing ui happy to help in any way.

-Raghu




On 26/04/17, 12:45 PM, "Kevin Waterson"  wrote:

>I have no real issues with Angular2, aside from possible technical debt
>with A3.
>I just do not feel it is the right tool for the job.
>A2 makes great single page applications, however, this is somewhat more
>modular.
>
>Kevin
>
>On Wed, Apr 26, 2017 at 4:38 PM, Raghu Mitra Kandikonda <
>r...@hortonworks.com> wrote:
>
>> Kevin,
>>
>> HTML5/d3 is a great choice for any charting/dashboard kind of requirements
>> but it might be a retro fit for a ui that has predominantly tables and
>> forms.
>>
>> Currently metron-ui is designed only to manage parser configurations. We
>> can still use d3 for our charting needs. Do you see any particular issues
>> with angular2 ?.
>>
>> -Raghu
>>
>>
>>
>> On 26/04/17, 11:41 AM, "moshe jarusalem"  wrote:
>>
>> >I may reserve some time to put some effort to your project. But first,
>> need
>> >to clarify some points; I have seen alert interface at the following link
>> >is it separate project from yours ?
>> >
>> >http://imgur.com/a/KMTKN
>> >
>> >If you may tell us work you need support for the project I will consider
>> if
>> >I can help.
>> >
>> >Regards,
>> >
>> >
>> >On Wed, Apr 26, 2017 at 2:43 AM, Waterson, Kevin <
>> >kevin.water...@team.telstra.com> wrote:
>> >
>> >> Of course, I should disclaim, this is my own project, and not a Telstra
>> >> related initiative.
>> >>
>> >> Kev
>> >>
>> >> -Original Message-
>> >> From: Waterson, Kevin [mailto:kevin.water...@team.telstra.com]
>> >> Sent: Wednesday, 26 April 2017 8:37 AM
>> >> To: dev@metron.apache.org
>> >> Subject: RE: metron UI
>> >>
>> >> HTML5/D3
>> >>
>> >> Kev
>> >>
>> >> -Original Message-
>> >> From: Ryan Merriman [mailto:merrim...@gmail.com]
>> >> Sent: Wednesday, 26 April 2017 8:24 AM
>> >> To: dev@metron.apache.org
>> >> Subject: Re: metron UI
>> >>
>> >> Kevin,
>> >>
>> >> Would be interested in learning more about the issues you have with
>> >> Angular.  Which technology would you prefer?
>> >>
>> >> Ryan
>> >>
>> >> On Tue, Apr 25, 2017 at 5:16 PM, Waterson, Kevin <
>> >> kevin.water...@team.telstra.com> wrote:
>> >>
>> >> > The use of Angular has me writing my own UI for Metron
>> >> >
>> >> > Kev
>> >> >
>> >> > -Original Message-
>> >> > From: Justin Leet [mailto:justinjl...@gmail.com]
>> >> > Sent: Tuesday, 25 April 2017 12:19 AM
>> >> > To: dev@metron.apache.org
>> >> > Subject: Re: metron UI
>> >> >
>> >> > To elaborate on Ryan's reply a bit, the UI is a fairly new component
>> >> > in Metron and was merged into master not long ago.  It's definitely
>> >> > open to being iterated and improved on, and getting feedback on
>> >> > direction users would like it to go would be a great contribution in
>> >> > and of itself (especially as it gets exercised and people want new
>> >> > features or improvements).
>> >> >
>> >> > Justin
>> >> >
>> >> > On Mon, Apr 24, 2017 at 9:59 AM, Ryan Merriman 
>> >> > wrote:
>> >> >
>> >> > > What features would you like to see included?
>> >> > >
>> >> > > On Mon, Apr 24, 2017 at 8:51 AM, moshe jarusalem 
>> >> > wrote:
>> >> > >
>> >> > > > Hi All,
>> >> > > > I have recently run metron UI. I am a bit surprised because it has
>> >> > > > very
>> >> > > few
>> >> > > > features such as configuring some ingestions and topologies.
>> >> > > >
>> >> > > > Have I not configured it properly or its features are limited at
>> >> > > > this
>> >> > > time
>> >> > > > ?
>> >> > > >
>> >> > > > Regards,
>> >> > > >
>> >> > >
>> >> >
>> >>
>>

Re: metron UI

[Kevin Waterson]

I have no real issues with Angular2, aside from possible technical debt
with A3.
I just do not feel it is the right tool for the job.
A2 makes great single page applications, however, this is somewhat more
modular.

Kevin

On Wed, Apr 26, 2017 at 4:38 PM, Raghu Mitra Kandikonda <
r...@hortonworks.com> wrote:

> Kevin,
>
> HTML5/d3 is a great choice for any charting/dashboard kind of requirements
> but it might be a retro fit for a ui that has predominantly tables and
> forms.
>
> Currently metron-ui is designed only to manage parser configurations. We
> can still use d3 for our charting needs. Do you see any particular issues
> with angular2 ?.
>
> -Raghu
>
>
>
> On 26/04/17, 11:41 AM, "moshe jarusalem"  wrote:
>
> >I may reserve some time to put some effort to your project. But first,
> need
> >to clarify some points; I have seen alert interface at the following link
> >is it separate project from yours ?
> >
> >http://imgur.com/a/KMTKN
> >
> >If you may tell us work you need support for the project I will consider
> if
> >I can help.
> >
> >Regards,
> >
> >
> >On Wed, Apr 26, 2017 at 2:43 AM, Waterson, Kevin <
> >kevin.water...@team.telstra.com> wrote:
> >
> >> Of course, I should disclaim, this is my own project, and not a Telstra
> >> related initiative.
> >>
> >> Kev
> >>
> >> -Original Message-
> >> From: Waterson, Kevin [mailto:kevin.water...@team.telstra.com]
> >> Sent: Wednesday, 26 April 2017 8:37 AM
> >> To: dev@metron.apache.org
> >> Subject: RE: metron UI
> >>
> >> HTML5/D3
> >>
> >> Kev
> >>
> >> -Original Message-
> >> From: Ryan Merriman [mailto:merrim...@gmail.com]
> >> Sent: Wednesday, 26 April 2017 8:24 AM
> >> To: dev@metron.apache.org
> >> Subject: Re: metron UI
> >>
> >> Kevin,
> >>
> >> Would be interested in learning more about the issues you have with
> >> Angular.  Which technology would you prefer?
> >>
> >> Ryan
> >>
> >> On Tue, Apr 25, 2017 at 5:16 PM, Waterson, Kevin <
> >> kevin.water...@team.telstra.com> wrote:
> >>
> >> > The use of Angular has me writing my own UI for Metron
> >> >
> >> > Kev
> >> >
> >> > -Original Message-
> >> > From: Justin Leet [mailto:justinjl...@gmail.com]
> >> > Sent: Tuesday, 25 April 2017 12:19 AM
> >> > To: dev@metron.apache.org
> >> > Subject: Re: metron UI
> >> >
> >> > To elaborate on Ryan's reply a bit, the UI is a fairly new component
> >> > in Metron and was merged into master not long ago.  It's definitely
> >> > open to being iterated and improved on, and getting feedback on
> >> > direction users would like it to go would be a great contribution in
> >> > and of itself (especially as it gets exercised and people want new
> >> > features or improvements).
> >> >
> >> > Justin
> >> >
> >> > On Mon, Apr 24, 2017 at 9:59 AM, Ryan Merriman 
> >> > wrote:
> >> >
> >> > > What features would you like to see included?
> >> > >
> >> > > On Mon, Apr 24, 2017 at 8:51 AM, moshe jarusalem 
> >> > wrote:
> >> > >
> >> > > > Hi All,
> >> > > > I have recently run metron UI. I am a bit surprised because it has
> >> > > > very
> >> > > few
> >> > > > features such as configuring some ingestions and topologies.
> >> > > >
> >> > > > Have I not configured it properly or its features are limited at
> >> > > > this
> >> > > time
> >> > > > ?
> >> > > >
> >> > > > Regards,
> >> > > >
> >> > >
> >> >
> >>
>