[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077509#comment-16077509
 ] 

ASF GitHub Bot commented on METRON-939:
---

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/619
  
Do we not have to re-write the ambari service?


> Upgrade ElasticSearch and Kibana
> 
>
> Key: METRON-939
> URL: https://issues.apache.org/jira/browse/METRON-939
> Project: Metron
>  Issue Type: Improvement
>Reporter: Jon Zeolla
>
> Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this).  Among 
> other benefits, this allows us to use periods in field names 
> (https://github.com/elastic/elasticsearch/pull/19937/files), which has been 
> available as of 5.0 and 2.4, and the ability to index an IPv6 address 
> properly 
> (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1021) increment metron version number to 0.4.1 in poms etc

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077507#comment-16077507
 ] 

ASF GitHub Bot commented on METRON-1021:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/637
  
The site/current-book stuff should still refer to current project as 0.4.0?


> increment metron version number to 0.4.1 in poms etc
> 
>
> Key: METRON-1021
> URL: https://issues.apache.org/jira/browse/METRON-1021
> Project: Metron
>  Issue Type: Task
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> It is good practice to increment the build version in POMs immediately after 
> a release, so that builds with new stuff cannot be mistaken for builds of the 
> release version. Consistent with METRON-1020 I suggest incrementing the minor 
> rather than the major version number for now, before we know what we're going 
> to release next and when.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1021) increment metron version number to 0.4.1 in poms etc

[Matt Foley (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077363#comment-16077363
 ] 

Matt Foley commented on METRON-1021:


For future reference, the following files had version numbers to change:
{code}
metron-analytics/metron-maas-common/pom.xml
metron-analytics/metron-maas-service/pom.xml
metron-analytics/metron-profiler-client/README.md
metron-analytics/metron-profiler-client/pom.xml
metron-analytics/metron-profiler-common/pom.xml
metron-analytics/metron-profiler/pom.xml
metron-analytics/metron-statistics/pom.xml
metron-analytics/pom.xml
metron-deployment/Kerberos-manual-setup.md
metron-deployment/amazon-ec2/conf/defaults.yml
metron-deployment/inventory/full-dev-platform/group_vars/all
metron-deployment/inventory/quick-dev-platform/group_vars/all
metron-deployment/packaging/ambari/metron-mpack/pom.xml
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/mpack.json
metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
metron-deployment/packaging/docker/rpm-docker/pom.xml
metron-deployment/playbooks/docker_probe_install.yml
metron-deployment/pom.xml
metron-deployment/roles/ambari_master/defaults/main.yml
metron-deployment/roles/ambari_master/tasks/main.yml
metron-deployment/roles/metron_pcapservice/defaults/main.yml
metron-docker/pom.xml
metron-interface/metron-config/package.json
metron-interface/metron-config/pom.xml
metron-interface/metron-config/scripts/package.json
metron-interface/metron-rest-client/pom.xml
metron-interface/metron-rest/pom.xml
metron-interface/metron-rest/src/main/resources/application.yml
metron-interface/pom.xml
metron-platform/README.md
metron-platform/elasticsearch-shaded/pom.xml
metron-platform/metron-api/pom.xml
metron-platform/metron-common/pom.xml
metron-platform/metron-data-management/pom.xml
metron-platform/metron-elasticsearch/pom.xml
metron-platform/metron-enrichment/pom.xml
metron-platform/metron-hbase/pom.xml
metron-platform/metron-indexing/pom.xml
metron-platform/metron-integration-test/pom.xml
metron-platform/metron-management/pom.xml
metron-platform/metron-parsers/pom.xml
metron-platform/metron-pcap-backend/pom.xml
metron-platform/metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh
metron-platform/metron-pcap/pom.xml
metron-platform/metron-solr/pom.xml
metron-platform/metron-storm-kafka-override/pom.xml
metron-platform/metron-storm-kafka/pom.xml
metron-platform/metron-test-utilities/pom.xml
metron-platform/metron-writer/pom.xml
metron-platform/pom.xml
metron-stellar/pom.xml
metron-stellar/stellar-common/3rdPartyStellar.md
metron-stellar/stellar-common/README.md
metron-stellar/stellar-common/pom.xml
pom.xml
site-book/pom.xml
{code}

> increment metron version number to 0.4.1 in poms etc
> 
>
> Key: METRON-1021
> URL: https://issues.apache.org/jira/browse/METRON-1021
> Project: Metron
>  Issue Type: Task
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> It is good practice to increment the build version in POMs immediately after 
> a release, so that builds with new stuff cannot be mistaken for builds of the 
> release version. Consistent with METRON-1020 I suggest incrementing the minor 
> rather than the major version number for now, before we know what we're going 
> to release next and when.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (METRON-1021) increment metron version number to 0.4.1 in poms

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley reassigned METRON-1021:
--

Assignee: Matt Foley

> increment metron version number to 0.4.1 in poms
> 
>
> Key: METRON-1021
> URL: https://issues.apache.org/jira/browse/METRON-1021
> Project: Metron
>  Issue Type: Task
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> It is good practice to increment the build version in POMs immediately after 
> a release, so that builds with new stuff cannot be mistaken for builds of the 
> release version. Consistent with METRON-1020 I suggest incrementing the minor 
> rather than the major version number for now, before we know what we're going 
> to release next and when.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1022) Elasticsearch REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077212#comment-16077212
 ] 

ASF GitHub Bot commented on METRON-1022:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/636#discussion_r126018829
  
--- Diff: 
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
 ---
@@ -60,96  61,8 @@ public ElasticsearchWriter 
withOptionalSettings(Map optionalSett
   @Override
   public void init(Map stormConf, TopologyContext topologyContext, 
WriterConfiguration configurations) {
 Map globalConfiguration = 
configurations.getGlobalConfig();
--- End diff --

This is a great idea, to standardize all this!


> Elasticsearch REST endpoint
> ---
>
> Key: METRON-1022
> URL: https://issues.apache.org/jira/browse/METRON-1022
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> We need a "search" endpoint that will allow basic lucene-style searches with 
> sorting and pagination options.  This endpoint should have a light 
> abstraction on top to make it simpler to consume and possibly allow different 
> search engines to be used in the future.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1022) Elasticsearch REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077193#comment-16077193
 ] 

ASF GitHub Bot commented on METRON-1022:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/636
  
@merrimanr, I'm obviously feeling a little confused about what seems like 3 
related things going on.  I don't mean to say this is the wrong approach.  
Would you feel comfortable opening a [DISCUSS] thread on REST and or Real Time 
Search abstraction?


> Elasticsearch REST endpoint
> ---
>
> Key: METRON-1022
> URL: https://issues.apache.org/jira/browse/METRON-1022
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> We need a "search" endpoint that will allow basic lucene-style searches with 
> sorting and pagination options.  This endpoint should have a light 
> abstraction on top to make it simpler to consume and possibly allow different 
> search engines to be used in the future.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1022) Elasticsearch REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077158#comment-16077158
 ] 

ASF GitHub Bot commented on METRON-1022:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/636
  
I feel like this and the alert UI are building the transcontinental 
railroad from each end of the county, and are going to miss each other by a 
state.

How does this relate to that effort?  Have we done any coordination?

From that PR has discussion about NOT having an api direct to the realtime 
store, but rather abstracting that.  Should this endpoint not be an abstraction?




> Elasticsearch REST endpoint
> ---
>
> Key: METRON-1022
> URL: https://issues.apache.org/jira/browse/METRON-1022
> Project: Metron
>  Issue Type: New Feature
>Reporter: Ryan Merriman
>
> We need a "search" endpoint that will allow basic lucene-style searches with 
> sorting and pagination options.  This endpoint should have a light 
> abstraction on top to make it simpler to consume and possibly allow different 
> search engines to be used in the future.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1022) Elasticsearch REST endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077148#comment-16077148
 ] 

ASF GitHub Bot commented on METRON-1022:


GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/636

METRON-1022: Elasticsearch REST endpoint

## Contributor Comments
The PR adds a simple Elasticsearch REST endpoint that can execute simple 
queries.  This will serve as the foundation for building more complex 
Elasticsearch functions.

This can be tested with integration tests or in full dev.  To test is full 
dev, navigate to Swagger through Ambari and look for the "search-controller".  
Here you can perform simple queries against data in Elasticsearch.  For 
example, posting this query should give back correct search results:
```
{
  "from": 0,
  "indices": [
"bro", "snort"
  ],
  "query": "*",
  "size": 10,
  "sort": [
{
  "field": "timestamp",
  "sortOrder": "desc"
}
  ]
}
```
Note:  Swagger gives an example query with sortOrder = 'DESC'.  As of now 
this needs to be lowercase but I'm considering changing it to accept both.  

I still need to come up with a good description and add it to the README so 
that is still pending.  Still wanted to put this up for review as it is fairly 
large and the first time any Elasticsearch-related code has been added to REST. 
 Looking forward to the community's feedback.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying 
Builds?show-miniview) for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1022

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/636.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #636


commit 672e086f7905071f316cca51fd752f764ea46f11
Author: merrimanr 
Date:   2017-07-06T20:02:36Z

initial commit

commit e70533a150d2664f7221fba23ffdc82769fa0b0c
Author: merrimanr 
Date:   

[jira] [Commented] (METRON-1001) Allow metron to ingest parser metadata along with data

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077100#comment-16077100
 ] 

ASF GitHub Bot commented on METRON-1001:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/621
  
Merging master in to get the unit tests to run.


> Allow metron to ingest parser metadata along with data
> --
>
> Key: METRON-1001
> URL: https://issues.apache.org/jira/browse/METRON-1001
> Project: Metron
>  Issue Type: Improvement
>Reporter: Casey Stella
>Assignee: Casey Stella
>
> Currently, we only ingest data in Metron.  Often, there is valuable metadata 
> constructed up-stream of Metron that is relevant to enrichment and cross-cuts 
> many data formats.  Take, for instance, a multi-tenancy case where multiple 
> sources come in and you'd like to tag the data with the customer ID.  In this 
> case you're stuck finding ways to add the metadata to each data source's 
> format.  Rather than do that, we should allow metadata to be ingested along 
> with the data associated with it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (METRON-1022) Elasticsearch REST endpoint

[Ryan Merriman (JIRA)]

Ryan Merriman created METRON-1022:
-

 Summary: Elasticsearch REST endpoint
 Key: METRON-1022
 URL: https://issues.apache.org/jira/browse/METRON-1022
 Project: Metron
  Issue Type: New Feature
Reporter: Ryan Merriman


We need a "search" endpoint that will allow basic lucene-style searches with 
sorting and pagination options.  This endpoint should have a light abstraction 
on top to make it simpler to consume and possibly allow different search 
engines to be used in the future.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1019) Metron 0.4.0 manual installation guide for CentOS 6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076843#comment-16076843
 ] 

ASF GitHub Bot commented on METRON-1019:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/631
  
 1.  I believe this is positioned with the correct expectations, and 
content wise has been worked out on the list ( where this contribution was 
indeed solicited ).

Would like to hear from @JonZeolla before committing.



> Metron 0.4.0 manual installation guide for CentOS 6 
> 
>
> Key: METRON-1019
> URL: https://issues.apache.org/jira/browse/METRON-1019
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> This guide will how to install Metron 0.4.0 with HDP 
> 2.5 bare-metal install on Centos 6 with MySQL for Metron REST and Apache 
> NiFi for log ingestion. Basically, it's a manual install of the full dev 
> environment with some additions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1019) Metron 0.4.0 manual installation guide for CentOS 6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076802#comment-16076802
 ] 

ASF GitHub Bot commented on METRON-1019:


Github user lvets commented on the issue:

https://github.com/apache/metron/pull/631
  
Changed the wording in the README.md file to manage expectations.
Added month + year.
There's a version number in the raw source of "Manual_Install_CentOS6.md".



> Metron 0.4.0 manual installation guide for CentOS 6 
> 
>
> Key: METRON-1019
> URL: https://issues.apache.org/jira/browse/METRON-1019
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> This guide will how to install Metron 0.4.0 with HDP 
> 2.5 bare-metal install on Centos 6 with MySQL for Metron REST and Apache 
> NiFi for log ingestion. Basically, it's a manual install of the full dev 
> environment with some additions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076670#comment-16076670
 ] 

ASF GitHub Bot commented on METRON-1018:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/635#discussion_r125928629
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
 ---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties 
topologyProperties) {
 topologyProperties.setProperty("es.clustername", "metron");
 topologyProperties.setProperty("es.port", "9300");
 topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
   }
 
   @Override
   public String cleanField(String field) {
 return field;
   }
--- End diff --

I would feel better keeping the template with the code and doing a resource 
copy into the ambari template area, hoping someone else will chime in


> Integration tests should reference flux yaml and property files deployed by 
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990.  During the review of that Jira it was 
> discovered that our integration tests are not referencing the actual flux 
> yaml and property files that are used in a Metron installation.  This means 
> the integration tests must also be maintained separately with no automated 
> protection against regression in cases where flux yaml or property files 
> change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1019) Metron 0.4.0 manual installation guide for CentOS 6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076619#comment-16076619
 ] 

ASF GitHub Bot commented on METRON-1019:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/631#discussion_r125919953
  
--- Diff: metron-deployment/other-examples/README.md ---
@@ -0,0 +1,3 @@
+# Other Example Deployments
--- End diff --

What I would want to see in this readme is some kind of disclaimer, but I'm 
not sure what it would be.

"these examples have been vetted and tested by the metron community as of 
the time of entry.  Things may change however, and if there are problems with 
these steps, please log a jira"

Also, should there be a date on each example, to show when it was entered?
Should there be a revision log?


> Metron 0.4.0 manual installation guide for CentOS 6 
> 
>
> Key: METRON-1019
> URL: https://issues.apache.org/jira/browse/METRON-1019
> Project: Metron
>  Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>
> This guide will how to install Metron 0.4.0 with HDP 
> 2.5 bare-metal install on Centos 6 with MySQL for Metron REST and Apache 
> NiFi for log ingestion. Basically, it's a manual install of the full dev 
> environment with some additions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076603#comment-16076603
 ] 

ASF GitHub Bot commented on METRON-1018:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/635#discussion_r125916203
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
 ---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties 
topologyProperties) {
 topologyProperties.setProperty("es.clustername", "metron");
 topologyProperties.setProperty("es.port", "9300");
 topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
   }
 
   @Override
   public String cleanField(String field) {
 return field;
   }
--- End diff --

this will be the first time pulling things out of ambari ( or 
metron-deployment really ) into the project for test or other.  I think we 
should think about alternative approaches


> Integration tests should reference flux yaml and property files deployed by 
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990.  During the review of that Jira it was 
> discovered that our integration tests are not referencing the actual flux 
> yaml and property files that are used in a Metron installation.  This means 
> the integration tests must also be maintained separately with no automated 
> protection against regression in cases where flux yaml or property files 
> change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076578#comment-16076578
 ] 

ASF GitHub Bot commented on METRON-1018:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/635#discussion_r125913966
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
 ---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties 
topologyProperties) {
 topologyProperties.setProperty("es.clustername", "metron");
 topologyProperties.setProperty("es.port", "9300");
 topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
   }
 
   @Override
   public String cleanField(String field) {
 return field;
   }
--- End diff --

I think it's a valid concern.  I'm also not sure what the ideal alternative 
is.  At this point I feel like Metron has succumbed to being tightly coupled 
with Ambari.


> Integration tests should reference flux yaml and property files deployed by 
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990.  During the review of that Jira it was 
> discovered that our integration tests are not referencing the actual flux 
> yaml and property files that are used in a Metron installation.  This means 
> the integration tests must also be maintained separately with no automated 
> protection against regression in cases where flux yaml or property files 
> change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076554#comment-16076554
 ] 

ASF GitHub Bot commented on METRON-1018:


Github user merrimanr commented on a diff in the pull request:

https://github.com/apache/metron/pull/635#discussion_r125910236
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
 ---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties 
topologyProperties) {
 topologyProperties.setProperty("es.clustername", "metron");
 topologyProperties.setProperty("es.port", "9300");
 topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
   }
 
   @Override
   public String cleanField(String field) {
 return field;
   }
+
--- End diff --

This was there previously and not something I added.  It's an 
implementation of an abstract method in IndexingIntegrationTest and it looks 
like this implementation differs from the one in SolrIndexingIntegrationTest.  

I don't see a way to move this to a util class without significantly 
changing IndexingIntegrationTest.


> Integration tests should reference flux yaml and property files deployed by 
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990.  During the review of that Jira it was 
> discovered that our integration tests are not referencing the actual flux 
> yaml and property files that are used in a Metron installation.  This means 
> the integration tests must also be maintained separately with no automated 
> protection against regression in cases where flux yaml or property files 
> change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076534#comment-16076534
 ] 

ASF GitHub Bot commented on METRON-1018:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/635#discussion_r125906152
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
 ---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties 
topologyProperties) {
 topologyProperties.setProperty("es.clustername", "metron");
 topologyProperties.setProperty("es.port", "9300");
 topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
   }
 
   @Override
   public String cleanField(String field) {
 return field;
   }
+
--- End diff --

This could be in test utils?


> Integration tests should reference flux yaml and property files deployed by 
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990.  During the review of that Jira it was 
> discovered that our integration tests are not referencing the actual flux 
> yaml and property files that are used in a Metron installation.  This means 
> the integration tests must also be maintained separately with no automated 
> protection against regression in cases where flux yaml or property files 
> change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076530#comment-16076530
 ] 

ASF GitHub Bot commented on METRON-1018:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/635#discussion_r125905841
  
--- Diff: 
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
 ---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties 
topologyProperties) {
 topologyProperties.setProperty("es.clustername", "metron");
 topologyProperties.setProperty("es.port", "9300");
 topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name", 
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
   }
 
   @Override
   public String cleanField(String field) {
 return field;
   }
--- End diff --

I am not sure about having things live in ambari vs. things being 
referenced by ambari. That doesn't seem right, like a violation of a separation 
of concerns.  I know we do this other places too ( like the ES Templates, 
although I'm trying to move away from that ).   I am not sure I have a 
different way to suggest, but I want to bring up the point.


> Integration tests should reference flux yaml and property files deployed by 
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990.  During the review of that Jira it was 
> discovered that our integration tests are not referencing the actual flux 
> yaml and property files that are used in a Metron installation.  This means 
> the integration tests must also be maintained separately with no automated 
> protection against regression in cases where flux yaml or property files 
> change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076487#comment-16076487
 ] 

ASF GitHub Bot commented on METRON-1018:


GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/635

METRON-1018: Integration tests should reference flux yaml and property 
files deployed by Ambari

## Contributor Comments
This PR makes a couple of improvements to our integration tests.  

Enrichment and integration tests now set properties using the same 
templates as the Ambari MPack.   This will make our tests easier to maintain 
because there is now a single source of properties.  Any 
addition/subtraction/modification to those properties will be detected in the 
tests.

The EnrichmentIntegrationTest now uses a single flux file instead of having 
a dedicated test flux file.  This was problematic because a) two flux files 
needed to be maintained for one topology and b) the test flux file could easily 
become stale and no longer accurately represent the enrichment topology (this 
happened to me on METRON-990).  The introduction of Stellar (this test was 
written a while ago) makes it possible to simulate an error so a modified 
topology/flux file with an error generating bolt is no longer required.

As a result of adding an error-generating Stellar statement I encountered a 
bug in the GenericEnrichmentBolt.  The exception thrown by the Stellar adapter 
was being reported twice: once on line 235 and again on line 275.  I attempted 
to fix this by removing the "error" flag and just reporting the error right 
when it happens.  There are likely multiple ways to do this so I want to 
highlight this class for reviewers.  Suggestions and scrutiny needed here.

Testing this is simple:  all tests should pass without issue.  Any change 
to the templates or enrichment flux file should break the tests.  I also 
ensured full dev starts up without issue. 

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1018

Alternatively you can review and apply 

[jira] [Commented] (METRON-1017) Ambari components should be separate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076438#comment-16076438
 ] 

ASF GitHub Bot commented on METRON-1017:


GitHub user merrimanr opened a pull request:

https://github.com/apache/metron/pull/634

METRON-1017: Ambari components should be separate

## Contributor Comments
This PR separates the various configurations in Ambari into different 
*-env.xml files.  Doing this allows us to create more granular dependencies so 
that a configuration change only requires a restart of the affected components 
and not the whole service.  Previously any change to any configuration would 
require every Metron component to be restarted.  There was also a small amount 
of refactoring done to the enrichment and indexing topics in MPack scripts to 
make them more clear and consistent.

This can be tested by spinning up full dev and changing various 
configurations.  A configuration change should now only prompt to restart the 
affected components instead of all components.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-1017

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/634.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #634


commit 85608e971fc15846d49c6701e1196f2c18e0ec9f
Author: merrimanr 
Date:   2017-07-05T16:31:36Z

initial commit

commit 4f15918c13bc55e70144ca70f067cc4d17845f26
Author: merrimanr 
Date:   2017-07-06T12:38:07Z

Merge remote-tracking branch 'mirror/master' into METRON-1017

commit c64cd752869f49309b8bee17a1a8740e25b4d4a9
Author: merrimanr 
Date:   2017-07-06T12:42:45Z

newline cleanup




> Ambari components should be separate
> 
>
> Key: METRON-1017
> URL: https://issues.apache.org/jira/browse/METRON-1017
> Project: Metron