date:20180129

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
FYI @cestella I submitted a fix against your PR branch that should address 
the issue with the embedded handlebars in the Ambari response. 


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

2018-01-29 Thread mmiklavc

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/915
  

https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java#L89


---

[GitHub] metron issue #912: METRON-1431 Adding REGEX_REPLACE(input, regex, value) to ...

2018-01-29 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/912
  
please remember to take care of your jira


---

[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

2018-01-29 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/912


---

[GitHub] metron issue #912: METRON-1431 Adding REGEX_REPLACE(input, regex, value) to ...

2018-01-29 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/912
  
+1 by inspection.  Thank you for the contribution



---

[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

2018-01-29 Thread 1havran

Github user 1havran commented on a diff in the pull request:

https://github.com/apache/metron/pull/912#discussion_r164570247
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RegExFunctions.java
 ---
@@ -100,4 +100,39 @@ public Object apply(List list) {
   return matcher.group(groupNumber);
 }
   }
+
+  @Stellar(name = "REGEXP_REPLACE",
+  description = "Replace all occurences of the regex pattern within 
the string by value",
+  params = {
+  "string - The input string",
+  "pattern - The regex pattern to be replaced. Special characters 
must be escaped (e.g. d)",
+  "value - The value to replace the regex pattern"
+  },
+  returns = "The modified input string with replaced values")
+  public static class RegexpReplace extends BaseStellarFunction {
+
+@Override
+public Object apply(List list) {
+  if (list.size() != 3) {
+throw new IllegalStateException(
+"REGEXP_REPLACE expects three args: [string, pattern, value]"
++ " where pattern is a regexp pattern");
+  }
+  String str = (String) list.get(0);
+  String stringPattern = (String) list.get(1);
+  String value = (String) list.get(2);
+
--- End diff --

Thanks for a hint, it is fixed now.


---

[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

2018-01-29 Thread 1havran

Github user 1havran commented on a diff in the pull request:

https://github.com/apache/metron/pull/912#discussion_r164570355
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RegExFunctionsTest.java
 ---
@@ -68,4 +68,19 @@ public void testRegExGroupVal() throws Exception {
   Assert.assertTrue("Did not fail on wrong number of 
parameters",false);
 }
   }
+
+  @Test
+  public void testRegExReplace() throws Exception {
+final Map variableMap = new HashMap() 
{{
+  put("numbers", "12345");
+  put("numberPattern", "\\d(\\d)(\\d).*");
+  put("letters", "abcde");
+  put("empty", "");
+}};
+
--- End diff --

Indeed, more tests included.


---

Re: Enrichment and indexing routing mechanism

No, I haven't yet. I have checked the source code and it seems I should be
able to modify source.type in Stellar in the parser config.

Regards,
Ali

On 29 Jan. 2018 23:37, "Otto Fowler"  wrote:

The source.type is set before the stellar transformations, so I think if
you change it in stellar
it should work.

Have you tried and failed?


On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote:

Yes, exactly.

On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
wrote:

> Are you trying to change the source.type to generate multiple sources
from
> a single feed?
>
>
> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Flow is:
>
> Parser (including the parser class, and all transformations, including
> stellar transformations) -> Kafka (enrichments)
>
> Kafka (enrichments) -> Enrichment topology with all it’s Stellary
goodness
> -> Kafka (indexing)
>
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured
based
> on the indexing config named the same as source.type -> wherever the
> indexer tells it to be.
>
> Simon
>
> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> >
> > Thanks, Simon. When will it apply for the enrichment? Is that after
> parser
> > and post-parser Stellar implementation? I am trying to understand If I
> > change it in post-parser Stellar, will it be overwritten at the last
step
> > of Parser topology or not?
> >
> > Cheers,
> > Ali
> >
> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Yes, it is.
> >>
> >> Sent from my iPhone
> >>
> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how the routing mechanism works in Metron currently.
> Can
> >>> somebody please explain how Enrichment Storm topology understands a
> >> single
> >>> event is related to which Metron feed? What about indexing? is that
> based
> >>> on "source.type" field?
> >>>
> >>> Cheers,
> >>> Ali
> >>
> >
> >
> >
> > --
> > A.Nazemian
>



-- 
A.Nazemian

[GitHub] metron pull request #915: METRON-1433: Only emit debugging timing fields in ...

2018-01-29 Thread cestella

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/915

METRON-1433: Only emit debugging timing fields in enrichment when debugging 
is turned on

## Contributor Comments
Right now we always emit performance debugging fields in the split/join 
bolts.  We should only do that when debug logging is turned on for 
`org.apache.metron.enrichment.bolt`

Test:
Spin up full-dev and ensure that `adapter.*.begin.ts` and 
`adapter.*.end.ts` fields do not show up unless you set debug logging for 
`org.apache.metron.enrichment.bolt`


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron turnoff_debugging

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #915


commit ba241c9c9624e8b02a52952d8733414450bc7f15
Author: cstella 
Date:   2018-01-29T20:55:51Z

Only emit logging fields when debug is turned on.




---

[GitHub] metron pull request #914: METRON-1397 Support for JSON Path and complex docu...

2018-01-29 Thread ottobackwards

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/914

METRON-1397  Support for JSON Path and complex documents in JSONMapParser

It would be useful for implementors to have the ability to ingest more 
complex documents without having to stand up a NiFi node/cluster.

This PR adds support for splitting multiple messages from a single message 
by way of using JSON Path statements to select the List element in the document.

Since the parse() interface already returns a List the scope of 
this change is in the parser.

Example:

```json
{
"foo" :
[
{ "name" : "foo1", "value" : "bar", "number" : 1.0 },
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
]
}
 ```

with JSON Path statement "$.foo" will result in two messages emitted from 
the parser

```json
 { "name" : "foo1", "value" : "bar", "number" : 1.0 }
```

```json
{ "name" : "foo2", "value" : "baz", "number" : 2.0 }
```
Support in the configuration for a new configuration parameter: jsonpQuery 
has been added.
An example of the new configuration:

```json
{
  "parserClassName":"org.apache.metron.parsers.json.JSONMapParser",
  "sensorTopic":"jsonMapQuery",
  "parserConfig": {"jsonpQuery":"$.foo"}
}
```

In order to integration test both with and without the query present, a new 
jsonMapQuery parser configuration has been added, with test data.


##Testing
- tests should run
- deployment should work


### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:


- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron jsonp-support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/914.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #914


commit 414611b9ea6eb06b79cba0cc86e63ea062335884
Author: Otto Fowler 
Date:   2018-01-29T16:58:16Z

JSON Path support for JSONMapParser
Tests, Integration Tests, Doc

commit ad03ac4d3e7c98fcf571f3f3906efbb915deb1f9
Author: Otto Fowler 
Date:   2018-01-29T17:13:10Z

checkstyle fixes

commit 7c76c5f5b7507ac55861ed44ad131700c1e6f124
Author: Otto Fowler 
Date:   2018-01-29T19:36:16Z

account for new config in rpm

fix handle empty




---

[GitHub] metron issue #913: METRON-1432 JDK Install Fails on Ubuntu Development Envir...

2018-01-29 Thread mmiklavc

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/913
  
+1 by inspection


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

2018-01-29 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/907
  
For 2, that's the same error @mmiklavc hit on centos.  It's very 
interesting that I'm not hitting it.


---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
For (2), the problem is that the HDFS configuration in Ambari has embedded 
'mustache handlebars' (see `{{major_stack_version}}`) that when returned from 
an API call confuses Ansible.  We need some way to strip that out or ignore it.

![screen shot 2018-01-29 at 2 31 44 
pm](https://user-images.githubusercontent.com/2475409/35530186-5072f3ae-0501-11e8-8bd9-93096da51f02.png)



---

Re: [REQUEST] Add Ian as an Assignee in JIRA

Ian should be all set now.


On January 29, 2018 at 13:28:18, Otto Fowler (ottobackwa...@gmail.com)
wrote:

I would, but I cannot.  I need a current admin to give me rights


On January 29, 2018 at 12:41:59, zeo...@gmail.com (zeo...@gmail.com) wrote:

Ian Abreu

Re: [REQUEST] Add Ian as an Assignee in JIRA

I would, but I cannot.  I need a current admin to give me rights


On January 29, 2018 at 12:41:59, zeo...@gmail.com (zeo...@gmail.com) wrote:

Ian Abreu

[GitHub] metron pull request #913: METRON-1432 JDK Install Fails on Ubuntu Developmen...

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/913

METRON-1432 JDK Install Fails on Ubuntu Development Environment

The Ansible role used to install the JDK does not work correctly on Ubuntu. 
 This fixes the problem and ensures that the JDK can be installed on either 
Ubuntu or CentOS.

## Testing

1. Launch the Ubuntu development environment. 
* Run the Metron Service Check
* Ensure data is visible within the Alerts UI

1. Launch the CentOS development environment.
* Run the Metron Service Check
* Ensure data is visible within the Alerts UI

## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1432

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/913.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #913


commit bbc27bf6337067d51e2d6d7d03bcb19203e35f59
Author: Nick Allen 
Date:   2018-01-29T18:15:32Z

METRON-1432 JDK Install Fails on Ubuntu Development Environment




---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/907
  
Running this up on Ubuntu, I ran into two issues.

(1) JDK Install Failed on Ubuntu - I do not think this was caused by this 
PR.  This is something that I should have hit in #903, but the changes there 
never brought this bug to light.  Fortunately, it is an easy fix and I will 
open it as a separate PR.

(2) After the Ambari deployment completes, it begins to install the 
sensors.  It queries Ambari for information to configure the sensors and in one 
of those queries it hits this problem.  I am not sure exactly what the problem 
is yet.
```
TASK [ambari_gather_facts : Ask Ambari: hdfs_url] 
**
ok: [node1]

TASK [ambari_gather_facts : set_fact] 
**
fatal: [node1]: FAILED! =>{  
   "failed":true,
   "msg":"the field 'args' has an invalid value, which appears to include a 
variable that is undefined. The error was: {u'status': 200, u'content_type': 
u'text/plain', u'set_cookie': 
u'AMBARISESSIONID=5tlv0y7btgc24krm5ugdpdgj;Path=/;HttpOnly', u'expires': u'Thu, 
01 Jan 1970 00:00:00 GMT', u'vary': u'Accept-Encoding, User-Agent', u'user': 
u'admin', u'pragma': u'no-cache', u'x_frame_options': u'DENY', 
u'x_xss_protection': u'1; mode=block', u'url': 
u'http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED',
 u'changed': False, u'x_content_type_options': u'nosniff', u'content': u'{\\n  
\"href\" : 
\"http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED\",\\n
  \"items\" : [\\n{\\n  \"href\" : 
\"http://node1:8080/api/v1/clusters/metron_cluster/configurations?type=core-site=TOPOLOGY_RESOLVED\",\\n
  \"tag\" : \"TOPOLOGY_RESOLVED\",\\n  \"type\" : \"core-site\",\\n 
 \"version\
 " : 2,\\n  \"Config\" : {\\n\"cluster_name\" : 
\"metron_cluster\",\\n\"stack_id\" : \"HDP-2.6\"\\n  },\\n  
\"properties\" : {\\n\"fs.defaultFS\" : \"hdfs://node1:8020\",\\n   
 \"fs.trash.interval\" : \"360\",\\n
\"ha.failover-controller.active-standby-elector.zk.op.retries\" : \"120\",\\n   
 \"hadoop.custom-extensions.root\" : 
\"/hdp/ext/{{major_stack_version}}/hadoop\",\\n
\"hadoop.http.authentication.simple.anonymous.allowed\" : \"true\",\\n
\"hadoop.proxyuser.hbase.groups\" : \"*\",\\n
\"hadoop.proxyuser.hbase.hosts\" : \"*\",\\n
\"hadoop.security.auth_to_local\" : \"DEFAULT\",\\n
\"hadoop.security.authentication\" : \"simple\",\\n
\"hadoop.security.authorization\" : \"false\",\\n
\"hadoop.security.key.provider.path\" : \"\",\\n
\"io.compression.codecs\" : 
\"org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.Snapp
 yCodec\",\\n\"io.file.buffer.size\" : \"131072\",\\n
\"io.serializations\" : 
\"org.apache.hadoop.io.serializer.WritableSerialization\",\\n
\"ipc.client.connect.max.retries\" : \"50\",\\n
\"ipc.client.connection.maxidletime\" : \"3\",\\n
\"ipc.client.idlethreshold\" : \"8000\",\\n\"ipc.server.tcpnodelay\" : 
\"true\",\\n\"mapreduce.jobtracker.webinterface.trusted\" : 
\"false\",\\n\"net.topology.script.file.name\" : 
\"/etc/hadoop/conf/topology_script.py\"\\n  },\\n  
\"properties_attributes\" : {\\n\"final\" : {\\n  
\"fs.defaultFS\" : \"true\"\\n}\\n  }\\n}\\n  ]\\n}', 
u'connection': u'close', u'msg': u'OK (unknown bytes)', u'redirected': False, 
u'cache_control': u'no-store'}: 'major_stack_version' is undefined\n\nThe error 
appears to have been in 
'/Users/nallen/tmp/metron-pr907/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml':
 line 82, column 3, but may\nbe elsewhe
 re in the file depending on the exact syntax problem.\n\nThe offending line 
appears to be:\n\n\n- set_fact:\n  ^ here\n"
}
```




---

[REQUEST] Add Ian as an Assignee in JIRA

2018-01-29 Thread zeo...@gmail.com

Can someone add Ian Abreu as a potential assignee on JIRA?  He has a PR
 open against his
ticket  in the bro
plugin repo.  Thanks,

Jon
-- 

Jon

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-29 Thread merrimanr

Github user merrimanr closed the pull request at:

https://github.com/apache/metron/pull/911


---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

2018-01-29 Thread merrimanr

GitHub user merrimanr reopened a pull request:

https://github.com/apache/metron/pull/911

METRON-1419: Create a SolrDao

## Contributor Comments
This PR is an initial attempt at creating a SolrDao that implements the
IndexDao interface, is functionally equivalent to ElasticsearchDao and passes
all tests in SearchIntegrationTest and UpdateIntegrationTest.

A high level summary of the changes include:

- Upgraded the Solr client version to 6.6.0
- Updated the SolrComponent to work with Solr 6.6 and added a couple
convenience methods similar to ElasticsearchComponent
- Added a new SolrDao implementation with all IndexDao methods implemented
- Refactored the SearchIntegrationTest to work for both Solr and
Elasticsearch and added an Solr implementation (more detail below)
- Created an abstract UpdateIntegrationTest and added a Solr implementation
- Added Solr schemas for test data sets
- Added new tests to SearchIntegrationTest including filtering on fields
with different types, faceting on fields with different types, and faceting on
fields with missing types.
- Broke the IndexDao down in the SolrDao to smaller, easier to understand
classes. The ElasticsearchDao class has become very large so I attempted to
make the SolrDao more readable.

There were a couple areas where Elasticsearch and Solr behave slightly
different. I attempted to accommodated for that through the SolrDao
implementation, by adjusting existing tests, and by splitting out specific
tests:

- Column metadata is different between the 2 search engines so each
implementation has their own tests
- There are cases where the clients will return different types in search
results. I am handling this in SearchIntegrationTest by first converting the
types to strings and then comparing (other ideas?). For example, the ES client
returns an Integer for timestamp while Solr returns a Long.
- There are cases where ES throws an error under certain conditions while
Solr does not (and vice-versa). These were moved to either ES or Solr
SearchIntegrationTest implementations.
- There is no support in Solr for sorting group results so I am sorting
them client-side instead.

At this point the scope is limited to tests passing, meaning
metron-elasticsearch and metron-solr pass all tests. There are other PRs in
progress that are needed before automated testing with full dev can be done. I
am still actively working on manually testing in full dev and adding
documentation but this should get us started.

This PR is intended to be merged into the upgrade-solr feature branch but I
have it set to master temporarily so review is easier. We will need to merge
in master to the feature branch to get rid of the extra commits since this PR
is up to date with master.

I'm expecting a lengthy review and would request multiple reviewers and +1s.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```

- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant

Re: When things change in hdfs, how do we know

I have updated the jira as well


On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com)
wrote:

https://github.com/ottobackwards/hdfs-inotify-zookeeper

Re: When things change in hdfs, how do we know

POC is kind of complete, only thing not done is having the inotify listener
do a tree cache to pick up new configurations.

On January 28, 2018 at 22:39:18, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Btw: https://issues.apache.org/jira/browse/METRON-534

On January 26, 2018 at 15:03:11, Otto Fowler (ottobackwa...@gmail.com)
wrote:

https://github.com/ottobackwards/hdfs-inotify-zookeeper
Has the basics framed out, short of pushing to zookeeper, which I mocked
out at this time.
I’ll add pushing to zk and a cache notification listener to the test soon.

On January 26, 2018 at 08:48:54, Otto Fowler (ottobackwa...@gmail.com)
wrote:

In the future, when the ‘filter paths on the name node side for inotify’
lands in hdfs ( there is a jira from the summer that is not making progress
) we can
just use the paths to register.

On January 26, 2018 at 08:47:11, Otto Fowler (ottobackwa...@gmail.com)
wrote:

In the end, what I’m thinking is this:

We have an ambari service that runs the notification -> zookeeper
it reads the ‘registration area’ from zookeeper to get it’s state and what
to watch
post 777 when parsers are installed and registered it is trivial to have my
installer also register the files to watch

the notifications service also has a notification from zookeeper for new
registrations.

On notify event, the ‘notification node’ has it’s content set to the event
details and time
which the parser would pick up…. causing the reload
???
profit

This would work for the future script parser etc etc.

On January 26, 2018 at 08:30:32, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Interesting, so you have an INotify listener to filter events, and then on
given changes, propagate a notification to zookeeper, which then triggers
the reconfiguration event via the curator client in Metron. I kinda like it
given our existing zookeeper methods.

Simon

On 26 Jan 2018, at 13:27, Otto Fowler  wrote:

https://github.com/ottobackwards/hdfs-inotify-zookeeper

Working on a poc

On January 26, 2018 at 07:41:44, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Should we consider using the Inotify interface to trigger reconfiguration,
in same way we trigger config changes in curator? We also need to fix
caching and lifecycle in the Grok parser to make the zookeeper changes
propagate pattern changes while we’re at it.

Simon

> On 26 Jan 2018, at 03:16, Casey Stella  wrote:
>
> Right now you have to restart the parser topology.
>
> On Thu, Jan 25, 2018 at 10:15 PM, Otto Fowler 
> wrote:
>
>> At the moment, when a grok file or something changes in HDFS, how do we
>> know? Do we have to restart the parser topology to pick it up?
>> Just trying to clarify for myself.
>>
>> ottO
>>

Re: Enrichment and indexing routing mechanism

The source.type is set before the stellar transformations, so I think if
you change it in stellar
it should work.

Have you tried and failed?


On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote:

Yes, exactly.

On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
wrote:

> Are you trying to change the source.type to generate multiple sources
from
> a single feed?
>
>
> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Flow is:
>
> Parser (including the parser class, and all transformations, including
> stellar transformations) -> Kafka (enrichments)
>
> Kafka (enrichments) -> Enrichment topology with all it’s Stellary
goodness
> -> Kafka (indexing)
>
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured
based
> on the indexing config named the same as source.type -> wherever the
> indexer tells it to be.
>
> Simon
>
> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> >
> > Thanks, Simon. When will it apply for the enrichment? Is that after
> parser
> > and post-parser Stellar implementation? I am trying to understand If I
> > change it in post-parser Stellar, will it be overwritten at the last
step
> > of Parser topology or not?
> >
> > Cheers,
> > Ali
> >
> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Yes, it is.
> >>
> >> Sent from my iPhone
> >>
> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how the routing mechanism works in Metron currently.
> Can
> >>> somebody please explain how Enrichment Storm topology understands a
> >> single
> >>> event is related to which Metron feed? What about indexing? is that
> based
> >>> on "source.type" field?
> >>>
> >>> Cheers,
> >>> Ali
> >>
> >
> >
> >
> > --
> > A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

And I am trying to understand if I set a post-parser Stellar transformation
to change the value of "source.type" will it impact enrichment routing or
it will get overwritten by an internal method?

On Mon, Jan 29, 2018 at 11:22 PM, Ali Nazemian 
wrote:

> Yes, exactly.
>
> On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
> wrote:
>
>> Are you trying to change the source.type to generate multiple sources from
>> a single feed?
>>
>>
>> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Flow is:
>>
>> Parser (including the parser class, and all transformations, including
>> stellar transformations) -> Kafka (enrichments)
>>
>> Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness
>> -> Kafka (indexing)
>>
>> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured
>> based
>> on the indexing config named the same as source.type -> wherever the
>> indexer tells it to be.
>>
>> Simon
>>
>> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
>> >
>> > Thanks, Simon. When will it apply for the enrichment? Is that after
>> parser
>> > and post-parser Stellar implementation? I am trying to understand If I
>> > change it in post-parser Stellar, will it be overwritten at the last
>> step
>> > of Parser topology or not?
>> >
>> > Cheers,
>> > Ali
>> >
>> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
>> > si...@simonellistonball.com> wrote:
>> >
>> >> Yes, it is.
>> >>
>> >> Sent from my iPhone
>> >>
>> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
>> >>>
>> >>> Hi All,
>> >>>
>> >>> I was wondering how the routing mechanism works in Metron currently.
>> Can
>> >>> somebody please explain how Enrichment Storm topology understands a
>> >> single
>> >>> event is related to which Metron feed? What about indexing? is that
>> based
>> >>> on "source.type" field?
>> >>>
>> >>> Cheers,
>> >>> Ali
>> >>
>> >
>> >
>> >
>> > --
>> > A.Nazemian
>>
>
>
>
> --
> A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

Yes, exactly.

On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
wrote:

> Are you trying to change the source.type to generate multiple sources from
> a single feed?
>
>
> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Flow is:
>
> Parser (including the parser class, and all transformations, including
> stellar transformations) -> Kafka (enrichments)
>
> Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness
> -> Kafka (indexing)
>
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based
> on the indexing config named the same as source.type -> wherever the
> indexer tells it to be.
>
> Simon
>
> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> >
> > Thanks, Simon. When will it apply for the enrichment? Is that after
> parser
> > and post-parser Stellar implementation? I am trying to understand If I
> > change it in post-parser Stellar, will it be overwritten at the last step
> > of Parser topology or not?
> >
> > Cheers,
> > Ali
> >
> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Yes, it is.
> >>
> >> Sent from my iPhone
> >>
> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how the routing mechanism works in Metron currently.
> Can
> >>> somebody please explain how Enrichment Storm topology understands a
> >> single
> >>> event is related to which Metron feed? What about indexing? is that
> based
> >>> on "source.type" field?
> >>>
> >>> Cheers,
> >>> Ali
> >>
> >
> >
> >
> > --
> > A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

Are you trying to change the source.type to generate multiple sources from
a single feed?


On January 29, 2018 at 07:08:57, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Flow is:

Parser (including the parser class, and all transformations, including
stellar transformations) -> Kafka (enrichments)

Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness
-> Kafka (indexing)

Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based
on the indexing config named the same as source.type -> wherever the
indexer tells it to be.

Simon

> On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
>
> Thanks, Simon. When will it apply for the enrichment? Is that after
parser
> and post-parser Stellar implementation? I am trying to understand If I
> change it in post-parser Stellar, will it be overwritten at the last step
> of Parser topology or not?
>
> Cheers,
> Ali
>
> On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Yes, it is.
>>
>> Sent from my iPhone
>>
>>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
>>>
>>> Hi All,
>>>
>>> I was wondering how the routing mechanism works in Metron currently.
Can
>>> somebody please explain how Enrichment Storm topology understands a
>> single
>>> event is related to which Metron feed? What about indexing? is that
based
>>> on "source.type" field?
>>>
>>> Cheers,
>>> Ali
>>
>
>
>
> --
> A.Nazemian

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Simon Elliston Ball

Flow is:

Parser (including the parser class, and all transformations, including stellar 
transformations) -> Kafka (enrichments) 

Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness -> 
Kafka (indexing) 

Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based on 
the indexing config named the same as source.type -> wherever the indexer tells 
it to be.

Simon

> On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> 
> Thanks, Simon. When will it apply for the enrichment? Is that after parser
> and post-parser Stellar implementation? I am trying to understand If I
> change it in post-parser Stellar, will it be overwritten at the last step
> of Parser topology or not?
> 
> Cheers,
> Ali
> 
> On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> 
>> Yes, it is.
>> 
>> Sent from my iPhone
>> 
>>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
>>> 
>>> Hi All,
>>> 
>>> I was wondering how the routing mechanism works in Metron currently. Can
>>> somebody please explain how Enrichment Storm topology understands a
>> single
>>> event is related to which Metron feed? What about indexing? is that based
>>> on "source.type" field?
>>> 
>>> Cheers,
>>> Ali
>> 
> 
> 
> 
> -- 
> A.Nazemian

Re: Enrichment and indexing routing mechanism

Thanks, Simon. When will it apply for the enrichment? Is that after parser
and post-parser Stellar implementation? I am trying to understand If I
change it in post-parser Stellar, will it be overwritten at the last step
of Parser topology or not?

Cheers,
Ali

On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Yes, it is.
>
> Sent from my iPhone
>
> > On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >
> > Hi All,
> >
> > I was wondering how the routing mechanism works in Metron currently. Can
> > somebody please explain how Enrichment Storm topology understands a
> single
> > event is related to which Metron feed? What about indexing? is that based
> > on "source.type" field?
> >
> > Cheers,
> > Ali
>

-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Simon Elliston Ball

Yes, it is.

Sent from my iPhone

> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> 
> Hi All,
> 
> I was wondering how the routing mechanism works in Metron currently. Can
> somebody please explain how Enrichment Storm topology understands a single
> event is related to which Metron feed? What about indexing? is that based
> on "source.type" field?
> 
> Cheers,
> Ali

Enrichment and indexing routing mechanism