Re: Enrichment and indexing routing mechanism

No, I haven't yet. I have checked the source code and it seems I should be able to modify source.type in Stellar in the parser config. Regards, Ali On 29 Jan. 2018 23:37, "Otto Fowler" wrote: The source.type is set before the stellar transformations, so I think if you

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/915 https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java#L89 ---

[GitHub] metron issue #912: METRON-1431 Adding REGEX_REPLACE(input, regex, value) to ...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/912 please remember to take care of your jira ---

[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/912 ---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/907 For (2), the problem is that the HDFS configuration in Ambari has embedded 'mustache handlebars' (see `{{major_stack_version}}`) that when returned from an API call confuses Ansible. We need

[GitHub] metron pull request #915: METRON-1433: Only emit debugging timing fields in ...

GitHub user cestella opened a pull request: https://github.com/apache/metron/pull/915 METRON-1433: Only emit debugging timing fields in enrichment when debugging is turned on ## Contributor Comments Right now we always emit performance debugging fields in the split/join bolts.

[GitHub] metron issue #912: METRON-1431 Adding REGEX_REPLACE(input, regex, value) to ...

Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/912 +1 by inspection. Thank you for the contribution ---

[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

Github user 1havran commented on a diff in the pull request: https://github.com/apache/metron/pull/912#discussion_r164570247 --- Diff: metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RegExFunctions.java --- @@ -100,4 +100,39 @@ public Object

[GitHub] metron pull request #912: METRON-1431 Adding REGEX_REPLACE(input, regex, val...

Github user 1havran commented on a diff in the pull request: https://github.com/apache/metron/pull/912#discussion_r164570355 --- Diff: metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RegExFunctionsTest.java --- @@ -68,4 +68,19 @@ public void

[GitHub] metron pull request #914: METRON-1397 Support for JSON Path and complex docu...

GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/914 METRON-1397 Support for JSON Path and complex documents in JSONMapParser It would be useful for implementors to have the ability to ingest more complex documents without having to stand up a

[GitHub] metron issue #913: METRON-1432 JDK Install Fails on Ubuntu Development Envir...

Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/913 +1 by inspection ---

Re: [REQUEST] Add Ian as an Assignee in JIRA

Ian should be all set now. On January 29, 2018 at 13:28:18, Otto Fowler (ottobackwa...@gmail.com) wrote: I would, but I cannot. I need a current admin to give me rights On January 29, 2018 at 12:41:59, zeo...@gmail.com (zeo...@gmail.com) wrote: Ian Abreu

Re: [REQUEST] Add Ian as an Assignee in JIRA

I would, but I cannot. I need a current admin to give me rights On January 29, 2018 at 12:41:59, zeo...@gmail.com (zeo...@gmail.com) wrote: Ian Abreu

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user cestella commented on the issue: https://github.com/apache/metron/pull/907 For 2, that's the same error @mmiklavc hit on centos. It's very interesting that I'm not hitting it. ---

[REQUEST] Add Ian as an Assignee in JIRA

Can someone add Ian Abreu as a potential assignee on JIRA? He has a PR open against his ticket in the bro plugin repo. Thanks, Jon -- Jon

Enrichment and indexing routing mechanism

Hi All, I was wondering how the routing mechanism works in Metron currently. Can somebody please explain how Enrichment Storm topology understands a single event is related to which Metron feed? What about indexing? is that based on "source.type" field? Cheers, Ali

Re: Enrichment and indexing routing mechanism

Yes, it is. Sent from my iPhone > On 29 Jan 2018, at 09:33, Ali Nazemian wrote: > > Hi All, > > I was wondering how the routing mechanism works in Metron currently. Can > somebody please explain how Enrichment Storm topology understands a single > event is related to

Re: Enrichment and indexing routing mechanism

And I am trying to understand if I set a post-parser Stellar transformation to change the value of "source.type" will it impact enrichment routing or it will get overwritten by an internal method? On Mon, Jan 29, 2018 at 11:22 PM, Ali Nazemian wrote: > Yes, exactly. > >

Re: Enrichment and indexing routing mechanism

The source.type is set before the stellar transformations, so I think if you change it in stellar it should work. Have you tried and failed? On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote: Yes, exactly. On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler

Re: When things change in hdfs, how do we know

POC is kind of complete, only thing not done is having the inotify listener do a tree cache to pick up new configurations. On January 28, 2018 at 22:39:18, Otto Fowler (ottobackwa...@gmail.com) wrote: Btw: https://issues.apache.org/jira/browse/METRON-534 On January 26, 2018 at 15:03:11,

Re: Enrichment and indexing routing mechanism

Thanks, Simon. When will it apply for the enrichment? Is that after parser and post-parser Stellar implementation? I am trying to understand If I change it in post-parser Stellar, will it be overwritten at the last step of Parser topology or not? Cheers, Ali On Mon, Jan 29, 2018 at 8:55 PM,

Re: Enrichment and indexing routing mechanism

Flow is: Parser (including the parser class, and all transformations, including stellar transformations) -> Kafka (enrichments) Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness -> Kafka (indexing) Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS)

Re: Enrichment and indexing routing mechanism

Yes, exactly. On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler wrote: > Are you trying to change the source.type to generate multiple sources from > a single feed? > > > On January 29, 2018 at 07:08:57, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Flow

Re: Enrichment and indexing routing mechanism

Are you trying to change the source.type to generate multiple sources from a single feed? On January 29, 2018 at 07:08:57, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Flow is: Parser (including the parser class, and all transformations, including stellar transformations) -> Kafka

Re: When things change in hdfs, how do we know

I have updated the jira as well On January 29, 2018 at 08:22:34, Otto Fowler (ottobackwa...@gmail.com) wrote: https://github.com/ottobackwards/hdfs-inotify-zookeeper

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/911 ---

[GitHub] metron pull request #911: METRON-1419: Create a SolrDao

GitHub user merrimanr reopened a pull request: https://github.com/apache/metron/pull/911 METRON-1419: Create a SolrDao ## Contributor Comments This PR is an initial attempt at creating a SolrDao that implements the IndexDao interface, is functionally equivalent to

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/907 FYI @cestella I submitted a fix against your PR branch that should address the issue with the embedded handlebars in the Ambari response. ---

[GitHub] metron issue #907: METRON-1427: Add support for storm 1.1 and hdp 2.6

Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/907 Running this up on Ubuntu, I ran into two issues. (1) JDK Install Failed on Ubuntu - I do not think this was caused by this PR. This is something that I should have hit in #903, but the

[GitHub] metron pull request #913: METRON-1432 JDK Install Fails on Ubuntu Developmen...

GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/913 METRON-1432 JDK Install Fails on Ubuntu Development Environment The Ansible role used to install the JDK does not work correctly on Ubuntu. This fixes the problem and ensures that the JDK can