Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
No, I haven't yet. I have checked the source code and it seems I should be able to modify source.type in Stellar in the parser config. Regards, Ali On 29 Jan. 2018 23:37, "Otto Fowler" wrote: The source.type is set before the stellar transformations, so I think if you

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Otto Fowler
The source.type is set before the stellar transformations, so I think if you change it in stellar it should work. Have you tried and failed? On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote: Yes, exactly. On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
And I am trying to understand if I set a post-parser Stellar transformation to change the value of "source.type" will it impact enrichment routing or it will get overwritten by an internal method? On Mon, Jan 29, 2018 at 11:22 PM, Ali Nazemian wrote: > Yes, exactly. > >

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
Yes, exactly. On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler wrote: > Are you trying to change the source.type to generate multiple sources from > a single feed? > > > On January 29, 2018 at 07:08:57, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Flow

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Otto Fowler
Are you trying to change the source.type to generate multiple sources from a single feed? On January 29, 2018 at 07:08:57, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Flow is: Parser (including the parser class, and all transformations, including stellar transformations) -> Kafka

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Simon Elliston Ball
Flow is: Parser (including the parser class, and all transformations, including stellar transformations) -> Kafka (enrichments) Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness -> Kafka (indexing) Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS)

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
Thanks, Simon. When will it apply for the enrichment? Is that after parser and post-parser Stellar implementation? I am trying to understand If I change it in post-parser Stellar, will it be overwritten at the last step of Parser topology or not? Cheers, Ali On Mon, Jan 29, 2018 at 8:55 PM,

Re: Enrichment and indexing routing mechanism

2018-01-29 Thread Simon Elliston Ball
Yes, it is. Sent from my iPhone > On 29 Jan 2018, at 09:33, Ali Nazemian wrote: > > Hi All, > > I was wondering how the routing mechanism works in Metron currently. Can > somebody please explain how Enrichment Storm topology understands a single > event is related to

Enrichment and indexing routing mechanism

2018-01-29 Thread Ali Nazemian
Hi All, I was wondering how the routing mechanism works in Metron currently. Can somebody please explain how Enrichment Storm topology understands a single event is related to which Metron feed? What about indexing? is that based on "source.type" field? Cheers, Ali