Re: Enrichment and indexing routing mechanism

[Ali Nazemian]

No, I haven't yet. I have checked the source code and it seems I should be
able to modify source.type in Stellar in the parser config.

Regards,
Ali

On 29 Jan. 2018 23:37, "Otto Fowler"  wrote:

The source.type is set before the stellar transformations, so I think if
you change it in stellar
it should work.

Have you tried and failed?


On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote:

Yes, exactly.

On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
wrote:

> Are you trying to change the source.type to generate multiple sources
from
> a single feed?
>
>
> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Flow is:
>
> Parser (including the parser class, and all transformations, including
> stellar transformations) -> Kafka (enrichments)
>
> Kafka (enrichments) -> Enrichment topology with all it’s Stellary
goodness
> -> Kafka (indexing)
>
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured
based
> on the indexing config named the same as source.type -> wherever the
> indexer tells it to be.
>
> Simon
>
> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> >
> > Thanks, Simon. When will it apply for the enrichment? Is that after
> parser
> > and post-parser Stellar implementation? I am trying to understand If I
> > change it in post-parser Stellar, will it be overwritten at the last
step
> > of Parser topology or not?
> >
> > Cheers,
> > Ali
> >
> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Yes, it is.
> >>
> >> Sent from my iPhone
> >>
> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how the routing mechanism works in Metron currently.
> Can
> >>> somebody please explain how Enrichment Storm topology understands a
> >> single
> >>> event is related to which Metron feed? What about indexing? is that
> based
> >>> on "source.type" field?
> >>>
> >>> Cheers,
> >>> Ali
> >>
> >
> >
> >
> > --
> > A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

[Otto Fowler]

The source.type is set before the stellar transformations, so I think if
you change it in stellar
it should work.

Have you tried and failed?


On January 29, 2018 at 07:22:23, Ali Nazemian (alinazem...@gmail.com) wrote:

Yes, exactly.

On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
wrote:

> Are you trying to change the source.type to generate multiple sources
from
> a single feed?
>
>
> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Flow is:
>
> Parser (including the parser class, and all transformations, including
> stellar transformations) -> Kafka (enrichments)
>
> Kafka (enrichments) -> Enrichment topology with all it’s Stellary
goodness
> -> Kafka (indexing)
>
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured
based
> on the indexing config named the same as source.type -> wherever the
> indexer tells it to be.
>
> Simon
>
> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> >
> > Thanks, Simon. When will it apply for the enrichment? Is that after
> parser
> > and post-parser Stellar implementation? I am trying to understand If I
> > change it in post-parser Stellar, will it be overwritten at the last
step
> > of Parser topology or not?
> >
> > Cheers,
> > Ali
> >
> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Yes, it is.
> >>
> >> Sent from my iPhone
> >>
> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how the routing mechanism works in Metron currently.
> Can
> >>> somebody please explain how Enrichment Storm topology understands a
> >> single
> >>> event is related to which Metron feed? What about indexing? is that
> based
> >>> on "source.type" field?
> >>>
> >>> Cheers,
> >>> Ali
> >>
> >
> >
> >
> > --
> > A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

[Ali Nazemian]

And I am trying to understand if I set a post-parser Stellar transformation
to change the value of "source.type" will it impact enrichment routing or
it will get overwritten by an internal method?

On Mon, Jan 29, 2018 at 11:22 PM, Ali Nazemian 
wrote:

> Yes, exactly.
>
> On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
> wrote:
>
>> Are you trying to change the source.type to generate multiple sources from
>> a single feed?
>>
>>
>> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
>> si...@simonellistonball.com) wrote:
>>
>> Flow is:
>>
>> Parser (including the parser class, and all transformations, including
>> stellar transformations) -> Kafka (enrichments)
>>
>> Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness
>> -> Kafka (indexing)
>>
>> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured
>> based
>> on the indexing config named the same as source.type -> wherever the
>> indexer tells it to be.
>>
>> Simon
>>
>> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
>> >
>> > Thanks, Simon. When will it apply for the enrichment? Is that after
>> parser
>> > and post-parser Stellar implementation? I am trying to understand If I
>> > change it in post-parser Stellar, will it be overwritten at the last
>> step
>> > of Parser topology or not?
>> >
>> > Cheers,
>> > Ali
>> >
>> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
>> > si...@simonellistonball.com> wrote:
>> >
>> >> Yes, it is.
>> >>
>> >> Sent from my iPhone
>> >>
>> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
>> >>>
>> >>> Hi All,
>> >>>
>> >>> I was wondering how the routing mechanism works in Metron currently.
>> Can
>> >>> somebody please explain how Enrichment Storm topology understands a
>> >> single
>> >>> event is related to which Metron feed? What about indexing? is that
>> based
>> >>> on "source.type" field?
>> >>>
>> >>> Cheers,
>> >>> Ali
>> >>
>> >
>> >
>> >
>> > --
>> > A.Nazemian
>>
>
>
>
> --
> A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

[Ali Nazemian]

Yes, exactly.

On Mon, Jan 29, 2018 at 11:15 PM, Otto Fowler 
wrote:

> Are you trying to change the source.type to generate multiple sources from
> a single feed?
>
>
> On January 29, 2018 at 07:08:57, Simon Elliston Ball (
> si...@simonellistonball.com) wrote:
>
> Flow is:
>
> Parser (including the parser class, and all transformations, including
> stellar transformations) -> Kafka (enrichments)
>
> Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness
> -> Kafka (indexing)
>
> Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based
> on the indexing config named the same as source.type -> wherever the
> indexer tells it to be.
>
> Simon
>
> > On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> >
> > Thanks, Simon. When will it apply for the enrichment? Is that after
> parser
> > and post-parser Stellar implementation? I am trying to understand If I
> > change it in post-parser Stellar, will it be overwritten at the last step
> > of Parser topology or not?
> >
> > Cheers,
> > Ali
> >
> > On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> > si...@simonellistonball.com> wrote:
> >
> >> Yes, it is.
> >>
> >> Sent from my iPhone
> >>
> >>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >>>
> >>> Hi All,
> >>>
> >>> I was wondering how the routing mechanism works in Metron currently.
> Can
> >>> somebody please explain how Enrichment Storm topology understands a
> >> single
> >>> event is related to which Metron feed? What about indexing? is that
> based
> >>> on "source.type" field?
> >>>
> >>> Cheers,
> >>> Ali
> >>
> >
> >
> >
> > --
> > A.Nazemian
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

[Otto Fowler]

Are you trying to change the source.type to generate multiple sources from
a single feed?


On January 29, 2018 at 07:08:57, Simon Elliston Ball (
si...@simonellistonball.com) wrote:

Flow is:

Parser (including the parser class, and all transformations, including
stellar transformations) -> Kafka (enrichments)

Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness
-> Kafka (indexing)

Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based
on the indexing config named the same as source.type -> wherever the
indexer tells it to be.

Simon

> On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
>
> Thanks, Simon. When will it apply for the enrichment? Is that after
parser
> and post-parser Stellar implementation? I am trying to understand If I
> change it in post-parser Stellar, will it be overwritten at the last step
> of Parser topology or not?
>
> Cheers,
> Ali
>
> On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
>
>> Yes, it is.
>>
>> Sent from my iPhone
>>
>>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
>>>
>>> Hi All,
>>>
>>> I was wondering how the routing mechanism works in Metron currently.
Can
>>> somebody please explain how Enrichment Storm topology understands a
>> single
>>> event is related to which Metron feed? What about indexing? is that
based
>>> on "source.type" field?
>>>
>>> Cheers,
>>> Ali
>>
>
>
>
> --
> A.Nazemian

Re: Enrichment and indexing routing mechanism

[Simon Elliston Ball]

Flow is:

Parser (including the parser class, and all transformations, including stellar 
transformations) -> Kafka (enrichments) 

Kafka (enrichments) -> Enrichment topology with all it’s Stellary goodness -> 
Kafka (indexing) 

Kafka (indexing) -> Indexing topologies (ES / Solr / HDFS) configured based on 
the indexing config named the same as source.type -> wherever the indexer tells 
it to be.

Simon

> On 29 Jan 2018, at 11:53, Ali Nazemian  wrote:
> 
> Thanks, Simon. When will it apply for the enrichment? Is that after parser
> and post-parser Stellar implementation? I am trying to understand If I
> change it in post-parser Stellar, will it be overwritten at the last step
> of Parser topology or not?
> 
> Cheers,
> Ali
> 
> On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
> si...@simonellistonball.com> wrote:
> 
>> Yes, it is.
>> 
>> Sent from my iPhone
>> 
>>> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
>>> 
>>> Hi All,
>>> 
>>> I was wondering how the routing mechanism works in Metron currently. Can
>>> somebody please explain how Enrichment Storm topology understands a
>> single
>>> event is related to which Metron feed? What about indexing? is that based
>>> on "source.type" field?
>>> 
>>> Cheers,
>>> Ali
>> 
> 
> 
> 
> -- 
> A.Nazemian

Re: Enrichment and indexing routing mechanism

[Ali Nazemian]

Thanks, Simon. When will it apply for the enrichment? Is that after parser
and post-parser Stellar implementation? I am trying to understand If I
change it in post-parser Stellar, will it be overwritten at the last step
of Parser topology or not?

Cheers,
Ali

On Mon, Jan 29, 2018 at 8:55 PM, Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> Yes, it is.
>
> Sent from my iPhone
>
> > On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> >
> > Hi All,
> >
> > I was wondering how the routing mechanism works in Metron currently. Can
> > somebody please explain how Enrichment Storm topology understands a
> single
> > event is related to which Metron feed? What about indexing? is that based
> > on "source.type" field?
> >
> > Cheers,
> > Ali
>



-- 
A.Nazemian

Re: Enrichment and indexing routing mechanism

[Simon Elliston Ball]

Yes, it is.

Sent from my iPhone

> On 29 Jan 2018, at 09:33, Ali Nazemian  wrote:
> 
> Hi All,
> 
> I was wondering how the routing mechanism works in Metron currently. Can
> somebody please explain how Enrichment Storm topology understands a single
> event is related to which Metron feed? What about indexing? is that based
> on "source.type" field?
> 
> Cheers,
> Ali

Enrichment and indexing routing mechanism

[Ali Nazemian]

Hi All,

I was wondering how the routing mechanism works in Metron currently. Can
somebody please explain how Enrichment Storm topology understands a single
event is related to which Metron feed? What about indexing? is that based
on "source.type" field?

Cheers,
Ali