Ankit that this problem could actually be solved quite
>>>>> elegantly with Flink's state. If you can ingest the product/account
>>>>> information changes as a stream, you can keep the latest version of it in
>>>>> Flink state by using a co-map function [1,
incoming event.
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/
>>>> [2]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streami
t;
>>> On Tue, Jul 24, 2018 at 2:15 PM Harshvardhan Agrawal <
>>> harshvardhan.ag...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> Thanks for your responses.
>>>
>>> There is no fixed interval for the data being updated. It’s more like
>
mandates that change
>> will trigger the reference data to change.
>>
>> It’s not just the enrichment we are doing here. Once we have enriched the
>> data we will be performing a bunch of aggregations using the enriched data.
>>
>> Which approach wo
o, just based on this feature, flink doesn’t seem to add a lot of value
> on top of Kafka. As Jorn said below, you can very well store all the events
> in an external store and then periodically run a cron to enrich later since
> your processing doesn’t seem to require absolute real time.
Ankit
From: Jörn Franke
Date: Monday, July 23, 2018 at 10:10 PM
To: Harshvardhan Agrawal
Cc:
Subject: Re: Implement Joins with Lookup Data
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not scale.
Depending
ell store all the events
> in an external store and then periodically run a cron to enrich later since
> your processing doesn’t seem to require absolute real time.
>
>
>
> Thanks
>
> Ankit
>
>
>
> *From: *Jörn Franke
> *Date: *Monday, July 23, 2018 at 10:10 PM
:
Subject: Re: Implement Joins with Lookup Data
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not scale.
Depending on when you need to do the enrichment you could also first store the
data and enrich it later as part
n top of Kafka. As Jorn said below, you can very well store all the events
> in an external store and then periodically run a cron to enrich later since
> your processing doesn’t seem to require absolute real time.
>
>
>
> Thanks
>
> Ankit
>
>
>
> *From: *Jörn Franke
> *Da
.
Thanks
Ankit
From: Jörn Franke
Date: Monday, July 23, 2018 at 10:10 PM
To: Harshvardhan Agrawal
Cc:
Subject: Re: Implement Joins with Lookup Data
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not scale
the product db updated? Based on that you can store
>>>>> product metadata as state in Flink, maybe setup the state on cluster
>>>>> startup and then update daily etc.
>>>>>
>>>>>
>>>>>
>>>>> Also, just based on this featur
;> Also, just based on this feature, flink doesn’t seem to add a lot of
>>>> value on top of Kafka. As Jorn said below, you can very well store all the
>>>> events in an external store and then periodically run a cron to enrich
>>>> later since your processing doesn’t
hen periodically run a cron to enrich
>>> later since your processing doesn’t seem to require absolute real time.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Ankit
>>>
>>>
>>>
>>> *From: *Jörn Franke
>>>
;>
>> Ankit
>>
>>
>>
>> *From: *Jörn Franke
>> *Date: *Monday, July 23, 2018 at 10:10 PM
>> *To: *Harshvardhan Agrawal
>> *Cc: *
>> *Subject: *Re: Implement Joins with Lookup Data
>>
>>
>>
>> For the first one (looku
cron to enrich later since
> your processing doesn’t seem to require absolute real time.
>
>
>
> Thanks
>
> Ankit
>
>
>
> *From: *Jörn Franke
> *Date: *Monday, July 23, 2018 at 10:10 PM
> *To: *Harshvardhan Agrawal
> *Cc: *
> *Subject: *Re: Implement Jo
well store all the events in an
external store and then periodically run a cron to enrich later since your
processing doesn’t seem to require absolute real time.
Thanks
Ankit
From: Jörn Franke
Date: Monday, July 23, 2018 at 10:10 PM
To: Harshvardhan Agrawal
Cc:
Subject: Re: Implement Joins
For the first one (lookup of single entries) you could use a NoSQL db (eg key
value store) - a relational database will not scale.
Depending on when you need to do the enrichment you could also first store the
data and enrich it later as part of a batch process.
> On 24. Jul 2018, at 05:25,
17 matches
Mail list logo
