RE: join 2 topic streams --> to another topic

[adrien ruffie]

You're right Matthias, I will take your suggestion of KTable-KTable join 

I suppose so that meant I don't need windowing ?


-Adrien


De : Matthias J. Sax <matth...@confluent.io>
Envoyé : lundi 9 avril 2018 20:15:17
À : users@kafka.apache.org
Objet : Re: join 2 topic streams --> to another topic

From what you describe I infer (but it might be a wild guess), that you
are actually trying to do KTable-KTable join?

It's all about the semantics of your input data... Note, that a Kafka
topic does not have semantics per-se; you apply semantics when you read
a topic either as stream or table though.

-Matthias

On 4/9/18 12:16 AM, adrien ruffie wrote:
> Hello Matthias,
>
> thank for your response. I will try to read the blog post today.
>
>
> For the keys, not really, In fact, keys are always the same "symbol" of a 
> STOCK example "YHOO", "INTC" and they never change.
>
> Do you with I need to use another key ? Or add a timestamp to the key ?
>
> For single stock join with multiple dividends, I didn't think about it before 
> ... is it possible ?
>
>
> For join depending of timestamps why not, is it possible with windowing ?
>
>
> Thank Matthias
>
> Adrien
>
> 
> De : Matthias J. Sax <matth...@confluent.io>
> Envoyé : dimanche 8 avril 2018 23:04:24
> À : users@kafka.apache.org
> Objet : Re: join 2 topic streams --> to another topic
>
> Check out this blog post that explain how the different joins work:
> https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
>
> It's hard to give a general answer -- it depends on the context of your
> application. Are keys unique? Do you want to get exactly one result or
> should a single stock join with multiple dividends? Do you want Stock
> and Dividend join depending the their timestamps?
>
>
> -Matthias
>
> On 4/8/18 1:34 PM, adrien ruffie wrote:
>> Hello all,
>>
>> I have 2 topics streamed by KStream<String, Stock> and one KStream<String, 
>> Dividend>
>>
>> I want to merge both object's informations (Stock & Dividend) and send to  
>> another topic
>>
>> with for example <String, DividendStockJoin>
>>
>>
>> The  key of 2 two topic is the same. I need to use, leftJoin, merge, 
>> KTable, ...
>>
>> what is the best solution ? What do you think ?
>>
>>
>> Moreover, how can I make sure the timestamp are pretty much the same? (for 
>> merging same key at the same time produced) Example:
>>
>> KStream<String, Stock> --> key: "YHOO",  and one timestamp of record 
>> 2018-04-08 19:56:30
>> KStream<String, Dividend> --> key: "YHOO",  and one timestamp of record 
>> 2018-04-08 19:55:27
>>
>>
>> I need a Windowing ?
>>
>>
>> Great thank & best regards,
>>
>>
>> Adrien
>>
>
>

RE: join 2 topic streams --> to another topic

[adrien ruffie]

Hello Matthias,

thank for your response. I will try to read the blog post today.


For the keys, not really, In fact, keys are always the same "symbol" of a STOCK 
example "YHOO", "INTC" and they never change.

Do you with I need to use another key ? Or add a timestamp to the key ?

For single stock join with multiple dividends, I didn't think about it before 
... is it possible ?


For join depending of timestamps why not, is it possible with windowing ?


Thank Matthias

Adrien


De : Matthias J. Sax <matth...@confluent.io>
Envoyé : dimanche 8 avril 2018 23:04:24
À : users@kafka.apache.org
Objet : Re: join 2 topic streams --> to another topic

Check out this blog post that explain how the different joins work:
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/

It's hard to give a general answer -- it depends on the context of your
application. Are keys unique? Do you want to get exactly one result or
should a single stock join with multiple dividends? Do you want Stock
and Dividend join depending the their timestamps?


-Matthias

On 4/8/18 1:34 PM, adrien ruffie wrote:
> Hello all,
>
> I have 2 topics streamed by KStream<String, Stock> and one KStream<String, 
> Dividend>
>
> I want to merge both object's informations (Stock & Dividend) and send to  
> another topic
>
> with for example <String, DividendStockJoin>
>
>
> The  key of 2 two topic is the same. I need to use, leftJoin, merge, 
> KTable, ...
>
> what is the best solution ? What do you think ?
>
>
> Moreover, how can I make sure the timestamp are pretty much the same? (for 
> merging same key at the same time produced) Example:
>
> KStream<String, Stock> --> key: "YHOO",  and one timestamp of record 
> 2018-04-08 19:56:30
> KStream<String, Dividend> --> key: "YHOO",  and one timestamp of record 
> 2018-04-08 19:55:27
>
>
> I need a Windowing ?
>
>
> Great thank & best regards,
>
>
> Adrien
>

Re: join 2 topic streams --> to another topic

[Ted Yu]

The blog is an interesting reading material.

There was a minor typo:

bq. an event arrives for either the left of right input stream

'of' above should be 'or'.

Cheers

On Sun, Apr 8, 2018 at 2:04 PM, Matthias J. Sax 
wrote:

> Check out this blog post that explain how the different joins work:
> https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/
>
> It's hard to give a general answer -- it depends on the context of your
> application. Are keys unique? Do you want to get exactly one result or
> should a single stock join with multiple dividends? Do you want Stock
> and Dividend join depending the their timestamps?
>
>
> -Matthias
>
> On 4/8/18 1:34 PM, adrien ruffie wrote:
> > Hello all,
> >
> > I have 2 topics streamed by KStream and one
> KStream
> >
> > I want to merge both object's informations (Stock & Dividend) and send
> to  another topic
> >
> > with for example 
> >
> >
> > The  key of 2 two topic is the same. I need to use, leftJoin,
> merge, KTable, ...
> >
> > what is the best solution ? What do you think ?
> >
> >
> > Moreover, how can I make sure the timestamp are pretty much the same?
> (for merging same key at the same time produced) Example:
> >
> > KStream --> key: "YHOO",  and one timestamp of record
> 2018-04-08 19:56:30
> > KStream --> key: "YHOO",  and one timestamp of record
> 2018-04-08 19:55:27
> >
> >
> > I need a Windowing ?
> >
> >
> > Great thank & best regards,
> >
> >
> > Adrien
> >
>
>

Re: join 2 topic streams --> to another topic

[Matthias J. Sax]

Check out this blog post that explain how the different joins work:
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/

It's hard to give a general answer -- it depends on the context of your
application. Are keys unique? Do you want to get exactly one result or
should a single stock join with multiple dividends? Do you want Stock
and Dividend join depending the their timestamps?


-Matthias

On 4/8/18 1:34 PM, adrien ruffie wrote:
> Hello all,
> 
> I have 2 topics streamed by KStream and one KStream Dividend>
> 
> I want to merge both object's informations (Stock & Dividend) and send to  
> another topic
> 
> with for example 
> 
> 
> The  key of 2 two topic is the same. I need to use, leftJoin, merge, 
> KTable, ...
> 
> what is the best solution ? What do you think ?
> 
> 
> Moreover, how can I make sure the timestamp are pretty much the same? (for 
> merging same key at the same time produced) Example:
> 
> KStream --> key: "YHOO",  and one timestamp of record 
> 2018-04-08 19:56:30
> KStream --> key: "YHOO",  and one timestamp of record 
> 2018-04-08 19:55:27
> 
> 
> I need a Windowing ?
> 
> 
> Great thank & best regards,
> 
> 
> Adrien
> 



signature.asc
Description: OpenPGP digital signature