Re: Is it possible to connect multiple streams

2022-09-21 Thread Deepak kumar Gunjetti
Thank you Gentlemen,
@yaroslav - I was very happy to see code on ppt. I checked on union and
join operators.
@shamoon - Thank you for the link I went through.
@Schwalbe - [Keyed]MultipleInputTransformation looks promising for me, as I
have more than 3 streams of different types and key type being the same. I
will check this further.

On Wed, Sep 21, 2022 at 4:03 PM Schwalbe Matthias <
matthias.schwa...@viseca.ch> wrote:

> Hi Deepak,
>
>
>
> Coming back to your original question, you’ve got a number of option (some
> of them already mentioned:
>
>- You can connect/join 2 streams of *different* types at a time by
>means of s1.connect(s2).
>   - (your example does not work directly as written (3 streams))
>- You can connect many streams of *same* type by means of s1.union(s2,
>s3, …).
>- The third option is new and not yet documented (marked as
>@experimental):
>   - Connect/join many streams of different types by means of a
>   [Keyed]MultipleInputTransformation
>  - If keyed: input streams need to either
> - be keyed on the same key type, or
> - be non-keyed (most likely) broadcast streams
>  - the API is still a little elaborate, a good starting point
>   could be this test case: [1]
>
>
>
> If of any interest, feel free to ask for clarifications …
>
>
>
>
>
> Thias
>
> PS: see also my next email in a couple of minutes …
>
>
>
>
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorBatchExecutionTest.java#L441
>
>
>
> *From:* Shammon FY 
> *Sent:* Wednesday, September 21, 2022 6:29 AM
> *To:* user@flink.apache.org
> *Cc:* Deepak kumar Gunjetti ; Yaroslav Tkachenko <
> yaros...@goldsky.com>
> *Subject:* Re: Is it possible to connect multiple streams
>
>
>
> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>
>
>
> Hi
>
>
>
> Thanks @yaroslav .
>
> And @deepakgd79 here is the document for datastream:
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-transformations
> You can find examples for union, connect, join and other transformations
>
>
>
>
>
> On Wed, Sep 21, 2022 at 11:55 AM Yaroslav Tkachenko 
> wrote:
>
> Hi Deepak,
>
>
>
> You can use a union operator. I actually gave a talk on creating an
> advanced join using the union operator and multiple streams:
>
> -
> https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
>
> - https://www.youtube.com/watch?v=tiGxEGPyqCg
>
>
>
> I hope this helps.
>
>
>
> On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti <
> deepakg...@gmail.com> wrote:
>
> Hi,
>
> My name is Deepak, I am a new user to apache flink. It is one of the best
> open source i have used. I want to thank the community for developing such
> a wonderful product.
>
>
>
> I have one query.
>
> Is it possible to connect multiple streams, like
>
> stream1.connect(stream2).connect(stream3).flatmap(new
> RickCoFlatMapFunctionHandler())
>
>
>
> Can someone please let me know how I can achieve this.
>
> Thanks,
>
> Deepak
>
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>


RE: Is it possible to connect multiple streams

2022-09-21 Thread Schwalbe Matthias
Hi Deepak,

Coming back to your original question, you’ve got a number of option (some of 
them already mentioned:

  *   You can connect/join 2 streams of different types at a time by means of 
s1.connect(s2).
 *   (your example does not work directly as written (3 streams))
  *   You can connect many streams of same type by means of s1.union(s2, s3, 
…).
  *   The third option is new and not yet documented (marked as @experimental):
 *   Connect/join many streams of different types by means of a 
[Keyed]MultipleInputTransformation
*   If keyed: input streams need to either
   *   be keyed on the same key type, or
   *   be non-keyed (most likely) broadcast streams
 *   the API is still a little elaborate, a good starting point could be 
this test case: [1]

If of any interest, feel free to ask for clarifications …


Thias
PS: see also my next email in a couple of minutes …


[1] 
https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorBatchExecutionTest.java#L441

From: Shammon FY 
Sent: Wednesday, September 21, 2022 6:29 AM
To: user@flink.apache.org
Cc: Deepak kumar Gunjetti ; Yaroslav Tkachenko 

Subject: Re: Is it possible to connect multiple streams

⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠


Hi

Thanks @yaroslav .
And @deepakgd79 here is the document for datastream: 
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-transformations
 You can find examples for union, connect, join and other transformations


On Wed, Sep 21, 2022 at 11:55 AM Yaroslav Tkachenko 
mailto:yaros...@goldsky.com>> wrote:
Hi Deepak,

You can use a union operator. I actually gave a talk on creating an advanced 
join using the union operator and multiple streams:
- 
https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
- https://www.youtube.com/watch?v=tiGxEGPyqCg

I hope this helps.

On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti 
mailto:deepakg...@gmail.com>> wrote:
Hi,
My name is Deepak, I am a new user to apache flink. It is one of the best open 
source i have used. I want to thank the community for developing such a 
wonderful product.

I have one query.
Is it possible to connect multiple streams, like
stream1.connect(stream2).connect(stream3).flatmap(new 
RickCoFlatMapFunctionHandler())

Can someone please let me know how I can achieve this.
Thanks,
Deepak
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet 
unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von 
e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine 
Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser 
Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per 
e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche 
unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng 
verboten.

This message is intended only for the named recipient and may contain 
confidential or privileged information. As the confidentiality of email 
communication cannot be guaranteed, we do not accept any responsibility for the 
confidentiality and the intactness of this message. If you have received it in 
error, please advise the sender by return e-mail and delete this message and 
any attachments. Any unauthorised use or dissemination of this information is 
strictly prohibited.


Re: Is it possible to connect multiple streams

2022-09-20 Thread Shammon FY
Hi

Thanks @yaroslav .
And @deepakgd79 here is the document for datastream:
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-transformations
You can find examples for union, connect, join and other transformations


On Wed, Sep 21, 2022 at 11:55 AM Yaroslav Tkachenko 
wrote:

> Hi Deepak,
>
> You can use a union operator. I actually gave a talk on creating an
> advanced join using the union operator and multiple streams:
> -
> https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
> - https://www.youtube.com/watch?v=tiGxEGPyqCg
>
> I hope this helps.
>
> On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti <
> deepakg...@gmail.com> wrote:
>
>> Hi,
>> My name is Deepak, I am a new user to apache flink. It is one of the best
>> open source i have used. I want to thank the community for developing such
>> a wonderful product.
>>
>> I have one query.
>> Is it possible to connect multiple streams, like
>> stream1.connect(stream2).connect(stream3).flatmap(new
>> RickCoFlatMapFunctionHandler())
>>
>> Can someone please let me know how I can achieve this.
>> Thanks,
>> Deepak
>>
>


Re: Is it possible to connect multiple streams

2022-09-20 Thread Yaroslav Tkachenko
Hi Deepak,

You can use a union operator. I actually gave a talk on creating an
advanced join using the union operator and multiple streams:
-
https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
- https://www.youtube.com/watch?v=tiGxEGPyqCg

I hope this helps.

On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti 
wrote:

> Hi,
> My name is Deepak, I am a new user to apache flink. It is one of the best
> open source i have used. I want to thank the community for developing such
> a wonderful product.
>
> I have one query.
> Is it possible to connect multiple streams, like
> stream1.connect(stream2).connect(stream3).flatmap(new
> RickCoFlatMapFunctionHandler())
>
> Can someone please let me know how I can achieve this.
> Thanks,
> Deepak
>